(4)OVFtool to Copy VMs from one ESXi host to another.

(1)How to increase the time limit on snapshot consolidation (2146270) 

Purpose

You can increase the maximum snapshot consolidation time if you want to stun the virtual machine for an extended period of time. 

 

              For example, you may want to increase the consolidation time to resolve an issue when the asynchronous consolidation fails after 10 iterations . By default, the snapshot consolidation time is set to 6 seconds.

Resolution

To increase the time limit of the snapshots consolidation, change the snapshot.maxConsolidateTime configuration parameter in the virtual machine to a higher value, such as 30 seconds:

Note: It is recommended to set this value to an optimum value of 30 seconds as this is usually sufficient to complete the required operations.

1. Shut down the virtual machine.
2. Right-click the virtual machine and click Edit Settings. 
3. Click the Options tab. 
4. Under Advanced, click General.
5. Click Configuration Parameters and add snapshot.maxConsolidateTime = 30.

(2)How to monitor snapshot deletion using esxtop command (2146232) 

Purpose

This article provides information on monitoring the snapshot deletion using esxtop when time stamps are not updating for virtual-mode RDM base disks. 

Resolution

Monitor using esxtop in ESXi 4.x/5.x

1. Log in as root to the ESX host using SSH. 
2. Run the esxtop command.
   
   Note: This command works only if the virtual machine is powered on. 
   
   
3. Press V to see only running virtual machines.
   
   Note: This is not the same as using the v option.
   
   
4. Find the virtual machine running the consolidation.
5. Type e to expand.
6. Enter the Group World ID (value from GID column).
7. Press Enter.
8. Make a note of the World ID (ID column) of the snapshot consolidation process:
   
   
   • In ESX/ESXi 4.x, the process is called SnapshotVMXCombiner.
   • In ESXi 5.x, the process is called vmx-SnapshotVMX.
     
9. Type u to display the disk device statistics.
10. Type e to expand and enter the device where the snapshot consolidation process is writing to.
    
    For example: 
    
    naa.xxx value
    
    Notes:
    • For a regular vmdk file, the device is the datastore that the flat file is located.
    • For a flat vmdk, identifying the datastore device ID can be done by running esxcfg-scsidevs -m.
    • For RDM, the vmkfstools -q against the pointer file reveals the vml ID, which needs to be correlated with the output of ls -l /vmfs/devices/disks/ to get the device ID.
      
11. Identify the Group World ID from step 6.
    
    Note: You may need to sort by MBREAD/s ( press R) or MBWRTN/s (press T) to see the process at the top of the screen.
    
    
12. Look at the number IOPS and throughput for the Consolidation process (WRITES/s and MBWRTN/s columns) to ensure that there is activity and the process is working.

(3)VMware Tools fails with the error: vthread-3, Exception 0xC0000005 access violation has occured.

Symptoms

• VMware Tools crashes on Windows 2003 terminal server
• Vmware tools throws pop up error message, you see error message similar to:

vThread-3, exception 0xC0000005 access violation has occurred.

Cause

This issue occurs while accessing a RPCout channel pointer when it is NULL.

Resolution

This is a known issue affecting ESXi 5.1 and ESXi 5.5.

To resolve this issue, upgrade VMware tools.

 

For ESXi 5.1

This issue is resolved in ESXi 5.1 Patch 04, available at VMware Downloads. For more information, see VMware ESXi 5.1, Patch Release ESXi510-201404001 (2070666)

For ESXi 5.5

 

This issue is resolved in ESXi 5.5 Patch 04, available at VMware Downloads. For more information, see VMware ESXi 5.5, Patch Release ESXi550-201501001 (2099265)

 

 

 

(4)OVFtool to Copy VMs from one ESXi host to another.

In today’s post, we will see ow to use OVFtool to copy VMs from one host to another without having shared datastore between them.

The idea behind this article came from one of the questions that were directed to me recently regarding how to copy a VM from one host to another without having shared storage between them.

Now you could perform this from the vCenter Server if both the source and destination hosts are registered to the vCenter Server.
And yes, this operation cannot be performed when the Virtual Machine is in a power ON state even from the vCenter Server.

But if the two hosts were to be standalone hosts, then you could use another VMware product like VMware Converter, but it has its own set of issues.

Another method would be to download the files of the Virtual Machine to your local desktop and upload it to the local datastore of the destination host.

Now you can imagine the complexities and efforts that the above two tasks can take, which is when I came across the ovftool utility and was surprised to see that this can be done with no efforts and is the easiest way to probably do this.

And another good thing about the ovftool is that it is available for Windows, Linux, and MAC Operating Systems.

In our demo here, I will be making use of the Windows msi file to install on a Windows 8.1 machine which has access to both my source and destination hosts.

To download the latest version of the ovftool as of today, use the below link.

Download OVFTool

In our example, I have two ESXi hosts with the IP Addresses 192.168.1.15 and 192.168.1.16. I have a VM called DSL which I am interested in copying from one host to another.

The VM DSL is currently running on host 192.168.1.15 residing on the local datastore of that host (LocalESXiH)
First, you will open a command prompt to navigate to the directory where the ovftool is installed.

Run the below command to connect to the ESXi host remotely.

ovftool.exe vi://root@192.168.1.15

We are making the connection using the root account.

As you can see above, it is listing the VMs registered on the host and Test is one of them.

Now, we just need to specify the source ESXi host and the destination ESXi host as well as the datastore using the -ds option using the ovftool.

The actual copy of the VMs happens through the NFC protocol.
Here is an example of using ovftool to export the VM from one ESXi host to another ESXi host:

ovftool.exe -ds=LocalESXiF vi://root@192.168.1.15/DSL vi://root@192.168.1.16

There we go, you can see that the copy of the VM was successful from the above message.
But a couple of things to remember when you try this.

• If you have snapshots running on the VM, then it will not copy over the delta files.
• If you have thin provisioned disk, it will convert to thick provisioned on the destination datastore by default.

If you wish to learn more about the ovftool, kindly use the -h parameter to list out the available options.

 

 

 

 

 

 

 

Create and Manage AD Users and Computers

             Create and Manage AD Users and Computers

Now we have the understanding of how and what an Active Directory Domain is and the terminology that is used and what are the roles and functionalities of the Domain Controller.

         Next up, we will look at the objects that can be part of the Domain, like Users and Computers.


   
In today’s objective, we will be looking at:

• Create, Copy, Configure and Delete Users and Computers.
• Configure Templates.
• Configure User Rights.
• Automate the creation of AD accounts.
• Manage Inactive and Disabled accounts.
• Perform Bulk AD operations.
• Offline Domain Join.

 

Create, Copy, Configure and Delete Users and Computers

Users and Computers are one of the most important objects of the Active Directory Domain.
In this section, we will see the various ways available to create them.

Using Active Directory Users and Computers

To launch ADUC on your Domain Controller, go to Server Manger > Tools > Active Directory Users and Computers.
You can also go to Run and type dsa.msc to launch ADUC.
Right Click on any OU > New > User.

Provide the basic details like the First Name, Last Name, and the User logon name to create the user account.
Similarly, you can create a Computer account, by right clicking on an OU > New > Computer.

You just have to provide the computer name to add the object to the domain.

Using Active Directory Administrative Center

Administrators when using Windows Server 2003 and Windows 2008 had only the option of ADUC when managing objects in the AD domain.
From Windows Server 2008 R2 onwards, Microsoft introduces Active Directory Administrative Center, which is another of managing the Domain and is completely built on PowerShell.
I really like this tool as it is very modern and has some really cool features.

As you can see, the design is very modern and it is a little more intuitive than the ADUC.
Similarly, you can create both Users and Computers using this tool.

Using dsadd

dsadd is a command line utility available from Windows Server 2008. This utility will only be available if the ADDS Server Role is installed.

Dsadd

 
Applies To: Windows Server 2003, Windows Server 2008, Windows Server 2003 R2, Windows Server 2012, Windows Server 2003 with SP1, Windows 8

Adds specific types of objects to the directory.
Dsadd is a command-line tool that is built into Windows Server 2008. It is available if you have the Active Directory Domain Services (AD DS) server role installed. To use dsadd, you must run the dsadd command from an elevated command prompt. To open an elevated command prompt, click Start, right-click Command Prompt, and then click Run as administrator.

Commands

Command	Description
Dsadd computer	:Adds a single computer to the directory.
Dsadd contact	:Adds a single contact to the directory.
Dsadd group	:Adds a single group to the directory.
Dsadd ou	:Adds a single organizational unit to the directory.
Dsadd user	:Adds a single user to the directory.
Dsadd quota	:Adds a quota specification to a directory partition.

dsadd user CN=AdilArif,CN=NewUsers,DC=enterprisedaddy,DC=com -pwd Password -disabled No

This will create a user called AdilArif in the NewUsers OU with the above password.
By default, the user is created and disabled, hence we are providing the parameter as NO.
Similarly, you can create computer account as well.

dsadd computer CN=Server1,CN=NewComputers,DC=enterprisedaddy,DC=com

Using PowerShell

Microsoft has made it very easy to administer most of the services of Windows Server using PowerShell.

How to create Active Directory Users with Powershell

 his is going to very interesting post as I will telling you different ways of creating users within Powershell. First things first, you need to make sure that you have Powershell installed on your machine. And also try and have Powershell 3.0, because it has some cool features from its predecessors. You can download Powershell 3.0 from here.
 
Once done, verify the version of Powershell opening up a Powershell window from start menu. Make sure you open the window by right-clicking and Run as Administrator. If you don’t, then trust me friends, BAD things happen!


You should see something similar to the below image.

Now that we have taken care of the prerequisites, lets get to the meat of the article.
The cmdlet that helps you with creating a new user in an Active Directory domain is New-ADUser. I am sure that was difficult to guess. No it wasn’t!!
Trust me guys, the Powershell team has made it extremely easy for us to understand and use this cmdlets. They follow the Verb-Noun naming convention.
I always recommend that once you hear or know a new cmdlet, get the help files related to it and spend some time understanding it! The way you get the help is by simply typing the below:
Get-Help New-ADUser
You can also add different parameters at the end to get different forms of help file. Simply append -Examples to the above line and see what happens.
Voila! It gives a complete info along with a few examples as to how you can use that cmdlet.
Now lets go ahead and add a real user!
New-ADUser Ronnie
Assume that you want to see the user that you just created now. Do not worry my friends, it could not have got any simpler, just type the below:
Get-ADUser Ronnie

Powershell will show everything that it has related to Ronnie. By default it shows only a few properties related to the user.
Let us say that you want to see all the properties associated with the user.
Get-ADUser Ronnie -Property *
The user Ronnie does not have many properties associated with him. You can see that there are lot of places that are left blank. That is because when we create Ronnie, we just gave his First Name.
Let us add his Last Name and give him a description so that you can easily find anything you want about Ronnie in future.
For this we will be using the Set-ADUser cmdlet. But you can also use Set-ADObject if you like to.
Again, I highly recommend that you go through the help files for each cmdlet that I have introduces in this post.
Get-ADuser Ronnie | Set-ADuser -Description "Ronnie is from the Marketing Team" -SurName "Hopkins" 
This was fun until you have couple of users to add. Imagine you get a list from someone saying that you have add 200 users today! I am sure that is going to be a pain.
Most often people will give the list of users with a csv file. Powershell lets you play with the csv files easily and make changes as per your wish. Assume that I have a csv file named newusers.csv
Make sure the first row of the csv file contains only the properties that are associated with the New-ADUser cmdlet.

#View users from CSV
Import-CSV ".\users.csv" | Out-GridView

Out-Grid View is just going to show the list on my screen in a nice format!
# Import users from CSV
Import-CSV ".\newusers.csv" | New-ADUser

We just imported the users and added them to the domain.

Before we continue, let us create a new Organizational Unit at the root of the domain.
New-ADOrganizationalUnit NewUsers
# Import users from CSV, set password, enable

Import-CSV ".\newusers.csv" |
New-ADUser `
-Enabled $True `
-AccountPassword $(ConvertTo-SecureString "P@55word" -AsPlainText -Force) `
-Company 'Enterprise Daddy.' `
-Path 'OU=NewUsers,DC=enterprisedaddy,DC=com'

In the above example we started to add the properties to the User accounts on the fly, the properties that were not mentioned in the csv file. Powershell gives you that flexibility and complete control.

Also note the character ` which you will find below the Esc key, this helps you make your code look clean and continue to the next line. Powershell will consider as though the code is continuing and is in the same line.

Configure Templates

In some cases, you are asked to create single user accounts, but they contain so many attributes, that it can be a time-consuming process.
We saw above that we can speed this up using various methods like dsadd.exe and New-ADUser cmdlet.
But there is another method in which you can create a user template.
A user template is a standard user account containing the most common attributes within the organization. You would usually start the name of the with the underscore.
Now if you have to create a user based on the template, you simply have  to right click on the template from ADUC and click on Copy.

Now you can enter the required information and enable the account.

Configure User Rights

To configure User Rights on a single machine, from Server Manager > Tools > Local Security Policies > User Rights management.

As mentioned before, the above mentioned is used for a single computer.
To configure for a whole of computers, we need to use Group Policy, which we will be discussing in the upcoming articles.

Automate the creation of AD Accounts

Some of the old ways of doing this are:

Using csvde.exe

A command line utility that can create new AD DS objects by importing from a CSV file.
The ‑i parameter specifies import mode; without it, the default mode of CSVDE is export. The ‑f parameter identifies the file name to import from or export to. The ‑k parameter is useful during import operations because it instructs CSVDE to ignore errors.
The syntax is as below:

csvde.exe -i -f <filename.csv> [-k]

Using ldif.exe 

Like csvde.exe but with more functionality, LDIFDE is a utility that can import ADDS information and use it to add, delete or modify objects.

anage InActive and Disabled Accounts

For inactive accounts, the old way of doing it was to check the last logon date. This is when the user would have accessed the domain.
You can use PowerShell to achieve this.

Get-ADUser -Filter * -Properties lastLogonDate | Format-Table Name, lastLogonDate

To check disabled account, the old way is to check the account was enabled using the Get-ADuser cmdlet.

Get-ADUser -Filter {enabled -ne $true}

In the new OS, we have a new cmdlet to easily find inactive and disabled accounts.

Search-ADAccount -AccountDisabled

The above will list both the Users and Computer objects that are disabled within the domain.

Search-ADAccount -AccountDisabled -UsersOnly

Only Disabled user accounts will be displayed.
Similarly for Inactive accounts, you can check as below.

Search-ADAccount -AccountInActive

For Password related issues.

Search-ADAccoun t -PasswordExpired
Search-ADAccount -PasswordNeverExpires
Search-ADAccount -LockedOut

Perform Bulk AD Operations

As discussed in the above section, csvde.exe, ldif.exe, and PowerSehll were a few ways wherein one can automate the process of AD object creation, deletion and modification.

Offline Domain Join

During an offline domain join, a computer is configured to join a domain without contacting a domain controller.
This makes it easy to join the computers to the domain where there is no network connectivity.
I have written a complete guide on how to perform this procedure which you can find below.

Offline Domain Join – Add computers to domain

Windows Server 2008 R2 and Windows 7 introduce a new option for joining computers to a domain, called offline domain join.

As the name suggests, this features lets you join a computer to the domain if there is no network connectivity or the computer cannot contact the domain controller.
For this, we will be using a command called Djoin.exe on a computer which is part of a workgroup with the information required to join to the domain.

When would you use Offline Domain join feature?

This is an important that can be used for datacenters, virtualized desktop environments, where the machines are built and provisioned on demand.
It can also be used when the machine is built and used in a lab environment usually disconnected from the actual network. So when the machine is first started up when part of the network, it will already be a member of the domain. This also helps apply the required Group Policy at the start up.

What are the steps to be performed for Offline Domain Join?

Basically there are four major steps that need to be performed to join a computer to the domain using the Offline Domain Join method.

1. Log on to the Windows Server 2008 R2 running the Active Directory Domain Services or Windows 7 machines running RSAT tools with the account having permission to add computers to the domain. (Domain Admins group has this permission by default)
2. Use the DJoin command to provision a computer for offline domain join. This step prepopulates Active Directory with the information that Active Directory needs to join the computer to the domain, and exports the information called a blob to a text file.
3. At the offline computer that you want to join the domain use DJoin to import the blob into the Windows directory.
4. When you start or restart the computer, it will be a member of the domain.

Now that you understand the requirements to perform the Offline Domain join of a computer, ;ets dig in further to see how will you go about doing the same.

NOTE: The user who will perform this actions need to be a part of Domain Admins Group or should have equivalent permissions to add computers to domain.

Provision a Computer in Active Directory for Offline Domain Join

Run Djoin.exe from an elevated Command Prompt to provision the computer account in Active Directory. The basic syntax of DJoin is as follows:
djoin.exe /provision /domain DomainDNSName /machine ComputerName /savefile Filename
where
/provison parameter creates new computer account in Active Directory. You can also /reuse parameter if the computer account already exists in AD.
DomainDNSName is the DNS name of the domain. In our case it will be enterprisedaddy.com
ComputerName will be the name of the computer to be created or reused.
Filename will be the path and name of the File that we will output the blob to.
Now let us see an example of the command that we will be using in our demo here.
djoin.exe /provision /domain enterprisedaddy.com /machine CLIENT02 /savefile C:\CLIENT02_Join.txt /machineOU “OU=NewComputers,dc=enterprisedaddy,dc=com”
Note that I have given an extra parameter called machineOU which will create a computer account in the OU called NewComputers under root domain.
Similarly you can use switches like /dcname domainControllerName to define which DC you want to create the account in.
Moving ahead, the computer account called CLIENT02 will be created in NewComputers OU and information will be exported to the path C:\CLIENT02_Join.txt
So we have completed the steps that need to performed in Active Directory. Now this information needs to be injected in the computer that has to joined by the offline method.

Perform an Offline Domain Join

The account metadata that was exported in a blob to a text file by using Djoin.exe /provision can be imported to a computer, after which the computer will become a domain member at the next startup.

The command that you will run on the computer and its syntax can be seen below:
djoin.exe /requestODJ /loadfile Filename /windowspath %SystemRoot% /localos
where:
/requestODJ specifies that you want to perform an offline domain join operation.

Filename is the path and file name of the text file that contains the account metadata blob. This is the file that you created by using Djoin.exe /provision.

%SystemRoot% is the built-in Windows variable that represents the directory in which Windows is installed.

/localos specifies that you are injecting the domain join information into the local computer.

In our case, the command will be below assuming we have copied the file to the C drive of the local machine:
djoin.exe /requestODJ /loadfile C:\CLIENT02_join.txt /windowspath %SystemRoot% /localos

That’s it! Now when the computer is connected to the domain and is started, it will automatically add itself to the domain and be a member of the domain.

Something

About Windows

Windows

Windows Common Task with Run Command

To Access…	Run Command
Accessibility Controls	access.cpl
Accessibility Wizard	accwiz
Add Hardware Wizard	hdwwiz.cpl
Add/Remove Programs	appwiz.cpl
Administrative Tools	control admintools
Adobe Acrobat (if installed)	acrobat
Adobe Photoshop (if installed)	photoshop
Automatic Updates	wuaucpl.cpl
Bluetooth Transfer Wizard	fsquirt
Calculator	calc
Check Disk Utility	chkdsk
Clipboard Viewer	clipbrd
Command Prompt	cmd
Component Services	dcomcnfg
Computer Management	compmgmt.msc
Control Panel	control
Date and Time Properties	timedate.cpl
Device Manager	devmgmt.msc
Direct X Control Panel (if installed)*	directx.cpl
Direct X Troubleshooter	dxdiag
Disk Cleanup Utility	cleanmgr
Disk Defragment	dfrg.msc
Disk Management	diskmgmt.msc
Disk Partition Manager	diskpart
Display Properties	control desktop
Display Properties	desk.cpl
Display Properties (w/Appearance Tab Preselected)	control color
Driver Verifier Utility	verifier
Event Viewer	eventvwr.msc
Files and Settings Transfer Tool	migwiz
File Signature Verification Tool	sigverif
Findfast	findfast.cpl
Firefox (if installed)	firefox
Folders Properties	folders
Fonts Folder	fonts
Group Policy Editor (XP Prof)	gpedit.msc
Hearts Card Game	mshearts
Help and Support	helpctr
HyperTerminal	hypertrm
Internet Connection Wizard	icwconn1
Internet Explorer	iexplore
Internet Properties	inetcpl.cpl
Internet Setup Wizard	inetwiz
IP Configuration (Display Connection Configuration)	ipconfig /all
IP Configuration (Display DNS Cache Contents)	ipconfig /displaydns
IP Configuration (Delete DNS Cache Contents)	ipconfig /flushdns
IP Configuration (Release All Connections)	ipconfig /release
IP Configuration (Renew All Connections)	ipconfig /renew
IP Configuration (Refreshes DHCP & Re-Registers DNS)	ipconfig /registerdns
IP Configuration (Display DHCP Class ID)	ipconfig /showclassid
IP Configuration (Modifies DHCP Class ID)	ipconfig /setclassid
Keyboard Properties	control keyboard
Local Security Settings	secpol.msc
Local Users and Groups	lusrmgr.msc
Logs You Out Of Windows	logoff
Microsoft Chat	winchat
Microsoft Excel (if installed)	excel
Mouse Properties	control mouse
Mouse Properties	main.cpl
Nero (if installed)	nero
Netmeeting	conf
Network Connections	control netconnections
Network Connections	ncpa.cpl
Network Setup Wizard	netsetup.cpl
Notepad	notepad
ODBC Data Source Administrator	odbccp32.cpl
On Screen Keyboard	osk
Outlook Express	msimn
Paint	pbrush
Password Properties	password.cpl
Performance Monitor	perfmon.msc
Performance Monitor	perfmon
Phone and Modem Options	telephon.cpl
Power Configuration	powercfg.cpl
Printers and Faxes	control printers
Printers Folder	printers
Regional Settings	intl.cpl
Registry Editor	regedit
Registry Editor	regedit32
Remote Desktop	mstsc
Removable Storage	ntmsmgr.msc
Resultant Set of Policy (XP Prof)	rsop.msc
Scanners and Cameras	sticpl.cpl
Scheduled Tasks	control schedtasks
Security Center	wscui.cpl
Services	services.msc
Shared Folders	fsmgmt.msc
Shuts Down Windows	shutdown
Sounds and Audio	mmsys.cpl
Spider Solitare Card Game	spider
SQL Client Configuration	cliconfg
System Configuration Editor	sysedit
System Configuration Utility	msconfig
System File Checker Utility (Scan Immediately)	sfc /scannow
System File Checker Utility (Scan Once At The Next Boot)	sfc /scanonce
System File Checker Utility (Scan On Every Boot)	sfc /scanboot
System File Checker Utility (Return Scan Setting To Default)	sfc /revert
System File Checker Utility (Purge File Cache)	sfc /purgecache
System File Checker Utility (Sets Cache Size to size x)	sfc /cachesize=x
System Information	msinfo32
System Properties	sysdm.cpl
Task Manager	taskmgr
TCP Tester	tcptest
Telnet Client	telnet
Tweak UI (if installed)	tweakui
User Account Management	nusrmgr.cpl
Utility Manager	utilman
Windows Backup Utility (if installed)	ntbackup
Windows Explorer	explorer
Windows Firewall	firewall.cpl
Windows Magnifier	magnify
Windows Management Infrastructure	wmimgmt.msc
Windows Media Player	wmplayer
Windows Messenger	msmsgs
Windows Picture Import Wizard (need camera connected)	wiaacmgr
Windows System Security Tool	syskey
Windows Update Launches	wupdmgr
Windows Version (to show which version of	winver

VMware

Virtual Machine Management Commands

1. Command: #Services.sh restart

This command will restart all the management agents and services running on the host. You should see each service stopping and then being started.

2. Command: #vim-cmd vmsvc

This is the most useful command for emergency maintenance and takes several extensions.

3. Command: #vim-cmd vmsvc/getallvms

This command will display the list of virtual machines registered on the ESX/ESXi host (Powered Off or Powered On). The list contains additional information such as vmid, name of the virtual machine, Guest OS type, VM HW version, location of the virtual machine configuration file and annotations.

Vmid Name File Guest OS Version Annotation

4. Command: #vim-cmd vmsvc/power.getstate <vmid>

This command will display the power status of the virtual machine in question.

5. Command: #vim-cmd vmsvc/power.off <vmid>

This command will power off the virtual machine in question.

6. Command: #vim-cmd vmsvc/power.on <vmid>

This command will power on the specified virtual machine and boot the Guest OS

7. Command: #vim-cmd vmsvc/power.reboot <vmid>

This command will reboot the guest operating system.

8. Command: #vim-cmd vmsvc/power.reset <vmid>

This command will reset the virtual machine in question.

9. Command: #vim-cmd vmsvc/power.shutdown <vmid>

This command will shutdown the guest operating system running inside the virtual machine.

10. Command: #vim-cmd vmsvc/power.suspend <vmid>

This command will suspend the virtual machine in question.

1. Command: #vim-cmd vmsvc/power.suspendresume <vmid>

This command will revert the virtual machine from suspended state to normal state.

2. Command: #vim-cmd vmsvc/power. Hibernate <vmid>

This command will place the guest operating system in standby mode.

3. Command: #vim-cmd vmsvc/snapshot.create

Usage: snapshot.create <vmid> [snapshotName] [snapshotDescription] [includeMemory] [quiesced]

This command is used to take snapshot of the virtual machine.

Virtual Networking Commands

1. How to create a standard virtual switch.

#esxcfg-vswitch -a <vswitch name>

EX: [root@server root]# esxcfg-vswitch –a vSwitch1

2. How to add a port group to the virtual switch

#esxcfg-vswitch -A <Portgroup name> <vswitch name>

EX: [root@server root]# esxcfg-vswitch –A “Service Console” vSwitch1

3. How to check whether the specified virtual switch exists or not

#esxcfg-vswitch -c <vswitch name>

4. How to check whether the specified portgroup exists or not

#esxcfg-vswitch -C <portgroup name>

Output: Will print 1 if the virtual switch exists and 0 if not

5. How to remove the virtual switch

#esxcfg-vswitch -d <vswitch name>

Output: Remove the virtual switch. This command fails if any ports on the virtual switch are in use by vmkernel networks or virtual machines.

6. How to remove a port group from virtual switch?

#esxcfg-vswitch -D <port group> <vswitch name>

Output: Removes the port group. This command fails if any port group on the virtual switch is in use

7. How to add an uplink adapter to the virtual switch ?

#esxcfg-vswitch -L <physical_nic> <vswitch_name>

Des: Running the command with this option attaches a new unused physical network adapter to a virtual switch

EX: [root@server root]# esxcfg-vswitch –L vmnic1 vSwitch1

8. How to list all virtual switches and their port groups?

#esxcfg-vswitch -l

9. How to remove an uplink adapter from the virtual switch?

#esxcfg-vswitch -U <physical_nic> <vswitch_name>

Des: Remove an uplink adapter from a virtual switch. An uplink adapter corresponds to a physical Ethernet adapter to which the virtual switch is connected. If you remove the last uplink adapter, you lose physical network connectivity for that switch.

10. How to set VLAN ID for a specific port group?

#esxcfg-vswitch -v <vlan_id> --pg <portgroup_name> <vswitch_name>

EX: [root@server root]# esxcfg-vswitch -v <VLAN> -p “Service Console” vSwitch0

Des: Set the VLAN ID for a specific port group of a virtual switch. Setting the option to 0 disables the VLAN for this port group. If you specify this option, you must also specify the --portgroup option.

esxcfg-vmknic

The esxcfg-vmknic command adds, deletes, and modifies VMkernel network interfaces. In vSphere 5, equivalent ESXCLI commands are available.

11. How to add a vmkernel nic to the system?

#esxcfg-vmknic -a -i <ip_address> -n <netmask> <portgroup_name>

EX: esxcfg-vmknic -a –i 192.168.5.6 –n 255.255.255.0 "VMkernel"

Des: Add a VMkernel NIC to the system. When the command completes successfully, the newly added VMkernel NIC is enabled.

12. How to remove a vmkernel nic?

#esxcfg-vmknic -d <vmknic_name>

EX: esxcfg-vmknic -d "VMkernel"

13. How to disable a specified vmkernel nic?

#esxcfg-vmknic --D --interface-name=<nic> --enabled=[true| false]

14. How to enable a specified vmkernel nic?

#esxcfg-vmknic -e <vmknic_name>

15. How to list all vmkernel nics?

#esxcfg-vmknic -l

EX: esxcfg-vmknic -lInterface  Port Group          IP Address      Netmask         Broadcast       MAC Address       MTU     TSO MSS   Enabled
vmk0       VMkernel            192.168.5.5    255.255.255.0   192.168.5.255  00:51:47:3c:20:cc 1500    40960     true

Esxcfg-route

Setting the vmkernel Default Gateway

16. How to set vmkernel default gateway?

#esxcfg-route -a 19.2.168.100.0 255.255.255.0 192.168.0.1

Or

#esxcfg-route -a default 192.168.0.1

17. How to delete a vmkernel gateway?

#esxcfg-route -d 192.168.100.0/24 192.168.0.1

18. How to list vmkernel gateway configured?

#esxcfg-route -l

Esxcfg-vswif

Used to configure service console networking in 4.x systems. Not needed and not available in the ESXi 5.x ESXi Shell

19. How to display service console interface?

#esxcfg-vswif -l

EX: esxcfg-vswif -l
Name     Port Group          IP Address       Netmask          Broadcast        Enabled   DHCP
vswif0   Service Console     192.168.5.5     255.255.255.0    192.168.5.255   true      false

20. How to add a new service console networking interface?

#esxcfg-vswif -a -p “service console” -i <ip address> -n <netmask> vswif0

21. How to delete a service console networking interface?

#esxcfg-vswif -d vswif0

Interview Questions

szddxf

Interview questions

1)What is service console?
A) The service console is developed based up on Redhat Linux Operating system,it is used to manage the vmkernel.

2)What are the basic commands to troubleshoot connectivity between vsphere client/vcenter to esx server?
(A)--->service mgmt-vmware restart
           (restarts host agent(vmware-hostd)on vmware esx server))
   
     ---->service vmware-vpxa restart(restarts vcenter agent service)

      ----->service network restart(restarts management networks on esx)


(3)what is vcenter agent?
(A)vc agent is an agent installed on esx server which enables communication between vc and esx server.This agent will be installed on esx/esxi will be done when you try to add the esx host in vcenter.

(4)what is the command used to restart SSH,NTP&vmware web access?
(A)
        (1)service sshd restart
        (2)service ntpd restart
        (3)service vmware-webaccess restart

(5)what are the types of port groups in esx?
(A)There are 3 types of port groups in esx

         (1)service console port group
         (2)vmkernel port group
         (3)virtual machine port group
      
                  There are only 2 types of port group in esxi
               
                      (1)vmkernel port group
                      (2)virtual machine port group

(6)what is vmkernel?
(A) vmkernel is a proprietary kernel of vmware and is not based on any of the flavors of linux operating systems.vmkernel requires an os to boot and manage the kernel.A service console is being provided when vmware kernel is booted.only service console is based up on redhat linux os not vmkernel.

(7)what is the use of service console port?
(A)service console port group required to manage the esx server and it acts
as the management network for the esx.vcenter/vsphere clent uses the service console ip's to communicate with the esx server.

(8)what is the use of vmkernel port?
(A)vmkernel port is used by esx/esxi for vmotion,iscsi&nfs communications.esxi uses vmkernel as the management network.since it dont have service console built in it.

(9)what is the use of virtual machine port group?
(A)virtual machine port group is used by virtual machine communication.

(10)How virtual machine communicates to another servers in network?
 (A)All the virtual machines which are configured i vm port group are able to connect to the other machines on the network.So this port group enables communication between Vswitch and physical switch by the use of uplink associated with  the port group.

(11)What is the default number of ports configured with the virtual switch?
 (A) When the time of virtual switch created,vswitch is creted with 56 ports by default.We can extend the no of ports by editing the vswitch properties.

(12)What are the different types of partitions in esx server?
(A)  /-root
       Swap
       /var
       /var/core
      /opt
       /home
      /tmp 

Vsphere Troubleshooting

After unexpacted host reboot, Powering on a RDM attached virtual machine fails with the error: Incompatible device backing specified for device '0'

Last week one of our host unexpectedly got restarted and once the host came online we were unable to power on a VM (a passive cluster node) due to an error like,

Incompatible device backing specified for device '0'

HA didn’t restart this VM due to a VM to host-Must DRS rule.

This error occurs when LUN is not consistently mapped on hosts where primary/secondary hosts are running however here when crosschecked found everything correct (LUN Number/naa.id) on affected host.

As this was a passive node so we removed the affected drive from this VM and started this node and then started investigating the issue.

On checking the vml.id of this LUN on both hosts, found it different but the strange thing was its correct on the host in question but wrong on all other hosts in cluster. To share a LUN with different nodes, it should be consistently mapped on all hosts and should have consistent unique vml.id (VMware Legacy id) but here its different so seems the RDM disk pointer file meta data got corrupted.

You can find the vml.id of LUN as follows,

First note down/copy the identifier of LUN (naa.id) and then fire this cmd,  

#esxcli storage core device list -d naa.id

Now to fix this issue what we can do is,  remove the affected RDM disk from the both nodes and then delete the RDM pointer file from Datastore (this doesn’t affect your actual data on LUN). Now after re-scanning the hosts for Datastores, re-add the LUN as RDM drive on both nodes. Now you would be able to power on the affected node.

If due to any reason above doesn’t work then as above after removing the affected RDM drives from both nodes, follow these steps,

1. Note the NAA_ID of the LUN.
2. Detach RDM using vSphere client.
3. Un-present  the LUN from host on storage array. 
4. Rescan host storage. 
5. Remove LUN from detached list using these commands:
   
   #esxcli storage core device detached list
   #esxcli storage core device detached remove -d naa.id
6. Rescan the host storage. 
7. Re-present LUN to host. 
8. Now again rescan the hosts for datastores

If the LUN has been flagged as perennially reserved, this can prevent the removal from succeeding.

Run this command to remove the flag:

#esxcli storage core device setconfig -d naa.id --perennially-reserved=false

Now the command to remove the device should work.

# esxcli storage core device detached remove -d naa.id

Now cross check the vml.id on hosts and it should be same and after adding the RDM drive on nodes you will be able to power on the VM nodes.

Reference: VMware KB# 1016210

That’s it… :) 

Thursday, Jul 21, 2016

How to deal with unresponsive windows service, like vCenter svc

You might have seen this, where you tried to restart a windows service and it got stuck on stopping or in some cases starting. Recently same thing happened with me when tried to restart vCenter service, it got stuck on stopping service.

Here what we can do is, first note down the service name by going to its properties,

Here for vCenter service its ‘vpxd’

Now open windows command prompt in elevated mode and run this cmd,

C:\> sc queryex vpxd 

This will give show you the detailed info/status of intended service, note down the PID of respective service.

        SERVICE_NAME: vpxd
        TYPE               : 10  WIN32_OWN_PROCESS
        STATE              : 3  STOP_PENDING
                                (STOPPABLE, NOT_PAUSABLE, ACCEPTS_SHUTDOWN)
        WIN32_EXIT_CODE    : 0  (0x0)
        SERVICE_EXIT_CODE  : 0  (0x0)
        CHECKPOINT         : 0x0
        WAIT_HINT          : 0x493e0
        PID                : 4061
        FLAGS              :

Now run this cmd,

C:/> taskkill /f /pid xxxx

Here the PID is 4061 so,

C:/> taskkill /f /pid 4061

this will terminate the service immediately, once done then you can start the service either from GUI (services console) or from cmd itself by running this command, 

C:/> sc start vpxd

That’s it... :)

ESXi 5.x host not accessible/showing as inaccessible after reboot

You might have seen this issue where you rebooted an host and even after waiting for a long time it didn't come up and showing as inaccessible in vCenter inventory. Tried  to reconnect the host but end up with this error, 
"Cannot contact the specified host (ESXi1.mylab.com). The host may not be available on the network, a network configurati-on problem may exist, or the manageme-nt services on this host may not be responding."

Then I tried to ping this host but no luck, however when connected to host using physical server remote management tool, found the host up.

This is something that I had seen earlier, in this case what you need to check is, login to DCUI from server remote management console (hp iLO or Dell iDRAC or IBM IMM or whatever) and then check IP configuration of the host as well as test the management network.

Most of the time when you test the management network, you would see something like, host is not reaching to gateway/DNS or sometimes strangely DNS is not reachable but name resolution is happening.

In this case most of the time, fix of this issue as simple as restarting the management network.

  
Sometimes you may need to restart the management network more than once.

As soon as management network restart, host would be accessible (start pinging) again however sometimes you may also need to restart host management agents to make it available/accessible from vCenter console.

That's it... :) 

Thursday, jul 21, 2016

VM has network connectivity but network card showing as disconnected, having red cross on it

In this post I will talk about this minor issue where, System has network connectivity but network card showing as disconnected, having red cross on it.

Earlier I had seen this problem and drafted a blog post about it but never posted until recently when one of my friend faced the same issue but couldn’t find its cause/fix.

Now coming to the point, you might have seen this where you deployed a VM from template or rebooted a system and when the VM came up, you are able to connect to it but its network card in notification area having red cross on it (appearing disconnected).

If you would check from connectivity point of view, you wouldn’t find anything wrong here however most of the time you would like to get rid of this red cross sign.

If you would further investigate about the issue, then you will find this in Network and Sharing Center,

Here to fix this, you just need to check the status of  “Network Location Awareness service”, in this case either its stopped or set to disabled. This service should be set to automatic and in started state. 

As this issue was caused by a windows services so I don't think its specific to VM only, you might see this issue on a physical Windows system as well.

Note: “Network Location Awareness service” is responsible to collect and stores configuration information for the network and notifies programs when this information is modified. If this service is stopped, configuration information might be unavailable. If this service is disabled, any services that explicitly depend on it will fail to start. 

That’s it… :)

Unexpected AD Account Lockouts When Logging On to Outlook

Today one of my friend asked me a AD account lockout related question, i.e. "there are two users and when they open outlook on their system, their AD accounts getting locked out automatically".

In such kind of cases, most of the time the issue is caused by a saved password, where user recently changed his account password but somehow outlook is not asking for new password and trying to connect using the old saved credentials instead.

To fix this account lockout issue, what one need to do is....remove the saved credentials from the system where your outlook account is configured.

To do so, Go to Start=> Control Panel => Credential Manager 

or open, Run => control keymgr.dll , this will also open the Credential Manager

You will find any saved credentials here,

Here Edit or Remove the saved outlook credentials and this will fix the AD account lockout issue. 

Alternatively we can do the same by using this method,

Click Start, Run => rundll32.exe keymgr.dll, KRShowKeyMgr (case sensitive)

 This will open the Stored User Names and Passwords window,

from here you can Edit/Remove or Add new credentials as well as Backup or restore the saved 
credentials.

Update: As per my discussion with a fellow Sys admin, Removing the credentials from Credentials Manager doesn't necessarily remove them from Key manager so sometimes we also need to check and remove the same from Stored User Names and Passwords. I am not sure if it's true as never seen this behaviour.

That's it... :)

Thursday,jul 21, 2016

Windows 7 VM auto suspend issue- reason and fix

Last day one of my colleague built a new VM with Windows 7 guest OS and after some time he noticed, once he logged off from this VM, its being automatically suspended after some time.
As he couldn't figure out the issue so asked me to take a look at this newly created VM to figure out why its being suspended automatically every now and then. When I checked, found the VM in suspended state, when taken a look at related task & events from vSphere client, couldn’t find anything in tasks however found suspend events under event tab.

However I couldn’t find any specific reason for this auto suspend from VMware end, then I logged in on to the Guest windows 7 OS  to check from inside the VM and started digging  System event logs and came across this,

Here it clearly mentioned, system was entered in sleep mode because it was ideal…..so what’s now…. Here we need to check from inside the guest, if the power plan for the Windows 7 is set to move the system to sleep if system is ideal for a specified time.

Now go to power options, Control Panel => Power Options => Click on Change when the computer sleeps

as you can see  here, this VM was set to enter in sleep mode if the system is ideal for 15 minutes.

Now what we need to do is,  set "Put the computer to sleep" to "Never" and you are done.

Note: I believe the same will be applicable for Windows 8 or any other client OS VM too.

This power plan setting is something that's by default set to never on server operating systems and that is why we never faced such issue with Server VMs.

That's it... :)

Wednesday, March 2, 2016

Uploading ESXi image to vSphere Update Manager failing with the error: Failed to import data

When it comes to ESXi host patching, I am a big fan of VMware Update Manager as it makes the upgrade process flawless. Using VUM, you just need to create upgrade baseline, download patches then attach baseline to host and finally stage/remediate the patches then update manger will initiate the patch installation and reboot the host once done.

Now coming to the point, we had few ESXi 5.0 hosts and wanted to upgrade them to ESXi5.1 update3 so I thought of upgrading them using VUM instead of command line.

As most of us know, to upgrade ESXi on an host using VUM, one first need to download ESXi iso image and import it to VUM ESXi image repository, here while trying to import the ESXi5.1 image to update manager5.5’s ESXi image repository, ESXi image import failed with below error,
"Failed to import data, The uploaded upgrade package cannot be used with VMware vSphere Update Manager"

I was aware about the fact that one need to upgrade the version of VUM before importing a newer image to update manager repository as the minimum version of VUM required should match the version of the ESXi image you are trying to import but here I was trying to upload ESXi5.1 image to VMware Upgrade Manager 5.5 and unable to do so. I tried with different ESXi5.1 images like Cisco or HP’s customized images as well as standard VMware ESXi5.1 image but the end result was same however when tried to import ESXi5.5 image to VUM, was able import it without any issue.

This is something that was very strange for me so started looking for the cause of this issue and my search end on the VMware KB articles# 2009812, if you would see the Note section of this KB, its clearly mentioned there,

“ You must use the same version of vSphere Update Manager to upgrade to the corresponding version of ESXi”

Later I checked with VMware support about this and they also confirmed the same.

Related KB article: 2097168

That’s it… :)

Saturday, January 30, 2016

vCenter Server shows ESXi host as not responding

In our environment we have two - three hosts located at different sites, and in every few days we was getting one or other host listed in vCenter inventory as “not accessible” and the VMs running on that particular host listed as disconnected in vCenter inventory. (These hosts have Esxi 5.5)

As a first step to troubleshoot this issue, tried to ping the host as well as access the VMs with success,  then connected to the host over ssh using putty and restarted the management agents and waited for some time to for host to respond on vCenter console but that didn’t happen.

Then I tried to reconnect the host to vCenter but end up with the error “cannot contact the specified host (EsxiHost0xxxx). The host may not be available on the network, a network configuration problem may exist, or the management service on this host may not be responding".When tried to connect to the host directly using vSphere client, I was able to connect to the host without any issue.
As no other host was having this issue except two- three remote hosts that means the issue is not related to vCenter server firewall/port blocking.

On checking vpxd logs I found few missed heartbeats entries as well as this kind of entries for affected host ,

As vpxd log clearly shows, this issue is related to vCenter to host connectivity and that could be due to congested network. Here what we can do as a work around to avoid this issue is, we can increase the host to vCenter heartbeat response timeout limit from 60 seconds to 120 seconds (by default Esxi host sends a heartbeat to vCenter in every 10 seconds and vCenter has time window of 60 seconds to receive it). Please remember Increasing the timeout is a short-term solution until the network issues can be resolved.

To do so, Using vSphere Client:

Connect to vCenter, Administration => vCenter Server Settings => select Advanced Settings

Now in the Key field, type: config.vpxd.heartbeat.notRespondingTimeout

In the Value field, type: 120

Click Add and then OK.

Restart the VMware vCenter Server service for changes to take effect.

Using vSphere Web Client:

Connect to vCenter Server using vSphere Web client and navigate to the vCenter Server instance

Select the Manage tab, 

and then select Advanced Settings and click on Edit, this will popup a new window,
Now in the Key field, type: config.vpxd.heartbeat.notRespondingTimeout

In the Value field, type: 120

Click Add, OK

Restart the VMware vCenter Server service for changes to take effect.

Reference: Related KB#1005757

That’s It… :)

Thursday, January 21, 2016

Newly presented LUNs are not visible on Esxi host

Today when I was migrating two MSCS cluster VMs from one host to another (cold migration), found 5 LUNs missing on target host so asked the Storage admin to present these LUNs on target host. Once the Storage admin confirmed the same, we re-scanned the Storage/HBA adapters for datastore/LUNS, after the rescan when checked for the newly added LUNs, was surprised to see on one host only two LUNs were visible while on another host all five LUNs were visible. Then I checked with storage team, they confirmed that everything is fine from their end and I might need to restart the host to make the LUNs visible.

After evacuating all the VMs, rebooted the host but even after reboot LUNs were unavailable. Later when I further investigated, the issue turned out to be related to max storage path, yes this host was already having the max storage paths that is why newly mapped LUNs were not visible.

Note: Local storage, including CD-ROMs, are counted in your total paths.

You can see how many paths are being used on a specific host by Selecting the host, going to Configuration => Storage Adapters => Storage Adapter

As most of us would be aware, the VMware vSphere Host storage path limit as of vSphere 5.x is 1024 and the maximum LUNs per host is 256  (refer to configuration maximums) and as this host was already having the max supported paths (552+471+1= 1024), thus was unable to add new LUNs/Paths. 

Fix: To fix this issue, ask the storage team to reduce the number of paths per LUN so that there are fewer than the 1024 total limit, or reduce the number of LUNs presented to the host.

Note: put the host in maintenance mode during the storage path correction and once done, re-scan the host/storage adapters for datastore/LUNs.

Reference: David Ring's blog post on host path limit, VMware KB#1020654.

That’s it... :)

Monday, January 18, 2016

VM not accessible/lost network connectivity after reboot

This past week as part some activity we powered off some Virtual Machines and after some time when powered on those VMs again, I was surprised when the network would not come up for two VMs. I tried to ping them but VMs were not reachable so I logged-in on one of the virtual machine via VM console to check the IP configuration etc and found Network Card was showing limited connectivity (yellow sign on NIC icon) however the IP information was correct, then tried to ping from inside the VM without success. I also the rebooted the VMs again but that didn’t fix the connectivity issue.

I did remember few months back we had faced a similar issue however at that time default gateway was not turned on and ipconfig was showing an APIPA address like 169.254.x.x. To fix that issue we had to reconnect the VM network card so tried the same here too and it worked.

To fix the issue what you need to do is, Select the affected VM and go to Edit VM settings => select the vNIC adaptor =>Deselect Connected => Now click OK to apply the setting

Now Navigate again to the Edit VM Settings => select the vNIC adaptor =>Select Connected => And click OK to apply the setting.

Once settings applied, VM came back to network again.

On another VM just to test if a cold reboot would work here, I powered off the VM and once it powered off, powered it on again and voila VM was accessible again. 

Thus we can fix this issue either by a cold reboot  or by reconnecting the virtual network card.

Update: Today I came across the same issue again, this time it was a MS Server 2012 R2 OS VM and a cold reboot didn't fix the issue, one more thing sometime in order to fix the issue, you may need to repeat the vNIC disconnect process.

Update2_25/02/2016: Sometime the above wouldn't work at all, then what you can do is: login to the affected server via VM console => Go to Network card properties and disable, re-enable the network card from inside the OS and hopefully server would come back to network again.
The other thing you can do is Change the IP assignment setting from Static to Dynamic, it would pick an IP from APIPA, then change it back to Static and it should work now.

Note: For further detail about the issue, you may refer to the related VMware KB#2012646.

That’s it... :)

Saturday, January 16, 2016

ESXi host stuck "in progress" when exiting maintenance mode

This was first time when I came across such issue where exit from maintenance mode was taking a long time and appearing like stuck on 15% (waited for at least 20 minutes). To see whats going on the host, connected to the host using Putty and checked the maintenance status of host using the vim-cmd command,
#vim-cmd hostsvc/hostsummary |grep inMaintenceMode  

And I was amazed to see the output, it was clearly showing the host has exited from maintenance mode while from GUI it was still showing in progress.

Then I thought vSphere client might not refreshed the task status so closed the connection and then reconnected to vC again. This time there was some progress, but still it was taking too long to Exit from maintenance mode.

Whenever we see this kind of unusual issues , we look at restart of host management agents. 

I have bad experience in past with restarting all the agents at once using #services.sh restart (it takes a long time to complete), so prefer to restart host and vCenter agents individually using below commands,

#/etc/init.d/hostd restart
#/etc/init.d/vpxa restart

Now coming back to the point, this fixed the issue however the host was showing like the HA agent didn’t installed correctly so again I put the host back in to the maintenance mode and once the task completed, exited from maintenance and this time there was no issue.

That’s it… :)

Wednesday, January 6, 2016

VM not accessible, you might also need to check datastore for space

Last weekend I got a support call from database team to check, why they lost connectivity to a database server VM. Firstly I tried to ping  this VM but it was not reachable so have to login to vCenter to see what’s wrong with that VM. First thing that I noted, VM was powered on but had a message sign on it and when tried to open VM console for further investigation, got a datastore space related popup question,

Clicked on retry but the pop-up came up again.

As this popup question suggests, there was not enough space in datastore for the VM to breath and this is because the VM was running on Snapshot (******-000001.vmdk).

Even you would see the same question in VM summary page.

When I checked the datastores where the VM disks reside, I was amazed to see this,

Here one may ask why this happened, do you not having Storage Cluster/SDRS and the answer is no we don’t (that’s a different story). Thin provisioning is also not the case here but the Snapshot of this large VM is (this VM having one or more TB disk attached). This snapshot was created by VM backup tool during backup but at the same time there was some activity going on the server so it grows unexpectedly, eaten all available datastore space and cause this issue.

(if you are thinking why the hell we are taking image level backup of database VM drives, please don’t bother to ask me as I also couldn’t find the logic of that)

So to fix this what we need to do is, check all the datastores where VM disks reside (in VM summary you would see the datastore in question with space error/warning alert), create space for VM to breath and once you are done, go to VM summary,

Select Retry option in VM question and click ok.

VM should be accessible now (now you might also take a look at backup server to see if backup was completed, if so but the snapshot is still there then delete the snapshot and if the backup job is still in running state then no worries snapshot would automatically delete once the backup complete).

Note: You may also see an open VM console MKS error during open VM console (like /vmx file not accessible or unable to open) due to space crunch in datastore.

That's it... :)

Thursday,Jul 21, 2016

How to add RDM to Microsoft Cluster nodes without downtime

This is something that we do once in months so you might forget the process and then come across errors few time before recalling the right process at least this have happened with me more than twice so thought of making a note of the process of adding RDMs to already up and running MSCS cluster nodes.

Adding RDM to Microsoft Cluster nodes is little different from adding RDM LUN to an independent Virtual Machine.
The first part is same in both cases, you can add RDM disk while VM is powered on however in case of MSCS node one need to power off the VM before initiating the same on secondary node.

Open VM Settings => Click on Add Hardware, Select Hard disk => select Disk type as RAW device mapping =>now this the screen you can select the intended LUN,

On next screen Select, where you want to store the RDM pointer files, Next => on this screen you would select the RDM compatibility, it could be Physical or Virtual (description of both is available in screenshot), Select Physical

On next screen you would select controller => then you would see summary here, click finish and you are done.

Again go to the fist node's VM settings and select the newly added RDM drive and copy/right down the path of RDM Pointer file, do the same for any other RDM...

Now power off the node if its not already powered off(if this was the primary node, cluster resources would automatically move to another available node or move the resources to secondary node manually and power off the VM).

If you wouldn't power off the first node and try to add the rdm to secondary VM, you will get following error,

And if you power off the secondary node then you would able to add the RDMs to it however when you would try to power it on, you will end up with this error,

So if you are not already aware you would wonder, what;s the right way to do that. Here it is:

Power off the node where you have already added the RDM LUNs(if not already) and Now Add the RDMs to the Secondary Node while its a active cluster host and online.

Open VM Settings => Click on Add Hardware, Select Hard disk => now this the screen you need to select add an existing disk

On Next screen you need to provide/browse the path earlier noted/copied RDM Pointer file, 

On next screen you would select controller => then you would see summary , click finish and you are done.

That's it... :)

Thursday,jul 21,2016

Snapshot Disk Consolidation fails with a file lock error message

This is common in an environment where you are using VM backup solutions like vRanger, Veem backup, Avamar etc which takes backup at Esxi's level and uses hot-add technology to take back up of a VM.

Cause: During the backup of the VM a snapshot was made, then the base disk of target VM was hot-added to the VM that handles the backup (vRanger/Veem or any other backup solution uses hot add technology). Now the backup was made. After the backup however the backup solution somehow did not manage to hot-remove the disks from the backup software VM. This meant the base disks of the VM being backed up were still locked, hence the failure when trying to consolidate.

In VM summary you would see this,

And when you try to consolidate the snapshot disks, you would get this error,

One can fix this issue by using one of these methods:
1. Go to the settings of your backup software Virtual Machine's settings and check for the attached disk, you would find some extra disks there (affected VMs disks), now you need to unlink your vmdk files from his virtual machine (Do Not Delete the Disks, Only remove them from this Virtual Machine).

Once disks are unlinked from the backup software VM then you need to run Snapshot consolidation on affected and this time it will consolidate all snapshots to base disk and complete without any issue.

2. In this method Storage vMotion the affected VM to another datastore (Right click on VM => Migrate => now you need choose migrate disks to another available datastore – this will clear lock on disk files of the virtual machine.
Now run the Snapshot consolidation and it will complete without any issue.

Note: However the Snapshot consolidation completed successfully but you may still need to remove the base disk from Backup Software Virtual Machine(Do Not Delete the Disks, Only remove them from this Virtual Machine).

Related Issue: VM backup failed with an error like, "one or more disks from virtual machine ******* is alreadymounted to the backup software VM. You must unmount these disks before attempting to bakup the virtual machine".

 
The fix of this issue lies in the first part of above solution.

That's it... :)

Thursday, jul 21, 2016

vSphere Web client and latest version of Chrome, IE11

Last week one of my friend pinged me to take a look at vSphere Web client as he was not able to open virtual machine console, this option was greyed out,

My first question was to him, have he installed wcb client client integration plugin before trying to open VM console using Chrome Browser and he said yes. On checking I found pop-up blocker was enabled but even after disabling, VM console option was still greyed out.

We re-installed client integration plugin, restarted browser but nothing worked.

Then I suggested it seems a browser issue lets try with ie, he had ie 11 installed when he logged in using IE11 browser, we were amazed to see we had all vm folders but no VM was available when clicking on VM folder.

 Its then when I started looking about this issue.

Lets start with Chrome:- After some digging we found VMware kb# 2114800, As per this kb, after updating Google Chrome to Version 42 or later, vSphere Web Client Integration Plugin 5.x no longer functions.
This issue occurs because the Netscape Plug-in API (NPAPI) is deprecated(no longer available) in Google Chrome 42 and later. The NPAPI is deprecated by all modern browsers.

Resolution:- This is a known issue affecting VMware vSphere Web Client 5.x, this issue is resolved in vCenter Server 5.5 Update 3a. 

Currently, there is no resolution for 5.0 and 5.1.

To work around this issue:

• Enabling the NPAPI feature within Google Chrome 42 through 44.
  
  Note: These steps are not applicable to Google Chrome 45 and later.
  
  To enable NPAPI in Google Chrome:

1. Open a new browser tab in Google Chrome
2. In the address bar, enter this:
   
   chrome://flags/#enable-npapi
3. Under the Enable NPAPI section, click Enable.
4. Refresh or launch the vSphere Web Client landing page and attempt to use the CIP features.

• Use Mozilla Firefox version 39.x or later in to access the CIP-integrated features of the vSphere Web Client.
• Use the vSphere Client.

Note: For virtual machine control due to the deprecation of NPAPI, the VMware Remote Console (VMRC) is released to take its place. 

IE11:-  I was not able to reproduce the issue for ie11 in my homelab as he had vSphere web client version 5.5.0 build 2414847 and I have a different build.

Microsoft Internet Explorer 11 is Supported in vSphere 5.5 Update 1 and later versions, for later versions,

Fixing the client integration issue for ie11 is quite simple, After logging to the vSphere web client, hit the Alt key within your IE11 session and select “Tools” followed by “Compatibility view settings”, and add the vSphere url to the approved compatibility view list. Then click OK and its all set.

Now vSphere web client integration for ie should work.

Note: For virtual machine control due to the deprecation of NPAPI, the VMware Remote Console (VMRC) is released to take its place. To open with VM remote console, go to VM Summary 

It would work for any browser.

Reference: kb# 2114800, 2005083

That's it... :)

Thursday, jul 21, 2016

Server has a weak ephemeral Diffie-Hellman public key error in Chrome/Firefox

I believe anyone who is using vSphere Web client on version 5.1 would be aware about this error, we get this error when try to connect to web client or any other site having certain SSL Ciphers using latest versions of Chrome/Mozila (so far I didn't see this issue with ie),

Note:- This is a known issue affected the vSphere Web Client 5.1,  it is resolved in vSphere Web Client 5.1 Update 3e and later
This issue occurs due to changes to the web browser containing a fix to combat an unrelated vulnerability that consequently disables certain SSL Ciphers.

When I was looking for how to avoid this for web client or any other site giving this error, I came across the thread about the related issue on Google Chrome Help Forum and the summary is, so far Chrome itself doesn't have any option to disable related setting to allow the sites having relatively week security.

If a secure website gets the error ERR_SSL_WEAK_SERVER_EPHEMERAL_DH_KEY, it means the website is trying to set up a secure connection, but it is actually IN-secure because the SSL/TLS uses a Diffie-Hellman group size smaller than 1024-bit.

This is the problem in the Logjam vulnerability, which affects both browsers and servers:   https://weakdh.org 

In this case, the website/webserver needs to be fixed.  Google Chrome won't use insecure connections in order to protect your privacy.

In my case I am using self signed certificate instead of certificate authority signed certificate.

Resolution:- Google Chrome:- As I earlier said the there is no option available within Chrome to enable you to access less secure sites over https however as a way around we can use IE Tab Chrome Extension it will allows us to open vSphere web client within Chrome.

To use this extension, first go to Chrome Web Store and add IE Tab extension to chrome, now go to your url, you will again get the "Server has a weak ephemeral Diffie-Hellman public key error" Now all you have to do is click on the IE Tab icon which you will find in the right corner of the Chrome window (Highlighted in Blue),

And once you would click on IE Tab icon,

Though it's not an official fix, it still works and would allow you view the web pages without any issues.

In Mozilla Firefox we have an option to disable it by going to following url,

about:config

Here in this config page, you will find a list of boolean entries. Search for below two entries,

security.ssl3.dhe_rsa_aes_128_sha

security.ssl3.dhe_rsa_aes_256_sha 

By default, these are set to TRUE. But you have to set them to FALSE in order to allow the less secured pages.

Reference:  kb# 2125607,

That's it... :)