Saturday, June 19, 2010

Enabling Jumbo frames on ESXi through vSphere CLI

If you want to enable jumbo frames or install drivers use the CLI option. Dont enable SSH and go to the unsupported console.

Download and install vSphere CLI on the desktop or on the laptop you used to manage the ESXi servers.

Once installed , go to Start - programs - vmware- vSphere CLI. This will open the command prompt and cd to bin directory. dir will list all the commands for you

C:\Program Files\VMware\VMware vSphere CLI\bin>

1) the below command will list all the switches and MTU size

C:\Program Files\VMware\VMware vSphere CLI\bin> vicfg-vswitch.pl --server vm4 -l

Enter username: root
Enter password:
Switch Name     Num Ports       Used Ports      Configured Ports    MTU     Uplinks
vSwitch1                 64                      1                  64                     1500

2) Run the below command to change the MTU size of the vswitch1

C:\Program Files\VMware\VMware vSphere CLI\bin> vicfg-vswitch.pl --server vm4 -m 9000 vSwitch1

3) To list the changes , run the command

C:\Program Files\VMware\VMware vSphere CLI\bin> vicfg-vswitch.pl --server vm4 -l
Enter username: root
Enter password:
Switch Name     Num Ports       Used Ports      Configured Ports    MTU     Uplinks
vSwitch1                     64                  1                   64                     9000

4) The below command will list the MTU of the vmnic 

C:\Program Files\VMware\VMware vSphere CLI\bin> esxcfg-nics.pl -server vm4 -l

Hope this helps someone. You can do this through vSphere CLI,or  vMA. But don't do this in unsupported way by enabling SSH.
-------------------------------------------------------------------------------------------

ESXi - Enabling SSH and Jumbo frames.

Enabling Jumbo frames on the ESXi server took me some time as most people suggest me to enable SSH and the login to the unsupported console and enable the jumbo frames. This is not supported by vmware. If you have a evaluation version running on a test machine you can try this.
We decided to go with ESXi installation on our Sun servers and HP servers and we enabled Jumbo frames on the switches and the Netapp storages. We enable on the real production machine through vSphere CLI.  I will write about that in next session.

1) Enable SSH on the ESXi hosts
  a) Go to the console of the ESXi host and press ALt+F1
  b) Type "unsupported" on the window, this text wont be echoed.
  c) Enter the root password, you will get the prompt ~ #
  d) Edit the file /etc/inetd.conf
         ~ # vi /etc/inetd.conf
  e) Remove the # from the line ssh.
   ssh     stream  tcp     nowait  root    /sbin/dropbearmulti     dropbear  ++min=0,swap,group=shell -i -K60

  f) Determine the process id of inetd from the command
           ~ # ps | grep inetd
  g) Restart the inetd
           ~ # kill -HUP

2) Enabling Jumbo frames

  a) Login to the server using SSH
  b) esxcfg-vswitch -l  , This will lists the current vswitches including the MTU sizes

~ # esxcfg-vswitch -l

Switch Name    Num Ports   Used Ports  Configured Ports  MTU     Uplinks
vSwitch0       64          3           64                1500    vmnic2

  PortGroup Name      VLAN ID  Used Ports  Uplinks
  VM Network          0        0           vmnic2
  Management Network  0        1           vmnic2

  c) Set the Jumbo frames for the vswitch0

     ~# esxcfg-vswitch -m 9000 vSwitch0

  d) Check the changes
  
~ # esxcfg-vswitch -l
 
Switch Name    Num Ports   Used Ports  Configured Ports  MTU     Uplinks
vSwitch0       64          3           64                9000    vmnic2

  PortGroup Name      VLAN ID  Used Ports  Uplinks
  VM Network          0        0           vmnic2
  Management Network  0        1           vmnic2



  e) Verify that that the NIC has the new value

     ~ # esxcfg-nics -l

      vmnic2  41:00.00 bnx2        Up   1000Mbps  Full   18:a9:05:68:cd:d0 9000

Enabling Jumbo frames on nxge interface or on T5140 server

To enable jumbo frames on the nxge interface you need to edit the below file. There is option to enable frames per interface, but this didnt work out for me so I enable on all the interface.

/platform/sun4v/kernel/drv/nxge.conf

#------- Jumbo frame support ---------------------------------
# To enable jumbo support for all nxge interfaces,
 accept_jumbo = 1;
#
# To disable jumbo support for all nxge interfaces,
# accept_jumbo = 0;
#
# Default is 0.  See the example at the end of this file for
# enabling or disabling jumbo for a particular nxge interface.
#
#

root@husky # dladm show-link
nxge0           type: non-vlan  mtu: 9194       device: nxge0
nxge1           type: non-vlan  mtu: 9194       device: nxge1
nxge2           type: non-vlan  mtu: 9194       device: nxge2
nxge3           type: non-vlan  mtu: 9194       device: nxge3