ESXi, Installing the CNA card for 10GBE and FCoE

Last week I installed the Q-logic CNA card on the ESXi server through vSphere CLI. It was a Live production machine.
1) Migrated all the VMs to the other machine on the cluster and put the ESXi on maintenance mode.
2) Then shutdown the machine and install the CNA adapter.
3) Power-on the server and it came back to maintenance mode.
4) Download the driver for 10GB Ethernet and FCoE from the Q-logic site ( it will re-direct to vmware site).    
(we need 2 drivers here, one for Ethernet and another for FCOE. The drivers are in ISO format.)
5) Burn the driver to the cd or use an ISO mounter on your Laptop/PC where the vSphere CLI is installed. ( here I mounted on drive H)
6) Start-Program-vmware-vSphere CLI, launch the CLI and install the driver by below command

C:\Program Files\VMware\VMware vSphere CLI\bin> vihostupdate.pl --server vm3 --install --bundle H:\offline-bundle\qlgc-qlge-100.2.1-offline_bundle-261179.zip
Enter username: root
Enter password:
Please wait patch installation is in progress ...
The update completed successfully, but the system needs to be rebooted for the changes to be effective.

7) Install the driver for FCoE (qlg.831.k1.23vmw-offline_bundle-207277.zip) also the same way. reboot the server and it will be back to maintenance mode
Check the adapter by click on configuration-network adapters.

Enabling Jumbo frames on ESXi through vSphere CLI

If you want to enable jumbo frames or install drivers use the CLI option. Dont enable SSH and go to the unsupported console.

Download and install vSphere CLI on the desktop or on the laptop you used to manage the ESXi servers.

Once installed , go to Start - programs - vmware- vSphere CLI. This will open the command prompt and cd to bin directory. dir will list all the commands for you

C:\Program Files\VMware\VMware vSphere CLI\bin>

1) the below command will list all the switches and MTU size

C:\Program Files\VMware\VMware vSphere CLI\bin> vicfg-vswitch.pl --server vm4 -l

Enter username: root
Enter password:
Switch Name     Num Ports       Used Ports      Configured Ports    MTU     Uplinks
vSwitch1                 64                      1                  64                     1500

2) Run the below command to change the MTU size of the vswitch1

C:\Program Files\VMware\VMware vSphere CLI\bin> vicfg-vswitch.pl --server vm4 -m 9000 vSwitch1

3) To list the changes , run the command

C:\Program Files\VMware\VMware vSphere CLI\bin> vicfg-vswitch.pl --server vm4 -l
Enter username: root
Enter password:
Switch Name     Num Ports       Used Ports      Configured Ports    MTU     Uplinks
vSwitch1                     64                  1                   64                     9000

4) The below command will list the MTU of the vmnic 

C:\Program Files\VMware\VMware vSphere CLI\bin> esxcfg-nics.pl -server vm4 -l

Hope this helps someone. You can do this through vSphere CLI,or  vMA. But don't do this in unsupported way by enabling SSH.
-------------------------------------------------------------------------------------------

ESXi - Enabling SSH and Jumbo frames.

Enabling Jumbo frames on the ESXi server took me some time as most people suggest me to enable SSH and the login to the unsupported console and enable the jumbo frames. This is not supported by vmware. If you have a evaluation version running on a test machine you can try this.
We decided to go with ESXi installation on our Sun servers and HP servers and we enabled Jumbo frames on the switches and the Netapp storages. We enable on the real production machine through vSphere CLI.  I will write about that in next session.

1) Enable SSH on the ESXi hosts
  a) Go to the console of the ESXi host and press ALt+F1
  b) Type "unsupported" on the window, this text wont be echoed.
  c) Enter the root password, you will get the prompt ~ #
  d) Edit the file /etc/inetd.conf
         ~ # vi /etc/inetd.conf
  e) Remove the # from the line ssh.
   ssh     stream  tcp     nowait  root    /sbin/dropbearmulti     dropbear  ++min=0,swap,group=shell -i -K60

  f) Determine the process id of inetd from the command
           ~ # ps | grep inetd
  g) Restart the inetd
           ~ # kill -HUP

2) Enabling Jumbo frames

  a) Login to the server using SSH
  b) esxcfg-vswitch -l  , This will lists the current vswitches including the MTU sizes

~ # esxcfg-vswitch -l

Switch Name    Num Ports   Used Ports  Configured Ports  MTU     Uplinks
vSwitch0       64          3           64                1500    vmnic2

  PortGroup Name      VLAN ID  Used Ports  Uplinks
  VM Network          0        0           vmnic2
  Management Network  0        1           vmnic2

  c) Set the Jumbo frames for the vswitch0

     ~# esxcfg-vswitch -m 9000 vSwitch0

  d) Check the changes
  
~ # esxcfg-vswitch -l
 
Switch Name    Num Ports   Used Ports  Configured Ports  MTU     Uplinks
vSwitch0       64          3           64                9000    vmnic2

  PortGroup Name      VLAN ID  Used Ports  Uplinks
  VM Network          0        0           vmnic2
  Management Network  0        1           vmnic2


  e) Verify that that the NIC has the new value

     ~ # esxcfg-nics -l

      vmnic2  41:00.00 bnx2        Up   1000Mbps  Full   18:a9:05:68:cd:d0 9000

Enabling Jumbo frames on nxge interface or on T5140 server

To enable jumbo frames on the nxge interface you need to edit the below file. There is option to enable frames per interface, but this didnt work out for me so I enable on all the interface.

/platform/sun4v/kernel/drv/nxge.conf

#------- Jumbo frame support ---------------------------------
# To enable jumbo support for all nxge interfaces,
 accept_jumbo = 1;
#
# To disable jumbo support for all nxge interfaces,
# accept_jumbo = 0;
#
# Default is 0.  See the example at the end of this file for
# enabling or disabling jumbo for a particular nxge interface.
#
#

root@husky # dladm show-link
nxge0           type: non-vlan  mtu: 9194       device: nxge0
nxge1           type: non-vlan  mtu: 9194       device: nxge1
nxge2           type: non-vlan  mtu: 9194       device: nxge2
nxge3           type: non-vlan  mtu: 9194       device: nxge3

Sendmail disabling help, vrfy, expn and version information.

We have a security audit and auditor asked me to disable the help, VRFY and EXPN  command  and  disable the version information displayed while login on the sendmail server.

1) To disable the version information from displaying, edit the sendmail.cf file. Make a backup copy of the file.

# vi /etc/mail/sendmail.cf

look for the line " GreetingMessage"
remove the " Sendmail $v/$Z " after $j. After editing the line looks like below.

O SmtpGreetingMessage=$j  $b

Restart the service

2) To disable the VRFY and EXPN ,look for  "PrivacyOptions" line in the /etc/mail/sendmail.cf file. Edit the PrivacyOptions line and append "novrfy,noexpn" to the line as below. Make sure you made a backup copy of the file before editing.

# privacy flags
O PrivacyOptions=authwarnings,novrfy,noexpn

Restart the service

3) To disable the help, go to /etc/mail/ , move the helpfile to helpfile.old and touch an empty helpfile

#pwd

/etc/mail

# mv helpfile helpfile.old

# touch helpfile

Restart the service

# /etc/init.d/sendmail restart

or

# svcadm refresh sendmail

After editing the configuration file telnet to port 25 will give the result as below
------------------------------------------------------------------------------
# telnet myhost 25
220 myhost ESMTP Mon, 8 Mar 2010 09:15:21 +0300 (AST)
help
214 2.0.0 End of HELP info
vrfy
252 2.5.2 Cannot VRFY user; try RCPT to attempt delivery (or try finger)
expn
502 5.7.0 Sorry, we do not allow this operation
quit
221 2.0.0 myhost closing connection
Connection to host lost.
----------------------------------------------------------------------

vSphere iommu error on dell optiplex 760

Once the installation of the vSphere is completed I got the following error on the console.
While loading its shows "Failed to load iommu driver"

cpu1:....: Initialization for vtd failed with -1"

To fix this disable the VT for Direct I/O from the BIOS

1) Reboot the esx host, press F12 and go to BIOS

disable "Virtualization support --VT for Direct I/O"

save and reboot

vSphere on dell optiplex 760, Not enough memory to load vmkernel

vSphere required minimum 2GB of memory ti install. I had a dell optiplex 760 with 2GB installed. While installing vSphere the installer stops and throws errors like
" vmnix...
 there is not enough memory to load vmkernel. 2040384 KB of RAM
was detected. We require at least 2064384 KB of RAM to boot.
Aborting boot "

I swapped 1GB from other server and installed the vphere. Once the installation is completed you can edit the memory required meory value to desired value.

1) Login to the esx server through console or via putty and edit the below file

# vi /etc/vmware/init/init.d/00.vmnix

2) Find the below line and edit with required value

RequiredMemory=2064384

3) Shutdown the server and remove the extra 1gb memory added and boot the esx host

I'm yet to try on ESXi host. This is my test environment to study vSphere. I'm not sure about any other workaround.