Working with ESX Log files

Source: http://www.simongreaves.co.uk/working-with-esxi-log-files/

Good article written by Simon Greaves. I grabbed things I really liked and pasted below: See his article for original content and some extra stuff.

Working with ESX(i) log files is important when troubleshooting issues within the virtual environment. You can view and search log files in ESX(i) and in vCenter Server using a few different methods.

Methods
Using the vSphere client
The direct console user interface (DCUI)
A web browser
A syslog or vMA appliance
An SSH connection to the host
PowerCLI using the Get-Log command

When using SSH, use the following commands to view and search the log files.
Use more to page through the log files one page at a time
Use tail to view the end of the log files
Use grep to search
Use pipe | to link commands together
Use pipe | to grep to search through files
Use cat to concatenate & use grep to search
Use find | print | grep filename to search for a file

Example
Cat hostd.log | grep search variable | more

vCenter log files
vCenter log files are in a vpxd-xx.log format where xx is a numerical value that increases when each log file is 5MB in size.
The log file numbers rotate when the vpxd service is started or when the log reaches 5MB in size.
The log files are located in the c:programdataVMwareVMware virtual centerlogs

Other log files include
Vpxd-alert-x.log
Vpxd-profile-x.log

Esx logs
/var/log/VMkernel – VMkernel messages
/var/log/vmkwarning
/var/log/vmksummary
/var/log/vmware/hostd.log
/var/log/messages – service console
/var/log/vmware/vpx/vpxa.log – vSphere client agent
/var/log/aam/VMware__xxx.log- HA
/var/log/vmkiscsid.log – iSCSI
/var/log/boot-logs/sysboot.log – boot log

vi commands
a – append
i – insert
O/o – open new line – O is line above, o is line below
r – replace
: – search or save options
/ – search
wq – write and quit
x – delete individual characters
dd – delete line
$ – go to the end of the line
ESC – break out of current mode

Log bundles
Log bundles can be accessed through the VMware folder on the start menu, by clicking generate vCenter server log bundle. This runs the vc-support windows scripting file located at c:program filesVMwarevirtual infrastructurevirtual centrescriptsvc-support.wsf and cscript.

You can also download it through the vSphere client and by connecting to the ESX(I) server using scp with Veeam FastSCP or WinSCP. To do this you have to enable tech support mode first.
An alternative way of generating log bundles is through the vm-support command run through an SSH connection to the COS or through the vMA.  Running vm-support will generate a tar compressed file.

Procedure
[root@esxhost]#/usr/bin/vm-support

With ESXi it is possible to place log files on shared storage.  To set this open the vSphere client connection to the host, click configuration>advanced settings>syslog select local and enter the path to the shared storage.  Enter the log file location as [datastorename]/logfiles/hostname.log.

vilogd
vilogd is a service that performs log collections.
You can manage it with the vilogger commands.  vilogger is used to enable and disable or configure the log collections with these commands.

To use vilogger, first ensure that vi-fastpass is enabled using vifp list server to list out the current vi-fastpass enabled servers, if no servers are listed use vifp addserver servername and vifptarget -sservername to add again.

Commands
vilogger enable
vilogger list
vilogger update policy

Control the vilogd service with etc/init.d/vmware-vilogd start|stop|restart

vilogger has several parameters available, an example of which are
–numrotation number of files to collect
–maxfilesize specified in MB
–collectionperiod how often to poll, specified in seconds

Example
vilogger enable –server servername –numrotation 20 –maxfilesize 10 –collectionperiod 10

This command will collect the following logs from the ESXi host
hostd.log
messages.log
vpxa.log

To scroll through the log files one page at a time use the more command.

Example
more hostd.log

Configure vMA as a Syslog Server
You can configure the vMA as a syslog receiver to collect log files from the ESX and ESXi server.  Run the commands listed below to configure.

vMA
#sudo service rsyslog stop
#sudo nano /etc/sysconfig/rsyslog

This will open nano so you can edit the following information
change SYSLOGD_OPTIONS=”-m 0? to SYSLOGD_OPTIONS=”-r -m 0?

Save and exit the file
#sudo service rsyslog start
#sudo iptables -I INPUT -i eth0 -p udp –dport 514 -j ACCEPT
#sudo nano /etc/rc.local

Edit the file to add the iptables line below to the end of the rc.local file
iptables -I INPUT -i eth0 -p udp –dport 514 -j ACCEPT

ESX
To configure ESX to use vMA as a syslog server add the IP address of the vMA to the /etc/syslog.conf  file.
#vi /etc/syslog.conf

Add the following lines to the bottom of the file
# Send all syslog traffic to vMA
*.* @<IP_Address_Of_vMA>

Open the firewall with
#/usr/sbin/esxcfg-firewall -o 514,udp,out,syslog

Finally restart the syslog service with
#sbin/services syslog restart