XenServer 6.5 Performance Issues

During a recent migration of virtual servers from a XenServer 6.2 pool to a XenServer 6.5 pool severe performance issues were noticed.  These issues were most apparent on XenDesktop workstation with client applications that connected to back end SQL servers.

The VM servers and workstations were copied from the XenServer 6.2 pool to the 6.5 pool and the XenTools upgraded.  It was also verified that there were no ghost nics on the servers or workstations.  It was also verified that the TCP Offload had been disabled on the NICs of the virtual workstations and servers.

The performance issue was not as apparent during file copies from server to server or from server to workstation.  The performance issue with clients applications caused a delay in processes that normally took 5 – 10 seconds to take several minutes if they ever completed.

Starting in XenServer 6.5 the Generic Receive Offload (GRO) setting for the phyiscal network interfaces on each XenServer are enabled by default.  Previously the GRO setting had been disaled by default.  The default enabling of GRO can be found in page 3 of the release notes found here.

In the event that the network adapters on the machine running XenServer do not support GRO a severe performance impact can result.  The following steps can be completed to list all pifs on the XenServer and disable GRO.

To list all PIFs on the host utilize the following command. (Note: replace # with the UUID of the host. The uuid for the host can be found in XenCenter or via the cli)

xe pif-list host-uuid=#

To disable GRO run the following command for each network adapter listed after running the above command.

xe pif-param-set uuid=<pif-uuid> properties:gro=off

It is important to note that no reboot is required after performing the above commands. The above commands should be completed for all network interfaces on all XenServers in the environment that do not support GRO.

The Citrix article outlining the above steps can be found here.