0 0

Storage Offloading: Let the Array Do All the Work

Storage management and utilization is an important function of the hypervisor. Without management and utilization of storage, virtualization would have gone nowhere fast.

Management of storage infrastructure can be a resource-intensive task. The hypervisor is responsible for performing most storage functions: creation of files, reading and writing files, processing SCSI commands, file copy, etc. Depending on the virtual workload, this can consume significant CPU and memory resources on the host, resulting in fewer resources available for the guest VM.

Consider the creation of a thick 100GB disk. The thick disk is created in full capacity on the storage subsystem — all 100GB of 0s. Those 0s come from the hypervisor, traverse the storage fabric or network (for iSCSI, NFS and FCoE), and is written to the storage subsystem. Depending on fabric or network conditions and configuration, this can take significant time.

Another example involves a file copy. The hypervisor reads the file and sends the same bytes back over the fabric or network to the storage subsystem (again). This imposes a double file-size IO hit on the environment as the hypervisor is doing all the work.

As virtualization initiatives took off, it became apparent that the load focused on storage management was just too high. Arrays are highly specialized devices that do one thing really well: storage. What if the hypervisor did not have to do all that work? Oh, what a world that would be. Offloading storage functions to the storage subsystem would reduce the load on the hypervisor and place the work on a device designed to handle such requests.

The need was identified, and with the introduction of VMware vSphere 4.1, vStorage APIs for Array Integration (VAAI) technology was introduced. Since then, we have seen inclusion of VAAI functions in more and more storage subsystems from various storage vendors in the marketplace. However, storage offloading is not unique to VMware products. Microsoft introduced support for offloaded data transfer (ODX) functionality in Hyper-V v3.0 (available with Windows Server 2012). Storage vendors are adapting ODX functions, and some functionality should be available with Windows Server 2012 GA.

The following functions are now being offloaded to the storage subsystems and away from the hypervisor:

  • Move: This offloads the IO involved in copying and moving files from one location on the array to another location on the array. Rather than the hypervisor reading and writing the data to the same device, the device is aware of the source and destination and performs the move without the data leaving the array on the fabric or network. Expect to see this function provide benefits in storage migration functions on the same array.
  • Locking: When virtual machine files are written to a block-based datastore, the files must be locked. In a high-IO environment, contention can appear as the hypervisor can only lock and unlock so quickly. This is also seen in high-density block datastores (lots of VMs on a single datastore). This function will offload locking of the files to the storage array. The storage array has quicker access to the files and can lock them significantly faster than the hypervisor. This function will provide benefits for high VM-to-datastore density environments and high-IO environments.
  • Initialization and creation: Creation of large empty files consumes time and resources. The hypervisor is responsible for filling the file with data. Upon creation of the file, this includes sending blank data (0s) to the file. For a large file (Example: 100GB), the hypervisor is responsible for sending 100GB of 0s at the rate of the fabric or network infrastructure available. This uses valuable CPU and fabric or network resources for the duration of the operation. Expect to see this function create files significantly faster as the data does not need to travel the fabric and is kept on the storage subsystem.
  • Virtual hard disk creation, expansion and merging: While similar to the initialization or creation functionality, this allows specific manipulation of virtual hard disk structures, not just creation functions. Extension of the disk happens at the storage level without needing to send the data to and from the storage system from the hypervisor.

Storage offloading improves with every hypervisor iteration; support for NAS (file-based data stores), snapshot functionality offloading, and even more functions. With the new focus on NAS storage from hypervisor providers, storage offloading is making NAS on-par, if not more feature-rich than block storage:

  • File cloning: Offloading copying of specified files to the NAS. The NAS has direct access to the files and can perform the operation on its own without impacting network and hypervisor performance.
  • Snapshot offloading: The NAS performs snapshotting functions internally vs. relying on the hypervisor. Plus, in the case of VMware's Linked Clones (highly used in their View product), linked cloning uses NAS-provided snapshots vs. traditional redo-log based snapshots. This provides significant performance benefits for the VDI environment and users.

While the functionality is valuable, perhaps one the most impressive functions involves enabling the functionality. Storage offload functionality for vSphere and Hyper-V is enabled by connecting the hypervisor to the storage subsystem. No check boxes, no configuration files, or anything like that. When the storage is connected to the hypervisor, the hypervisor negotiates available functionality. The hypervisor "asks" the storage subsystem if it supports the offload functions. If the subsystem supports the functions, they are enabled automatically and the virtualization environment can take advantage of it. In the event the subsystem does not support the functionality, a subsystem code update may add it.

The hypervisor still needs to perform some storage functions. But by offloading some functions to the array, hypervisor resources are significantly less stressed and performance is increased. Talk about a win/win for the virtualization environment.



Bill Hill / Sep 05, 2012

Bill is an industry recognized vExpert. By day Bill manages the IT infrastructure for a multinational logistics company. By night he blogs on virtualbill....more