Big Data Requires Big Datastores

I recently wrote an article on the improper use of “best practices” due to a number of factors. Chief among them is the idea that as time progresses, technology changes at a much more rapid pace, making it difficult to let any particular practice steep in an environment long enough to rise up as the “best” one. At its heart, the idea of a best practice is one grown in the soil of past mistakes and triumphs and is meant to guide a way forward with as little fuss as possible.

At some point, however, a best practice becomes dated and works against the spirit of technology. Can you imagine if you were still following an older “best practice” of no more than 25 VMs per datastore? You’d be swimming in tiny datastores, or large ones with a vast amount of empty, wasted space. Or, more to the point in this article – what if you were still using small, 500 GB datastores and trying to balance your VMs among them. I’d even go so far as to say 1TB to 2TB datastores can be a hassle to juggle, even if you had compression and deduplication to rely upon.

So if the idea is to use fewer “big” datastores, there are a few questions to answer.

  • What about replication?
  • How are backups handled?
  • Do recoveries take longer?

I think these are all good points that absolutely must be addressed. Realistically, most of these issues are resolved when we stop treating the storage as a large, clunky bucket and start focusing on the VMs themselves. Storage systems should allow you to replicate, backup and recover VMs, and not just entire datastores based on LUNs and volumes, which are fast becoming legacy technology in the new world of virtualization. Historically, I designed and managed NFS exports in the 5TB+ size range for VM workloads, so I know these were issues I faced even then.

From a replication standpoint, many products today focus on replicating the LUN or volume. The technology doesn’t care what’s on it, but certainly will ensure that all those bits are at both sites. There are definitely times when this works – it requires a lot of planning, administration, and change control to ensure that things on this special LUN continue to be things we care about replicating. When the focus shifts towards the VM, suddenly the “bigness” of a datastore that is many TBs in size no longer matters. Being granular to the VM or VMDK has removed datastore size from the conceptual design’s list of constraints.

The trickledown effect is that the same granularity principle applies to backups and recoveries and is not limited to replication. It frees up the design to allow for large pools of storage to be presented. VMware has made it very clear that Storage DRS, an excellent technology that I highly advocate, addresses many problems around balancing workloads and figuring out what spindles are actually serving up a datastore (via the SIOC Injector and VASA). Why not take it a step further and remove the issue entirely, by using large datastores that support VM-aware storage management features to reduce the number of objects you need to manage but still provides the per-VM granularity of storage management that you need?

Just make sure not to call it a best practice!