0 0

Ask Rick: Measuring Storage Usage the Easy Way

Rick Ehrhart is Tintri’s Developer Evangelist, Engineering. In his column, Rick solves common IT problems with one simple solution: Tintri storage.

How do I measure the amount of storage and IOPS I use for my private cloud infrastructure?

As a service provider or a manager of private cloud infrastructure, you may want to create reports or charge your clients based on the amount of storage used, as well as the IOPs used to obtain that storage. Fortunately, Tintri VMstore has those statistics for you at a VM level, not just in the UI, but also via APIs. This makes it easy to charge storage usage on a per-VM basis directly to the client.

One use case collects statistics at a certain time once a day. This will yield daily numbers which the service provider can use in any desired calculations, ranging from daily averages to charging on a per-day basis.

Another use case is to charge by the hour. Again, Tintri APIs can be used to either collect statistics once an hour or review historical statistics daily.

VMstore currently collects two types of statistics: historic and real time. Historic statistics are collected every 10 minutes on the VMstore. Seven days of historical statistics are kept on the VMstore, while 30 days are kept on Tintri Global Center (TGC). Real time statistics are collected at request time. The minimal request time for real time statistics is every 10 seconds. Real time statistics are collected from VMstore only.

There are 3 ways to collect VM statistics:

  1. /v310/vm/{vmId}/statsHistoric : can set from and to time
  2. /v310/vm/{vmId/statsRealtime : a one shot real time invoke
  3. /v310/vm : returns the last available historic statistics

The Data Transfer Object (DTO) is the same for all three cases. Let's look at some of the collected statistics.

  • spaceUsedLiveGiB: the logical space used for VM data, hypervisor snapshots, disk extent files, and swap files.
  • spaceUsedGib: the logical space used by spaceUsedLiveGiB and Tintri snapshots.
  • operationsTotalIops: the total number of I/Os per second.
  • normalizedToalIops: the total number of I/Os per second normalized to 8 KiB blocks
  • latencyTotalMs: the total latency consisting of host, network, storage, and throttle latencies. Storage latency consists of contention, flash, and disk latencies.

I have uploaded an API example on Tintri GitHub that can be used to collect statistics in a one shot fashion using /v310/vm . It can be used to collect statistics once an hour or once a day. The overall example flow is to collect all the VM information, and display the desired statistics. It works for both TGC and VMstore.

This example uses the following modules: tintri, available on Tintri's GitHub site, json, sys, and prettytable. It displays the preferred version, and collects all the VM information. The code below shows a pagination example collecting VM information. Here the query parameters are initialized to return only live VMs, start off at offset 0, and set the page size accordingly.

def get_vms(session_id):

    page_size = 25  # default    # dictionary of VM objects
    vms = {}

    # Get a list of VMs a page size at a time
    get_vm_url = "/v310/vm"
    count = 1
    vm_paginated_result = {'live' : "TRUE",
                           'next' : "offset=0&limit=" + str(page_size)}

    # While there are more VMs, go get them
    while 'next' in vm_paginated_result:
        url = get_vm_url + "?" + vm_paginated_result['next']
        print_debug("Next GET VM URL: " + str(count) + ": " + url)

        # Invoke the API
        r = tintri.api_get(server_name, url, session_id)

This snippet of code uses the API next property for looping. If there are more VMs to process, the /v310/vm API returns the next property which can be used as part of the URL to invoke the API. No counting pages or VMs, just rely on next

Later in the loop, the statistics are collected and stored in an object for later retrieval. Note the vm_stats = line. To collect the statistics, sortedStats is a list, and therefore [0] must be used. There is only one item in the sortedStats list.

        # Get and store the VM items and save in a VM object.
        items = vm_paginated_result["items"]
        for vm in items:
            vm_name = vm["vmware"]["name"]
            vm_uuid = vm["uuid"]["uuid"]
            vm_stats = VmStat(vm_name, vm_uuid, vm["stat"]["sortedStats"][0])
            print_debug(str(count) + ": " + vm_name + ", " + vm_uuid)
            count += 1

            # Store the VM stats object keyed by VM name.
            vms[vm_name] = vm_stats

The rest of code displays the VMs by name and their associated statistics in a table form. The line:

stat_fields = ['spaceUsedGiB', 'operationsTotalIops', 'latencyTotalMs']

determines which statistics fields to display. The field names are located in VirtualMachineStat DTO and can be found the API documentation. See my blog post on how to download the API documentation.

If integrating into your own reporting process, a database of client to VM names could be kept, or the VM name itself would indicate which client it belonged to. Also some hypervisors allow VM tags which would allow a client tag.

Enjoy spelunking,


Rick Ehrhart / Apr 15, 2015

Rick Ehrhart is Tintri’s Developer Evangelist with a strong software development background focusing on APIs. Rick has assisted many companies with API integrations including VMware, NetApp, HP, ...more