0 0

Tintri VMstore™ with VMware® vSphere and Microsoft Exchange Best Practice Guide

Intended Audience

This Best Practices Guide for running Microsoft Exchange Server 2013 (virtualized on VMware vSphere) on Tintri VMstore™ systems is intended to assist IT Administrators and Architects who are responsible for deploying and managing Microsoft Exchange Server within their virtual infrastructures powered by VMware vSphere and Tintri VMstore storage appliances. This guide focuses specifically on Exchange Server 2013 and vSphere 5.5 – and therefore all tests, instructions and screenshots are on those versions.

Executive Summary

Exchange Server is the market-leading mail server, calendar and contact manager, developed by Microsoft. In the past decade, storage systems have gone through revolutionary change—driven by flash, virtualization and cloud deployments. Unfortunately, traditional storage has changed far less, often hindering virtualization efforts. Specifically, Exchange Servers have notoriously been a challenge to virtualize, in great part because of limitations created by traditional storage systems, which impose a deal-breaking combination of performance issues and management complexity.

Tintri challenges the convention and the ‘storage quo’ with purpose built VMstore storage solutions for virtualized environments and cloud deployments. For organizations that have prioritized virtualization, Tintri VMstore systems provide an all-flash level of performance, VM-level visibility and simplified management.

This guide will walk you through the process of setting up Microsoft Exchange Server 2013, virtualized in a VMware vSphere 5.5 environment, using Tintri VMstore™ 3.1 for storage and provide best practice guidelines for successful deployments. This document provides guidelines so you can now not only confidently virtualize Exchange Servers without fear of end-user complaints or creating a management nightmare, but also eliminate cost and complexity as your Exchange deployments scale.

We designed and validated this Deployment and Best Practices around a 16,000-user Exchange Server environment, which would be a typical large enterprise deployment. However, almost all the best practice guidelines will also apply to smaller and larger environments – all the calculations that are done around the number of users are provided in enough detail that smaller or larger user numbers can be substituted.

Assumptions

This document assumes you are working with a fully configured virtual infrastructure.

It also assumes that a fully configured Active Directory infrastructure is in place – note that here Microsoft recommends the use of 64-bit Active Directory domain controllers to increase directory service performance for Exchange 2013.

Microsoft Exchange Server 2013 Overview

Exchange 2013 brings a number of changes, most notably in the architecture (elimination of the option for dedicated Hub Transport servers and Unified Messaging servers – these now run on Mailbox and Client Access servers), and management interface (The Exchange Management Console and the Exchange Control Panel have been replaced by the web-based Exchange Admin Center).

Exchange Server 2013 Architecture

Figure 1 - Exchange Server 2013 Architecture (credit: https://products.office.com/en-us/previous-versions/microsoft-exchange-2013)

For more details on the changes brought by Exchange Server 2013, please check Microsoft’s “Compare Exchange versions” table in https://products.office.com/en-us/previous-versions/microsoft-exchange-2013.

Consolidated List of Practices

The list below includes the recommended practices in this document. Click the text on any of the recommendations to jump to the section that corresponds to each recommendation for additional information.


DO: Use fully licensed vSphere servers – Standard, Enterprise or Enterprise Plus – to host your Virtual Exchange Servers.

DO: Use LSI Logic SAS virtual controllers in your Exchange VMs.

DON’T: Use PVSCSI virtual controllers in your Exchange VMs.

DO: Deploy a minimum of two Edge Transport Servers in your Exchange environment

DO: Distribute your Mailbox and Client Access Exchange Server VMs across a minimum of two ESXi hosts

DO: Use Tintri’s space efficient VM-level cloning to create the Client Access and Edge Transport Servers after creating a template for these servers.

DO: Use the latest version of Windows Server 2012 R2 to host Exchange Servers. Use full installtion of Windows Server rather than the Server Core mode.

DO: Make sure your base server VMs have all the prerequisites and updates in them before you clone them

DON’T: Clone VMs that already have Exchange fully installed in them; once you finalize an Exchange installation, the host name cannot be changed, not even through VMware Customization.

DON’T: Join to an AD Domain any VMs that you are going to install the Edge Transport Server role.

DON’T: Worry about Flash Hit % in Exchange Server 2013 environments; a lower Flash Hit % has a neglible performance impact on the Exchange Servers given their workload and IO patterns.

DO: Use LSI Logic SAS controllers for Exchange deployments on Tintri VMstore systems.

DON’T: Create more Mailbox Databases than strictly necessary (we recommend four per Exchange Mailbox Server, with each Database limited to 15 TiB). With Tintri VMstore, there is no benefit to or reason for having more than four Mailbox databases per server if each database will always be 15 TiB or smaller.

DO: Run LoadGen before you take an Exchange environment to production – and plan for plenty of time to do so.

DO: Use Tintri SnapVM snapshots to protect the Exchange database and the datafiles.

DO: Install and update to the latest version of Use VMware Tools

DO: Use a modern Backup tool that supports VADP and can perform Synthetic Full Backups (“synthesizes” a full backup from incremental data you already have on the backup repository)

DON’T: Run LoadGen on a production environment, since it will interfere with performance significantly.


Prerequisites

Licensing

VMware

Exchange deployments have become business critical for enterprises. Therefore, we do not recommend the use of the free ESXi edition (now named “vSphere Hypervisor”) as the hypervisor platform.


DO: Use fully licensed vSphere servers – Standard, Enterprise or Enterprise Plus – to host your Virtual Exchange Servers. VMware vSphere licenses are required for all servers on top of which Exchange Server will be installed.


As to which specific edition to use, the vSphere editions are compared on this page from VMware:

http://www.vmware.com/products/vsphere/compare

This Deployment and Best Practices Guide assumes a vSphere Standard license and does not cover any of the advanced features offered by the other vSphere editions.

VMware vCenter comes in two editions: vCenter Server Foundation – for small environments with 3 vSphere Servers or less – and vCenter Server Standard – for any environments larger than 3 vSphere servers.

Tintri VMstore

No special VMstore licenses are required for an Exchange deployment – unless advanced features (like replication) are required.

Microsoft Exchange Server 2013

An overview of the Exchange Server 2013 licensing options is available in

http://products.office.com/en-us/exchange/microsoft-exchange-server-licensing-licensing-overview

For more details and quotes, we recommend the Microsoft License Advisor, available through

http://mla.microsoft.com/default.aspx

Sizing Microsoft Exchange Server 2013 on vSphere

Compute Requirements

For determining the Compute Requirements for a Microsoft Exchange Server 2013 deployment, running on vSphere and a Tintri VMstore, we recommend the excellent “Microsoft Exchange 2013 on VMware Design and Sizing Guide”, available from https://www.vmware.com/content/dam/digitalmarketing/vmware/en/pdf/solutions/business-critical-apps/exchange/exchange-2013-on-vmware-design-and-sizing-guide.pdf

The above document provides all the necessary guidelines for compute sizing, and three sizing examples (for 12,000, 24,000 and 50,000 users).

To illustrate the compute sizing process, here’s a sizing example: 16,000 users in a single role server design:

Example Parameters for Sizing

  • Total mailboxes – 16,000

  • Mailbox quota – 2048MB

  • Average daily send/receive – 150 messages

  • Average message size of 75KB

  • High availability – Is out of scope for this document. HA using Exchange Database Availability Groups will be covered in a separate paper.

  • Database copies – 1

  • Sites – one site

  • Processor architecture – 16-core processor with a SPECint2006 rating of 70 per core (AMD OpteronTM 6380, 2.5 GHz, 16 cores - SPECint®_rate2006 = 1120)

Processor Core Requirements for Mailbox Servers

As per Table 1 in the VMware Design and Sizing Guide, for an average daily send/receive of 150 messages, 3 Megacycles per (active) mailbox are required.

Also according to the same document, the adjusted megacycles per core for our example CPU are (70 * 2500 / 18.75 =) 9,333 adjusted megacycles per core.

A 16,000 mailbox deployment will require 16,000 * 3 = 48,000 megacycles.

For 35% peak utilization, servers should be sized for 48,000 / 0.35 = 137,143 megacycles.

That comes out at (137,143 / 9,333 =) about 15 cores needed to keep it <35% utilization at peak, or, to round it up, 16 cores (which will yield about 32% CPU utilization at peak).

To put it another way: about 1 core per 1,000 users for the Mailbox Servers.

Memory Requirements for Mailbox Servers

Memory Requirements are much easier to calculate.

For 150 Messages Sent or Received per day, the guideline is 9 MB of cache per mailbox – for a total of 144 GB of cache needed.

According to Table 4 of the VMware Sizing Guide, that amount of cache is provided by a total server RAM of (144/92*128=) 200 GB.

That comes out at 12.5 GB per 1,000 users.

Requirements for Client Access Servers

Client Access Servers are stateless and have less complex needs. In fact, Microsoft recommends that, in physical deployments, the Client Access Role be simply added to the Mailbox server. In virtual deployments, however, VMware recommends, in their Design and Sizing Guide, Single Role servers to improve visibility and manageability.

For Single Role Client Access Servers, 4 cores and 8 GB of memory are recommended for each server. The Microsoft Exchange 2013 Server Role Requirements calculator – available from http://aka.ms/E2013Calc – can be used to calculate how many of these servers will be necessary for your environment. In our case of 16,000 users, the calculator came up with 3.

Network Requirements

Page 14 of the VMware Design and Sizing Guide explains that the very standard network requirements for Exchange on vSphere. In our test environment we met or exceeded those recommendations: we had two vNICs in each server VM, on top of ESXi hosts that have 4 physical 10 GbE network adapters teamed for redundancy and performance.

Mailbox Server Storage Requirements

There are two aspects to calculating the storage requirements: requirements defined by capacity needs and requirements defined by performance needs.

On the latter, both the VMware Design and Sizing Guide and the Microsoft requirements calculator where designed with legacy (HDD-based) storage, and all its performance limitations and concerns. However, with Tintri none of these problems exist; according to this Microsoft article with Exchange 2013 and our example of 150 messages per mailbox per day, only about 0.101 IOPS per mailbox are produced – totaling 1,616 IOPS for the whole deployment.

Given the high performance of VMstore systems, one of the key benefits of using Tintri for your Exchange storage needs is that performance is simply not a concern, whatever model you chose.

Storage capacity requirements are what really determine the VMstore model you’ll need for your Exchange deployments.

For that, we used the Microsoft Exchange 2013 Server Role Requirements calculator again, which suggests the following numbers for the 16,000-user example we’ve used:

  • Database Storage – 60 TiB

  • Log Storage – 5 TiB

Given Tintri’s architecture, those storage requirements of 60 TiB can be met with a single VMstore T880.

For flexibility purposes, and to allow for HA, we decided to spread the CPU and Memory requirements across 4 Mailbox Servers – handling 4,000 users each. Given the storage requirements of 60TiB, each VM needs 15 TiB for Exchange Database storage and 1.25 TiB for Log storage.

For better visibility and performance management, as well as more reliable backups and faster restores, we recommend 4 databases per Mailbox Server, each in its own VMDK. The Log Storage should also be split 4-ways, and kept in separate VMDKs, for better visibility and easier management. We further recommend that those 8 VMDKs (4 for Databases and 4 for Logs) be attached to their own Virtual Storage Controller.

As we will show in our Jetstress testing section, this configuration actually provides for optimal performance.

To store these 65 TiB (including database and log files), Client Access Servers, any temporary additional needs for Database restores, snapshots and other data resiliency needs, the ideal Tintri model for a 16,000 user Exchange deployment is therefore the VMstoreT880, which provides 100 TB of Effective Capacity in a 4RU rack space.

That brings us to discussing one final consideration with storage: what Virtual SCSI adapter to use in each of the Virtual Machines you’ll be building for your Exchange 2013 deployment.

VMware covers that in detail in page 15 of their Exchange 2013 on VMware Best Practices document:

“Exchange 2013 has greatly reduced the amount of I/O generated to access mailbox data, however storage latency is still a factor. In environments supporting thousands of users per Mailbox server, PVSCSI might prove beneficial. The decision on whether to use LSI Logic SAS or PVSCSI should be made based on Jetstress testing of the predicted workload using both adapters. Additionally, organizations must consider any management overhead an implementation of PVSCSI might introduce. Because many organizations have standardized on LSI Logic SAS, if the latency and throughput difference is negligible with the proposed configuration, the best option might be the one with the least impact to the current environment.“

As we will see during the Jetstress section of this document, using PVSCSI not only provided no benefit, performance was actually worse, so we recommend using the LSI Logic SAS virtual controllers.

Exchange Mailbox Server Virtual Machine

Figure 2 - Exchange Mailbox Server Virtual Machine Configuration


DO: Use LSI Logic SAS virtual controllers in your Exchange VMs.

DON’T: Use PVSCSI virtual controllers in your Exchange VMs.


Edge Transport Servers

A role in Exchange Server 2013 we haven’t yet covered in this sizing section is the “Edge Transport Server”.

Edge Transport servers minimize the attack surface by handling all Internet-facing mail flow, which provides SMTP (Simple Mail Transfer Protocol) relay and smart host services for your Exchange organization. Agents running on the Edge Transport server provide additional layers of message protection and security. These agents provide protection against viruses and spam and apply transport rules to control mail flow.

Because the Edge Transport server is installed in the perimeter network, it's never a member of your organization's internal Active Directory forest and doesn't have access to Active Directory information. However, the Edge Transport server requires data that resides in Active Directory—for example, connector information for mail flow and recipient information for antispam recipient lookup tasks. This data is synchronized to the Edge Transport server by the Microsoft Exchange EdgeSync service (EdgeSync). EdgeSync is a collection of processes run on an Exchange 2013 Mailbox server to establish one-way replication of recipient and configuration information from Active Directory to the Active Directory Lightweight Directory Services (AD LDS) instance on the Edge Transport server. EdgeSync copies only the information that's required for the Edge Transport server to perform anti-spam configuration tasks and to enable end-to-end mail flow. EdgeSync performs scheduled updates so the information in AD LDS remains current.

You can install more than one Edge Transport server in the perimeter network. Deploying more than one Edge Transport server provides redundancy and failover capabilities for your inbound message flow. You can load balance the SMTP traffic to your organization among Edge Transport servers by defining more than one MX record with the same priority value for your mail domain. You can achieve consistency in the configuration among multiple Edge Transport servers by using cloned configuration scripts.

Because Edge Transport Servers are designed to handle Internet-facing mail flow, their sizing will depend on your organization’s volume of Internet-facing email.

They are essentially stateless, so the sizing is really on the compute side – their need for storage is essentially for OS and fairly static application files, much like the Client Access Servers.

These Servers do compute-intensive work, such as Anti-spam and Antivirus protection, but unfortunately, the only guidance Microsoft provides for their sizing is for Exchange 2010 and is fairly vague.

The only recommendation they provide is that a minimum of two Edge Transport servers should be deployed for redundancy and to ensure uninterrupted service in case of planned or unplanned server downtime.


DO: Deploy a minimum of two Edge Transport Servers in your Exchange environment


Additionally, we recommend that each server should have the same Virtual Hardware configuration of the Client Access Servers, for consistency and ease of management.

Summary of Example Guest Virtual Machine Requirements

The above sizing resulted in the following Virtual Machine Configuration:

Exchange Role

Resources per Server

Mailbox Server – 4 Servers

- CPU – 4 vCPU (32% max utilization)
- Memory – 50 GiB
- OS and Application File Storage – 100 GiB (OS and

application files)
- Database Storage – 15 TiB (divided in 4x 3.75 TiB

VMDKs)
- Log Storage – 1.25 TiB (divided in 4x 320 GiB VMDKs) - Restore Drive – 3.75 TiB (Thin Provisioned)
- Network – 10 Gbps

Client Access Server – 3 Servers

 - CPU – 4 vCPU
- Memory – 8 GiB

- OS and Application File Storage – 80 GiB (OS and application files)

- Network – 10 Gbps

Edge Transport Server – 2 Servers

 - CPU – 4 vCPU
- Memory – 8 GiB

- OS and Application File Storage – 80 GiB (OS and application files)

- Network – 10 Gbps

Virtual Machine Distribution

To build availability into the architecture, ideally each of the mailbox servers should be in its own ESXi host – along with one of the Client Access Servers (i.e., 4 hosts for the above 7 or 8 VMs). However, given the associated costs – licensing and hardware – splitting over 2 hosts will in most cases provide almost the same level of availability while being more cost efficient; in our testing, we did use only 2 hosts – one with 2 of the Mailbox Servers and 1 of the Client Access Servers, and another with the remaining 2 Mailbox Servers and 2 Client Access Servers.


DO: Distribute your Mailbox and Client Access Exchange Server VMs across a minimum of two ESXi hosts


The Edge Transport Servers will need to be hosted in the DMZ, and, also for redundancy purposes, each in a different ESXi host.

Configuration

This section provides a walkthrough of building the example Exchange Server 2013 environment as sized and architected in the previous sections.

It is assumed that you’ve completed some prerequisites, including:

  1. Have the necessary vSphere servers with the necessary licenses.

  2. Have a fully configured vCenter Server managing all applicable vSphere servers, with the necessary licenses.

  3. Have configured the Tintri VMstore (a T880, in our example) as a datastore for all applicable vSphere servers.

a. We recommend that the VMstore be connected to the vSphere servers with dual active-active 10GbE connections (leveraging the VMstore’s dual controllers, each with dual 10GbE connections) as per Figure 2, below.

  1. Have procured the necessary Exchange Server 2013 licenses.

  2. Have a fully configured and functional Active Directory environment.

  3. Have collected all the necessary information to configure Exchange Server 2013, as outlined in the Exchange Server Deployment Assistant.

Network Configuration for vSphere servers and VMstore

Figure 3 - Network Configuration for vSphere servers and VMstore

Creating the Virtual Machines for the Exchange Servers

The first step is to create the necessary Virtual Machines. Note that Exchange Server 2013 does require a 64-bit Server Operating System, so bear that in mind when configuring the Virtual Machine Guest OS Family.

First up are the four Mailbox Servers.

In our example, the Mailbox Servers need to be created using the vSphere Web Client, since 4 of their Virtual Disks are bigger than 2 TiB; using the Web Client brings that and other benefits in terms of newer Virtual Hardware:

Mailbox Servers need to be created using the vSphere Web Client

Creation of Example Mailbox Server Virtual Machine

Figure 4 - Creation of Example Mailbox Server Virtual Machine

Continuation of creation of Example Mailbox Server Virtual Machine: Virtual Hardware

Figure 5 – Continuation of creation of Example Mailbox Server Virtual Machine: Virtual Hardware

The Client Access Servers and Edge Transport Servers do not need to be created using the Web Client (since they have small Virtual Hard Disks), but, again, using the Web Client allows you to use the newest Virtual Hardware.

Creation of Example Client Access Server or Edge Transport Server Virtual Machine

Figure 6 - Creation of Example Client Access Server or Edge Transport Server Virtual Machine

Our recommendation is that you only create one of each of the types of Virtual Machines (one Mailbox Server and one Client Access / Edge Transport Server) using the above process – and then simply clone them as many times as necessary after you’ve installed the OS and all prerequisites in each of them.


DO: Use Tintri’s space efficient VM-level cloning to create the Client Access and Edge Transport Servers after creating a template for these servers.


Install Operating System in the Virtual Machines

We recommend that Microsoft’s latest Server Operating System – Windows Server 2012 R2 – be used to host all Exchange Servers. Furthermore, Microsoft does not support the installation of Exchange 2013 on a computer that’s running in Windows Server Core mode. The computer must be running the full installation of Windows Server.

Exchange Server 2013 also supports Windows Server 2008 R2 and Windows Server 2012, as long as they are full (not core) installations.


DO: Use the latest version of Windows Server 2012 R2 to host Exchange Servers. Use full installtion of Windows Server rather than the Server Core mode.


Run Exchange Server 2013 installer prerequisite check in each base VM

Before Cloning the two base VMs you just created, run the Exchange Server 2013 installer so it goes through the “Readiness Check” section. That will install most prerequisite options and inform you of the only one that it won’t (the Microsoft Unified Communications Managed API 4.0, Core Runtime 64-bit).

Launching Exchange Setup to install prerequisites

Figure 7 - Launching Exchange Setup to install prerequisites

Microsoft Exchange Server 2013 configuring all the prerequisites

Figure 8 - Microsoft Exchange Server 2013 configuring all the prerequisites

Once the prerequisite configuration is finished you will get the following Readiness Check screen – you should abort the setup at this point:

Readiness Check Analysis Screen

Figure 9 - Readiness Check Analysis Screen

Proceed to install Microsoft Unified Communications Managed API 4.0, Core Runtime 64-bit from the provided URL:

Microsoft UCM Installer

Figure 10 - Microsoft UCM Installer

Run Windows Update in each base VM

Once the prerequisites are installed, run Windows Update in each base VM and reboot/rerun until all updates are applied.


DO: Make sure your base server VMs have all the prerequisites and updates in them before you clone them 


Clone VMs (with OS) as necessary

Once you have a base VM for the Mailbox Servers and a base VM for the Client Access / Edge Transport Servers, with a OS, prerequisites and updates installed, it is time to clone them as necessary, leveraging the VMware Customization Wizard to make sure they have unique SSIDs, names and IPs (note that you can’t change a server’s name once you’ve installed Exchange in it).


DON’T: Clone VMs that already have Exchange fully installed in them; once you finalize an Exchange installation, the host name cannot be changed, not even through VMware Customization.


In our case, we cloned the larger Mailbox Server VM three times (yielding 4 VMs) and the smaller VM four times (yielding 5 VMs – 3 for Client Access and 2 for Edge Transport).

NOTE: It is important to remember the Edge Transport Servers cannot be part of an Active Directory domain. As you run the VMware Customization Wizard do make sure the two VMs you are electing for the ETS role are part of a Workgroup, NOT an AD Domain.

DON’T: Join to an AD Domain any VMs that you are going to install the Edge Transport Server role.

On the other hand, all the Mailbox and Client Access Servers must be part of the same Active Directory domain.

VMs for Example Scenario

Figure 11 - VMs for Example Scenario

Jetstress Testing

Now that the VMs are built, this is a good point for testing the storage performance (before Exchange is actually installed, just in case adjustments need to be made).

According to Microsoft:

“Use Jetstress to verify the performance and stability of a disk subsystem prior to putting an Exchange server into production. Jetstress helps verify disk performance by simulating Exchange disk Input/Output (I/O) load. Specifically, Jetstress simulates the Exchange database and log file loads produced by a specific number of users. You use Performance Monitor, Event Viewer, and ESEUTIL in conjunction with Jetstress to verify that your disk subsystem meets or exceeds the performance criteria you establish. After a successful completion of the Jetstress Disk Performance and Stress Tests in a non-production environment, you will have ensured that your Exchange disk subsystem is adequately sized (in terms of performance criteria you establish) for the user count and user profiles you have established."

The details of how we performed Jetstress testing in our environment are in Appendix A – Jetstress Testing Details, in page 43.

Jetstress Test Results

The Exchange configuration passed Jetstress testing with flying colors.

Latency, as measured, by Jetstress was very low (about 1.1 ms read and 3.4 ms write, both well below the pass threshold) and the testing showed our configuration could actually handle up to 1 IOPS per mailbox, which is 10x the specified scenario (and would correspond to each and every user getting about 100 MB in emails a day).

The full test results are in Appendix A, in page 43, including screenshots from the Tintri VMstore GUI.

Considerations on Flash Hit %

One of the notable things we’ve observed during Jetstress testing is that the Flash Hit %, as reported by the Tintri GUI, had absolutely no impact on the Jetstress results: the one Mailbox Server that experienced very low Flash Hit % (80% or lower, caused by the very large logical footprint of the test databases – 15 TiB total in each server – and widely random read/write patterns that Jetstress generates), reported almost the same low latency (only a 0.2 ms increase) that the Mailbox Server that had 100% Flash Hit.

As we’ll cover in the section “Backups to Disk or Tape”, on page 40, it is also important to note that full Exchange backups will typically result in temporary decreases in Flash Hit %.


DON’T: Worry about Flash Hit % in Exchange Server 2013 environments; a lower Flash Hit % has a neglible performance impact on the Exchange Servers given their workload and IO patterns.


Considerations on SCSI controllers

As explained in the “Mailbox Server Storage Requirements” section, we had to use Jetstress to assess if the best practice was to use one controller for each VMDK, or one controller for all VMDKs, and if that controller was to be the standard LSI Logic SAS controller or the PVSCSI controller.

So besides the test we described above, we tested 3 other combinations: one LSI controller for each VMDK, one PVSCSI controller for all Exchange-specific VMDKs, and one PVSCSI controller for each of the 3 Exchange-specific VMDKs.

The first alternative scenario basically yielded exactly the same results – in other words, giving each VMDK its own LSI controller resulted in the same exact latency and performance, both reads and writes.

Use of PVSCSI controller increased both read latency (increased to over 5ms) and write latency (increased to 16 ms).

Therefore, our recommendation is to use the LSI Logic SAS controller for your Exchange 2013 deployments.


DO: Use LSI Logic SAS controllers for Exchange deployments on Tintri VMstore systems.


Install Exchange Server 2013 in each VM

Now that all the VMs are up and running, and we have validated that the architecture we have chosen will provide great performance, it is time to go into each of them actually run the Exchange Server 2013 installer to its completion, selecting the right role in each instance:

Server Role Selection

Figure 12 - Server Role Selection

This time the Readiness Check should give you an “install” button:

Successful Readiness Check

Figure 13 - Successful Readiness Check

... and after pressing that button the install should quickly complete:

Successful Exchange 2013 Setup Screen

Figure 14 - Successful Exchange 2013 Setup Screen

A quick and easy way to verify the installation was in fact successful is to open the Exchange Management shell and run “Get-ExchangeServer”:

Get-ExchangeServer command

Figure 15 - Get-ExchangeServer command

Configure Edge Subscriptions in the Edge Transport Servers

As explained above, because the Edge Transport Servers are outside the AD Domain the Mailbox and Client Access Servers are part of, they need to use an out-of-band synchronization mechanism, called EdgeSync. Setting up “Edge Subscriptions” configures this:

First step is to create the corresponding XML file in each ETS:

Creating Edge Subscription XML file in each ETS

Figure 16 - Creating Edge Subscription XML file in each ETS

Second step is to load each of the XML files into one of the CAS:

Loading Edge Subscription XML file in one of the CAS

Figure 17 - Loading Edge Subscription XML file in one of the CAS

Third Step is to kick off EdgeSync:

Kicking Off Edge Synchronization

Figure 18 - Kicking Off Edge Synchronization

License and Configure Exchange 2013

Once all the Exchange VMs are up and running, it is time to access the Exchange Admin Center through one of the Client Access Servers (via https://yourcasname/ecp/):

EAC Login Screen

Figure 19 - EAC Login Screen

First place you should go in the EAC is the “Servers” tab to verify all the servers are up and visible:

Servers Tab in EAC

Figure 20 - Servers Tab in EAC

The first thing that we need to do in this tab is to license each of the servers; that can be done by selecting each server individually and clicking the “Edit” button (a pencil icon):

Editing a Server

Figure 21 - Editing a Server

You will then get this screen, where you can input the license:

Entering License Key for Each Exchange Server

Figure 22 - Entering License Key for Each Exchange Server

The next step is to configure the Mailbox and Log Databases.

The reason you need to do this is that unfortunately during setup Exchange 2013 does not ask you where to put its Mailbox and Logs – it will simply, by default, install them in the same drive where the application files are installed.

You can correct this by going to the “databases” sub-tab under the “servers” tab and clicking the plus sign:

Adding new databases

Figure 23 - Adding new databases

This will bring you to this screen:

New Database Dialog Box

Figure 24 - New Database Dialog Box

In our example, the (large) drive for the first Mailbox Database is E:, and the (smaller) drive for the corresponding Logs is F:.

The Mailbox Servers, in our case, are named EXCH2013-1, EXCH2013-2, EXCH2013-3 and EXCH2013-4. We also deleted the default databases (that were residing in drive C: in each server).


DON’T: Create more Mailbox Databases than strictly necessary (we recommend four per Exchange Mailbox Server, with each Database limited to 15 TiB). With Tintri VMstore, there is no benefit to or reason for having more than four Mailbox databases per server if each database will always be 15 TiB or smaller.


The result is below:

Example Database Configuration

Figure 25 - Example Database Configuration

After the new mailbox databases are created, they need to be further configured, by pressing the pencil button. This will bring you to this dialog:

Mailbox Database configuration

Figure 26 - Mailbox Database configuration

In our case, as per our example, we configured each individual mailbox to be limited to 2 GB, with a warning sent to the user at 1.9 GB.

Load Generator 2013 Testing

Previous sections showed the entire configuration that was needed.

The next testing stage is to run Load Generator 2013 (LoadGen) against the fully configured deployment.

From Microsoft’s description of LoadGen:

“Use Microsoft Exchange Load Generator 2013 (LoadGen) as a simulation tool to measure the impact of MAPI, OWA, ActiveSync, IMAP, POP and SMTP clients on Exchange 2013 servers. LoadGen allows you to test how a server running Exchange 2013 responds to e-mail loads. To simulate the delivery of these messaging requests, you run LoadGen tests on client computers. These tests send multiple messaging requests to the Exchange server, thereby causing a mail load. LoadGen is a useful tool for administrators who are sizing servers and validating a deployment plan. Specifically, LoadGen helps you determine if each of your servers can handle the load to which they are intended to carry. Another use for LoadGen is to help validate the overall solution.

Important! LoadGen should be used only in laboratories that have no connection to the production environment. This tool should not be used in any way in a production environment or an environment that is mission critical or contains important information of any kind anywhere in the network. “

LoadGen is a complementary tool to Jetstress, and is mostly designed to test compute, not storage. In fact, Microsoft specifically says, in the LoadGen documentation:

“LoadGen will not replicate the correct IOPs per user. For IOP testing, we recommend you use Microsoft Exchange Server Jetstress.”

We highly recommend that you run LoadGen before deploying Exchange in production, because:

  1. Validates the compute sizing calculations

  2. Allows you to better measure the impact of Database Maintenance operations, as we will discuss in the section “Database Maintenance Considerations”.


DO: Run LoadGen before you take an Exchange environment to production – and plan for plenty of time to do so.


The LoadGen tool, and its documentation, is available from:

http://www.microsoft.com/en-us/download/details.aspx?id=40726

Details on performing LoadGen testing are included in Appendix B – LoadGen Testing Details, in page 54.

Database Maintenance Considerations

Exchange has always had the capacity to perform background maintenance. Since Exchange 2010 Extensible Storage Engine (ESE) maintenance carried out on internal databases is done on a 24 x 7 basis by default rather than within a predefined time window. However, with Exchange 2010, that background process consumed about 5 MB/s per database, but Exchange 2013 only consumes 1 MB/s.

Maintenance operations are essential for an Exchange database because they:

  • Remove items and mailboxes from the database (a hard delete) after their retention time expires

  • Discover pages that were previously occupied by deleted items and mailboxes, and free up these pages for reuse by the database

  • Validate checksums on pages to ensure that they aren’t corrupt

 

The non-essential operation that Exchange automatically performs as part of maintenance is online defragmentation – which was important to maintain performance when storage was hard drive-based, but is actually counterproductive with flash based storage. Unfortunately, Microsoft doesn’t give users the option of disabling defragmentation separately from the essential maintenance operations.

Background (24x7) Maintenance vs. Scheduled Maintenance

Leaving the default of 24x7 maintenance alone is clearly the best option on traditional HDD based storage systems; it could be problematic to rely on a specific time window for completing the maintenance tasks given the performance constraints of such systems. This problem grows linearly as database grows, so as the database gets bigger the only solution is to assign a larger time window in hopes that you keep pace with the work.

Even though the background database maintenance can be disabled (and instead scheduled) any mention or discussion of this in the Exchange Server 2013 documentation has been removed – essentially, 24x7 background maintenance is a given for Microsoft in Exchange Server 2013.

Enabling/Disabling 24x7 background database maintenance

Figure 27 – Enabling/Disabling 24x7 background database maintenance

With the Tintri VMstore, however, the choice is not as clear; due to the copious amounts of spare performance, maintenance will take very little time so a short window will be enough, and the continuous impact of 24x7 maintenance may be a concern for some end users. Having said that, according to our testing, running 24x7 maintenance will increase latency by only 0.2 ms.

This increase in latency comes mostly from lower flash hit % – the background maintenance process will be working on older data that has likely read from HDDs. As we’ve discussed in the “Considerations on Flash Hit %” section, a 0.2 ms increase in latency doesn’t really translate into a perceivable loss of performance by users.

One of the major advantages of the architecture we are presenting as a Best Practice, is that by using 4 databases per Mailbox server and, more importantly, putting each of these databases in their own VMDK (and the Log Files in yet another VMDK), performance statistics – including latency – can be monitored at a very granular level using the Tintri UI. This is illustrated in Figure 28, below.

Monitoring Latency with the Tintri UI at a very granular level

Figure 28 - Monitoring Latency with the Tintri UI at a very granular level

Exchange Protection and Backup Options

The Tintri VMstore provides two VM protection tools: SnapVM and ReplicateVM. Tintri SnapVM creates instantaneous snapshot backups, and Tintri ReplicateVM replicates snapshots to another VMstore, providing the necessary second offsite copy of the snapshot backups.

Detailed best practices for Tintri VMstore backups are covered in the white paper Backup and Recovery Best Practices with Tintri VMstore.

Exchange Protection and Backup Options

Figure 29 - Exchange Protection and Backup Options

VMware Snapshots

VMware snapshot technology preserves the state of a virtual machine at a point in time. The snapshot includes the virtual machine’s state and the files that make up the virtual machine.

However, a virtual machine snapshot is not a complete copy of the original VMDK disk files, rather VMware maintains changes to the VMDK in the snapshot. On high-transaction virtual machines, such as database servers, the existence of a snapshot will affect the performance of the virtual machine. For this reason VMware recommends that no single snapshot exist for more than 24-72 hours and they be deleted.

More information on VMware snapshots can be found in the following VMware KB articles:

Understanding virtual machine snapshots in VMware ESXi and ESX (KB Article 1015180)
Best practices for virtual machine snapshots in the VMware environment (KB Article 1025279)

Tintri SnapVM™ Snapshots

Tintri’s SnapVM snapshots provide a simple and powerful on-line backup tool for Exchange VM and databases. The SnapVM snapshot technology is unique in being able to operate on individual Virtual Machines to create point-in-time, read-only copies of individual VMs and the associated VMDKs. Tintri SnapVM snapshots make it easy to backup, clone, restore, and mirror entire Exchange databases.

Tintri’s SnapVM snapshot technology is:

  • Granular - SnapVM snapshots apply to individual VMs, not the entire storage appliance.

  • High performance - SnapVM snapshots do not affect the performance of the VM.

  • Space efficient - Storage is consumed only when the VM writes changes its VMDK file(s).

  • Flexible - Restoring a snapshot doesn’t delete newer, or older snapshots. All recovery points continue to be available.

  • Automated – Configured hourly, daily, weekly, monthly, on-demand or called via a script.

Tintri supports two types of SnapVM snapshots: VM-Consistent and Crash Consistent. VM-Consistent snapshots leverage VMware Tools to quiesce the Exchange database and the virtual machine before creating the snapshot. Crash-Consistent snapshots are created instantly and ignore the state of the virtual machine.

When applied to an Exchange database, SnapVM snapshots quickly create point-in-time copies of the entire Exchange VM and associated VMDK files, including all database executables and database datafiles. Consider using the SnapVM snapshots in the following ways:

  • Use VM-Consistent snapshots to create a “golden copy” of a production database prior to upgrading Exchange database executables.

  • Use VM-Consistent snapshots to create a source snapshot for making clones of the database.

  • Use Crash-Consistent snapshots to create on-line point-in-time copies of the database, which provide multiple on-line restore options. The SnapVM snapshots incur no performance penalty so they can remain on the storage for long periods of time with no effect on the production database.


DO: Use Tintri SnapVM snapshots to protect the Exchange database and the datafiles.


Tintri SnapVM VM-Consistent Snapshots

A VM-Consistent snapshot is taken when the virtual machine has been quiesced. The ability to quiesce a virtual machine is made possible when VMware Tools is installed in the virtual machine. A quiesce operation will flush in-guest memory buffers and quiesce the storage I/O in the virtual machine, making it possible to use a storage-based snapshot to capture the complete state of the virtual machine and all of its data files.

VM-Consistent snapshots can be scheduled or invoked from the Tintri VMstore UI. When the Tintri SnapVM command is initiated, the following actions are executed:

  1. A VMware snapshot is taken with the “Quiesce Guest File System” option selected.

  2. A Tintri SnapVM snapshot is created.

  3. The VMware snapshot is deleted so there is no performance impact on the virtual machine.

Snapshot Dialog, showing the VM-consistent snapshot option, which quiesces the Guest File System

Figure 30 - Snapshot Dialog, showing the VM-consistent snapshot option, which quiesces the Guest File System


DO: Install and update to the latest version of Use VMware Tools


Tintri SnapVM Crash-Consistent Snapshots

When Tintri SnapVM Crash-Consistent snapshots are used, no attempt is made to flush in-memory I/O buffers for guest applications or guest operating systems. When this type of snapshot is used with an active Exchange database the resulting snapshot will contain an image of the database that is equivalent to an Exchange database that has crashed.

Exchange databases are designed to be recoverable from a crashed state and experience shows that SnapVM Crash-Consistent will be completely recoverable and usable. This value of this type of snapshot comes from the fact that it executes very quickly and does not disrupt the I/O to the production database. It executes quickly because it does not freeze the I/O on the virtual machine, nor does it execute the pre-freeze or post-freeze scripts.

Because they execute so quickly, Exchange administrators can take advantage of its non-disruptive nature to create multiple snapshot copies of the database each day using a round-robin schedule. These snapshots provide an online safety net and provide multiple on-line recovery points for the production database.

Tintri ReplicateVM™ for VM-Level Replication

Tintri’s ReplicateVM performs WAN-efficient replication of a VM from one VMstore to another. If a VM is configured for replication, any snapshots created by that VM would be automatically replicated to the destination VMstore. Only changed blocks between snapshots are replicated after deduplication and compression reducing the WAN bandwidth utilization by up to 95%. A replicated VM is called a synthetic VM on the destination VMstore and can be materialized into a virtual machine from any of the snapshots available.

When ReplicateVM is used to replicate a production Exchange database, the result is a powerful disaster recovery solution that is simple to setup.

Information on ReplicateVM is found at: http://www.tintri.com/products/replicatevm

The Tintri VM Protection Dialog Box

Figure 31 – The Tintri VM Protection Dialog Box

Backups to Disk or Tape (Image-Level or Guest-Level)

Image-Level and Guest-Level backups are discussed at length in the Backup and Recovery Best Practices with Tintri VMstore whitepaper.

Microsoft Exchange presents a particular case since performing regular Image-Level or Guest-Level backups is considered by many compulsory – given it is the only way to clear out accumulating Logs, outside of using Circular Logging.

Performing traditional backups usually means performing regular full backups, which does present two huge performance challenges:

  1. As Exchange databases grow, the amount of data that needs to be moved (read and written) grows too, and full Backups become longer and resource consuming (storage I/O and network bandwidth).

  2. A full Backup means reading all the databases in entirety, which, given Tintri’s architecture, will result in a temporary impact on Flash Hit % as data is read from HDDs.

For these two reasons, we strongly recommend that our customers use a modern backup solution, such as Veeam Backup & Replication, that supports VMware vStorage API for Data Protection (VADP, that provides features like Changed Block Tracking) and doesn’t require actual full backups. These backup applications can perform a “Synthetic Full Backup” from base line full backup and subsequent incremental backups.

When you perform synthetic full backup, Veeam Backup & Replication does not read all VM data from the source datastore. Instead, it “synthesizes” a full backup from data you already have on the backup repository. Veeam Backup & Replication accesses the previous full backup file and a chain of subsequent increments on the backup repository, consolidates VM data from these files and writes consolidated data into a new full backup file. As a result, the created synthetic full backup file contains the same data you would have if you created a full backup in a regular manner.


DO: Use a modern Backup tool that supports VADP and can perform Synthetic Full Backups (“synthesizes” a full backup from incremental data you already have on the backup repository)


If using a tool like Veeam Backup & Replication is not an option, we recommend that you pick your backup windows very carefully – to match the absolute lowest user traffic periods – and monitor performance particularly carefully; as the database grows, backup windows will quickly become the points in time with the highest storage I/O and latency.

Conclusion

Tintri VMstore was designed from the ground up for virtualization and cloud workloads, and purpose- built to take full advantage of flash technology. Tintri’s legendary ease of use and low TCO extends to Microsoft Exchange Server too: you can have your Exchange environment up and running in record time, with recording breaking simplicity – no need to dozens or even hundreds of LUNs, like with legacy solutions – small footprint (4RUs in storage for 16,000 users!), low maintenance and therefore an unbeatable low TCO.

These Tintri best practices provide the information required for a successful deployment of a virtualized Exchange Servers with the Tintri VMstore. Tintri fully supports the use of virtualized Exchange databases so administrators can leverage Tintri’s native per virtual machine data management capabilities for deploying, maintaining and troubleshooting.

The Tintri VMstore’s feature rich UI provides a virtualized data center administrator the tools to effectively and efficiently monitor, manage and troubleshoot virtual machines. The ability to visualize performance bottlenecks across all layers of the infrastructure, the ability to track performance and capacity utilization on a per virtual machine and per virtual disk basis gives virtualization and database administrators the tools necessary to simplify and scale Exchange Server deployments.

The use of Tintri VMstore’s SnapVM™, CloneVM™, and ReplicateVM™ functionalities ensures that virtualized Exchange Servers can be protected locally and remotely between multiple VMstores and across data centers. Given the hypervisor agnostic nature of Tintri VMstore systems, the same VM-level data management functionality can be used across multiple hypervisors deployed on a single Tintri VMstore. We hope the information in this document and the best practice guidelines helps you get the most from your Exchange deployments on Tintri VMstore systems.

Thank you for choosing Tintri. We hope the options and guidance presented in this guide enable you to leverage this new technology. 

References

Tintri

Tintri VMstore System Administration Manual

VMware

Microsoft Exchange 2013 on VMware Best Practices Guide

Microsoft Exchange 2013 on VMware Design and Sizing Guide

Microsoft

Exchange Server 2013 Deployment Assistant

Microsoft Exchange Server 2013 Documentation

Jetstress 2013

LoadGen 2013

Appendix A – Jetstress Testing Details

Configuring Jetstress

Jetstress is a fairly easy tool to setup and use. All we had to do was install a copy of Jetstress (available from http://www.microsoft.com/en-us/download/details.aspx?id=36849) in each of our elected mailbox servers (along with and run the test scenarios set by our example:

Jetstress Welcome Screen

Figure 32 - Jetstress Welcome Screen

Setting the Jetstress XML Configuration File

Figure 33 - Setting the Jetstress XML Configuration File

Our objective was to test our example Exchange mailbox profile, and therefore we chose that option in Jetstress:

Define Test Scenario in Jetstress

Figure 34 - Define Test Scenario in Jetstress

The next screen is about defining the Exchange Mailbox Profile itself. Here, two things to bear in mind:

  • Because we are running a Jetstress instance in each of the 4 Mailbox Servers, each instance should only test (16,000 / 4 =) 4,000 mailboxes

  • As we saw above, in our scenario Exchange 2013 will only be generating 0.101 IOPS per mailbox. However, we wanted to test a little breathing room so we we configured the test for 0.18 IOPS/mailbox.

Exchange Mailbox Profile to be tested

Figure 35 - Exchange Mailbox Profile to be tested

In the Test Type Screen we selected the appropriate options:

Jetstress test type

Figure 36 - Jetstress test type

We configured for a 2 hour test run:

Test Run settings

Figure 37 - Test Run settings

The next configuration part of Jetstress is about the Database Configuration:

Jetstress Database Configuration

Figure 38 - Jetstress Database Configuration

Next, it was time to generate the databases:

Select Database Source

Figure 39 - Select Database Source

... and to actually run the test; note that we ran the (identical) tests on all 4 mailbox servers simultaneously.

Select Database Source

Jetstress Test Report

Microsoft Exchange Jetstress 2013

Performance Test Result Report

Test Summary

Overall Test Result Pass

Machine Name EXCH2013-1

Test Description

Test Start Time 1/22/2015 10:29:25 AM

Test End Time 1/23/2015 7:57:52 AM

Collection Start Time 1/23/2015 3:46:38 AM

Collection End Time 1/23/2015 5:46:16 AM

Jetstress Version 15.00.0995.000

ESE Version 15.00.0847.030

Operating System Windows Server 2012 R2 Standard (6.2.9200.0)

Performance Log C:\Program Files\Exchange Jetstress\Tuning_2015_1_23_3_30_38.blg      C:\Program Files\Exchange Jetstress\Performance_2015_1_23_3_46_20.blg

Database Sizing and Throughput 

Achieved Transactional I/O per Second 958.599

Target Transactional I/O per Second 720

Initial Database Size (bytes) 8589968146432

Final Database Size (bytes) 8595613679616

Database Files (Count) 4

Jetstress System Parameters

Thread Count 2

Minimum Database Cache 128.0 MB

Maximum Database Cache 1024.0 MB

Insert Operations 40%

Delete Operations 20%

Replace Operations 5%

Read Operations 35%

Lazy Commits 70%

Run Background Database Maintenance  True

Number of Copies per Database 1

 

Database Configuration

Instance11824.1 Log path: I:\Jetstress Log 1 Database: H:\Jetstress001001.edb

Instance11824.2 Log path: K:\Jetstress Log 2 Database: J:\Jetstress002001.edb

Instance11824.3 Log path: M:\Jetstress Log 3 Database: L:\Jetstress003001.edb

Instance11824.4 Log path: O:\Jetstress Log 4 Database: N:\Jetstress004001.edb

 

Transactional I/O Performance

Transactional I/O Performance

Background Database Maintenance I/O Performance

Background Database Maintenance I/O Performance

Log Replication I/O Performance

Log Replication I/O Performance

Total I/O Performance

Total I/O Performance

Host System Performance

Host System Performance
Host System Performance

 

Test Log 1/22/2015 10:29:25 AM -- Preparing for testing... 1/22/2015 10:29:25 AM -- Creating H:\Jetstress001001.edb.

1/22/2015 10:29:25 AM -- Database cache settings: (minimum: 32.0 MB, maximum: 256.0 MB) 1/22/2015 10:29:25 AM -- Database flush thresholds: (start: 2.5 MB, stop: 5.1 MB)
1/22/2015 5:17:24 PM -- 100.0% of 2.0 TB complete (683518984 records inserted).
1/22/2015 5:17:24 PM -- 100.0% of 2.0 TB complete (683519677 records inserted).

1/22/2015 5:17:27 PM -- Duplicating 4 databases:
1/23/2015 3:29:44 AM -- 100.0% of 7.8 TB complete (7.8 TB duplicated).
1/23/2015 3:29:49 AM -- Attaching databases...
1/23/2015 3:29:49 AM -- Preparations for testing are complete.
1/23/2015 3:29:49 AM -- Starting transaction dispatch..
1/23/2015 3:29:49 AM -- Database cache settings: (minimum: 128.0 MB, maximum: 1.0 GB)
1/23/2015 3:29:49 AM -- Database flush thresholds: (start: 10.2 MB, stop: 20.5 MB)
1/23/2015 3:29:53 AM -- Database read latency thresholds: (average: 20 msec/read, maximum: 100 msec/read).
1/23/2015 3:29:53 AM -- Log write latency thresholds: (average: 10 msec/write, maximum: 100 msec/write). 1/23/2015 3:29:53 AM -- Attaining prerequisites:
1/23/2015 3:30:38 AM -- \MSExchange Database(JetstressWin)\Database Cache Size, Last: 983896100.0 (lower bound: 966367600.0, upper bound: none)
1/23/2015 3:30:41 AM -- Performance logging started (interval: 5000 ms).
1/23/2015 3:30:41 AM -- Automatic tuning is starting...
1/23/2015 3:31:31 AM -- Instance11824.1 has 0.01 for read latency slope.
1/23/2015 3:31:31 AM -- Instance11824.2 has 0.04 for read latency slope.
1/23/2015 3:31:31 AM -- Instance11824.3 has 0.03 for read latency slope.
1/23/2015 3:31:31 AM -- Instance11824.4 has 0.01 for read latency slope.
1/23/2015 3:34:50 AM -- 1576 batch transactions/sec and 16 sessions have 6591 IOPS.
1/23/2015 3:34:50 AM -- 16 sessions have actual 6591 IOPS (target IOPS: 720)
1/23/2015 3:34:50 AM -- JetstressWin/Instance11824.1 has 2.4 for I/O Database Reads Average Latency.
1/23/2015 3:34:50 AM -- JetstressWin/Instance11824.2 has 2.4 for I/O Database Reads Average Latency.
1/23/2015 3:34:50 AM -- JetstressWin/Instance11824.3 has 2.1 for I/O Database Reads Average Latency.
1/23/2015 3:34:50 AM -- JetstressWin/Instance11824.4 has 2.3 for I/O Database Reads Average Latency.
1/23/2015 3:34:50 AM -- JetstressWin/Instance11824.1 has 1.7 for I/O Log Writes Average Latency.
1/23/2015 3:34:50 AM -- JetstressWin/Instance11824.1 has 0.7 for I/O Log Reads Average Latency.
1/23/2015 3:34:50 AM -- JetstressWin/Instance11824.2 has 1.7 for I/O Log Writes Average Latency.
1/23/2015 3:34:50 AM -- JetstressWin/Instance11824.2 has 0.6 for I/O Log Reads Average Latency.
1/23/2015 3:34:50 AM -- JetstressWin/Instance11824.3 has 1.6 for I/O Log Writes Average Latency.
1/23/2015 3:34:50 AM -- JetstressWin/Instance11824.3 has 0.7 for I/O Log Reads Average Latency.
1/23/2015 3:34:50 AM -- JetstressWin/Instance11824.4 has 1.6 for I/O Log Writes Average Latency.
1/23/2015 3:34:50 AM -- JetstressWin/Instance11824.4 has 0.7 for I/O Log Reads Average Latency.
1/23/2015 3:34:50 AM -- Operation mix: Sessions 8, Inserts 40%, Deletes 20%, Replaces 5%, Reads 35%, Lazy Commits 70%.
1/23/2015 3:35:37 AM -- Instance11824.1 has 0.01 for read latency slope.
1/23/2015 3:35:37 AM -- Instance11824.2 has 0.00 for read latency slope.
1/23/2015 3:35:37 AM -- Instance11824.3 has 0.01 for read latency slope.
1/23/2015 3:35:37 AM -- Instance11824.4 has 0.01 for read latency slope.
1/23/2015 3:38:46 AM -- 1080 batch transactions/sec and 8 sessions have 4265 IOPS.
1/23/2015 3:38:46 AM -- 8 sessions have actual 4265 IOPS (target IOPS: 720)

1/23/2015 3:38:46 AM -- JetstressWin/Instance11824.1 has 1.7 for I/O Database Reads Average Latency.
1/23/2015 3:38:46 AM -- JetstressWin/Instance11824.2 has 1.6 for I/O Database Reads Average Latency.
1/23/2015 3:38:46 AM -- JetstressWin/Instance11824.3 has 1.6 for I/O Database Reads Average Latency.
1/23/2015 3:38:46 AM -- JetstressWin/Instance11824.4 has 1.7 for I/O Database Reads Average Latency.
1/23/2015 3:38:46 AM -- JetstressWin/Instance11824.1 has 1.3 for I/O Log Writes Average Latency.
1/23/2015 3:38:46 AM -- JetstressWin/Instance11824.1 has 0.4 for I/O Log Reads Average Latency.
1/23/2015 3:38:46 AM -- JetstressWin/Instance11824.2 has 1.3 for I/O Log Writes Average Latency.
1/23/2015 3:38:46 AM -- JetstressWin/Instance11824.2 has 0.4 for I/O Log Reads Average Latency.
1/23/2015 3:38:46 AM -- JetstressWin/Instance11824.3 has 1.3 for I/O Log Writes Average Latency.
1/23/2015 3:38:46 AM -- JetstressWin/Instance11824.3 has 0.5 for I/O Log Reads Average Latency.
1/23/2015 3:38:46 AM -- JetstressWin/Instance11824.4 has 1.3 for I/O Log Writes Average Latency.
1/23/2015 3:38:46 AM -- JetstressWin/Instance11824.4 has 0.4 for I/O Log Reads Average Latency.
1/23/2015 3:38:46 AM -- Operation mix: Sessions 4, Inserts 40%, Deletes 20%, Replaces 5%, Reads 35%, Lazy Commits 70%.

1/23/2015 3:39:32 AM -- Instance11824.1 has 0.01 for read latency slope.
1/23/2015 3:39:32 AM -- Instance11824.2 has 0.01 for read latency slope.
1/23/2015 3:39:32 AM -- Instance11824.3 has 0.01 for read latency slope.
1/23/2015 3:39:32 AM -- Instance11824.4 has 0.01 for read latency slope.
1/23/2015 3:42:35 AM -- 668 batch transactions/sec and 4 sessions have 2506 IOPS.
1/23/2015 3:42:35 AM -- 4 sessions have actual 2506 IOPS (target IOPS: 720)
1/23/2015 3:42:35 AM -- JetstressWin/Instance11824.1 has 1.4 for I/O Database Reads Average Latency.
1/23/2015 3:42:35 AM -- JetstressWin/Instance11824.2 has 1.3 for I/O Database Reads Average Latency.
1/23/2015 3:42:35 AM -- JetstressWin/Instance11824.3 has 1.4 for I/O Database Reads Average Latency.
1/23/2015 3:42:35 AM -- JetstressWin/Instance11824.4 has 1.4 for I/O Database Reads Average Latency.
1/23/2015 3:42:35 AM -- JetstressWin/Instance11824.1 has 1.2 for I/O Log Writes Average Latency.
1/23/2015 3:42:35 AM -- JetstressWin/Instance11824.1 has 0.2 for I/O Log Reads Average Latency.
1/23/2015 3:42:35 AM -- JetstressWin/Instance11824.2 has 1.2 for I/O Log Writes Average Latency.
1/23/2015 3:42:35 AM -- JetstressWin/Instance11824.2 has 0.3 for I/O Log Reads Average Latency.
1/23/2015 3:42:35 AM -- JetstressWin/Instance11824.3 has 1.2 for I/O Log Writes Average Latency.
1/23/2015 3:42:35 AM -- JetstressWin/Instance11824.3 has 0.2 for I/O Log Reads Average Latency.
1/23/2015 3:42:35 AM -- JetstressWin/Instance11824.4 has 1.2 for I/O Log Writes Average Latency.
1/23/2015 3:42:35 AM -- JetstressWin/Instance11824.4 has 0.3 for I/O Log Reads Average Latency.
1/23/2015 3:42:35 AM -- Operation mix: Sessions 2, Inserts 40%, Deletes 20%, Replaces 5%, Reads 35%, Lazy Commits 70%.

1/23/2015 3:43:20 AM -- Instance11824.1 has 0.00 for read latency slope.
1/23/2015 3:43:20 AM -- Instance11824.2 has 0.01 for read latency slope.
1/23/2015 3:43:20 AM -- Instance11824.3 has 0.01 for read latency slope.
1/23/2015 3:43:20 AM -- Instance11824.4 has 0.00 for read latency slope.
1/23/2015 3:46:20 AM -- 374 batch transactions/sec and 2 sessions have 915 IOPS.
1/23/2015 3:46:20 AM -- 2 sessions have actual 915 IOPS (target IOPS: 720)
1/23/2015 3:46:20 AM -- JetstressWin/Instance11824.1 has 1.1 for I/O Database Reads Average Latency.
1/23/2015 3:46:20 AM -- JetstressWin/Instance11824.2 has 1.1 for I/O Database Reads Average Latency.
1/23/2015 3:46:20 AM -- JetstressWin/Instance11824.3 has 1.1 for I/O Database Reads Average Latency.
1/23/2015 3:46:20 AM -- JetstressWin/Instance11824.4 has 1.2 for I/O Database Reads Average Latency.
1/23/2015 3:46:20 AM -- JetstressWin/Instance11824.1 has 0.9 for I/O Log Writes Average Latency.
1/23/2015 3:46:20 AM -- JetstressWin/Instance11824.1 has 0.2 for I/O Log Reads Average Latency.
1/23/2015 3:46:20 AM -- JetstressWin/Instance11824.2 has 0.8 for I/O Log Writes Average Latency.
1/23/2015 3:46:20 AM -- JetstressWin/Instance11824.2 has 0.1 for I/O Log Reads Average Latency.
1/23/2015 3:46:20 AM -- JetstressWin/Instance11824.3 has 0.8 for I/O Log Writes Average Latency.
1/23/2015 3:46:20 AM -- JetstressWin/Instance11824.3 has 0.1 for I/O Log Reads Average Latency.
1/23/2015 3:46:20 AM -- JetstressWin/Instance11824.4 has 0.9 for I/O Log Writes Average Latency.
1/23/2015 3:46:20 AM -- JetstressWin/Instance11824.4 has 0.1 for I/O Log Reads Average Latency.
1/23/2015 3:46:20 AM -- Performance logging has ended.

1/23/2015 3:46:20 AM -- Automatic tuning succeeded.
1/23/2015 3:46:23 AM -- Operation mix: Sessions 2, Inserts 40%, Deletes 20%, Replaces 5%, Reads 35%,

Lazy Commits 70%.
1/23/2015 3:46:23 AM -- Performance logging started (interval: 15000 ms).
1/23/2015 3:46:23 AM -- Attaining prerequisites:
1/23/2015 3:46:23 AM -- \MSExchange Database(JetstressWin)\Database Cache Size, Last: 1064706000.0 (lower bound: 966367600.0, upper bound: none)
1/23/2015 5:46:24 AM -- Performance logging has ended.
1/23/2015 7:57:51 AM -- JetInterop batch transaction stats: 102226, 102225, 102225 and 102225. 1/23/2015 7:57:51 AM -- Dispatching transactions ends.
1/23/2015 7:57:51 AM -- Shutting down databases...
1/23/2015 7:57:52 AM -- Instance11824.1 (complete), Instance11824.2 (complete), Instance11824.3 (complete) and Instance11824.4 (complete)
1/23/2015 7:57:52 AM -- C:\Program Files\Exchange Jetstress\Performance_2015_1_23_3_46_20.blg has 478 samples.
1/23/2015 7:57:52 AM -- Creating test report...
1/23/2015 7:57:57 AM -- Instance11824.1 has 1.1 for I/O Database Reads Average Latency.
1/23/2015 7:57:57 AM -- Instance11824.1 has 0.8 for I/O Log Writes Average Latency.
1/23/2015 7:57:57 AM -- Instance11824.1 has 0.8 for I/O Log Reads Average Latency.
1/23/2015 7:57:57 AM -- Instance11824.2 has 1.1 for I/O Database Reads Average Latency.
1/23/2015 7:57:57 AM -- Instance11824.2 has 0.8 for I/O Log Writes Average Latency.
1/23/2015 7:57:57 AM -- Instance11824.2 has 0.8 for I/O Log Reads Average Latency.
1/23/2015 7:57:57 AM -- Instance11824.3 has 1.1 for I/O Database Reads Average Latency.
1/23/2015 7:57:57 AM -- Instance11824.3 has 0.8 for I/O Log Writes Average Latency.
1/23/2015 7:57:57 AM -- Instance11824.3 has 0.8 for I/O Log Reads Average Latency.
1/23/2015 7:57:57 AM -- Instance11824.4 has 1.2 for I/O Database Reads Average Latency.
1/23/2015 7:57:57 AM -- Instance11824.4 has 0.8 for I/O Log Writes Average Latency.
1/23/2015 7:57:57 AM -- Instance11824.4 has 0.8 for I/O Log Reads Average Latency.
1/23/2015 7:57:57 AM -- Test has 0 Maximum Database Page Fault Stalls/sec.
1/23/2015 7:57:57 AM -- The test has 0 Database Page Fault Stalls/sec samples higher than 0.
1/23/2015 7:57:57 AM -- C:\Program Files\Exchange Jetstress\Performance_2015_1_23_3_46_20.xml has 477 samples queried.

Screenshots from Tintri GUI while running Jetstress

Latency

Latency

IOPS

IOPS

Throughput

Throughput

Appendix B – LoadGen Testing Details

A couple of important notes:

  • In order to perform LoadGen testing, one more test client VMs need to be built – LoadGen is run on a client, not a server. In our case, we built 32 Windows 7 x64 VMs for this purpose (a 64-bit OS is required for LoadGen to install).

  • Full LoadGen testing can take an extremely long time to initialize for larger deployments, if a realistic simulation is to be performed. Even with copious client CPU resources, it only generates less than 1 TiB of email a day during the initialization process; in our example scenario we have 60 TiB of mailbox data – something LoadGen will therefore take months to initialize.

  • LoadGen will create and manage temporary user accounts in Exchange, so if you are testing, say 1,000 users, it will create 1,000 user accounts (with mailboxes).


DON’T: Run LoadGen on a production environment, since it will interfere with performance significantly.


LoadGen Test Configuration

Once the LoadGen client is up and running, and the LoadGen installer has been run, the test can be performed by using the LoadGen GUI.

In the first screen, you need to configure the basic simulation parameters as well as your environmental (domain and credential) settings. Do not however that the “total length of the simulation” does not include the one-time initialization, when the test mailboxes are actually built.

LoadGen GUI

Figure 40 - LoadGen GUI

LoadGen will automatically access your configured Exchange environment and get its topology. It will then allow you to specify the number of users you’d like to test per discovered mailbox server. In our case:

LoadGen User Settings

Figure 41 - LoadGen User Settings

It will also allow you to generate other types of recipients, like Distribution Lists and Contacts, or even External Recipients:

LoadGen Advanced Recipient Settings

Figure 42 - LoadGen Advanced Recipient Settings

LoadGen will then generate the required recipients, configured in the above two screens:

LoadGen Recipient Creation

Figure 43 - LoadGen Recipient Creation

Once that’s done (which happens remarkably quickly), come the options to have multiple LoadGen client machines generating load:

LoadGen Remote Configurations

Figure 44 - LoadGen Remote Configurations

... and then the type of tests you want to run; in our case, we wanted to simulate the example we provided, with 150 emails per day and 2GB mailboxes per user:

LoadGen Test User configuration

Figure 45 - LoadGen Test User configuration

You finally get the summary and the option to initialize the environment and run the simulation:

LoadGen Configuration Summary

Figure 46 - LoadGen Configuration Summary

LoadGen Run

Note, again, that the initialization is required the first time but can take a long, long time, since it is during that process that the actual test mailboxes are populated with emails:

LoadGen Run

The initialization process does generate little load on the storage, some load on the LoadGen client and significant load on the server compute:

Load on VMstore during LoadGen Initialization

Figure 47 - Load on VMstore during LoadGen Initialization

Load on LoadGen client

Figure 48 - Load on LoadGen client

Server Compute Load during LoadGen initialization

Figure 49 - Server Compute Load during LoadGen initialization

Once the initialization is complete, the simulation itself starts – which is bound by the time specified in the first screen.

LoadGen Simulation in Progress

Figure 50 - LoadGen Simulation in Progress 

 

 

 

 

 

Temporary_css