robotUser-prof7.pngHow does MAPI over HTTP really compare with RPC over HTTP in term of end-user experience?


In our first article, we saw how MAPI/HTTP differs from RPC/HTTP and the value proposition we intend to deliver.

You can read it here: A brief history of MAPI over HTTP benefits

Now let’s see how we conducted our tests and what the results of our performance tests were.

Our Experiment

We achieved 3 different use cases running MAPI/ HTTP versus RPC/ HTTP.

  • A first deployment was on Outlook 2016 client with Windows 10
  • The second was on Outlook 2016 with Windows 10 but in a network that experienced issues (packet loss, latency, etc.)
  • And the last one was on Outlook 2013 with Windows 7, as we know that many companies still use this configuration.

Let’s now move on to the data.

So how do we collect this data?

As explained in other articles, GSX provides Robot Users that perform end-user experience tests as well as network latency checks. Their actions aim at collecting metrics about the end-user experience and the underlying network. Theses end-user scenarios are specific to each suite of Office 365 services.

In this article, we will only speak about Microsoft Exchange.

Read more about GSX Robot Users here >>

We are then using PowerBI to display the data to report and analyze the information.

In order to have more information about the type of graph we are using in PowerBI and how to read them, please read our dedicated blog here >>

So let’s come back to the results.

Results in a clean environment with Outlook 2016 and Windows 10

5.png

The dashboard we see here provides the results of the 2 first Robot Users we set up in order to compare the performance between RPC/HTTP (yellow statistics) and MAPI/HTTP (blue statistics).

This dashboard gathers several days of these metrics. First, we looked at the open mailbox that performs a full auto discover to retrieve all the different settings, connect to that mailbox and authenticate to access the mailbox.

But checking an “Open Mailbox” action is not enough. We have seen in other RoboTech articles that performance can greatly vary, for example between an Open Mailbox and a Free/Busy lookup. Therefore, our tests simulate multiple scenarios with the GSX Robot Users to assure the most accurate results.

For these tests we add the actions: download an attachment, perform a free/busy lookup and create a meeting.

We expected to see better overall performance with MAPI as the protocol can better reconnect to a session. Indeed, we see this with the Open Mailbox.

7.png

As you can see here, there is a gain of about 25% performance in establishing the connection between the client and the mailbox when using MAPI over HTTP.

However, when comparing this to the performance of the other actions, we then realize that they are 5 to 20% slower when using the MAPI protocol.

8.png

How can we explain this discrepancy?
First, it needs to be understood that we are looking at the raw protocol performance in a very static configuration. This means that these scenarios are run on virtual machines that are not being disconnected from the network, not running on wireless, and not suffering from any problems that could arise when working at home or at a cafe.

Secondly, no matter the performance, it is important to consider their stability. On the right side of the dashboard you can see some statistics calculations (Variance and Standard Deviation) that provide some insight about the stability of performances.

If you want more information about Variance and Standard Deviation please read our blog here >>

 

9.png

Here you can see the dramatic difference between MAPI/HTTP and RPC. The connection is much more stable with MAPI, and does not vary as much on MAPI than RPC.

To sum up these findings, MAPI brings more stability and speed when connecting to the mailbox, but does not necessarily improve performance for other daily actions that are performed by an Outlook client.

Let’s see our second set of tests.

 

 

 

 

 

 

MAPI/HTTP versus RPC/HTTP with Outlook 2013 on Windows 7 machines

 10.png

As with the previous tests, we have one Robot User using MAPI/ HTTP (Grey statistics) and the second one running RPC/ HTTP (Green statistics).

On the performance side, we saw surprisingly contrasting sets of results. The time taken for connections and other actions varied a lot.

Depending on the timeframe we analyzed, sometime it seemed as if RPC was faster when initiating the connection to the mailbox and slower on the other actions, while sometimes MAPI seemed to have a better performance…

However, we can confirm the former conclusion that MAPI is much more stable by checking the statistics calculations on the right side of the dashboard. 

11.pngIn the next phase, we aimed to see how both the protocol
react when the network is in trouble.

 

 

 

 

 

 

 

 

 

 

MAPI/HTTP versus RPC/HTTP while running on poor network performance

For this test we set up different network environments with packet loss, latency, throttling, packet duplication, and packets out of order, to see how the protocol would respond. The purpose was to simulate realistic situations that could happen in the everyday life of a company with multiple remote locations and people working from home or on the road.

Results showed 5% packet loss and 50 milliseconds of added latency, knowing that it is already going across the Atlantic (Robot User in Philadelphia accessing a tenant in Europe).

 12.png

Once again it confirmed the first set of data we presented.

The connection time was always slower for RPC. The difference was not extreme (10%) but further confirmed our first findings.

From most articles, one would think that in these environments MAPI/ HTTP would perform much better than RPC.

While MAPI/HTTP was designed for better resiliency, which it does provide, it did not make much difference in other areas.

In order to find a solution, we started to take down a router for about 2 second out of every 6, and then we put a proxy server in the middle that was throwing away packets.

But no matter what we tried, we struggled to actually come up with a scenario where, other than being better at reconnecting the mailbox, MAPI/HTTP actually had a better and faster overall performance.

Data Volume

When we looked at the 2 Robot Users in Philadelphia and isolated the amount of data coming out, we saw that there were definitely fewer TCP sessions established, but that they also were sending large amounts of data when compared to the RPC protocol, occasionally even twice as much data volume.

Despite these interesting results, it was expected to be the contrary, as MAPI is supposed to be a thinner protocol.

Even if the MAPI/HTTP protocol sometimes had to move twice the volume of data in contrast to the RPC protocol, it only had a maximum of 10% difference in performance compared to the RPC protocol. When moving the data, MAPI is very efficient, but that doesn’t necessarily mean a smaller volume of data.

This corresponds to MAPI’s design, which is to improve connection resiliency, not to send a smaller amount of data.

Conclusion of our tests

The shift to MAPI is already in motion. If you are on Office 365, you’re most likely already using this protocol, and if you move from on-premises to Exchange 2016 that is definitely the protocol that you will want to use.

From a protocol standpoint, MAPI/ HTTP is more in line with the modern world, as we are always connecting from different networks and locations nowadays.

As usual, before implementing any change in your environment, you need to do some careful planning.

To plan your change to MAPI/ HTTP, you need to gather metrics about the current performance and what the future performance will be to understand the consequences on the user experience and, in the end, on the number of tickets that will be generated.

And that is what the GSX Robot Users are designed for.

MAPI is intended to provide a better connection and reconnection to the mailbox, which is what we found while running our tests. It has not been designed to improve the performance of every action you can perform with an Outlook client in your day-to-day life. You should check the impact of this new protocol on your end-user experience before rolling it out.

From a data perspective, you should expect a greater amount of data flowing on your network when moving to MAPI.

Of course, the amount of increase all depends on your network configuration and the way your messaging environment is used. Once again, it is a good idea to measure it in order to anticipate any bandwidth consumption increase in the future.

Finally, moving to a new protocol is not necessarily only a question of performance. One of the biggest benefits of MAPI, besides its resiliency, is better, more modern authentication. So if you’re doing multi-factor authentication, you still may want to take a look at it.