GSX Blog

Office 365 Migration: User Satisfaction is the Key Factor to Measure Success

Posted by Jean-Francois Piot on Thu, Nov 02, 2017

illuBlog-business-Case-vert-1.pngGSX Solutions is the specialist of Office 365 services performance optimization. We’ve worked with many Enterprise customers measuring their end-user experience thanks to our Robot Users. We have the expertise and want to share the knowledge with you and explain real business case that we faced with our customers.

Context of the Company

  • Largest financial company of Canada.
  • Moving from Exchange 2013 to Office 365 Exchange Online.
  • First batch included 90,000 mailboxes.
  • The bank will deploy Skype for Business Online later, SharePoint will stay on-premise.
  • Final Architecture will be Hybrid with Azure AD Connect, ADFS and 15 Exchange servers left on-premises.
  • The email security appliance is IronPort with multiple boxes on-premises.
  • Branch offices are spread across multiple continents and thousands of locations.
  • Route to Office 365 was the normal Internet (no express) and the connections were made without any proxy.

Primary need:

The IT management decided that one of the primary factors to determine the success of the migration was the end-user satisfaction. Users that complain generally open a ticket, and cost a lot of money at the end.

The purposes of the project were:

  • Understand the difference of the end-user experience between Exchange on-premises and Exchange Online.
  • Anticipate any performance issues.
  • Prove that the performance is stabilized and predictable at the end of the migration keeping an acceptable latency that would not impact the end-users.

For that, it was decided to measure the end-user experience of Exchange on-premises from several critical locations, and compare it with the new Office 365 service delivery during the whole migration process.

 How GSX was put in place

This large financial institution contacted GSX during Spring 2015.

GSX worked with Office 365 migration project manager to install the solution prior to the first batch of moves.

Before June, 4 main GSX Robot Users were installed at what would represent key locations to test the end-user experience: One in Canada, one in Germany, a third one in Brazil and a final one in the main USA offices.

GSX Robot Users were all installed in one day and connected to the central Robot User management & reporting console.

For each Robot Users, key Exchange scenarios were set up (open mailbox, create email, free/busy lookup) as well as Mail routing between offices to test the Exchange 2013 on-premises environment.

On top of these end-user actions, the Robot Users also tested and captured key metrics on the network availability and performance between the users and the Exchange 2013 data center.

Read more about GSX Robot Users >>

The purpose was to determine potential current bottleneck that would also degrade the further Office 365 quality of service.

The mail routing was setup to test the response time of the infrastructure, including IronPort to deliver email from one office to the other.

Again, that point of control would be key during the migration as it is the first service that the users use when it comes to Office 365 Exchange Online.

This initial phase was meant to establish current end-user experience baseline that will then be the point of comparison with the Office 365 services.

After this initial end-user experience monitoring configuration, GSX experts worked with the IT operation to determine what statistics they would expect in monthly reports that would be sent to the IT management to follow the success of the migration.

After two weeks of on-premises end-user experience monitoring, the monitoring was extended to the rising Office 365 infrastructure.

On-premises measurement continued but the Office 365 end-user experience monitoring was added to the Robot Users. That is a particularity of GSX, as our Robot User are able, at the same time, to perform end-user & network latency tests both on-premises and to the Cloud.

The monitoring of Cisco IronPort was also kept and Microsoft Azure AD Connect & ADFS were added to the mix in order to detect any potential issue with end-user authentication along the process.

This second phase lasted the whole migration, about 6 months during which the operational team could be alerted in case of any end-user experience issues, even before the user could notice it.

At the end of each month, all the data were analyzed by GSX experts that organized a recurring meeting with the customer management team to expose the results.

The outcome for the Company

The initial tests of the on-premises environments (Exchange 2013, IronPort, Gateways) already detected significant differences in end-user experience between the office tested.

Without any surprise, the Brazilian office was the one that experienced repentantly latency issue during the week. The overall performance wasn’t that different from the German or North American offices, but what was striking was the number of latency peak during the week and especially during the morning. We also discovered that the route to the datacenter was clearly inconsistent as the number of hops were clearly unstable.

Therefore, a first action of the main IT office was to investigate their local network to decrease these potential bottlenecks that would certainly also affect the Office 365 services.

The root cause was found thanks to the constant DNS resolution time test that the Robot Users are performing. Clearly, the DNS was not able to support the load when users first connected during morning. Once this issue was solved, the migration continued and results were pretty interesting.

As expected, the overall end-user experience of the Office 365 Exchange Online services was below the one on-premises. This was not a surprise. Security is the main concern of the Cloud services, not performance. But even if the performance was under what was experienced before, it was still in the acceptable range that was defined.

From a mail routing perspective, the performance was exactly as it was before. Reports showed over the months several other network bottlenecks (inconsistent route, local IT that had to be updated, internet latencies) that were fixed during the migration period.

At the end, with the evidence provided by GSX Solutions and its Robot Users, the project was proved as a success. The number of ticket has been limited and handled proactively. The end-user complaints were rare and anticipated. The company decided to keep the solution and expanded the number of its Robot Users to now more than 100. Every month, the end-user experience is justified through reports directly to the management.

Office 365 End-user Experience Monitoring.png

Tags: Microsoft Exchange, Cloud, Hybrid Monitoring, Migration, Hybrid, Migration to Office 365, Exchange, Exchange Online, Microsoft Office 365, End user Experience, Use cases