GSX Blog

How GSX displays and analyzes the Office 365 end-user experience

Posted by Jean-Francois Piot on Tue, Oct 24, 2017


At GSX we spend our time trying to make your life easier. For that, we’ve developed the only technology that is able to measure the end-user experience of your Office 365 full cloud or hybrid deployment.

We’ve seen in a recent post how the GSX Robot User constantly performs actions on Office 365 services and the Microsoft on-premises server (for hybrid deployment) and then collects its performance statistics to give you clarity on the services that are really delivered.

In this post we will explain how to read the PowerBI dashboard that we build from the statistics we collect. You can access some of the dashboard we created for our customers on this page.

The rise of PowerBI has been quick and powerful. The amazing work done by Microsoft on this analytic tool has continuously delivered new features on a weekly basis. There are plenty of ways to show information on PowerBI dashboards to show you critical insights.

Let’s explore this dashboard, which is composed of many customizable parts: 

Exchange Online FreeBusy.png

1. List of Robot Users

 Exchange Online Robot Users.png

This part displays the list of all Robot Users that are used to feed this dashboard in statistics. Clicking on any selection box adds or removes the statistics of the designated Robot User from the dashboard.

2. Linear Graph Area

Exchange Online Linear Graph.png

This shows the performance over time of each Robot User in a linear graph. Each color represents one of the Robot Users doing the action specified (here free/busy lookups). The time to accomplish each action is displayed on the Y axis in milliseconds. The X axis shows the date of the action (which are performed in repeat intervals depending on the configuration of the Robot User). Clicking on any line of the graph shows the exact measure and time for that particular point.

3. Square Graph

Exchnage Online square graph.png

Each square represents a Robot User performing an end-user simulation (here again the free/busy lookup). The purpose of the graph is to quickly identify which Robot User has the worst (or best) performance over a certain period of time on average.

The surface of each square is defined by the average performance, in milliseconds, of the Robot User performing  the action  compared to the other Robot Users: The ratio of its performance divided by the total performance of all the other Robot Users.

This graph makes it easy to identify:

  • Which  locations are having issues.
  • How bad the situation is for the locations experiencing issues

Here, for example, we see that the grey Robot User from Philadelphia has more trouble than the purple Robot User from Nice.

4. Statistics Computation

The middle section of the dashboard gathers an important selection of statistics computations.Exchange Online Access Time.jpg

The Min of Access Time is the minimum value that has been detected to complete the action across all selected Robot Users for the selected period of time.

The Max of Access Time is the maximum value that has been detected to complete the action across all selected Robot Users for the selected period of time.

The Average of Access Time is the average value of the time needed to complete the action across all Robot Users for the selected period of time.

The Median of Access Time is the value separating the higher half of the data set from the lower half. The median is not as skewed by extreme high or low outliers, giving a better view of the “typical” value of the time it takes to complete the action on Office 365.

For example, we can see here that the median is lower than the Average of Access Time, so 0.5 milliseconds can be considered as the typical value for the completion of the Free/busy lookup across all selected Robot Users over the selected period of time.

The Standard deviation of Access Time PowerBI.jpg

The Standard deviation of Access Time represents the average difference between the average time and the actual amount of time needed. The higher this value is, the more often the real time to complete the action differs from its average. When you look at end-user experience, a high standard deviation means that the performance delivered often differs from one moment to another. The end-user experience is not very consistent and this can cause complaints from consumers.

The Variance of Access Time calculates the degree to which each point differs from the average statistic. When you measure Office 365 end-user experience, a large variance indicates that when the time to complete an action is not around its average value, it is much higher than its average value. Essentially, when it is slow, it is really slow! You can expect complaints when this is figure is high, too.

5. Over Threshold Action Count

For a specific statistic over time, it is easy to calculate how many times this statistic has been over a certain value using PowerBI.

Count of access time PowerBI.jpg

The value we choose for our tests is usually in line with what our customers have told us about “user acceptance threshold,” which means the time a user is willing to wait for a certain action before getting angry, complaining, and finally opening a ticket.

In this example we see 5 seconds. Imagine a scenario where you are trying to schedule a meeting with five people, so you want to check the schedules of these five people to make sure you can make the meeting at the right time you want. If you need to wait 5 seconds to check each calendar, you will need to wait 25 seconds just to be able to know if the attendees are free at the time you want to create the meeting.

Here in the “38 counts” example, there were 38 times where the Free/Busy search took more than 5 seconds to perform.

Exchnage Online Free Busy Lookups.jpg

This measure is important because it is generally pretty well related with the number of tickets that are opened for a particular situation. Below is another way of looking at that statistic, with a 3 second threshold.

6. Time Period

Time period PowerBI.jpg

To choose the time period in the Power BI dashboard you just have to select a start and end point on the horizontal “time period” bar. This time range is visible on every dashboard.

7. Performance Map

This map shows the locations of the selected Robot Users.

Performance map Power BI.jpg

It also shows a circle for each with a surface area proportional to the average value of a chosen statistic. This statistic can be the average time to complete an action on Office 365 or any other network test, such as a traceroute. In this example, the surface area of the circles represents the time taken to complete a traceroute. So we see that the Robot User in Philadelphia is much slower than the one in Nice, France.

At GSX, we assess and analyze current Office 365 user experience issues and establish the future performance of a service. This assessment clearly outlines which experience you deliver to your users, comparing service quality and detecting bottlenecks, service-by-service. You will be able to visualize this data with PowerBI custom dashboards and compare the results across your critical locations.

Test your environment and find out how to deliver a faster Office 365 experience!Office 365 Assessment.jpg

Tags: hybrid cloud, Cloud, GSX Robot User, End user Experience, office 365 reporting, office 365 performance, Office 365 Hybrid, PowerBI Dashboard, PowerBI