robotUser-prof7.png

Skype for Business Online key metrics to watch to assess your
end-user experience before, during and after any deployment.


In this article, we will define the meaning of every metric that are compulsory to measure the ability of each of your locations in order to provide a descent Skype for Business end-user experience.

As we mention in the RoboTech article, the GSX Robot User is the perfect tool to collect these data the way Microsoft recommends to.

For more information about how the Robot Users are providing you with end-user experience metrics, please read this article >>

 

Latency one-way or ping in millisecond and round-trip latency
(in millisecond too)

a1.png

► It measures the time taken to send a data packet from point A to B and come back. It is tied to physical distance between the two points and the speed of transmission and the overhead taken by the routers in between.
► The latency impacts the smoothness of the conversation between two people
► Too much latency provides unnatural pauses during the conversation
► People also report that it feels like using a satellite phone
► In the end, it leads to having people talking at the same time
► Everybody has already experienced that

Packet loss: the value is in percent

a2.png 
 
► It represents the amount of packet lost for 15 seconds (for example if 1000 packets are sent in 15 seconds and 50 are lost, it will generate a 5% packet loss).
► This measure is extremely important in VoIP as it used as one of the elements to determine the MOS that we will explain right after.
► A high packet loss will lead to a moment of silence during a call (if you have a period of sustained packet loss during a call) or to a degradation of the voice quality giving people ‘robot-voices.’

Jitter (also called packet inter-arrival jitter: in millisecond)

A3.png
 
► Audio packets are sent at regular intervals on the network. But that doesn’t mean that they are received with the same regularity (usually because of network latency). That is why a buffer is needed, it waits for all the packets before reconstructing them in the correct order.
► The Jitter is the size of the buffer that is needed to store packets before reconstructing them in the correct order. It can be compared to an audio packet waiting room. The value of the Jitter is calculated over every period of 15 seconds.
► A low Jitter number means that the connection to the call is good and solid.
► A large Jitter buffer provides additional delay in calls. It is the sign of a congestion of the network.
► The shrink or the expansion of the buffer will provide audio distortion during the call like speeding or slowing down of the speech.
► As for packet loss, the Jitter value is used in the Network MOS determination.
 

The Network MOS: Network “Mean Opinion Score”

a4.png

This concept needs a little bit more of an explanation.

The MOS is usually a score that is based on a questionnaire sent to the users, like the one you have after each call on Skype or WhatsApp: “Please rate the quality of the call from One to Five stars” (5 being excellent, 1 being poor).

That is why you can’t have a MOS superior to 5. The problem is that when you want to assess your environment or even during migration you don’t have any or enough feedback to get this score.

That is why Microsoft has worked with other network specialists on the definition of Network MOS metric that can predict what will be the value of the Wideband Listening Quality Mean Opinion Score (MOS-LQ).

To calculate that, multiple factors are taken in account such as the latency, the packet loss, Jitter, the codec used, etc.

As for the real MOS, the Network MOS ranges from 1 to 5. But because of the compulsory impact of the audio codec, the highest score is usually around 4.4.

How to use the MOS value?

The Network MOS is a perfect tool to identify if the network conditions are impacting the end-user audio quality experience. It can be used to identify a wide range of issues.

The first way to analyze it is to compare it with a previous average value in time to understand if you are dealing with a degradation of the audio quality or not.

Then, you can combine this information with the packet loss or Jitter to understand what is causing that degradation and in which location. And that is why you need to constantly measure this parameter from all your sensitive locations.

The best example of root cause is the LAN Congestion.

a55.png

When your LAN start to be overloaded, the rates of packet loss and amount of Jitter is increasing for all calls going through your LAN.

This will automatically be seen in the MOS score.

If you trend your MOS score, thanks to GSX Robot User for example, you’ll be able to understand exactly at what time and where the LAN was congested, determining your peak time and adjusting your network to prevent this issue from happening again.

We see here how important it is to constantly monitor all these specificities to be able to take proactive actions before users start complaining and opening tickets.

As we’ll see later, the GSX Robot Users are constantly calculating the Network MOS value and alert in any case of decrease, providing you time to work on the issue before it becomes a problem for your users.

 

How packet loss and Jitter are related to the Network MOS score

The packet loss plays a very important role in the Network MOS calculation. Over 20% of packet loss, your MOS cannot be higher than 2.5.

The latency round trip won’t affect the MOS if it stays under 100ms.

If the Jitter stays under 50ms, the MOS will be as high as it can. When the Jitter rises, the MOS degrades quickly.

 

Basic recommendation on Network MOS

As the MOS is a prediction of the end-user experience of audio quality, it is really important to constantly measure it. Jitter and the Packet loss should also be measured frequently. Make sure that the Jitter value stays under 50ms, the packet loss as close as 0% as you can and of course the latency under the 100 milliseconds.

Now that we’ve seen what kind of statistics you should collect, let’s see what is recommended as a collection process.

 

Interpretation of the GSX Robot User measures and alerts

Thanks to the GSX Robot Users, you have an easy way to test and collect every statistic you need to assess the readiness of your environment to deploy Skype for Business Online.

You also have a way to easily store and analyze the statistics thanks to our PowerBI reports.

Now, let’s see how to analyze them!  First, what should be the results if you want to pass the exam?

Microsoft’s recommended results for readiness assessment

Microsoft recommends that you perform the test from 2 different “exit points” to analyze what part of your network might cause issue.

For each location you should have a Robot User performing your tests:

► Directly from where the clients sit to the Microsoft Edge
► From your infrastructure’s Edge to Microsoft Edge
 
a6.png
 

This will give you a pretty good understanding of how your infrastructure is decreasing the call quality of the Skype for Business services.

Here are the recommended values for each of them:

Name of the Metric

Client to Microsoft Edge

Customer’s Edge to Microsoft Edge

Latency (on way)

< 50ms

< 30ms

Latency (RTT / Round trip time)

< 100ms

< 60ms

Burst packet Loss

< 10% during any 200ms interval

< 1% during any 15s interval

Packet loss

<1% during any 15s interval

< 0.1 % during any 15s interval

Jitter

< 30ms during any 15s interval

< 15ms during any 15 interval

 Network MOS*

>4

>4

*Network MOS: The MOS score target is usually always the same.  A MOS > 4 will provide good audio quality. The closer you are to 4, the smaller distortions you will experience but it will be acceptable.

Between 3 and 4, you can still use the audio (the closer to 4 the better of course) but some users will be impacted by all the issues we explained earlier. It means that you should look at the location to see if you can improve the network quality before deploying Skype for Business.

If you have location with a MOS under 3, you shouldn’t deploy Skype for Business Yet. It will create more tickets and frustration than anything.

A heavy look in your network is necessary before doing anything else.

What to do with these values?

Now that we have the targets, it is really important to compare them with your locations’ values over time.

For that you want to know what your peak times during the day and during the week are.

It is easy to recognize them with your PowerBI reports that draw all your statistics over days and weeks. You can then identify what are your key moments of the day/week and compare the results of your tests with the Microsoft recommendations.

Finally, as most of business activity also has a monthly pattern, it is important to keep monitoring this activity for several weeks.  You’ll be able to identify these patterns and make sure that you’ve discovered all your peak times.

You’ll see if your network can handle them. If not, you will know directly which location fails and exactly at what time and period of the month.

You’ll be fully prepared to improve the network or handle the users’ complaints.

Reaching the target

From a general point of view, to reach these targets you should make sure that:

► The latency one-way is stable during busy and non-busy hours
► The latency RTT is stable and stays close to twice the latency one way
► Packet loss are not increasing too much during busy hours (it is normal if it rises)
► Jitter do not increase too much as well
► Edge connection tests stay better than client connections ones

If all your locations pass the exam during busy and non-busy hours, you can be confident in Skype for Business services. User complaints and tickets should stay low, ensuring the best ROI you can have.

But as your infrastructure and the way to use it is in constant change, you should continue to monitor these metrics to prevent any crisis situation, and to predict potential issues at particular locations.

To do so, you just have to let your Robot User keep working for you.

 

What happens if a site fails?

Data

We already explained in the statistics description how each of them has an impact on the audio quality. Now, let’s see an example of a site failing the tests:

Metric

Target

Result Busy hour during a normal day

Result busy hour during the busiest day of the week

Latency one way

< 50ms

40ms

38ms

Latency RTT

< 100ms

81ms

77ms

Burst packet Loss

< 10% during any 200ms interval

12%

15%

Packet Loss

< 1% during any 15s interval

4%

5%

Jitter

< 30ms during any 15s interval

25ms

35ms

MOS

>4

3.7

3.3


Data interpretation

Basically, what’s going on here?

Most of the results exceed the target. Clearly, during the busiest day of the week, the network cannot handle the peak time. The site won’t be able to provide a good quality Skype for Business service.

So basically, the calls will have reduced quality (PSTN, peer-to-peer and conference):

► Users will experience a moment of silence during the call
► They will hear a robot voice for each of the participant
► The speech of the participant will suddenly slow down and then speed up

At the end, the quality will be poor, users will complain, you will have plenty of tickets coming from that location, increasing your administrations costs as well as your user dissatisfaction.

If you have a similar result, you should definitely look into improving your network before deploying Skype for Business Online.

Finally, with constant monitoring of each site you can easily rank each of your site and organize them by ROI.

Site

Number of users

Sensitivity

Results

Cost

ROI

Actions

Boston

1000

High

Pass

Zero

High

Monitoring

Geneva

500

Low

Fail

High

Low

Stand-By

Singapore

200

High

Fail

Medium

High

Network improvement

For each site it is very important to understand the ROI of your network investment.

Sometimes small sites have highly sensitive people and even if the network improvement will be important, the ROI will still be high (like in our Singapore example).

Some other time costs are too high to help remote sites compare to the benefit you will get. In our case, the Geneva costs outweigh the benefits. 

It all depends of your organization.

But before going through extensive network design and investment, maybe a first set of cheap improvements could enable your organization to provide a decent service across most of your sites.

For that please read our RoboTech article: How to improve your network to meet Skype for Business online requirement.

If you want to consider buying an ExpressRoute, then please read our article >>