Edges and Bandwidth and Performance Measurement: What you need to know about...

 


Around 2 months ago when doing a Speedtest check of my links I experienced a substancial reduction of measured UPlink and DOWNlink speed.

As I only have a single link to the Internet using Cable Modem connectivity which is a shared linktype, after some retries I assumed the problem to be at the Service Provider and some congestion there. I contacted the Service Provider but he assured me, that this is not the case.

Then I found the real "culprit".

It was my VMware SD WAN Edge which drastically reduced the bandwidth sent out via the Overlay.

When checking the Edge I found out that the last measured Bandwidth was around 40/10 instead of 180/40 Mbps.  

I was quite unsure about that measured  bandwidth and found out that according to this VMware knowledge base article ...

... on wired links the bandwidth test is only done when there is a Link Up event or after 7 days.

So what seemed have happened was, that the last bandwidth measurement took place at a time, where due to some external reasons the resulting bandwidth was much lower than usual. After that until rerunning  a new bandwidth measurement by navigating to 

Test&Troubleshoot --> WAN Link Bandwidth Test 

the lower measured and store bandwidth limited the bandwidth used by the Edge to reach destinations via Overlay. 

Note: Even checking the Dynamic Bandwidth Adjustment will not help as it never adjusts to a higher value than the originally measured one

 Therefore whenever you are concerned about performance and want to monitor performance of your Edges you should also include the Measured Uplink and Downlink Bandwidth values

  • bpsOfBestPathTx
  • bpsOfBestPathRx

in your monitoring and rerun a bandwidth Test when the values are below 90% of the expected or highest measured values

The values typically can be gathered via RestAPI ('metrics/getEdgeLinkMetrics' or 'livedata´)

ubuntu@SDwan-Monitor:/$ vcoclient.py --vco="10.8.1.254" edges_get_lm   --enterpriseid=1 --edgeid=1 --starttime="2022-01-26 12:00" --endtime="2022-01-26 12:30" --filters="bpsOfBestPath"

                       GE3        GE4
bpsOfBestPathRx   95518000  639905000
bpsOfBestPathTx  871032000  277411000

The above output uses a nice tool from Github  

 https://github.com/iddocohen/vcoclient

For an automatic measurement and email alerting of remaining important link values like Delay, Jitter and Packet Loss in case of a degradation, you also find a nice solution on Github

 https://github.com/kurtcms/vco-api-wan-anomaly-alert

 Here is an actual example of such email message:

 

Hope that will help you keep an eye on your SD-WAN Edge and Link performance.



Comments

Post a Comment

Popular posts from this blog

Orchestrator Upgrade to Version 5.2

Image
 Recently I upgraded my VMware SD-WAN lab from version 5.0 to 5.2. Since some previous versions the official documentation  recommends to ask TAC for help in upgrading.   Docu 5.2: Orchestrator Upgrade  But if you have an unsupported implementation, like I do, you need to do it on your own: Trying to copy the new version to the VCO failed and I found out that I needed to increase a disk volume How to increase disk volume on VCO is described in this blog:   VCO Upgrade to 4.5.x After increasing the disk size my copy succeeded   NOTE: I did not copy direct to the installation directory including a renaming, as once an upgrade started that file gets deleted and if there are any problems to have to redo the copy from outside.  vcadmin@vco-lab-254:~$ ls -l total 2504488 drwxr-x--- 2 root    root          4096 Aug 31  2022 20220831130007 -rw-rw---- 1 vcadmin vcadmin 2564587520 Aug 10 07:57 vco-debs-signed-5.2.0.3-R5203-20230809-GA-ff5cd1917e.tar vcadmin@vco-lab-254:~$ sudo cp vco-debs-signed
Read more

Deep Dive on DMPO and its Performance Features (available and missing) Part 1

Image
 In the last year I got quite deep insight into ther vendors SD-WAN implementation and also have seen what various vendors are critizising about the VMware DMPO features. Also there are a lot of misconceptions on the market around some of these features. So let´s try and evaluate and look a little bit deeper into typical performance features and the VMware implementation of them Features discussed in this Blog (Part 1) Per Packet Load Balancing Forward Error Correction (FEC) Packet Replication Latency Remediation The following Features will be discussed in Part 2 of my blog TCP Optimization Techniques Dejitter Buffering SAAS Application Monitoring   Per Packet Load Balancing  Unfortunately this term can be quite easily misinterpreted, when you Google for the term and get the following result:  This behaviour (often criticized by other vendors about VMware) as described above would only work quite well when you have very similar or equal connections. If for example 2 links have  both 10
Read more

Deep Dive on DMPO and its Performance Features (available and missing) Part 2

Image
  The following Features will be discussed in this second part of my blog TCP Optimization Techniques Dejitter Buffering SAAS Application Monitoring   Before diving into the mechanism here an important basic fact:     Remediation is never done for applications classified as LOW priority     TCP optimization TCP has some embedded traffic management capabilities for reliable traffic delivery  (window.size, slow-start, RTT handling,...)  But there are a bunch of factors which can negatively influence the performance, like Latency TCP-slow-start Last Mile network problems Out of Sequence packets Busty losses end host TCP limitations (missing fatures like SACK, windows scaling or timestamp options)  Typically TCP optimization helps on the transmission side and improve the download time for large data transfers over latency-high and lossy WAN links. But it can also be used for improving perfomance when on the receiver side the amount of buffers is limited and/or when the client is n
Read more