Crazy about VMware SD-WAN

SD-WAN has a kind of Plug&Play strategy when it comes to connectivity and routing. Therefore automatically a default route is established on every working WAN-Interface and on every Overlay tunnel. Additionally there is an automatic bidirectional redistribution on every Edge between Underlay, LAN  and Overlay routes. From a networking point of view a kind of routing nightmare with Mutual Redistribution in both direction on every Edge. BUT by using an internal clever preferencing algorithm, routing loops and blackholes are mostly avoided. A SD WAN administrator can avoid pitfalls and wrong paths by following some basic rules:  Only enable Dynamic Routing when it is really necessary Never run a Dynamic Routing Protocol direct between Edges If Dynamic Routing is needed then there are some further rules to consider: Prevent Redistribution form Overlay to Underlay (it is not needed in most cases) Restrict Redistribution from Underlay to Overlay to Edges (Hub) who are playing...

As already described in my Blog regarding  Lost OFC Routes  https://sd-wahn.blogspot.com/2022/01/where-have-all-ofc-routes-gone-or-my.html there were considerable routing behaviour changes starting with version 3.4.0 Excerpt from VMware SD-WAN Operator Guide Version 4.5 Configure Distributed Cost Calculation By default, the Orchestrator is actively involved in learning the dynamic routes. VMware SD-WAN Edges and Gateways rely on the Orchestrator to calculate initial route preferences and return them to the Edge and Gateway. The Distributed Cost Calculation feature enables you to distribute the route cost calculation to the Edges and Gateways. Note: Enabling Distributed Cost Calculation is recommended for all customers. This default method of involving the Orchestrator in both dynamic route calculation and the distribution of those routes to Edges and Gateways has drawbacks of significant higher route convergence time ...

Around 2 months ago when doing a Speedtest check of my links I experienced a substancial reduction of measured UPlink and DOWNlink speed. As I only have a single link to the Internet using Cable Modem connectivity which is a shared linktype, after some retries I assumed the problem to be at the Service Provider and some congestion there. I contacted the Service Provider but he assured me, that this is not the case. Then I found the real "culprit". It was my VMware SD WAN Edge which drastically reduced the bandwidth sent out via the Overlay. When checking the Edge I found out that the last measured Bandwidth was around 40/10 instead of 180/40 Mbps.   I was quite unsure about that measured  bandwidth and found out that according to this VMware knowledge base article ... ... on wired links the bandwidth test is only done when there is a Link Up event or after 7 days . So what seemed have happened was, that the last bandwidth measurement took place at a time, where due to s...

  If an Enterprise uses its own Orchestrator (VCO) the device is often placed inside the internal Datacenter (DC) network and has a private internal Eth0 address configured. Therfore the internet facing Firewall in the main Data Center will have a static 1:1 translation between the public IP address used by SD-WAN Edges and the private IP address used by the VCO. In my lab Edges are reaching the Orchestrator via 110.1.1.254 and the RO-230-IOL-Entry router will translate it to 10.8.1.254 the VCO address on Eth0. In such cases you need a specific additional system property set to make remote diagnostics to work  as you see you also need to set the network.portal.websocket.address Then Remote Diagnostic access from your browser using the internal private address and edges using the public address should work.  

When you set up new Orchestrator and Gateways it typically takes some time until you need to relogin via SSH or via Console in order to do for example an upgrade to a new version. Default password expiration is typically set to 90 days . Normally when for an account the password is expired you still can login via console and specifiy a new password afterwards. Unfortunately VMware SD WAN Orchestrator and Gateways have a specific setting, that completely disables login of vcadmin after the first 90 days without any password change even on console. In that case you are left with unreachable VCO and VCGs regarding CLI. So the only way to overcome is a password reset procedure: And there are as far as I have seen, 2 ways to do such a password reset:  Password Reset via GRUB Recovery Mode   Reconnect of Disk to separate Unix System and doing Password  Reset from there As the first method is much simpler you should always try that one, before using the 2nd one. WARNING : ...

In this blog I want to share some findings gathered in the last year regarding not so well documented traps when setting up and maintaining your own Orchestrator and Gateways  So let's start with a recent one: VCO Update 4.5 fails with OSError: [Errno 28] No space left on device  When upgrading to version 4.5.x a general error arises in case you never have enlarged the size of a specific LVM volume on the VCO vcadmin@vco-01:~$ sudo /opt/vc/bin/vco_software_update                         2021-10-12 09:39:13,817 - UPGRADE - WARNING - Verification key does not exist: /var/lib/velocloud/software_update/keys/software.key WARNING: failed to verify package identity. Proceed as untrusted [y/n] [n]: y 2021-10-12 09:39:17,908 - UPGRADE - WARNING - WARNING: installing untrusted package 2021-10-12 09:39:17,908 - UPGRADE - INFO - Loading manifest and extracting p...