"Where have all the OFC routes gone..." or my long-run search for lost OFC routes

 Last year i transferred my lab environment from my home grown ESXi environment to the company SC-lab environment and also did an update from 4.2 to 4.3 on Orchestrator, Gateways and Edges.

After the update the Overlay Flow Control (OFC table)  showed static, connected and gateway learned routes only.

all the dynamically learned routes were gone (23 ->8).

We looked into all the different configuration items to find about the problem, and even with the help of someone from VMware we did not find the reason.

I was also told that some other guys have a similar problem in their network.

The strange thing was, that the routes missing in the Orchestrator´s OFC table were available in the Edges local Routing Table

 and you can also see that routes are learned also via Cloud VPN (via the Overlay)

So for us it was clear that it is not a consequence of a wrong configuration. 

But now the question arose if that is a planned behaviour change in the routing or  a BUG. 

After 3.4 there were some fundamental changes implemented regarding the treatment and evaluation of routes.

Befor 3.4 routes from the Edge were forwarded to the Orchestrator via Gateway, and then the Orchestrator via OFC table distributed the learned prefixes and metric to the Edges.

As the Orchestrator can be a single point of failure and the communication is done only after receiving a Heartbeat from the Edge (default is every 30 seconds), Routing updates did not run fast and reliable enough.

Therefore from 3.4.0 onwards routing exchange is done directly between Edges and Gateways, the Gateways are playing the role of a Route Reflector (RR) and only a copy of the routes is sent to the Orchestrator. This also means that the Edge is doing it´s own decisions on which route to use, based on principle information regarding route preferences forwarded from the OFC part of the Orchestrator.

So again BUG or behaviour change as now we only see non dynamic learned prefixes via Overlay.

Then I also found that starting with 4.2 the documentation had important changes on the detailed description of the OFC and it´s routes.

Thee older version showed also dynamic routes and had a much wider definition of Route Type

Whereas the newer docu after 4.2 showed:

So due to that change in the wording of the documentation we assumed a planned behaviour change by the SD WAN developers.

But our assumption was wrong, after some further discussion with different people from VMware I got the information, that this was a BUG, but...

... according to the information it was because I did not correctly follow all necessary additional items when upgrading the Orchestrator from version 3.x to version 4.x 

The according to VMware PSC people missing item was to run a phyton routine after Orchestrator upgrade :

root@vco:/opt/vc/scripts# /opt/vc/scripts/vco_schema_update.sh -e

or from the vcadmin account with sudo

vcadmin@vco:#sudo  /opt/vc/scripts/vco_schema_update.sh -e

 And YES, after calling that python routine and rebooting the Orchestrator the system came up including all the missing routes in the OFC table

 The only stilll open item was that I never had used any version older than 4.0 and thus never did an upgrade from 3.x to 4.x, so against information from VMware it can also happen on other updates as well (in my case from version 4.2 to version 4.3)

 So as a resume:

Whenever after an Orchestrator upgrade you have missing routes in OFC, but those routes are seemingly visible on the Edges you can run the

/opt/vc/scripts/vco_schema_update.sh -e

to correct and update DB schema changes.