"Orchestrator Blues" Part 1: VCO Upgrade to 4.5.x

 

In this blog I want to share some findings gathered in the last year regarding not so well documented traps when setting up and maintaining your own Orchestrator and Gateways

 So let's start with a recent one:

VCO Update 4.5 fails with OSError: [Errno 28] No space left on device

 When upgrading to version 4.5.x a general error arises in case you never have enlarged the size of a specific LVM volume on the VCO

vcadmin@vco-01:~$ sudo /opt/vc/bin/vco_software_update                         2021-10-12 09:39:13,817 - UPGRADE - WARNING - Verification key does not exist: /var/lib/velocloud/software_update/keys/software.key

WARNING: failed to verify package identity. Proceed as untrusted [y/n] [n]: y

2021-10-12 09:39:17,908 - UPGRADE - WARNING - WARNING: installing untrusted package

2021-10-12 09:39:17,908 - UPGRADE - INFO - Loading manifest and extracting package content.

Traceback (most recent call last):

  File "/usr/lib/python3.6/tarfile.py", line 2173, in makefile

    copyfileobj(source, target, tarinfo.size, ReadError, bufsize)

  File "/usr/lib/python3.6/tarfile.py", line 252, in copyfileobj

    dst.write(buf)

OSError: [Errno 28] No space left on device

 even after a cleanup of some old file from previous upgrades the problem is still there

vcadmin@vco-01:~$ sudo docker system prune -af

Deleted Images:

untagged: vco-i18n:R420-20210122-GA-f8580b27e2

deleted: sha256:2354c0589193a08d13eeff2d47c631e3aafcabd665863ca2f031a87eda5ee486

deleted: sha256:05a9d892eb19dca804571ba10146f2bad65a78bbce7a5779d21d3b5d45a472c9

untagged: vco-i18n:R430-20210527-GA-de848500e8

Total reclaimed space: 62.42MB

 if you look into the disk usage

vcadmin@vco-01:~$ df -h

Filesystem               Size  Used Avail Use% Mounted on

udev                     7.9G     0  7.9G   0% /dev

tmpfs                    1.6G   11M  1.6G   1% /run

/dev/mapper/vols-root1    12G  6.8G  4.5G  61% /

tmpfs                    7.9G     0  7.9G   0% /dev/shm

tmpfs                    5.0M     0  5.0M   0% /run/lock

tmpfs                    7.9G     0  7.9G   0% /sys/fs/cgroup

/dev/mapper/store2-data   79G  2.2G   74G   3% /store2

/dev/mapper/store3-data  394G  1.2G  377G   1% /store3

/dev/mapper/store-data   394G  5.3G  373G   2% /store

/dev/mapper/vols-home    2.0G  6.1M  1.8G   1% /home

/dev/mapper/vols-var1    3.9G  2.4G  1.3G  65% /var

/dev/mapper/vols-log      25G  4.4G   19G  19% /var/log

/dev/mapper/vols-tmp     2.0G  6.1M  1.8G   1% /tmp

/dev/mapper/vols-audit   2.0G  715M  1.1G  39% /var/log/audit

/dev/mapper/vols-vartmp  2.0G  6.1M  1.8G   1% /var/tmp

tmpfs                    1.6G     0  1.6G   0% /run/user/1001

overlay                  3.9G  2.4G  1.3G  65% /var/lib/docker/overlay2/f7f2bba62fbb32cf18a78452fdfa521cc61cbc18925bddceab7ab5c229c5105b/merged


you see some disks at around 60% but nothing higher

But some of the Logical Volumes are rather small, so they can be get filled quickly.

After consulting PS from VMware they told me that i need to increase disk size of  the /var LVM

so by configuring

vcadmin@vco-01:~$ sudo lvextend -L +4G /dev/mapper/vols-var1

  Size of logical volume vols/var1 changed from 4.00 GiB (1024 extents) to 8.00 GiB (2048 extents).

  Logical volume vols/var1 successfully resized.

vcadmin@vco-01:~$ sudo resize2fs /dev/mapper/vols-var1

resize2fs 1.44.1 (24-Mar-2018)

Filesystem at /dev/mapper/vols-var1 is mounted on /var; on-line resizing required

old_desc_blocks = 1, new_desc_blocks = 1

The filesystem on /dev/mapper/vols-var1 is now 2097152 (4k) blocks long.

vcadmin@vco-01:~$ df -h /var

Filesystem             Size  Used Avail Use% Mounted on

/dev/mapper/vols-var1  7.9G  605M  6.9G   8% /var

vcadmin@vco-01:~$ sudo reboot

 the disk size was increased from 4 to 8 GB and then the upgrade worked without any problem.

 PS: i wrote this block because even after 2 months later I did not find according information, neither in Release Notes nor in any VMware KB document.

Other items which will follow in further parts of my Blog on "Orchestrator Blues"
  • remote diagnostics not working
  • login problems after 90 days
  • password recovery



Comments

Post a Comment

Popular posts from this blog

Orchestrator Upgrade to Version 5.2

Image
 Recently I upgraded my VMware SD-WAN lab from version 5.0 to 5.2. Since some previous versions the official documentation  recommends to ask TAC for help in upgrading.   Docu 5.2: Orchestrator Upgrade  But if you have an unsupported implementation, like I do, you need to do it on your own: Trying to copy the new version to the VCO failed and I found out that I needed to increase a disk volume How to increase disk volume on VCO is described in this blog:   VCO Upgrade to 4.5.x After increasing the disk size my copy succeeded   NOTE: I did not copy direct to the installation directory including a renaming, as once an upgrade started that file gets deleted and if there are any problems to have to redo the copy from outside.  vcadmin@vco-lab-254:~$ ls -l total 2504488 drwxr-x--- 2 root    root          4096 Aug 31  2022 20220831130007 -rw-rw---- 1 vcadmin vcadmin 2564587520 Aug 10 07:57 vco-debs-signed-5.2.0.3-R5203-20230809-GA-ff5cd1917e.tar vcadmin@vco-lab-254:~$ sudo cp vco-debs-signed
Read more

Deep Dive on DMPO and its Performance Features (available and missing) Part 1

Image
 In the last year I got quite deep insight into ther vendors SD-WAN implementation and also have seen what various vendors are critizising about the VMware DMPO features. Also there are a lot of misconceptions on the market around some of these features. So let´s try and evaluate and look a little bit deeper into typical performance features and the VMware implementation of them Features discussed in this Blog (Part 1) Per Packet Load Balancing Forward Error Correction (FEC) Packet Replication Latency Remediation The following Features will be discussed in Part 2 of my blog TCP Optimization Techniques Dejitter Buffering SAAS Application Monitoring   Per Packet Load Balancing  Unfortunately this term can be quite easily misinterpreted, when you Google for the term and get the following result:  This behaviour (often criticized by other vendors about VMware) as described above would only work quite well when you have very similar or equal connections. If for example 2 links have  both 10
Read more

Deep Dive on DMPO and its Performance Features (available and missing) Part 2

Image
  The following Features will be discussed in this second part of my blog TCP Optimization Techniques Dejitter Buffering SAAS Application Monitoring   Before diving into the mechanism here an important basic fact:     Remediation is never done for applications classified as LOW priority     TCP optimization TCP has some embedded traffic management capabilities for reliable traffic delivery  (window.size, slow-start, RTT handling,...)  But there are a bunch of factors which can negatively influence the performance, like Latency TCP-slow-start Last Mile network problems Out of Sequence packets Busty losses end host TCP limitations (missing fatures like SACK, windows scaling or timestamp options)  Typically TCP optimization helps on the transmission side and improve the download time for large data transfers over latency-high and lossy WAN links. But it can also be used for improving perfomance when on the receiver side the amount of buffers is limited and/or when the client is n
Read more