"Orchestrator Blues" Part 1: VCO Upgrade to 4.5.x

In this blog I want to share some findings gathered in the last year regarding not so well documented traps when setting up and maintaining your own Orchestrator and Gateways

 So let's start with a recent one:

VCO Update 4.5 fails with OSError: [Errno 28] No space left on device

 When upgrading to version 4.5.x a general error arises in case you never have enlarged the size of a specific LVM volume on the VCO

vcadmin@vco-01:~$ sudo /opt/vc/bin/vco_software_update                         2021-10-12 09:39:13,817 - UPGRADE - WARNING - Verification key does not exist: /var/lib/velocloud/software_update/keys/software.key

WARNING: failed to verify package identity. Proceed as untrusted [y/n] [n]: y

2021-10-12 09:39:17,908 - UPGRADE - WARNING - WARNING: installing untrusted package

2021-10-12 09:39:17,908 - UPGRADE - INFO - Loading manifest and extracting package content.

Traceback (most recent call last):

  File "/usr/lib/python3.6/tarfile.py", line 2173, in makefile

    copyfileobj(source, target, tarinfo.size, ReadError, bufsize)

  File "/usr/lib/python3.6/tarfile.py", line 252, in copyfileobj

    dst.write(buf)

OSError: [Errno 28] No space left on device

 even after a cleanup of some old file from previous upgrades the problem is still there

vcadmin@vco-01:~$ sudo docker system prune -af

Deleted Images:

untagged: vco-i18n:R420-20210122-GA-f8580b27e2

deleted: sha256:2354c0589193a08d13eeff2d47c631e3aafcabd665863ca2f031a87eda5ee486

deleted: sha256:05a9d892eb19dca804571ba10146f2bad65a78bbce7a5779d21d3b5d45a472c9

untagged: vco-i18n:R430-20210527-GA-de848500e8

Total reclaimed space: 62.42MB

 if you look into the disk usage

vcadmin@vco-01:~$ df -h

Filesystem               Size  Used Avail Use% Mounted on

udev                     7.9G     0  7.9G   0% /dev

tmpfs                    1.6G   11M  1.6G   1% /run

/dev/mapper/vols-root1    12G  6.8G  4.5G  61% /

tmpfs                    7.9G     0  7.9G   0% /dev/shm

tmpfs                    5.0M     0  5.0M   0% /run/lock

tmpfs                    7.9G     0  7.9G   0% /sys/fs/cgroup

/dev/mapper/store2-data   79G  2.2G   74G   3% /store2

/dev/mapper/store3-data  394G  1.2G  377G   1% /store3

/dev/mapper/store-data   394G  5.3G  373G   2% /store

/dev/mapper/vols-home    2.0G  6.1M  1.8G   1% /home

/dev/mapper/vols-var1    3.9G  2.4G  1.3G  65% /var

/dev/mapper/vols-log      25G  4.4G   19G  19% /var/log

/dev/mapper/vols-tmp     2.0G  6.1M  1.8G   1% /tmp

/dev/mapper/vols-audit   2.0G  715M  1.1G  39% /var/log/audit

/dev/mapper/vols-vartmp  2.0G  6.1M  1.8G   1% /var/tmp

tmpfs                    1.6G     0  1.6G   0% /run/user/1001

overlay                  3.9G  2.4G  1.3G  65% /var/lib/docker/overlay2/f7f2bba62fbb32cf18a78452fdfa521cc61cbc18925bddceab7ab5c229c5105b/merged


you see some disks at around 60% but nothing higher

But some of the Logical Volumes are rather small, so they can be get filled quickly.

After consulting PS from VMware they told me that i need to increase disk size of  the /var LVM

so by configuring

vcadmin@vco-01:~$ sudo lvextend -L +4G /dev/mapper/vols-var1

  Size of logical volume vols/var1 changed from 4.00 GiB (1024 extents) to 8.00 GiB (2048 extents).

  Logical volume vols/var1 successfully resized.

vcadmin@vco-01:~$ sudo resize2fs /dev/mapper/vols-var1

resize2fs 1.44.1 (24-Mar-2018)

Filesystem at /dev/mapper/vols-var1 is mounted on /var; on-line resizing required

old_desc_blocks = 1, new_desc_blocks = 1

The filesystem on /dev/mapper/vols-var1 is now 2097152 (4k) blocks long.

vcadmin@vco-01:~$ df -h /var

Filesystem             Size  Used Avail Use% Mounted on

/dev/mapper/vols-var1  7.9G  605M  6.9G   8% /var

vcadmin@vco-01:~$ sudo reboot

 the disk size was increased from 4 to 8 GB and then the upgrade worked without any problem.

 PS: i wrote this block because even after 2 months later I did not find according information, neither in Release Notes nor in any VMware KB document.

Other items which will follow in further parts of my Blog on "Orchestrator Blues"
  • remote diagnostics not working
  • login problems after 90 days
  • password recovery


Comments

Post a Comment

Popular posts from this blog

Deep Dive on DMPO and its Performance Features (available and missing) Part 1

Image
 In the last year I got quite deep insight into ther vendors SD-WAN implementation and also have seen what various vendors are critizising about the VMware DMPO features. Also there are a lot of misconceptions on the market around some of these features. So let´s try and evaluate and look a little bit deeper into typical performance features and the VMware implementation of them Features discussed in this Blog (Part 1) Per Packet Load Balancing Forward Error Correction (FEC) Packet Replication Latency Remediation The following Features will be discussed in Part 2 of my blog TCP Optimization Techniques Dejitter Buffering SAAS Application Monitoring   Per Packet Load Balancing  Unfortunately this term can be quite easily misinterpreted, when you Google for the term and get the following result:  This behaviour (often criticized by other vendors about VMware) as described above would only work quite well when you have very similar or equal connections. If for example 2 ...
Read more

Deep Dive on DMPO and its Performance Features (available and missing) Part 2

Image
  The following Features will be discussed in this second part of my blog TCP Optimization Techniques Dejitter Buffering SAAS Application Monitoring   Before diving into the mechanism here an important basic fact:     Remediation is never done for applications classified as LOW priority     TCP optimization TCP has some embedded traffic management capabilities for reliable traffic delivery  (window.size, slow-start, RTT handling,...)  But there are a bunch of factors which can negatively influence the performance, like Latency TCP-slow-start Last Mile network problems Out of Sequence packets Busty losses end host TCP limitations (missing fatures like SACK, windows scaling or timestamp options)  Typically TCP optimization helps on the transmission side and improve the download time for large data transfers over latency-high and lossy WAN links. But it can also be used for improving perfomance when on the receiver side the amount ...
Read more

Secure Edge CLI Access: Addditional Useful Commands and Parameter

Image
 additionally to the possible commands and paramteres described in the docu Secure Edge CLI: Output Examples V5.2 there are a lot of useful commands and procedures you can use  Here some of the commands which does not have a direct representation in the Remote Diagnostics debug --dec  <shows the actual detailed status and QOE for each path RX and TX>   debug --peer_stats all <shows packet counts and peer-is for all peers> the peer-id is needed for a detailed dump of peer statistics per peer diag PATHS_DUMP --peer <peer-id> <shows counter and QOE parameters for all paths to a specific peer> for a nice display you need a terminal window of 350 chars/line if you have priviledged shell access you can also use cd /opt/vc/bin in this directory you find a lot of internal but useful prgrams, phyton or shell scripts like...  ./dispcnt here an example for usage:   ./dispcnt -s  bgp -t 15 -z   shows all non-zero bgp counter updated e...
Read more