Orchestrator Upgrade to Version 5.2

 Recently I upgraded my VMware SD-WAN lab from version 5.0 to 5.2.

Since some previous versions the official documentation  recommends to ask TAC for help in upgrading.

 Docu 5.2: Orchestrator Upgrade

 But if you have an unsupported implementation, like I do, you need to do it on your own:

Trying to copy the new version to the VCO failed and I found out that I needed to increase a disk volume


How to increase disk volume on VCO is described in this blog:

 VCO Upgrade to 4.5.x

After increasing the disk size my copy succeeded  

NOTE: I did not copy direct to the installation directory including a renaming, as once an upgrade started that file gets deleted and if there are any problems to have to redo the copy from outside.

 vcadmin@vco-lab-254:~$ ls -l
total 2504488
drwxr-x--- 2 root    root          4096 Aug 31  2022 20220831130007
-rw-rw---- 1 vcadmin vcadmin 2564587520 Aug 10 07:57 vco-debs-signed-5.2.0.3-R5203-20230809-GA-ff5cd1917e.tar

vcadmin@vco-lab-254:~$ sudo cp vco-debs-signed-5.2.0.3-R5203-20230809-GA-ff5cd1917e.tar /var/lib/velocloud/software_update/vco_update.tar

But trying to upgrade failed


vcadmin@vco-lab-254:~$ sudo /opt/vc/bin/vco_software_update
2023-08-19 08:45:43,298 - UPGRADE - WARNING - Verification key does not exist: /var/lib/velocloud/software_update/keys/software.key
WARNING: failed to verify package identity. Proceed as untrusted [y/n] [n]: y
2023-08-19 08:45:47,218 - UPGRADE - WARNING - WARNING: installing untrusted package
2023-08-19 08:45:47,218 - UPGRADE - INFO - Loading manifest and extracting package content.
2023-08-19 08:45:47,222 - UPGRADE - ERROR - Not enough space for operation. Make sure there is at least 3077505024 bytes are available on /var/lib/velocloud/software_update/_staging

Note: you need this size FREE after the tar file has been again copied to var/lib/velocloud/software_update

 
vcadmin@vco-lab-254:~$ df -h
Filesystem               Size  Used Avail Use% Mounted on
udev                     7.9G     0  7.9G   0% /dev
tmpfs                    1.6G   13M  1.6G   1% /run
/dev/mapper/vols-root1    20G  6.7G   12G  36% /
tmpfs                    7.9G     0  7.9G   0% /dev/shm
tmpfs                    5.0M     0  5.0M   0% /run/lock
tmpfs                    7.9G     0  7.9G   0% /sys/fs/cgroup
/dev/mapper/store2-data   79G  1.5G   75G   2% /store2
/dev/mapper/store-data   394G  1.7G  377G   1% /store
/dev/mapper/store3-data  394G  631M  378G   1% /store3
/dev/mapper/vols-home    3.9G  2.4G  1.3G  66% /home
/dev/mapper/vols-tmp     2.0G   56K  1.8G   1% /tmp
/dev/mapper/vols-var1    3.9G  1.4G  2.3G  38% /var
/dev/mapper/vols-vartmp  3.9G   48K  3.7G   1% /var/tmp
/dev/mapper/vols-log      25G   15G  9.2G  61% /var/log
/dev/mapper/vols-audit   2.0G   50M  1.8G   3% /var/log/audit
tmpfs                    1.6G     0  1.6G   0% /run/user/1001
vcadmin@vco-lab-254:~$

So I had to extend the /var volume as well and find the correct LVM for this directory:

vcadmin@vco-lab-254:~$ findmnt -n -o SOURCE --target /var/lib/velocloud/software_update/_staging
/dev/mapper/vols-var1

The real problem can arise when both extends do not work as the physical device /dev/sda/ with the logical /dev/sda1/ and /dev/sda2 uses the 64GB already.

So you need to extend the disk size (in my case on vsphere) first, then extend sda, then extend sda2 and then extend the necessary vols-home and vols-var1 directory.

NOTE: extending disks does only work when you have no snapshots configured, so either make a backup or clone of your VM before starting your actions

Step 1: on vsphere got to edit settings and increase the harddisk 1 (in my case from 64GB to 96GB)

Step 2: using parted increase partition 2 (sda2) to maximum

vcadmin@vcolabdr:~$ sudo parted /dev/sda
GNU Parted 3.2
Using /dev/sda

(parted) unit s print free
Model: VMware Virtual disk (scsi)
Disk /dev/sda: 201326592s
Sector size (logical/physical): 512B/512B
Partition Table: msdos
Disk Flags:

Number  Start       End         Size        Type     File system  Flags
        63s         2047s       1985s                Free Space
 1      2048s       204799s     202752s     primary  fat32        boot
        204800s     206847s     2048s                Free Space
 2      206848s     134217727s  134010880s  primary               lvm
        134217728s  201326591s  67108864s            Free Space


(parted) resizepart 2
End?  [134217727s]? 201326591s
(parted) unit s print free
Model: VMware Virtual disk (scsi)
Disk /dev/sda: 201326592s
Sector size (logical/physical): 512B/512B
Partition Table: msdos
Disk Flags:

Number  Start    End         Size        Type     File system  Flags
        63s      2047s       1985s                Free Space
 1      2048s    204799s     202752s     primary  fat32        boot
        204800s  206847s     2048s                Free Space
 2      206848s  201326591s  201119744s  primary               lvm

(parted) quit
Information: You may need to update /etc/fstab.

vcadmin@vcolabdr:~$ lsblk
NAME            MAJ:MIN RM  SIZE RO TYPE MOUNTPOINT
sda               8:0    0   96G  0 disk
??sda1            8:1    0   99M  0 part
??sda2            8:2    0 95.9G  0 part

Step 3: resize the PV of sda2

vcadmin@vcolabdr:~$ sudo pvresize /dev/sda2
  Physical volume "/dev/sda2" changed
  1 physical volume(s) resized / 0 physical volume(s) not resized

Step 4: resize the logical volumes

vcadmin@vcolabdr:~$ sudo lvresize -r -L +20G /dev/mapper/vols-var1
  Size of logical volume vols/var1 changed from 4.00 GiB (1024 extents) to 24.00 GiB (6144 extents).
  Logical volume vols/var1 successfully resized.
resize2fs 1.44.1 (24-Mar-2018)
Filesystem at /dev/mapper/vols-var1 is mounted on /var; on-line resizing required
old_desc_blocks = 1, new_desc_blocks = 3
The filesystem on /dev/mapper/vols-var1 is now 6291456 (4k) blocks long.

Step 4: check sizes and reboot

vcadmin@vcolabdr:~$ sudo fdisk -l /dev/sda
Disk /dev/sda: 96 GiB, 103079215104 bytes, 201326592 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: dos
Disk identifier: 0xf1193695

Device     Boot  Start       End   Sectors  Size Id Type
/dev/sda1  *      2048    204799    202752   99M 83 Linux
/dev/sda2       206848 201326591 201119744 95.9G 8e Linux LVM

 

vcadmin@vcolabdr:~$ df -h
Filesystem               Size  Used Avail Use% Mounted on
udev                     7.9G     0  7.9G   0% /dev
tmpfs                    1.6G   12M  1.6G   1% /run
/dev/mapper/vols-root1    20G  6.7G   12G  36% /
tmpfs                    7.9G     0  7.9G   0% /dev/shm
tmpfs                    5.0M     0  5.0M   0% /run/lock
tmpfs                    7.9G     0  7.9G   0% /sys/fs/cgroup
/dev/mapper/store-data   394G  7.1G  371G   2% /store
/dev/mapper/store2-data   79G  1.8G   74G   3% /store2
/dev/mapper/store3-data  394G  423M  378G   1% /store3
/dev/mapper/vols-home     11G  148K   11G   1% /home
/dev/mapper/vols-var1     24G  1.4G   22G   6% /var
/dev/mapper/vols-tmp     2.0G   56K  1.8G   1% /tmp
/dev/mapper/vols-vartmp  3.9G   48K  3.7G   1% /var/tmp
/dev/mapper/vols-log      25G  8.3G   15G  36% /var/log
/dev/mapper/vols-audit   2.0G   53M  1.8G   3% /var/log/audit
tmpfs                    1.6G     0  1.6G   0% /run/user/1001
vcadmin@vcolabdr:~$ reboot

##############################################################

Now test that the VCO is working correctly before continuing with the upgrade

############################################################################

Another problem can happen, when you start the upgrade from an SSH session and the session times out during the upgrade procedure.

as some parts are already upgraded and some parts are still missing, a normal redo of the standard upgrade file can be dangerous, so you can without problems continue by using:

 sudo dpkg --configure -a

Always make sure that you rebooted the VCO after the upgrade has been finished, and be patient as after the upgrade the SSH login can be quite quick, but it can take some time before the system has all modules ready and some data converted before starting the web-server and allowing HTML access.

*************************************************************

In my next blog I will give you some infos about changed hierarchies regarding configuration and also about some nice new features... stay tuned

Comments

Post a Comment

Popular posts from this blog

Deep Dive on DMPO and its Performance Features (available and missing) Part 1

Image
 In the last year I got quite deep insight into ther vendors SD-WAN implementation and also have seen what various vendors are critizising about the VMware DMPO features. Also there are a lot of misconceptions on the market around some of these features. So let´s try and evaluate and look a little bit deeper into typical performance features and the VMware implementation of them Features discussed in this Blog (Part 1) Per Packet Load Balancing Forward Error Correction (FEC) Packet Replication Latency Remediation The following Features will be discussed in Part 2 of my blog TCP Optimization Techniques Dejitter Buffering SAAS Application Monitoring   Per Packet Load Balancing  Unfortunately this term can be quite easily misinterpreted, when you Google for the term and get the following result:  This behaviour (often criticized by other vendors about VMware) as described above would only work quite well when you have very similar or equal connections. If for example 2 ...
Read more

Deep Dive on DMPO and its Performance Features (available and missing) Part 2

Image
  The following Features will be discussed in this second part of my blog TCP Optimization Techniques Dejitter Buffering SAAS Application Monitoring   Before diving into the mechanism here an important basic fact:     Remediation is never done for applications classified as LOW priority     TCP optimization TCP has some embedded traffic management capabilities for reliable traffic delivery  (window.size, slow-start, RTT handling,...)  But there are a bunch of factors which can negatively influence the performance, like Latency TCP-slow-start Last Mile network problems Out of Sequence packets Busty losses end host TCP limitations (missing fatures like SACK, windows scaling or timestamp options)  Typically TCP optimization helps on the transmission side and improve the download time for large data transfers over latency-high and lossy WAN links. But it can also be used for improving perfomance when on the receiver side the amount ...
Read more

Secure Edge CLI Access: Addditional Useful Commands and Parameter

Image
 additionally to the possible commands and paramteres described in the docu Secure Edge CLI: Output Examples V5.2 there are a lot of useful commands and procedures you can use  Here some of the commands which does not have a direct representation in the Remote Diagnostics debug --dec  <shows the actual detailed status and QOE for each path RX and TX>   debug --peer_stats all <shows packet counts and peer-is for all peers> the peer-id is needed for a detailed dump of peer statistics per peer diag PATHS_DUMP --peer <peer-id> <shows counter and QOE parameters for all paths to a specific peer> for a nice display you need a terminal window of 350 chars/line if you have priviledged shell access you can also use cd /opt/vc/bin in this directory you find a lot of internal but useful prgrams, phyton or shell scripts like...  ./dispcnt here an example for usage:   ./dispcnt -s  bgp -t 15 -z   shows all non-zero bgp counter updated e...
Read more