Orchestrator Upgrade to Version 5.2
Recently I upgraded my VMware SD-WAN lab from version 5.0 to 5.2.
Since some previous versions the official documentation recommends to ask TAC for help in upgrading.
Docu 5.2: Orchestrator Upgrade
But if you have an unsupported implementation, like I do, you need to do it on your own:
Trying to copy the new version to the VCO failed and I found out that I needed to increase a disk volume
How to increase disk volume on VCO is described in this blog:
After increasing the disk size my copy succeeded
NOTE: I did not copy direct to the installation directory including a renaming, as once an upgrade started that file gets deleted and if there are any problems to have to redo the copy from outside.
vcadmin@vco-lab-254:~$ ls -l
total 2504488
drwxr-x--- 2 root root 4096 Aug 31 2022 20220831130007
-rw-rw---- 1 vcadmin vcadmin 2564587520 Aug 10 07:57 vco-debs-signed-5.2.0.3-R5203-20230809-GA-ff5cd1917e.tar
vcadmin@vco-lab-254:~$ sudo cp vco-debs-signed-5.2.0.3-R5203-20230809-GA-ff5cd1917e.tar /var/lib/velocloud/software_update/vco_update.tar
But trying to upgrade failed
vcadmin@vco-lab-254:~$ sudo /opt/vc/bin/vco_software_update
2023-08-19 08:45:43,298 - UPGRADE - WARNING - Verification key does not exist: /var/lib/velocloud/software_update/keys/software.key
WARNING: failed to verify package identity. Proceed as untrusted [y/n] [n]: y
2023-08-19 08:45:47,218 - UPGRADE - WARNING - WARNING: installing untrusted package
2023-08-19 08:45:47,218 - UPGRADE - INFO - Loading manifest and extracting package content.
2023-08-19 08:45:47,222 - UPGRADE - ERROR - Not enough space for operation. Make sure there is at least 3077505024 bytes are available on /var/lib/velocloud/software_update/_staging
Note: you need this size FREE after the tar file has been again copied to var/lib/velocloud/software_update
vcadmin@vco-lab-254:~$ df -h
Filesystem Size Used Avail Use% Mounted on
udev 7.9G 0 7.9G 0% /dev
tmpfs 1.6G 13M 1.6G 1% /run
/dev/mapper/vols-root1 20G 6.7G 12G 36% /
tmpfs 7.9G 0 7.9G 0% /dev/shm
tmpfs 5.0M 0 5.0M 0% /run/lock
tmpfs 7.9G 0 7.9G 0% /sys/fs/cgroup
/dev/mapper/store2-data 79G 1.5G 75G 2% /store2
/dev/mapper/store-data 394G 1.7G 377G 1% /store
/dev/mapper/store3-data 394G 631M 378G 1% /store3
/dev/mapper/vols-home 3.9G 2.4G 1.3G 66% /home
/dev/mapper/vols-tmp 2.0G 56K 1.8G 1% /tmp
/dev/mapper/vols-var1 3.9G 1.4G 2.3G 38% /var
/dev/mapper/vols-vartmp 3.9G 48K 3.7G 1% /var/tmp
/dev/mapper/vols-log 25G 15G 9.2G 61% /var/log
/dev/mapper/vols-audit 2.0G 50M 1.8G 3% /var/log/audit
tmpfs 1.6G 0 1.6G 0% /run/user/1001
vcadmin@vco-lab-254:~$
So I had to extend the /var volume as well and find the correct LVM for this directory:
vcadmin@vco-lab-254:~$ findmnt -n -o SOURCE --target /var/lib/velocloud/software_update/_staging
/dev/mapper/vols-var1
The real problem can arise when both extends do not work as the physical device /dev/sda/ with the logical /dev/sda1/ and /dev/sda2 uses the 64GB already.
So you need to extend the disk size (in my case on vsphere) first, then extend sda, then extend sda2 and then extend the necessary vols-home and vols-var1 directory.
NOTE: extending disks does only work when you have no snapshots configured, so either make a backup or clone of your VM before starting your actions
Step 1: on vsphere got to edit settings and increase the harddisk 1 (in my case from 64GB to 96GB)
Step 2: using parted increase partition 2 (sda2) to maximum
vcadmin@vcolabdr:~$ sudo parted /dev/sda
GNU Parted 3.2
Using /dev/sda
(parted) unit s print free
Model: VMware Virtual disk (scsi)
Disk /dev/sda: 201326592s
Sector size (logical/physical): 512B/512B
Partition Table: msdos
Disk Flags:
Number Start End Size Type File system Flags
63s 2047s 1985s Free Space
1 2048s 204799s 202752s primary fat32 boot
204800s 206847s 2048s Free Space
2 206848s 134217727s 134010880s primary lvm
134217728s 201326591s 67108864s Free Space
(parted) resizepart 2
End? [134217727s]? 201326591s
(parted) unit s print free
Model: VMware Virtual disk (scsi)
Disk /dev/sda: 201326592s
Sector size (logical/physical): 512B/512B
Partition Table: msdos
Disk Flags:
Number Start End Size Type File system Flags
63s 2047s 1985s Free Space
1 2048s 204799s 202752s primary fat32 boot
204800s 206847s 2048s Free Space
2 206848s 201326591s 201119744s primary lvm
(parted) quit
Information: You may need to update /etc/fstab.
vcadmin@vcolabdr:~$ lsblk
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
sda 8:0 0 96G 0 disk
??sda1 8:1 0 99M 0 part
??sda2 8:2 0 95.9G 0 part
Step 3: resize the PV of sda2
vcadmin@vcolabdr:~$ sudo pvresize /dev/sda2
Physical volume "/dev/sda2" changed
1 physical volume(s) resized / 0 physical volume(s) not resized
Step 4: resize the logical volumes
vcadmin@vcolabdr:~$ sudo lvresize -r -L +20G /dev/mapper/vols-var1
Size of logical volume vols/var1 changed from 4.00 GiB (1024 extents) to 24.00 GiB (6144 extents).
Logical volume vols/var1 successfully resized.
resize2fs 1.44.1 (24-Mar-2018)
Filesystem at /dev/mapper/vols-var1 is mounted on /var; on-line resizing required
old_desc_blocks = 1, new_desc_blocks = 3
The filesystem on /dev/mapper/vols-var1 is now 6291456 (4k) blocks long.
Step 4: check sizes and reboot
vcadmin@vcolabdr:~$ sudo fdisk -l /dev/sda
Disk /dev/sda: 96 GiB, 103079215104 bytes, 201326592 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: dos
Disk identifier: 0xf1193695
Device Boot Start End Sectors Size Id Type
/dev/sda1 * 2048 204799 202752 99M 83 Linux
/dev/sda2 206848 201326591 201119744 95.9G 8e Linux LVM
vcadmin@vcolabdr:~$ df -h
Filesystem Size Used Avail Use% Mounted on
udev 7.9G 0 7.9G 0% /dev
tmpfs 1.6G 12M 1.6G 1% /run
/dev/mapper/vols-root1 20G 6.7G 12G 36% /
tmpfs 7.9G 0 7.9G 0% /dev/shm
tmpfs 5.0M 0 5.0M 0% /run/lock
tmpfs 7.9G 0 7.9G 0% /sys/fs/cgroup
/dev/mapper/store-data 394G 7.1G 371G 2% /store
/dev/mapper/store2-data 79G 1.8G 74G 3% /store2
/dev/mapper/store3-data 394G 423M 378G 1% /store3
/dev/mapper/vols-home 11G 148K 11G 1% /home
/dev/mapper/vols-var1 24G 1.4G 22G 6% /var
/dev/mapper/vols-tmp 2.0G 56K 1.8G 1% /tmp
/dev/mapper/vols-vartmp 3.9G 48K 3.7G 1% /var/tmp
/dev/mapper/vols-log 25G 8.3G 15G 36% /var/log
/dev/mapper/vols-audit 2.0G 53M 1.8G 3% /var/log/audit
tmpfs 1.6G 0 1.6G 0% /run/user/1001
vcadmin@vcolabdr:~$ reboot
##############################################################
Now test that the VCO is working correctly before continuing with the upgrade
############################################################################
Another problem can happen, when you start the upgrade from an SSH session and the session times out during the upgrade procedure.
as some parts are already upgraded and some parts are still missing, a normal redo of the standard upgrade file can be dangerous, so you can without problems continue by using:
sudo dpkg --configure -a
Always make sure that you rebooted the VCO after the upgrade has been finished, and be patient as after the upgrade the SSH login can be quite quick, but it can take some time before the system has all modules ready and some data converted before starting the web-server and allowing HTML access.
*************************************************************
In my next blog I will give you some infos about changed hierarchies regarding configuration and also about some nice new features... stay tuned
Comments
Post a Comment