"Orchestrator Blues" Part 2: VCO and VCG password traps and recovery actions

 
When you set up new Orchestrator and Gateways it typically takes some time until you need to relogin via SSH or via Console in order to do for example an upgrade to a new version.

Default password expiration is typically set to 90 days.

Normally when for an account the password is expired you still can login via console and specifiy a new password afterwards.

Unfortunately VMware SD WAN Orchestrator and Gateways have a specific setting, that completely disables login of vcadmin after the first 90 days without any password change even on console.

In that case you are left with unreachable VCO and VCGs regarding CLI.

So the only way to overcome is a password reset procedure:

And there are as far as I have seen, 2 ways to do such a password reset:

1.  Password Reset via GRUB Recovery Mode 
2.  Reconnect of Disk to separate Unix System and doing Password  Reset from there

As the first method is much simpler you should always try that one, before using the 2nd one.

WARNING: Be careful and have a Snapshot or backup done before attempting the following procedures, the author takes no responsibility for the correctness of the following commands and sequences. 

Password Reset via GRUB Recovery Mode

a.) Connect to VCO (VCG) via Console

b1.) Send <CTRL+ALT+DEL> to Console

or

b2.) Reboot Virtual Machine

c.) press <SHIFT> during reboot until GRUB screen turns up

d.) by using the drop-down menu select the OS version stating recovery mode and press <Enter>

e.) after starting of the mode wait until it prompt you to select recovery mode

f.) when the root@<VCO> prompt is there remount the disk in RW mode by using

     mount -o remount,rw /

g.) now you can set a new password by entering passwd vcadmin

h.) finish by using  reboot to start the device in normal mode. 

ALTERNATION for method 1 after step e:
If the system asks you for a root password in maintenance mode or asks you to press <CTRL> D then:
1.reboot again
2.press <SHIFT>
3. when in GRUB menu highlight the recovery line but do not press <Enter>
4. Press e to go into edit mode
5. scroll down to a line starting with linux
6. near the end of the line replace ro with rw and at the end of line add init="/bin/bash"
7, press F10 or <CTRL> X to continue
8. when prompted enter passwd vcadmin and set the new password
9. boot the VCO again and all should be okay

Reconnect of Disk to separate Unix System and doing Password  Reset from there

a.) shutdown the device

b.) mount the disk on another linux system as second disk

c.) start that system 

d.) Activate the correct LVM partition of VCO or VCG

     pvscan --cache -aay /dev/nbd0p2

e.) Mount the root volume

     mount /dev/vols/root1 /mnt

f.) change root and set new root password

     chroot /mnt

     passwd root

g.) exit and unmount the disk

     umount /mnt

     vgchange -an vols 

h.) shutdown the linux system and reconnect disk to original VCO or VCG VM 

i.) start up the VCO or VCG and login via root and the specified password

j.) reset the vcadmin password lock and set new vcadmin password

     pam_tally2 --reset

     passwd -w -1 -x 99999 vcadmin

     passwd vcadmin

 

 

Both methods presented allow you to continue without needed to reapply and reconfigure the whole device.