Upgrading to 7.0.0

The workflow for upgrading Key Manager to 7.0.0 involves the following steps:

Stop the Key Manager system processes to prevent management jobs from running during update (Stopping the Key Manager System Processes).
Backing up the Key Manager Database (Optional, but highly recommended; Backing Up Recovery Data).
Upgrading Key Manager software packages, and the Key Manager Database (Upgrading Key Manager Software and Database).
Restarting Key Manager processes (Restarting Key Manager Processes).

These steps are described in more detail in the following subsections.

Preparing for Upgrade

Before upgrading Key Manager, please take the following into account:

Before upgrading your production Key Manager system, we strongly recommend that you perform a test upgrade of Key Manager using a test system. The test upgrade should be conducted in an environment created for testing and should not interfere with actual production systems.
During upgrade, you will be required to stop the Tectia SSH Server on all the Key Manager back ends. When Tectia SSH Server is stopped, you will be unable to establish new SSH connections to it. Ensure that you can maintain terminal access to your Key Manager back ends during upgrade. This can be accomplished by, for example:
- Opening a persistent SSH terminal session (such as a SSH terminal session over screen) to the Key Manager back ends before stopping Tectia SSH Server on them. The advantage of this approach is that you can resume upgrade procedures even if your SSH session disconnects abruptly.
  
  Note that it is important to establish the connection before Tectia SSH Server is stopped: When Tectia SSH Server is stopped, existing SSH connections remain open. However, it is not possible to establish new SSH connections to the Tectia SSH Server while it is stopped.
- Having another SSH server available on the Key Manager back ends during upgrade. The alternate SSH server must run on a different port than the Tectia SSH Server.
- Having physical terminal access to the Key Manager back ends during upgrade.
You must have the Key Manager installation package version 7.0.0 available on all the Key Manager Server machines. Also ensure that the package is intended for your Key Manager Server platform.

You can check the version and the platform from the name of the installation package, which should be similar to the following:

sshmgr-7.0.0-*.x86_64.tar
You must have a valid license for the Key Manager version you are upgrading to.
The Tectia SSH Server installations on Key Manager back ends must be upgraded to the version described in Key Manager Server Requirements. Installation packages for the required version of Tectia SSH Server are provided in the Key Manager 7.0.0 installation packages. Instructions for upgrading Tectia SSH Server are provided later.
Key Manager version 4.2 introduced Tectia SSH Server version 6.6.1, which can use Post Quantum Crypto graphic (PQC) algorithms. The use of PQC algorithms requires an upgrade to the Tectia license. Please contact customer support to acquire a new Tectia license.
Some Key Manager custom plugins may only work with newer versions of Key Manager. If your Key Manager deployment uses custom plugins, consult SSH Communications Security Corporation customer service about plugin compatibility with the Key Manager version you are upgrading to. You can contact SSH Communications Security Corporation customer service at https://support.ssh.com/.
All Key Manager Servers and User Portals that belong to the same deployment must be upgraded to the same target version.

Key Manager agents shipped with the supported source versions also function with the target version of the system. Note that while previous versions of Key Manager agent may work with newer versions of the system, they typically do not support the Key Manager features that were introduced in subsequent system versions.
The upgrade requires interaction throughout.

caution
Any hotfixes that have been installed on Key Manager servers must be removed before the start of the upgrade process.

For information about removing hotfixes from Key Manager versions 2.X, contact support at SSH Communications Security. If upgrading from version 3.0.0 to a newer version, see Removing a Hotfix from Key Manager for more information about removing hotfixes.

Stopping the Key Manager System Processes

Key Manager must not be running any worker processes during upgrade - in other words, there must not be any jobs in pending, queued or running state at upgrade time.

In addition, Key Manager processes must be stopped for the duration of the upgrade:

On all the Key Manager back ends, stop the Tectia SSH Server. This prevents Key Manager from starting worker processes for agent-based hosts during Key Manager upgrade.
```
# service ssh-server-g3 stop
```
caution
It is not possible to establish new SSH connections to an SSH server while it is down!
Make sure that you retain the ability to restart the SSH server while it is down. This can be accomplished by, for example, leaving an open SSH terminal connection to the Key Manager back end before stopping the SSH server (stopping the SSH server does not terminate existing SSH connections), by having another SSH server on the Key Manager back end (on another port), or by having direct access to the command-line client on the machine.
To prevent Key Manager from starting new worker processes for agentless hosts and other management jobs, set the maximum-process limit to 0:

Access the Key Manager GUI. On the Settings→General→Backend page, remember the current value of the setting Maximum processes (this value should be set back after upgrade), then set Maximum processes to 0. Click Apply to apply the changes.
On all the Key Manager back ends, wait for all running Key Manager worker processes to complete.
Currently running worker processes can be checked with the following command on the Key Manager back ends:
```
ps -fade | grep 'ssh_mgr_worker'
```
The command should only return the grep process itself after all the worker processes have finished.
To prevent internal jobs from being created, stop the back-end service on all Key Manager back ends.
```
# supervisorctl stop backend:
```
If any jobs are still queued or pending, use the Key Manager command-line client to terminate them.
```
$ ssh-mgr-client cancel-jobs
```
For more information about using command-line client, see the PrivX Key Manager Administrator Manual.
On all the Key Manager front ends, stop the Nginx web-server process, which serves the Key Manager GUI.
```
# service nginx stop
```
Stop the Key Manager Server services on all the Key Manager Servers.
```
# supervisorctl stop all
```
Then verify the status of the Key Manager Server services with:
```
# supervisorctl status
```
Also ensure that no celery, redis, or gunicorn processes are running. You can do this with the following command:
```
# ps -fade | grep "celery\|redis\|gunicorn"
```
Kill any processes returned by the command (except for the grep process itself) before proceeding.

Backing Up Recovery Data

We strongly suggest that you back up the data required for restoring the current version of Key Manager, in case you encounter problems during upgrade. To be able to restore the current version of Key Manager after upgrade, you must have the following backups:

A backup of the Key Manager Database.
A backup of all the Key Manager Server settings files.
A backup of any Key Manager Server files which have been modified.

Back up the Key Manager Database before upgrading your Key Manager system. The database backup is required in case you later need to restore the pre-upgrade version of Key Manager.

For information about backing up and restoring your database, please consult the documentation provided by your database vendor.
On each Key Manager Server, back up the Key Manager Server settings files:
- Environment variables at /etc/sysconfig/sshmgr
- Local settings at /opt/sshmgr/app/localsettings.py
- Settings file at /opt/sshmgr/app/settings.py
- Web-server-configuration file at /etc/nginx/sites-available/sshmgr (Key Manager front ends only).
Additionally, if you have made custom changes to any Key Manager Server files, these files must also be backed up.

note
Remember the file location of each file you back up. If you need to perform restoration at a later time, you must be able to restore all files to their former locations.

tip
If you are running Key Manager Servers on virtual appliances, you can create a snapshot of those virtual appliances, instead of manually backing up files on them.
Back up any custom plugins that you have installed to Key Manager. Plugins are located on Key Manager Servers, typically under the following path:

/opt/sshmgr/plugins

Upgrading Key Manager Software and Database

Unless otherwise stated, the instructions in this section must be performed on all the Key Manager Servers.

note

On Key Manager environments using Oracle databases, before upgrading Key Manager servers, you must copy the Oracle database timezone file to all Key Manager servers for succesful database connections.

Acquire the Oracle timezone file from your database Administrator. On the Oracle server you can find out which timezone file is being used by running the SQL commands in the following example:

SQL> select *
from v$timezone_file;
FILENAME VERSION CON_ID

timezlrg_32.dat 32 0

You can get this file from the database server in ORACLE_HOME\oracore\zoneinfo.

Copy the timezone file to all Key Manager servers in /opt/sshmgr/.runtime/lib/oracore/zoneinfo/ directory. The oracore directory, and directories and files under it need to be owned and readable by user sshmgr:tectia. Key Manager will use the timezone file with the largest number in the filename from this directory.

Navigate to the directory where you have the Key Manager installation package. Extract the Key Manager 7.0.0 installation package to the current directory:
```
# tar -xvf sshmgr-7.0.0-*.x86_64.tar
```
Navigate to the created upgrade-package directory:
```
# cd sshmgr-7.0.0-*/
```
These instructions assume that subsequent instructions are run from this location.
Install the OS-specific support packages required by Key Manager. The required packages differ depending on the OS on which you are installing Key Manager:
- For Red Hat 8.x / Rocky Linux 8.4 installations, install the following support packages:
```
# yum install os-support/rhel8/*.rpm
```
- For Red Hat 9.x / Rocky Linux 9.x installations, install the following support packages:
```
# yum install os-support/rhel9/*.rpm
```
The Key Manager installation package contains a version of Tectia SSH Server required for running Key Manager. To upgrade Tectia SSH Server using the provided packages, run:
```
# yum install ssh-tectia-server/ssh-tectia-common-*.rpm
# yum install ssh-tectia-server/ssh-tectia-server-*.rpm
```
If some of the support packages could not be installed due to conflicts, you must remove the conflicting packages, then try again to install the missing support packages.
The supervisord process manager must be running when Key Manager Server software is upgraded. Ensure that supervisord is running with the following command:
```
# service supervisord status
```
Start supervisord if it is not already running:
```
# service supervisord start
```
Upgrade the Key Manager Server software:
```
# yum install sshmgr-7.0.0-*.x86_64.rpm
```
On Key Manager environments using an Oracle database: You must run a database upgrade script on the database machine as the database Administrator. The database upgrade script is found in the following location in Key Manager server:
```
/opt/sshmgr/examples/oracle_upgrade/oracle_table_column_rename.sql
```
To update the component settings, run the upgrade script:
```
# /opt/sshmgr/sbin/upgrade
```
Depending on the exact source and destination versions, the upgrade-script output may describe additional commands that need to be run, such as database-migration commands. Check the output of the upgrade command and run any additional commands that are required.

note
In large Key Manager deployments, database-migration commands may take considerable time to execute. If you need to run any database-migration commands, we recommend you run them via a persistent terminal session (for example, using screen sessions). This is to ensure that the command continues running even if your terminal session disconnects during execution.

Restarting Key Manager Processes

Complete the upgrade by restarting the processes related to Key Manager.

On all the Key Manager back ends, restart the SSH server used by Key Manager
```
# service ssh-server-g3 restart
```
Restart the supervisord service to ensure that it is running:
```
# service supervisord restart
```
supervisord enables process-management commands that are later used for restarting Key Manager services.

On all the Key Manager back ends, start the PrivX Key Manager service daemon and redis:

# supervisorctl start backend:
# supervisorctl start redis

On all the Key Manager front ends, start the front-end services:

# supervisorctl start frontend:
# supervisorctl start graphupdate
# supervisorctl start redis
# service nginx start

Set the maximum-process limit to allow Key Manager to resume running management jobs. To do this, access the Key Manager GUI. On the Settings→General page, under the Backend category, set Maximum processes (for example, to the value specified before upgrade). Click Apply to save the changes.

The Key Manager system is now upgraded. On Key Manager back ends, you can check that the backend: and frontend: services are running with the following command:

# supervisorctl status

If the backend: service is not running, it is possible that your Key Manager license is not valid. In this case, run the following command:

# /opt/sshmgr/bin/ssh-mgr-controller --check-license

If Key Manager Server produces an error No valid licenses found, the back end will not start and the front end will only work partially. A new license can be imported via the frontend by navigating to Settings → Licenses. You must restart the frontend: service and start the backend: service after importing the license by giving the following commands:

# supervisorctl restart frontend:
# supervisorctl start backend:

Post-Upgrade Tasks

After you have verified that the Key Manager system has been upgraded successfully, perform the following tasks to enable the features introduced in the new Key Manager version.

Importing a valid license is intructed in Import the PrivX Key Manager Software License.

Upgrading Key Manager Agents

After you have verified that your Key Manager system has been upgraded successfully, you may upgrade the Key Manager agents installed on your agent-based hosts. Upgrading Key Manager agents enables support for features introduced by the new version of Key Manager.

Upgrading Key Manager agents is highly recommended, albeit optional. Previous versions of Key Manager agents typically support the management actions introduced in their respective versions.

For instructions about upgrading Key Manager agents, see Upgrading Key Manager Agents.

Updating Key Manager sudo-settings Files

On the hosts that are managed using sudo-privileged management accounts, you should update the sudosettings files to enable support for features introduced by the new version of Key Manager.

Updating sudo-settings files is highly recommended, albeit optional. Previous versions of sudo-settings files typically support the management actions introduced in their respective versions.

You can update the sudo-settings file using the example sudo-settings files. To do this, perform the following on all the hosts managed via sudo-privileged management accounts:

Optional: Back up the current Key Manager sudo-settings file. The file is located at the following path:

/etc/sudoers.d/ukm
Acquire a sudo-settings template from the following directory:

/opt/sshmgr/examples/sudoers

Make sure to take the sudo-settings template that matches to the platform of the host. Copy the sudo-settings template to /etc/sudoers.d/ukm on the managed host.
In /etc/sudoers.d/ukm, specify the name of the management account on the line that looks like the following (replace sshmgr with the name of the management account):
```
User_Alias  SSHUKM=sshmgr
```
For compatibility with the host, you may also have to modify some of the commands specified in /etc/sudoers.d/ukm

For more information about sudo setup for the management account, see the Key Manager Administrator Manual.

Preparing for Upgrade​

Stopping the Key Manager System Processes​

Backing Up Recovery Data​

Upgrading Key Manager Software and Database​

Restarting Key Manager Processes​

Post-Upgrade Tasks​

Upgrading Key Manager Agents​

Updating Key Manager sudo-settings Files​