Rolling Upgrades

A rolling upgrade involves upgrading multiple OneSpan Authentication Server instances while keeping the authentication service alive. Environments that require a rolling upgrade typically support high-availability services, where the authentication service absolutely cannot be taken offline.

An environment that requires a rolling upgrade typically has the following characteristics:

• There are multiple instances of OneSpan Authentication Server running on multiple servers.
• All OneSpan Authentication Server instances either use the same database as their data store, or each one instance has its own data store.
• The OneSpan Authentication Server upgrades involved will require a database schema update.
• User load distribution between all OneSpan Authentication Server instances is managed by a third-party application.

Different setup scenarios for rolling upgrades will be discussed in a separate topic (see Rolling upgrade scenarios).

General considerations

Schema updates and data migration

Depending on your database setup, upgrading will most likely involve a database schema update. This is not an issue when you are allowed to take your authentication service offline to upgrade, or if each instance has its own database.

However, when performing a rolling upgrade of multiple OneSpan Authentication Server instances using the same database, a database schema update raises the following issues:

• Both the old and new versions of OneSpan Authentication Server need the database to use the correct schema version.
• Any database writes that occur during the upgrade procedure must be persistent.
• Any database changes that occur during a schema update can cause data corruption.
• The database cannot be taken offline.

To address these issues, the database must initially be duplicated right before upgrading the first instance. The schema of the duplicate database will be upgraded to the required version as the first instance is upgraded. The original database will be used by the remaining instances still using the old database schema.

As each instance is upgraded, they should be configured to use the database with the updated schema. In addition, you will need to configure replication between servers. Replication ensures that no database changes are lost during the rolling upgrade or database schema update. For more information about setting up replication, see Replication. For more information about performing schema updates, see ODBC database manual setup.

After the schema update, OneSpan Authentication Server server data from the previous installation must be migrated to match the new database schema. To ensure that authentication services remain available at all times, data is migrated using two complementary mechanisms:

• On-the-fly data migration

  On-the-fly data migration means that data is migrated on demand whenever OneSpan Authentication Server receives a request (e.g. an authentication request) and accesses server data records. Only data records required to process the request are migrated, whereas data records which are newly created or have already been migrated will not be processed (a second time).

  On-the-fly data migration is only triggered upon updating or reading a data record; it is not initiated by queries (listing several data records).

• Task-based data migration

  In addition, to systematically migrate all server data from the old installation, you need to start or schedule a data migration task in the Administration Web Interface. This task will run in the background and migrate all database records one-by-one to the new schema, except for server data which has already been migrated on-the-fly.

  For rolling upgrades, this task can only run on one OneSpan Authentication Server instance, regardless of whether the instances share one database or use separate databases. In scenarios where multiple databases are used, the migrated server data is replicated to all other databases.

Each server data record is migrated only once, either on-the-fly or in the course of the data migration task.

To avoid replication queue issues, the data migration task needs to run after all OneSpan Authentication Server instances have been upgraded, and when replication between all instances is in place again.

For an overview of the OneSpan Authentication Server upgrade and server data migration process, refer to the OneSpan Authentication Server Product Guide. For instructions to create and schedule a data migration task after an OneSpan Authentication Server upgrade, refer to the OneSpan Authentication Server Installation Guide for Windows or the OneSpan Authentication Server Installation Guide for Linux.

Replication

During the rolling upgrade scenarios (see Rolling upgrade scenarios), you will be requested to do the following:

• Initially configure replication in a certain direction between OneSpan Authentication Server instances.

  The replication direction is crucial to ensure continued service, data integrity, and security throughout the rolling upgrade. If you are instructed to configure replication from server A to server B, then server B should not be able to replicate to server A (unless explicitly instructed).

• Break and restore replication between OneSpan Authentication Server at specific times.

  When asked to break or restore replication in a certain direction, do so at the network level. We recommend that you use the system firewall to break the replication.

  Do NOT disable the replication on the servers. Do NOT remove the replication configuration from the server. Otherwise, replication messages are omitted, leaving the server databases unsynchronized and in different states!

Breaking and restoring replication between two servers

To break replication between two servers during a rolling upgrade (from server A to server B)

1. Block incoming connections on server B from server A on the replication port (default 20003).

   You can verify the correct port in the Configuration Utility under Replication / Destination Servers.

2. Restart the OneSpan Authentication Server service on server B.
3. Verify the replication break with the Audit Viewer on server B.

To restore replication between two servers during a rolling upgrade

1. Unblock incoming connections to server B from server A on the replication port.
2. Restart the OneSpan Authentication Server service on server B (presumably, after upgrading server B).
3. Verify that the replication is restored correctly with the Audit Viewer on server A.

Replication queue monitoring

During the rolling upgrade, you will be requested to wait until the replication queue to a server is empty right before breaking the replication. You can verify this by using the Audit Viewer on the server you are replicating to and filter on replication events.

We recommend that you create a Filter in the Audit Viewer for all Audit Message Types and with Field Category equals Replication for easy viewing of the replication events. For more information, see System monitoring.

When you no longer see new messages appearing, the queue is empty.

Alternatively, you can verify the number of messages in the replication queue using the Administration Web Interface, i.e. in the Replication tab available via SYSTEM > Server Status.

Usability issues

During the rolling upgrade, the following problems can occur:

• Two-step authentications may fail if a server is taken offline before the authentication is completed.
• Software authenticator provisioning, auto-assignment, and Dynamic User Registration (DUR) may be made invalid if these features are used during the rolling upgrade.

To avoid these problems, disable the policies relating to these features before beginning the rolling upgrade process. These issues are valid whether or not all OneSpan Authentication Server instances share databases.

Load management

The procedures for the rolling upgrade scenarios (see Rolling upgrade scenarios) include steps to disable user load distribution to specific OneSpan Authentication Server instances. This step always precedes the shutdown and upgrade of the instance.

After disabling user load distribution to one particular OneSpan Authentication Server instance, always remember to wait for the instance to finish processing its entire load, i.e. write the necessary changes to database. Once this is done, it should be safe to disable and upgrade the OneSpan Authentication Server instance.

Specific instructions how to disable user load distribution are not provided in this guide, as it is assumed that a load-management solution is already in place for this purpose.

The instructions provided in Rolling upgrade scenario: Three OneSpan Authentication Server instances with individual databases begin with the upgrade of a single OneSpan Authentication Server instance. As soon as that instance is upgraded, it becomes the sole service provider, as you need to take the other instances offline to upgrade them.

As such, you should schedule the rolling upgrade to a time when the load is expected to be low enough to be manageable by just one OneSpan Authentication Server instance.