# Upgrading Cluster Software

## Version Compatibility

When releasing newer versions we take care to make sure we are backwardly compatible with older versions. While you will always be able to upgrade from version x to x+1, in order to continue to produce high quality software it may occasionally be necessary to drop compatibility with older versions.

There will always be an upgrade path from any series-2 release to any other series-2 release.

There are three approaches to upgrading your cluster software

- Complete Cluster Shutdown

- Rolling (node by node)

- Disconnect and Reattach

Each method has advantages and disadvantages, some of which are listed in the table below, and you should chose the one most appropriate to your needs.

### Summary of Upgrade Methodologies

Type | Available between all software versions | Service Outage During Upgrade | Service Recovery During Upgrade | Exercises Failover Logic/Configuration | Allows change of cluster stack type For example, switching from Heartbeat to Corosync. Consult the Heartbeat or Corosync documentation to see if upgrading them to a newer version is also supported
--- | --- | --- | --- | --- | ---
Shutdown | yes | always | N/A | no | yes
Rolling | no | always | yes | yes | no
Reattach | yes | only due to failure | no | no | yes

## Complete Cluster Shutdown

In this scenario one shuts down all cluster nodes and resources and upgrades all the nodes before restarting the cluster.

### Procedure

1. On each node:

   1. Shutdown the cluster stack (Heartbeat or Corosync)

   2. Upgrade the Pacemaker software. This may also include upgrading the cluster stack and/or the underlying operating system..

2. Check the configuration manually or with the `crm_verify` tool if available.

3. On each node:

   1. Start the cluster stack. This can be either Corosync or Heartbeat and does not need to be the same as the previous cluster stack.

## Rolling (node by node)

In this scenario each node is removed from the cluster, upgraded and then brought back online until all nodes are running the newest version.

This method is currently broken between Pacemaker 0.6.x and 1.0.x

Measures have been put into place to ensure rolling upgrades always work for versions after 1.0.0 If there is sufficient demand, the work to repair 0.6 -> 1.0 compatibility will be carried out. Otherwise, please try one of the other upgrade strategies. Detach/Reattach is a particularly good option for most people.

### Procedure

On each node:

1. Shutdown the cluster stack (Heartbeat or Corosync)

2. Upgrade the Pacemaker software. This may also include upgrading the cluster stack and/or the underlying operating system.

   1. On the first node, check the configuration manually or with the `crm_verify` tool if available.

3. Start the cluster stack. This must be the same type of cluster stack (Corosync or Heartbeat) that the rest of the cluster is using. Upgrading Corosync/Heartbeat may also be possible, please consult the documentation for those projects to see if the two versions will be compatible.

Repeat for each node in the cluster

### Version Compatibility

#### Version Compatibility Table

Version being Installed | Oldest Compatible Version
--- | ---
Pacemaker 1.0.x | Pacemaker 1.0.0
Pacemaker 0.7.x | Pacemaker 0.6 or Heartbeat 2.1.3
Pacemaker 0.6.x | Heartbeat 2.0.8
Heartbeat 2.1.3 (or less) | Heartbeat 2.0.4
Heartbeat 2.0.4 (or less) | Heartbeat 2.0.0
Heartbeat 2.0.0 | None. Use an alternate upgrade strategy.

### Crossing Compatibility Boundaries

Rolling upgrades that cross compatibility boundaries must be preformed in multiple steps. For example, to perform a rolling update from Heartbeat 2.0.1 to Pacemaker 0.6.6 one must:

1. Perform a rolling upgrade from Heartbeat 2.0.1 to Heartbeat 2.0.4

2. Perform a rolling upgrade from Heartbeat 2.0.4 to Heartbeat 2.1.3

3. Perform a rolling upgrade from Heartbeat 2.1.3 to Pacemaker 0.6.6

## Disconnect and Reattach

A variant of a complete cluster shutdown, but the resources are left active and re-detected when the cluster is restarted.

### Procedure

1. Tell the cluster to stop managing services. This is required to allow the services to remain active after the cluster shuts down.

   `crm_attribute -t crm_config -n is-managed-default -v false`

2. For any resource that has a value for is-managed, make sure it is set to false (so that the cluster will not stop it)

   `crm_resource -t primitive -r <rsc_id> -p is-managed -v false`

3. On each node:

   1. Shutdown the cluster stack (Heartbeat or Corosync)

   2. Upgrade the cluster stack program - This may also include upgrading the underlying operating system.

4. Check the configuration manually or with the `crm_verify` tool if available.

5. On each node:

   1. Start the cluster stack. This can be either Corosync or Heartbeat and does not need to be the same as the previous cluster stack.

6. Verify the cluster re-detected all resources correctly

7. Allow the cluster to resume managing resources again

   `crm_attribute -t crm_config -n is-managed-default -v true`

8. For any resource that has a value for is-managed reset it to true (so the cluster can recover the service if it fails) if desired

   `crm_resource -t primitive -r <rsc_id> -p is-managed -v false`

### Notes

Always check your existing configuration is still compatible with the version you are installing before starting the cluster.

The oldest version of the CRM to support this upgrade type was in Heartbeat 2.0.4