# Debugging Cluster Startup

## Corosync

### Prerequisites

#### Minimum logging configuration

      # /etc/init.d/openais start

      logging {
          to_syslog: yes
          syslog_facility: daemon
      }

Whatever other logging you have, these two lines are required for Pacemaker clusters

### Confirm Corosync Started

#### Expected output when starting openais

      # /etc/init.d/openais start

      Starting Corosync daemon (aisexec): starting... rc=0: OK

#### Expected log messages - startup

      # grep -e "openais.*network interface" -e "AIS Executive Service" /var/log/messages

      Aug 27 16:23:37 test1 openais[26337]: [MAIN ] AIS Executive Service RELEASE 'subrev 1152 version 0.80'
      Aug 27 16:23:38 test1 openais[26337]: [MAIN ] AIS Executive Service: started and ready to provide service.
      Aug 27 16:23:38 test1 openais[26337]: [TOTEM] The network interface [192.168.9.41] is now up.

The versions may differ, but you should see Corosync indicate it started and sucessfully attached to the machine's network interface

#### Expected log messages - membership

      # grep CLM /var/log/messages

      Aug 27 16:53:15 test1 openais[2166]: [CLM  ] CLM CONFIGURATION CHANGE
      Aug 27 16:53:15 test1 openais[2166]: [CLM  ] New Configuration:
      Aug 27 16:53:15 test1 openais[2166]: [CLM  ] Members Left:
      Aug 27 16:53:15 test1 openais[2166]: [CLM  ] Members Joined:
      Aug 27 16:53:15 test1 openais[2166]: [CLM  ] CLM CONFIGURATION CHANGE
      Aug 27 16:53:15 test1 openais[2166]: [CLM  ] New Configuration:
      Aug 27 16:53:15 test1 openais[2166]: [CLM  ]     r(0) ip(192.168.9.41)
      Aug 27 16:53:15 test1 openais[2166]: [CLM  ] Members Left:
      Aug 27 16:53:15 test1 openais[2166]: [CLM  ] Members Joined:
      Aug 27 16:53:15 test1 openais[2166]: [CLM  ]     r(0) ip(192.168.9.41)
      Aug 27 16:53:15 test1 openais[2166]: [CLM  ] got nodejoin message 192.168.9.41

The exact messages will differ, but you should see a new membership formed with the real IP address of your node

### Checking Pacemaker

Now that we have confirmed that Corosync is functional we can check the rest of the stack.

#### Expected Pacemaker startup logging for Corosync

      # grep pcmk_plugin_init /var/log/messages

      Aug 27 16:53:15 test1 openais[2166]: [pcmk ] info: pcmk_plugin_init: CRM: Initialized
      Aug 27 16:53:15 test1 openais[2166]: [pcmk ] Logging: Initialized pcmk_plugin_init
      Aug 27 16:53:15 test1 openais[2166]: [pcmk ] info: pcmk_plugin_init: Service: 9
      Aug 27 16:53:15 test1 openais[2166]: [pcmk ] info: pcmk_plugin_init: Local hostname: test1

If you don't see these messages, or some like them, there is likely a problem finding or loading the pacemaker plugin.

#### Expected process listing on a 64-bit machine

      # ps axf

      3718 ?        Ssl    0:05 /usr/sbin/aisexec
      3723 ?        SLs    0:00  \_ /usr/lib64/heartbeat/stonithd
      3724 ?        S      0:05  \_ /usr/lib64/heartbeat/cib
      3725 ?        S      0:21  \_ /usr/lib64/heartbeat/lrmd
      3726 ?        S      0:01  \_ /usr/lib64/heartbeat/attrd
      3727 ?        S      0:00  \_ /usr/lib64/heartbeat/pengine
      3728 ?        S      0:01  \_ /usr/lib64/heartbeat/crmd

On 32-bit systems the exact path may differ, but all the above processes should be listed.