6.2 My SQL Cluster

Owned by Jenni Wallin
Last updated: May 14, 2023 by Former user (Deleted)
2 min read

The troubleshooting options you have for the MySQL Cluster is to recover nodes, either automatically or manually, and check error logs and trace files. If a node fails, an alarm will be generated.

Automatic Node Recovery

The monitoring tool CMON performs automatic node and cluster recovery when needed.

Manual Node Recovery

Nodes can be restarted manually either from the CMON web interface, located at http://<[CZ] host name or IP address>/cmon, or by using a command line tool. 

If a manual recovery can not be performed, this will be indicated in the alarm generated on node failure. In that case, the node has to be restarted from the web interface.

From the Web Interface

To restart a node using the CMON web interface:

  1. In the CMON Monitoring Dashboard, click on the cluster you want to make a backup for in the menu to the left.
     

  2. Click on the Node mgmt option in the menu to the right.

    A column named Manage containing management buttons will now be displayed to the right in all the node tables.
     

  3. Click on theStart button for the node you want to restart.

    The node will be restarted.

From the Command Line Tool

To restart a node from the command line:

  1. Start a command line tool on [CZ], logged in as root, and enter the scripts directory:

    $ cd mysqlcluster-71/cluster/scripts

  2. Run the script for restarting the node:

    $ ./start-ndbd -nodeid=<nodeid of failed node> --hostname=<the hostname of the failed node> [--initial]

    The -initial flag should only be used for "escalating" the recovery procedure. If a node recovery does not succeed without the -initial flag, then you can try to specify the -initial flag.

    Note!

    Use the -initial flag with great care since it will clear and recreate the file system for the data node. If this is executed on several nodes, either simultaneously or in turn, data loss may occur.

Alarms and Logs

In the CMON web interface you can view alarms and logs. They often give a hint at what type of problem you may have:

  • NDB Filesystem Error - e g out of disk space or corrupt file system.

  • Pointer Too Large - usually a bug.

  • Arbitrator decided to shutdown node - node failed during startup or network partitioning.

Viewing Alarms

To view alarms in the CMON web interface:

  1. In the CMON Monitoring Dashboard, click on the cluster you want to make a backup for in the menu to the left.
     

  2. Click on the Alarms option in the menu to the right.
    The Alarms page will open displaying information about all the alarms that have been generated.

    CMON Alarms

  3. If you want to dismiss and remove the alarm from the list, click on the dismiss button.
     

  4. If you want to view more detailed information about a host alarm, click on the hostname.
    Statistics over the selected host will be displayed.


CMON Host Statistics
 

Viewing Logs

To view logs in the CMON web interface:

  1. In the CMON Monitoring Dashboard, click on the cluster you want to make a backup for in the menu to the left.
     

  2. Click on the Logs option, beneath the heading Troubleshooting, in the menu to the right.

    A list with log files for the MySQL Cluster will displayed with information about Hostname, Logfile, Size in bytes and Time for each entry.