Backing up and restoring the cluster

Automation Suite supports the backup and restore functionality to prevent data loss in various scenarios. You can configure a backup any time post-installation.

To use the backup and restore functionality, you must enable an NFS server, backup cluster, and restore cluster. These concepts are defined in the following section.

Terminology

NFS Server – The server that stores the backup data and facilitates the restoration. You can set up the NFS server on any machine or a PaaS service offered by cloud providers. Note that we do not support Windows-based NFS and Azure blob-based NFS.

Backup Cluster – The cluster you set up to install Automation Suite. This is the cluster where you will enable the backup.

Restore Cluster – The cluster where you restore all the data from the backup cluster. This becomes the new cluster where you run Automation Suite once the restore process is complete.

Prerequisites

Note: This setup will only enable a backup of the cluster, including cluster configuration and data stored as part of the in-cluster storage in /datadisk attached to server machines.

However, this will not enable the backup of any external data sources, such as the SQL database. You must enable the external data source backup separately.

If you want to add new server nodes after enabling the backup, make sure you meet the prerequisites and take the following steps to enable the backup on the new nodes:

1. Configure the NFS server to allow access to the new node. For details, see Allowing nodes to access NFS mount point.

2. Enable the backup on the new server node:

   Automated: Enabling a backup on the cluster

   Manual: Enabling the backup on the cluster

To set up the backup and restore functionality, you must meet the following requirements:

• You must use NFSv4 on Linux.
• You must set up the NFS server on a separate machine hosted outside of the backup and restore cluster.
• There must not be more than 10-millisecond Round Trip Time (RTT) latency between the NFS server and the backup and restore cluster.
• The cluster you want to back up and the NFS server must be in the same region.

• The NFS server must meet the following hardware requirements:

  CPU	RAM	Disk
  4(v-) CPU	16 GiB	10 TiB SSD (1100 IOPS)

• The NFS server must be reachable from all the cluster nodes.

• You must enable the following ports on the NFS server and all the nodes in the backup cluster. When restoring the cluster, the same ports must be open on all the nodes in the restore cluster.

  Port	Protocol	Purpose
  2049	TCP	Bidirectional communication between the NFS server and the backup and restore cluster. This is the port on which the NFS server will run.
  111	TCP	Bidirectional communication between the NFS server and the backup and restore cluster. This port is used for rpcbind between the NFS server and the backup and restore cluster.

Preparing the Backup Configuration

To enable the backup, you must configure the backup server and interval in a file called backup.json.

To do that, take the following steps:

1. Create a file named backup.json.

   {
     "backup": {
       "etcdBackupPath": "PLACEHOLDER",
       "nfs": {
         "endpoint": "PLACEHOLDER",
         "mountpath": "PLACEHOLDER"
       }
     },
     "backup_interval": "15"
   }

2. Fill out the file based on the following field definitions:

   Parameter	Configuration
   backup.etcdBackupPath	The relative path where the backup data will be stored on the NFS server. You can give it the name of cluster. Example: cluster0
   backup.nfs.endpoint	The endpoint of the NFS server (IP address or DNS name). This will be either the FQDN or IP address of the NFS machine. There must not be any protocol present in the endpoint. Example: nfs.automationsuite.mycompany.com or 20.224.01.66
   backup.nfs.mountpath	The path on the NFS server (endpoint). This is the location where you have attached the disk for storing the cluster backup. Example: /asbackup
   backup_interval	The backup time interval in minutes. This interval is the lead time between two consecutive backups. You can only restore the last successful backup, so you should carefully decide on this interval. The minimum backup interval can be as low as 15 minutes.

   Important:

   • If the backup interval is too short, for instance 30 minutes, the backup operations will be too frequent, forcing you to only store the data that was backed up in last 30 minutes. Similarly, if the backup interval is 1 week, this can cause data loss during the interval between the last backup and disaster. Therefore, it is recommended to keep the backup interval in line with your Recovery Point Objective (RPO) requirements.
   • When setting up the backup of the external SQL server, you should take the cluster backup interval into consideration. It is recommended to set up the same interval for both the external SQL server and the Automation Suite cluster.
   • When the backup is enabled in the cluster, regardless of the backup interval, Automation Suite will instantly trigger the backup. After that, the next backup will be scheduled based on the backup interval.
   • You can verify the backup by logging into the NFS server and navigating to the following path: /backup.nfs.mountpath/backup.etcdBackupPath. For example: /asbackup/cluster0.

Automation Suite does not make a backup of all the Persistent Volumes, such as the volumes attached to the training pipeline in AI Center. A backup is created only for a few Persistent Volumes such as Alert Manager, Prometheus, Docker Registry, MongoDB, RabbitMQ, Ceph Objectstore, and Insights.

Preparing the Restore Configuration

To restore the cluster from the backup data, you must provide the restore server configuration in a file called restore.json.

To do that, take the following steps:

1. Create a file named restore.json.

   {
     "restore": {
       "etcdRestorePath": "PLACEHOLDER",
       "nfs": {
         "endpoint": "PLACEHOLDER",
         "mountpath": "PLACEHOLDER"
       }
     }
   }

2. Fill out the file based on the following field definitions:

   Parameter	Configuration
   restore.etcdRestorePath	The path on the NFS server from where the data will be restored. It must be the same name as the one you have provided for backup.etcBackupPath in backup.json. Example: cluster0
   restore.nfs.endpoint	The endpoint of the NFS server. This will be either the FQDN or the IP address of the NFS machine. There must not be any protocol present in the endpoint. Example: nfs.automationsuite.mycompany.com or 20.224.01.66
   restore.nfs.mountpath	The mount path of the NFS server. This is the location where you have attached the disk for storing the cluster backup. Example: /asbackup