Automation Suite provides support for backup and restore functionality to prevent data loss in various scenarios. You can configure a backup any time post-installation.

To use the backup and restore functionality, you must enable an NFS server, backup cluster, and restore cluster. These concepts are defined in the following section.

Terminology

NFS Server – The server that stores the backup data and facilitates the restoration. You can set up the NFS server on any machine or a PaaS service offered by cloud providers.

Backup Cluster – The cluster you set up to install Automation Suite.

Restore Cluster – The cluster where you restore all the data from the backup cluster. This becomes the new cluster where you run Automation Suite once the restore process is complete.

Prerequisites

📘

Note:

This setup will only enable a backup of the cluster, including cluster configuration and data stored as part of the in-cluster storage in /datadisk attached to server machines.
However, this will not enable the backup of any external data sources, such as the SQL database. You must enable the external data source backup separately.

---

If you want to add new server nodes after enabling the backup, make sure you meet the prerequisites and take the following steps to enable the backup on the new nodes:

1. Configure the NFS server to allow access to the new node. For details, see Allowing nodes to access NFS mount point.
2. Enable the backup on the new server node:
   Automated: Enabling a backup on the cluster
   Manual: Enabling the backup on the cluster

To set up the backup and restore functionality, you must meet the following requirements:

• You must set up the NFS server on a separate machine hosted outside of the backup and restore cluster.

• There must not be more than 10-millisecond Round Trip Time (RTT) latency between the NFS server and the backup and restore cluster.

• The cluster you want to back up and the NFS server must be in the same region.

• The NFS server must meet the following hardware requirements:

• The NFS server must be reachable from all the cluster nodes.

• You must enable the following ports on the NFS server and all the nodes in the backup cluster. When restoring the cluster, the same ports must be open on all the nodes in the restore cluster.

Preparing the backup configuration

To enable the backup, you must configure the backup server and interval in a file called backup.json.

To do that, take the following steps:

1. Create a file named backup.json.

{
  "backup": {
    "etcdBackupPath": "PLACEHOLDER",
    "nfs": {
      "endpoint": "PLACEHOLDER",
      "mountpath": "PLACEHOLDER"
    }
  },
  "backup_interval": "15"
}

2. Fill out the file based on the following field definitions:

🚧

Important!

If the backup interval is too short, for instance 30 minutes, the backup operations will be too frequent, forcing you to only store the data that was backed up in last 30 minutes. Similarly, if the backup interval is 1 week, this can cause data loss during the interval between the last backup and disaster. Therefore, it is recommended to keep the backup interval in line with your Recovery Point Objective (RPO) requirements.

When setting up the backup of the external SQL server, you should take the cluster backup interval into consideration. It is recommended to set up the same interval for both the external SQL server and the Automation Suite cluster.

When the backup is enabled in the cluster, regardless of the backup interval, Automation Suite will instantly trigger the backup. After that, the next backup will be scheduled based on the backup interval.

You can verify the backup by logging into the NFS server and navigating to the following path: /backup.nfs.mountpath/backup.etcdBackupPath. For example: /asbackup/cluster0.

Automation Suite does not make a backup of all the Persistent Volumes, such as the volumes attached to the training pipeline in AI Center. A backup is created only for a few Persistent Volumes such as Alert Manager, Prometheus, Docker Registry, MongoDB, RabbitMQ, Ceph Objectstore, and Insights.

Preparing the restore configuration

To restore the cluster from the backup data, you must provide the restore server configuration in a file called restore.json.

To do that, take the following steps:

1. Create a file named restore.json.

{
  "restore": {
    "etcdRestorePath": "PLACEHOLDER",
    "nfs": {
      "endpoint": "PLACEHOLDER",
      "mountpath": "PLACEHOLDER"
    }
  }
}

2. Fill out the file based on the following field definitions: