This page provides an overview, with links to detailed instructions, of the process and procedure for backing-up and restoring your AI Center deployment.
As AI Center is running on Kubernetes cluster, there are four elements that need to be backed up and restored independently in order to fully recover your application. Depending on your strategy you may want to back up only some of it.
We recommend to regularly back up the four elements listed below. Overall for this process 3 VMs will be involved:
- Current AI Center VM
- New AI Center VM (can potentially be the same with the current one in case you just want to reset it)
- Backup VM (needed for Storage and registry)
In addition to that you need access to storage provider supported by Velero (see dedicated page) for Cluster state backup.
The process of restoring data works as follows:
- Create your new AI Center VM
- Install AI Center (both infrastructure and application)
- Restore all elements in the following order:
- Cluster state
- Run Post Restore Sanitization
- Change Orchestrator config to point to the new AI Center instance in case DNS/IP is not the same. Note that we highly recommend using DNS and simply switch your DNS to the new AI Center machine.
The first element is the storage of your Kubernetes cluster. This includes:
- ML Packages: all packages that you retrained or manually upload
- Pipeline artifacts (outputs of pipelines)
- Pipeline logs
See instructions on how to Backup and Restore your storage.
The third element is the state of your Kubernetes cluster. This takes a snapshot of all namespaces, deployment and jobs in your cluster at a given time, allowing you to recreate all of those easily. We are using Velero as a tool for this.
See instructions on how to Backup and Restore your cluster state.
The last element is the database, as this is simple SQL database (often the same with Orchestrator). We do not provide specific tool to backup the Database. We recommend to use same strategy as with Orchestrator to Backup and Restore your Database
In case you restore your database, the name of that server IP/DNS (depending on what you use to access it) needs to remain same.
Updated about a year ago