How to backup and restore your Airy Core instance.
All the application data, including all the configured sources, channels and templates, are stored in Kafka. As Kafka needs Zookeeper to run properly, the Kafka operational data is stored in Zookeeper. Both Kafka and Zookeeper are deployed as statefulsets in Kubernetes and all the data is stored in PersistentVolumes, provisioned by PersistentVolumeClaims. The location of the persistent volumes is dependent of your implementation. For example, in AWS the Elastic Block Store (EBS) is used to manage the volumes.
It is best to do backup of the Kafka volumes when the Kafka brokers are stopped, in order to avoid inconsistencies.
To view the PersistentVolumeClaims and PersistentVolumes run:
To Backup all your data, you should make periodical snapshots or backup of the PersistentVolumes.
All the credentials and configurations of the Airy Core instance, is stored in your
You can restore your Airy Core instance in an empty Kubernetes cluster, by restoring the Kafka and Zookeeper PersistentVolumes (PV) and PersistentVolumeClaims (PVC).
Once you have your volumes restored, either from a backup location or from snapshots, you can create the PVs using the following Kubernetes manifests.
Modify the file according to your infrastructure configuration by specifying VOLUME_ID, AVAILABILITY_ZONE and REGION.
Apply the Kubernetes manifest to create the PV resources
Run the following command to create the necessary PVCs:
Once the storage is restored, transform your
airy.yaml file to be compatible with
Helm and install the Helm chart for
Airy Core, with the desired version:
In rare cases, when the backup is performed on a running Kafka instance, there can be some inconsistencies in the internal topics, in the restored Kafka instance.
This can be resolved by resetting the Kafka streaming apps, to beginning or to latest, depending on the setting of the particular app. Refer to the Airy component reset guide.