Redis Probe Failure

Description

Redis probe can fail if the node ID file does not exist. This can happen if the pod is not yet bootstrapped.

There is a recovery job that automatically fixes this issue, and the following steps should not be performed while the job is running.

When a Redis Enterprise cluster loses contact with more than half of its nodes (either because of failed nodes or network split), then the cluster stops responding to client connections. The Pods also fail to rejoin the cluster.

Solution

1. Delete the Redis cluster and database using the following commands:

   kubectl delete redb -n redis-system redis-cluster-db --force --grace-period=0 &
   kubectl delete rec -n redis-system redis-cluster --force --grace-period=0 &
   kubectl patch redb -n redis-system redis-cluster-db --type=json -p '[{"op":"remove","path":"/metadata/finalizers","value":"finalizer.redisenterprisedatabases.app.redislabs.com"}]'
   kubectl patch rec redis-cluster -n redis-system --type=json -p '[{"op":"remove","path":"/metadata/finalizers","value":"redbfinalizer.redisenterpriseclusters.app.redislabs.com"}]'
   kubectl delete job redis-cluster-db-job -n redis-system

2. Go to the ArgoCD UI and sync the redis-cluster application.