Orchestrator inaccessible due to Redis pods on AI agent node

Restore Orchestrator access when Redis pods are scheduled on unhealthy AI agent nodes.

Description

In certain scenarios, Orchestrator may become inaccessible and return a "no healthy upstream" error, with Orchestrator pods entering a CrashLoopBackOff state.

The root cause is that the Redis cluster pods were scheduled on an AI agent node that is not functioning properly. Because Orchestrator depends on Redis, it cannot start while Redis is unavailable.

Solution

1. Identify the AI agent node where Redis pods are scheduled and confirm the node is not functioning properly.
2. Disable scheduling on the affected AI agent node.
3. Scale down the Redis StatefulSet, then scale it back up to trigger rescheduling on a healthy node.
4. Verify that Redis pods are now running on a healthy node.
5. Verify that Orchestrator pods recover and return to a running state.