Automation Suite
2021.10
false
- Overview
- Requirements
- Installation
- Post-installation
- Cluster administration
- Monitoring and alerting
- Migration and upgrade
- Product-specific configuration
- Best practices and maintenance
- Troubleshooting
- How to Troubleshoot Services During Installation
- How to Uninstall the Cluster
- How to clean up offline artifacts to improve disk space
- How to Disable TLS 1.0 and 1.1
- How to enable Istio logging
- How to manually clean up logs
- How to clean up old logs stored in the sf-logs bundle
- How to debug failed Automation Suite installations
- How to disable NIC checksum offloading
- Unable to run an offline installation on RHEL 8.4 OS
- Error in Downloading the Bundle
- Offline installation fails because of missing binary
- Certificate issue in offline installation
- SQL connection string validation error
- Failure After Certificate Update
- Automation Suite Requires Backlog_wait_time to Be Set 1
- Failure to Resize Objectstore PVC
- Rook Ceph or Looker pod stuck in Init state
- StatefulSet volume attachment error
- Failure to create persistent volumes
- Storage reclamation patch
- Cannot Log in After Migration
- Setting a timeout interval for the management portals
- Update the underlying directory connections
- Kinit: Cannot Find KDC for Realm <AD Domain> While Getting Initial Credentials
- Kinit: Keytab Contains No Suitable Keys for *** While Getting Initial Credentials
- GSSAPI Operation Failed With Error: An Invalid Status Code Was Supplied (Client's Credentials Have Been Revoked).
- Login Failed for User <ADDOMAIN><aduser>. Reason: The Account Is Disabled.
- Alarm Received for Failed Kerberos-tgt-update Job
- SSPI Provider: Server Not Found in Kerberos Database
- Failure to get the sandbox image
- Pods not showing in ArgoCD UI
- Redis Probe Failure
- RKE2 Server Fails to Start
- Secret Not Found in UiPath Namespace
- ArgoCD goes into progressing state after first installation
- Unexpected Inconsistency; Run Fsck Manually
- Missing Self-heal-operator and Sf-k8-utils Repo
- Degraded MongoDB or Business Applications After Cluster Restore
- Unhealthy Services After Cluster Restore or Rollback
- Using the Automation Suite Diagnostics Tool
- Using the Automation Suite Support Bundle Tool
- Exploring Logs
Rook Ceph or Looker pod stuck in Init state
Automation Suite Installation Guide
Last updated Apr 19, 2024
Rook Ceph or Looker pod stuck in Init state
Error code |
30001 |
Occasionally, on node restart, an issue causes the Looker or Rook Ceph pod to get stuck in Init state as the volume required for attaching the PVC to a pod is missing.
Verify if the problem is indeed related to Longhorn by running the following command:
kubectl get events -A -o json | jq -r '.items[] | select(.message != null) | select(.message | contains("cannot get resource \"volumeattachments\" in API group \"storage.k8s.io\""))'
kubectl get events -A -o json | jq -r '.items[] | select(.message != null) | select(.message | contains("cannot get resource \"volumeattachments\" in API group \"storage.k8s.io\""))'
If it is related to Longhorn, this command should return a list of pod names affected by the issue. If the command does not return anything, the cause of the problem is different.
Run the following script to fix the problematic pods if the previous command returns a non-empty output:
#!/bin/bash
function wait_till_rollout() {
local namespace=$1
local object_type=$2
local deploy=$3
local try=0
local maxtry=2
local status="notready"
while [[ ${status} == "notready" ]] && (( try != maxtry )) ; do
kubectl -n "$namespace" rollout status "$deploy" -w --timeout=600s;
# shellcheck disable=SC2181
if [[ "$?" -ne 0 ]];
then
status="notready"
try=$((try+1))
else
status="ready"
fi
done
if [[ $status == "notready" ]]; then
echo "$deploy of type $object_type failed in namespace $namespace. Plz re-run the script once again to verify that it's not a transient issue !!!"
exit 1
fi
}
function fix_pv_deployments() {
for pod_name in $(kubectl get events -A -o json | jq -r '.items[] | select(.message | contains("cannot get resource \"volumeattachments\" in API group \"storage.k8s.io\"")) | select(.involvedObject.kind == "Pod") | .involvedObject.name + "/" + .involvedObject.namespace' | sort | uniq)
do
POD_NAME=$(echo "${pod_name}" | cut -d '/' -f1)
NS=$(echo "${pod_name}" | cut -d '/' -f2)
controller_data=$(kubectl -n "${NS}" get po "${POD_NAME}" -o json | jq -r '[.metadata.ownerReferences[] | select(.controller==true)][0] | .kind + "=" + .name')
[[ $controller_data == "" ]] && error "Error: Could not determine owner for pod: ${POD_NAME}" && exit 1
CONTROLLER_KIND=$(echo "${controller_data}" | cut -d'=' -f1)
CONTROLLER_NAME=$(echo "${controller_data}" | cut -d'=' -f2)
if [[ $CONTROLLER_KIND == "ReplicaSet" ]]
then
controller_data=$(kubectl -n "${NS}" get "${CONTROLLER_KIND}" "${CONTROLLER_NAME}" -o json | jq -r '[.metadata.ownerReferences[] | select(.controller==true)][0] | .kind + "=" + .name')
CONTROLLER_KIND=$(echo "${controller_data}" | cut -d'=' -f1)
CONTROLLER_NAME=$(echo "${controller_data}" | cut -d'=' -f2)
replicas=$(kubectl -n "${NS}" get "$CONTROLLER_KIND" "$CONTROLLER_NAME" -o json | jq -r '.status.replicas')
unavailable_replicas=$(kubectl -n "${NS}" get "$CONTROLLER_KIND" "$CONTROLLER_NAME" -o json | jq -r '.status.unavailableReplicas')
if [ -n "$unavailable_replicas" ]; then
available_replicas=$((replicas - unavailable_replicas))
if [ $available_replicas -eq 0 ]; then
kubectl -n "$NS" scale "$CONTROLLER_KIND" "$CONTROLLER_NAME" --replicas=0
sleep 15
kubectl -n "$NS" scale "$CONTROLLER_KIND" "$CONTROLLER_NAME" --replicas="$replicas"
deployment_name="$CONTROLLER_KIND/$CONTROLLER_NAME"
wait_till_rollout "$NS" "deploy" "$deployment_name"
fi
fi
fi
done
}
fix_pv_deployments
#!/bin/bash
function wait_till_rollout() {
local namespace=$1
local object_type=$2
local deploy=$3
local try=0
local maxtry=2
local status="notready"
while [[ ${status} == "notready" ]] && (( try != maxtry )) ; do
kubectl -n "$namespace" rollout status "$deploy" -w --timeout=600s;
# shellcheck disable=SC2181
if [[ "$?" -ne 0 ]];
then
status="notready"
try=$((try+1))
else
status="ready"
fi
done
if [[ $status == "notready" ]]; then
echo "$deploy of type $object_type failed in namespace $namespace. Plz re-run the script once again to verify that it's not a transient issue !!!"
exit 1
fi
}
function fix_pv_deployments() {
for pod_name in $(kubectl get events -A -o json | jq -r '.items[] | select(.message | contains("cannot get resource \"volumeattachments\" in API group \"storage.k8s.io\"")) | select(.involvedObject.kind == "Pod") | .involvedObject.name + "/" + .involvedObject.namespace' | sort | uniq)
do
POD_NAME=$(echo "${pod_name}" | cut -d '/' -f1)
NS=$(echo "${pod_name}" | cut -d '/' -f2)
controller_data=$(kubectl -n "${NS}" get po "${POD_NAME}" -o json | jq -r '[.metadata.ownerReferences[] | select(.controller==true)][0] | .kind + "=" + .name')
[[ $controller_data == "" ]] && error "Error: Could not determine owner for pod: ${POD_NAME}" && exit 1
CONTROLLER_KIND=$(echo "${controller_data}" | cut -d'=' -f1)
CONTROLLER_NAME=$(echo "${controller_data}" | cut -d'=' -f2)
if [[ $CONTROLLER_KIND == "ReplicaSet" ]]
then
controller_data=$(kubectl -n "${NS}" get "${CONTROLLER_KIND}" "${CONTROLLER_NAME}" -o json | jq -r '[.metadata.ownerReferences[] | select(.controller==true)][0] | .kind + "=" + .name')
CONTROLLER_KIND=$(echo "${controller_data}" | cut -d'=' -f1)
CONTROLLER_NAME=$(echo "${controller_data}" | cut -d'=' -f2)
replicas=$(kubectl -n "${NS}" get "$CONTROLLER_KIND" "$CONTROLLER_NAME" -o json | jq -r '.status.replicas')
unavailable_replicas=$(kubectl -n "${NS}" get "$CONTROLLER_KIND" "$CONTROLLER_NAME" -o json | jq -r '.status.unavailableReplicas')
if [ -n "$unavailable_replicas" ]; then
available_replicas=$((replicas - unavailable_replicas))
if [ $available_replicas -eq 0 ]; then
kubectl -n "$NS" scale "$CONTROLLER_KIND" "$CONTROLLER_NAME" --replicas=0
sleep 15
kubectl -n "$NS" scale "$CONTROLLER_KIND" "$CONTROLLER_NAME" --replicas="$replicas"
deployment_name="$CONTROLLER_KIND/$CONTROLLER_NAME"
wait_till_rollout "$NS" "deploy" "$deployment_name"
fi
fi
fi
done
}
fix_pv_deployments