automation-suite
2021.10
false
- 概述
- 要求
- 安装
- 安装后
- 集群管理
- 监控和警示
- 迁移和升级
- 特定于产品的配置
- 最佳实践和维护
- 故障排除
如何清理存储在 sf-logs 存储桶中的旧日志
重要 :
请注意此内容已使用机器翻译进行了部分本地化。
新发布内容的本地化可能需要 1-2 周的时间才能完成。
不在支持范围内
Automation Suite 安装指南
上次更新日期 2024年11月21日
如何清理存储在 sf-logs 存储桶中的旧日志
某个错误可能会导致
sf-logs
对象存储存储桶中的日志累积。 要清理 sf-logs
存储桶中的旧日志,请按照有关运行专用脚本的说明进行操作。 请务必按照与您的环境类型相关的步骤操作。
要清理
sf-logs
存储桶中存储的旧日志,请执行以下步骤:
-
获取您的环境中可用的
sf-k8-utils-rhel
映像的版本:- 在离线环境中,运行以下命令:
podman search localhost:30071/uipath/sf-k8-utils-rhel --tls-verify=false --list-tags
- 在在线环境中,运行以下命令:
podman search registry.uipath.com/uipath/sf-k8-utils-rhel --list-tags
- 在离线环境中,运行以下命令:
-
相应地更新以下
yaml
定义中的第 121 行,以包含正确的图像标签:apiVersion: v1 kind: ConfigMap metadata: name: cleanup-script namespace: uipath-infra data: cleanup_old_logs.sh: | #!/bin/bash function parse_args() { CUTOFFDAY=7 SKIPDRYRUN=0 while getopts 'c:sh' flag "$@"; do case "${flag}" in c) CUTOFFDAY=${OPTARG} ;; s) SKIPDRYRUN=1 ;; h) display_usage exit 0 ;; *) echo "Unexpected option ${flag}" display_usage exit 1 ;; esac done shift $((OPTIND - 1)) } function display_usage() { echo "usage: $(basename "$0") -c <number> [-s]" echo " -s skip dry run, Really deletes the log dirs" echo " -c logs older than how many days to be deleted. Default is 7 days" echo " -h help" echo "NOTE: Default is dry run, to really delete logs set -s" } function setS3CMDContext() { OBJECT_GATEWAY_INTERNAL_HOST=$(kubectl -n rook-ceph get services/rook-ceph-rgw-rook-ceph -o jsonpath="{.spec.clusterIP}") OBJECT_GATEWAY_INTERNAL_PORT=$(kubectl -n rook-ceph get services/rook-ceph-rgw-rook-ceph -o jsonpath="{.spec.ports[0].port}") AWS_ACCESS_KEY=$1 AWS_SECRET_KEY=$2 # Reference https://rook.io/docs/rook/v1.5/ceph-object.html#consume-the-object-storage export AWS_HOST=$OBJECT_GATEWAY_INTERNAL_HOST export AWS_ENDPOINT=$OBJECT_GATEWAY_INTERNAL_HOST:$OBJECT_GATEWAY_INTERNAL_PORT export AWS_ACCESS_KEY_ID=$AWS_ACCESS_KEY export AWS_SECRET_ACCESS_KEY=$AWS_SECRET_KEY } # Set s3cmd context by passing correct AccessKey and SecretKey function setS3CMDContextForLogs() { BUCKET_NAME='sf-logs' AWS_ACCESS_KEY=$(kubectl -n cattle-logging-system get secret s3-store-secret -o json | jq '.data.OBJECT_STORAGE_ACCESSKEY' | sed -e 's/^"//' -e 's/"$//' | base64 -d) AWS_SECRET_KEY=$(kubectl -n cattle-logging-system get secret s3-store-secret -o json | jq '.data.OBJECT_STORAGE_SECRETKEY' | sed -e 's/^"//' -e 's/"$//' | base64 -d) setS3CMDContext "$AWS_ACCESS_KEY" "$AWS_SECRET_KEY" } function delete_old_logs() { local cutoffdate=$1 days=$(s3cmd ls s3://sf-logs/ --host="${AWS_HOST}" --host-bucket= s3://sf-logs --no-check-certificate --no-ssl) days=${days//DIR} if [[ $SKIPDRYRUN -eq 0 ]]; then echo "DRY RUN. Following log dirs are selected for deletion" fi for day in $days do day=${day#*sf-logs/} day=${day::-1} if [[ ${day} < ${cutoffdate} ]]; then if [[ $SKIPDRYRUN -eq 0 ]]; then echo "s3://$BUCKET_NAME/$day" else echo "###############################################################" echo "Deleting Logs for day: {$day}" echo "###############################################################" s3cmd del "s3://$BUCKET_NAME/$day/" --host="${AWS_HOST}" --host-bucket= --no-ssl --recursive || true fi fi done } function main() { # Set S3 context by setting correct env variables setS3CMDContextForLogs echo "Bucket name is $BUCKET_NAME" CUTOFFDATE=$(date --date="${CUTOFFDAY} day ago" +%Y_%m_%d) echo "logs older than ${CUTOFFDATE} will be deleted" delete_old_logs "${CUTOFFDATE}" if [[ $SKIPDRYRUN -eq 0 ]]; then echo "NOTE: For really deleting the old log directories run with -s option" fi } parse_args "$@" main exit 0 --- apiVersion: v1 kind: Pod metadata: name: cleanup-old-logs namespace: uipath-infra spec: serviceAccountName: fluentd-logs-cleanup-sa containers: - name: cleanup image: localhost:30071/uipath/sf-k8-utils-rhel:0.8 command: ["/bin/bash"] args: ["/scripts-dir/cleanup_old_logs.sh", "-s"] volumeMounts: - name: scripts-vol mountPath: /scripts-dir securityContext: privileged: false allowPrivilegeEscalation: false readOnlyRootFilesystem: true runAsUser: 9999 runAsGroup: 9999 runAsNonRoot: true capabilities: drop: ["NET_RAW"] volumes: - name: scripts-vol configMap: name: cleanup-script
apiVersion: v1 kind: ConfigMap metadata: name: cleanup-script namespace: uipath-infra data: cleanup_old_logs.sh: | #!/bin/bash function parse_args() { CUTOFFDAY=7 SKIPDRYRUN=0 while getopts 'c:sh' flag "$@"; do case "${flag}" in c) CUTOFFDAY=${OPTARG} ;; s) SKIPDRYRUN=1 ;; h) display_usage exit 0 ;; *) echo "Unexpected option ${flag}" display_usage exit 1 ;; esac done shift $((OPTIND - 1)) } function display_usage() { echo "usage: $(basename "$0") -c <number> [-s]" echo " -s skip dry run, Really deletes the log dirs" echo " -c logs older than how many days to be deleted. Default is 7 days" echo " -h help" echo "NOTE: Default is dry run, to really delete logs set -s" } function setS3CMDContext() { OBJECT_GATEWAY_INTERNAL_HOST=$(kubectl -n rook-ceph get services/rook-ceph-rgw-rook-ceph -o jsonpath="{.spec.clusterIP}") OBJECT_GATEWAY_INTERNAL_PORT=$(kubectl -n rook-ceph get services/rook-ceph-rgw-rook-ceph -o jsonpath="{.spec.ports[0].port}") AWS_ACCESS_KEY=$1 AWS_SECRET_KEY=$2 # Reference https://rook.io/docs/rook/v1.5/ceph-object.html#consume-the-object-storage export AWS_HOST=$OBJECT_GATEWAY_INTERNAL_HOST export AWS_ENDPOINT=$OBJECT_GATEWAY_INTERNAL_HOST:$OBJECT_GATEWAY_INTERNAL_PORT export AWS_ACCESS_KEY_ID=$AWS_ACCESS_KEY export AWS_SECRET_ACCESS_KEY=$AWS_SECRET_KEY } # Set s3cmd context by passing correct AccessKey and SecretKey function setS3CMDContextForLogs() { BUCKET_NAME='sf-logs' AWS_ACCESS_KEY=$(kubectl -n cattle-logging-system get secret s3-store-secret -o json | jq '.data.OBJECT_STORAGE_ACCESSKEY' | sed -e 's/^"//' -e 's/"$//' | base64 -d) AWS_SECRET_KEY=$(kubectl -n cattle-logging-system get secret s3-store-secret -o json | jq '.data.OBJECT_STORAGE_SECRETKEY' | sed -e 's/^"//' -e 's/"$//' | base64 -d) setS3CMDContext "$AWS_ACCESS_KEY" "$AWS_SECRET_KEY" } function delete_old_logs() { local cutoffdate=$1 days=$(s3cmd ls s3://sf-logs/ --host="${AWS_HOST}" --host-bucket= s3://sf-logs --no-check-certificate --no-ssl) days=${days//DIR} if [[ $SKIPDRYRUN -eq 0 ]]; then echo "DRY RUN. Following log dirs are selected for deletion" fi for day in $days do day=${day#*sf-logs/} day=${day::-1} if [[ ${day} < ${cutoffdate} ]]; then if [[ $SKIPDRYRUN -eq 0 ]]; then echo "s3://$BUCKET_NAME/$day" else echo "###############################################################" echo "Deleting Logs for day: {$day}" echo "###############################################################" s3cmd del "s3://$BUCKET_NAME/$day/" --host="${AWS_HOST}" --host-bucket= --no-ssl --recursive || true fi fi done } function main() { # Set S3 context by setting correct env variables setS3CMDContextForLogs echo "Bucket name is $BUCKET_NAME" CUTOFFDATE=$(date --date="${CUTOFFDAY} day ago" +%Y_%m_%d) echo "logs older than ${CUTOFFDATE} will be deleted" delete_old_logs "${CUTOFFDATE}" if [[ $SKIPDRYRUN -eq 0 ]]; then echo "NOTE: For really deleting the old log directories run with -s option" fi } parse_args "$@" main exit 0 --- apiVersion: v1 kind: Pod metadata: name: cleanup-old-logs namespace: uipath-infra spec: serviceAccountName: fluentd-logs-cleanup-sa containers: - name: cleanup image: localhost:30071/uipath/sf-k8-utils-rhel:0.8 command: ["/bin/bash"] args: ["/scripts-dir/cleanup_old_logs.sh", "-s"] volumeMounts: - name: scripts-vol mountPath: /scripts-dir securityContext: privileged: false allowPrivilegeEscalation: false readOnlyRootFilesystem: true runAsUser: 9999 runAsGroup: 9999 runAsNonRoot: true capabilities: drop: ["NET_RAW"] volumes: - name: scripts-vol configMap: name: cleanup-script -
将上述
yaml
定义的内容复制到名为cleanup.yaml
的文件中。 触发 Pod 以清理旧日志:kubectl apply -f cleanup.yaml
kubectl apply -f cleanup.yaml -
获取有关进度的详细信息:
kubectl -n uipath-infra logs cleanup-old-logs -f
kubectl -n uipath-infra logs cleanup-old-logs -f -
删除作业:
kubectl delete -f cleanup.yaml
kubectl delete -f cleanup.yaml