500 errors and rate limiting on S3 requests in ODF

Solutions for 500 errors and rate limiting on S3 requests in OpenShift Data Foundation, including NooBaa memory allocation issues.

Description

Services that send S3 requests through OpenShift Data Foundation (ODF) can encounter rate limiting or 500 Internal Server Error responses. In ODF, storage management is handled by NooBaa. When the number of requests surges beyond a threshold, NooBaa allocates additional memory. If that allocation exceeds the CPU or memory limits configured on the NooBaa deployment, the pod can be terminated by the out-of-memory (OOM) killer.

This termination causes service interruptions, request throttling, and error responses.

Solution

To address the issue, you must increase the CPU and memory limits and requests for the NooBaa deployment so it can handle workload spikes without being terminated. Adjusting limits is the main resolution, while increasing requests helps improve resource allocation.

Take the following steps:

1. Retrieve the relevant BackingStore by running the following command:

   oc get backingstores.noobaa.io -n openshift-storage
   

2. Patch the BackingStore to increase CPU and memory resource limits by running the following command:

   oc patch BackingStore -n openshift-storage <backing-store-name> --type='merge' -p '{
     "spec": {
       "pvPool": {
         "resources": {
           "limits": {
             "cpu": "1000m",
             "memory": "4000Mi"
           },
           "requests": {
             "cpu": "500m",
             "memory": "500Mi"
           }
         }
       }
     }
   }'
   

The BackingStore may take several minutes to return to an active state after applying the patch.