MastodonSidekiqProcessingHighErrorRate #

Meaning #

The Sidekiq background job processor for Mastodon is experiencing a high failure rate. More than 5% of processed jobs are failing, which can affect federation, media processing, and other background tasks.

Impact #

Failed Sidekiq jobs can cause delays in federating posts, processing media uploads, sending notifications, and other background tasks. While Mastodon will continue to function, users may experience degraded performance and delayed interactions.

Diagnosis #

Check the Sidekiq exporter metrics for the current failure rate:

rate(sidekiq_failed_jobs_total{namespace="mastodon"}[5m])
rate(sidekiq_processed_jobs_total{namespace="mastodon"}[5m])

Check the Sidekiq exporter pod status:

kubectl get pods -n mastodon -l app.kubernetes.io/component=sidekiq

Check the Sidekiq exporter logs for error messages:

kubectl logs -n mastodon -l app.kubernetes.io/component=sidekiq --tail=100

Verify Redis/Valkey connectivity, as Sidekiq depends on it:

kubectl exec -it $(kubectl get pods -n mastodon -l app.kubernetes.io/component=web -o name | head -1) -n mastodon -- redis-cli -h rfs-valkey -p 26379 ping

Mitigation #

If the issue is caused by a specific job type, check Mastodon logs for error details and consider temporarily disabling the problematic worker.

Restart the Sidekiq process to clear any stuck jobs:

kubectl rollout restart deployment -n mastodon -l app.kubernetes.io/component=sidekiq

Check for resource constraints by monitoring CPU and memory usage of the Sidekiq pod. If the pod is being throttled or OOM-killed, consider increasing resource limits.

If the issue persists, check for database connectivity problems or Redis/Valkey issues that may be causing job failures.