📌 Executive Summary

In a real-world setup using RabbitMQ + Celery + Docker, I encountered multiple production-level issues:

• SMTP credential mismatch (username & domain swapped)

• Environment variable confusion (CI/CD vs local volume)

• RabbitMQ connection failures

• Container restart issues

• Accidental .env deletion leading to application crash

These issues caused:

• Email failures

• Celery not connecting

• Application downtime

• Debugging confusion

This blog explains everything in detail — what, why, how, where, when, who — based on real debugging experience

🧠 What is RabbitMQ?

RabbitMQ is a message broker

👉 It acts like a middleman between services

🎯 Real-world example

User places order → email should be sent

Without RabbitMQ:

User waits ❌

With RabbitMQ:

Task stored → processed later ✅

🧠 Internal flow

Producer (Django) → RabbitMQ → Consumer (Celery worker)

🧠 What is Celery Worker?

Celery Worker is the executor

RabbitMQ → sends task → Celery executes

Example:

Send email → executed by Celery worker

🧠 What is Celery Beat?

Celery Beat is a scheduler

Runs tasks at intervals

Example:

Every 1 minute → check pending emails → send

🔥 Complete Architecture

User → Django → RabbitMQ → Celery Worker → SMTP
                     ↑
                Celery Beat

🚨 CELERY_BROKER_URL (CRITICAL CONCEPT)

❌ Wrong

CELERY_BROKER_URL=amqp://guest:guest@localhost:5672//
CELERY_BROKER_URL=amqp://guest:guest@127.0.0.1:5672//

🧠 Why wrong?

localhost = same container ❌

👉 In Docker:

Each container = separate machine

So:

celery container → localhost = celery itself

NOT RabbitMQ

✅ Correct

CELERY_BROKER_URL=amqp://guest:guest@rabbitmq:5672//

🧠 Why correct?

rabbitmq = docker-compose service name

👉 Docker provides internal DNS

🧠 Where is CELERY_BROKER_URL used in code?

settings.py

import os

CELERY_BROKER_URL = os.environ.get(
    "CELERY_BROKER_URL",
    "amqp://guest:guest@rabbitmq:5672//"
)

celery.py

from celery import Celery

app = Celery("project")
app.config_from_object("django.conf:settings", namespace="CELERY")

👉 Celery automatically reads:

CELERY_BROKER_URL → connects to RabbitMQ

🚨 SMTP USERNAME & DOMAIN MISMATCH (REAL ISSUE)

❌ Wrong .env

EMAIL_HOST=user@example.com
EMAIL_HOST_USER=smtp.example.com

🧠 Why wrong?

EMAIL_HOST = SMTP server
EMAIL_HOST_USER = login email

❌ Result

SMTP silently rejects ❌
Connection unexpectedly closed ❌

✅ Correct .env

EMAIL_HOST=smtp.example.com
EMAIL_HOST_USER=user@example.com
EMAIL_HOST_PASSWORD=xxxx
EMAIL_PORT=587
EMAIL_USE_TLS=True

🧠 How to identify in future

🔍 Logs

Connection unexpectedly closed
SendAsDenied

🔍 Manual test

docker exec -it celery-worker python manage.py shell

from django.core.mail import send_mail
send_mail("test","hello","user@example.com",["test@example.com"])

🔍 Network test

telnet smtp.example.com 587

🚨 ENVIRONMENT VARIABLE ISSUE (CRITICAL LESSON)

🔴 What I did initially Used GitLab CI variables Created .env in pipeline Used env_file

showing one stage in this docker compose file where using env_file to copy gitlab variable to .env inside working directory of the container but it didn't contain all the env variables like django SECRET_KEY but it was present in volume /home/app/.env in the server

rm -f .env

this is from gitlab-ci-yml

deploy_container:
  image: docker:cli
  stage: deploy
  script:
    - docker login -u "\(CI_REGISTRY_USER" -p "\)CI_REGISTRY_PASSWORD" "$CI_REGISTRY"
    - rm -f .env #(here .env removed)
    - echo "CELERY_BROKER_URL=$CELERY_BROKER_URL" > .env
    - echo "EMAIL_HOST=$EMAIL_HOST" >> .env
    - echo "EMAIL_PORT=$EMAIL_PORT" >> .env
    - echo "EMAIL_USE_TLS=$EMAIL_USE_TLS" >> .env
    - echo "EMAIL_HOST_USER=$EMAIL_HOST_USER" >> .env
    - echo "EMAIL_HOST_PASSWORD=$EMAIL_HOST_PASSWORD" >> .env

    - docker-compose down
    - docker-compose pull
    - docker-compose up -d

💥 Result

SECRET_KEY missing ❌ (django)
App crashed ❌
Developer panic ❌

In Django, the SECRET_KEY is a critical cryptographic key used internally to secure and sign sensitive data such as user sessions, CSRF tokens, password reset links, and signed cookies; it acts like a private signature that ensures data cannot be tampered with or forged. If the SECRET_KEY is missing, Django cannot perform these security operations and the application will fail to start, which is why your app crashed when the .env file containing it was removed. If the SECRET_KEY is changed, all existing sessions and tokens become invalid because they were signed with the old key. For security reasons, it should never be exposed publicly or hardcoded in code, and must be stored securely in environment variables (like .env, CI/CD variables, or a secrets manager), especially in Docker setups where the application reads it during container startup.

🧠 Why?

Because app was using:

👉 It reads from local volume

The rm -f .env command in the CI/CD pipeline only removed the .env file from the current working directory (of the container ) of the pipeline (for example, ./ .env), where variables like CELERY_BROKER_URL=amqp://guest:guest@rabbitmq:5672// were being injected, but this did not affect the actual environment file used by the container, which was mounted via Docker volume from /home/app/.env to /app/.env. Because of this, the application did not crash when rm -f .env was executed, since it was still reading all required variables (including SECRET_KEY) from the volume-based .env file; however, services like Celery and RabbitMQ could face issues if their required variables were missing from the pipeline-generated .env. The application itself crashed only after the volume mapping (/home/app/.env:/app/.env) was removed from docker-compose.yml and replaced with env_file, because at that point the container started relying entirely on the .env generated from CI/CD variables, and since those variables did not include the Django SECRET_KEY, the application started with incomplete configuration and failed to start. The application crash issue was resolved by reusing the volume mapping (/home/app/.env:/app/.env) and removing env_file from docker-compose.yml, allowing the container to again read the complete set of environment variables from the local .env file. After that, the remaining issue was resolved by adding CELERY_BROKER_URL=amqp://guest:guest@rabbitmq:5672// directly into the local volume-based .env file at /home/app/.env, which is mounted inside the container at /app/.env, ensuring that Celery could correctly connect to RabbitMQ using the Docker service name.

✅ Fix

Restored local .env

🎯 Why volume used?

Because:

All configs already present:
- DB
- SMTP
- SECRET_KEY
- APIs

🚨 CI/CD vs LOCAL ENV (MAJOR CONFUSION)

❌ Problem

Pipeline .env ≠ Volume .env

🧠 Lesson

Always check where .env is coming from

🚨 SMTP CREDENTIAL DECISION

I received:

Production SMTP credentials

🧠 Decision

Did NOT use them ❌

👉 Used working safe credentials ✅

🔥 Best Practice

Never use production credentials blindly

🚨 RESTART ISSUE (VERY IMPORTANT)

❌ What happened

Restarted only web container ❌

🧠 Why it failed?

Celery still using old environment ❌

👉 Each container has its own environment

✅ Fix

docker-compose down
docker-compose up -d

🧠 Why it worked?

All containers restarted → new env loaded everywhere

🚨 ALL PROBLEMS FACED (IN ORDER)

1. RabbitMQ connection failed

2. Wrong broker URL

3. .env confusion

4. Deleted .env

5. SMTP credentials expired

6. Username/domain mismatch

7. Restart mistake

8. SMTP permission issue

9. CI/CD vs local env confusion

🚀 BEST PRACTICES

🔥 1. Always know env source

CI vs volume vs env_file

🔥 2. Never delete .env blindly

🔥 3. Restart ALL containers after config change

🔥 4. Use service names

rabbitmq ✅
localhost ❌

🔥 5. Validate SMTP manually

🔥 6. Use secrets manager (future improvement)

🔥 7. Add health checks

🎯 FINAL LEARNING

DevOps is NOT setup
DevOps is debugging + understanding system behavior

🎉 FINAL RESULT

Email sent successfully ✅
Celery working ✅
RabbitMQ working ✅
System stable ✅

🔄 Task Flow Diagram

🚀 Final Architecture

User → Django → RabbitMQ → Celery → SMTP
                     ↑
                Celery Beat

🔥 Conclusion

This journey taught me:

• RabbitMQ → message broker

• Celery → executor

• Celery Beat → scheduler

• Docker → networking isolation

• Env management → critical

• SMTP → tricky debugging

• Restart → crucial

Here is the current working ci pipeline and Docker compose file

this is gitlab-ci.yml

variables:
  IMAGE_NAME: $CI_REGISTRY_IMAGE:latest
  VERSION_TAG: \(CI_COMMIT_REF_NAME-\)CI_COMMIT_SHORT_SHA

services:
  - docker:dind

stages:
  - build
  - deploy

# -------------------------------
# BUILD STAGE
# -------------------------------
build_image:
  image: docker:cli
  stage: build
  script:
    - docker login -u "\(CI_REGISTRY_USER" -p "\)CI_REGISTRY_PASSWORD" "$CI_REGISTRY"
    - docker build -t \(IMAGE_NAME -t \)CI_REGISTRY_IMAGE:$VERSION_TAG .
    - docker push $IMAGE_NAME
    - docker push \(CI_REGISTRY_IMAGE:\)VERSION_TAG
  only:
    - main
  tags:
    - docker-runner

# -------------------------------
# DEPLOY STAGE
# -------------------------------
deploy_container:
  image: docker:cli
  stage: deploy
  script:
    - docker login -u "\(CI_REGISTRY_USER" -p "\)CI_REGISTRY_PASSWORD" "$CI_REGISTRY"

    # NOTE:
    # Using .env from server volume instead of CI variables
    # Avoid mixing multiple env sources

    - docker-compose down
    - docker-compose pull
    - docker-compose up -d

    - sleep 15

    - echo "===== RUNNING CONTAINERS ====="
    - docker ps -a

    - echo "===== DOCKER COMPOSE STATUS ====="
    - docker-compose ps

    - echo "===== APPLICATION LOGS ====="
    - docker logs app-container || true

    - echo "===== CONTAINER STATE ====="
    - docker inspect app-container --format='{{.State.Status}}'

    - docker image prune -f

  only:
    - main
  tags:
    - docker-runner

In this pipeline, we are NOT creating a .env file inside CI/CD.

Instead, the application uses a .env file mounted from the server using Docker volumes.

This avoids conflicts between CI/CD variables and runtime environment configuration.

Docker Compose file

version: '3.8'

services:

  web:
    image: '\({CI_REGISTRY_IMAGE}:\){VERSION_TAG}'
    container_name: app-web
    restart: always
    ports:
      - "8000:8000"
    volumes:
      # Mount environment file
      - /opt/app/.env:/app/.env

      # Application data (generic paths)
      - /opt/app/data:/app/data
      - /opt/app/logs:/app/logs

      # Optional override file (for configs/scripts)
      - /opt/app/config_override.py:/app/config_override.py

    depends_on:
      - rabbitmq

  rabbitmq:
    image: rabbitmq:3-management
    container_name: rabbitmq
    restart: always
    ports:
      - "5672:5672"
      - "15672:15672"

  celery:
    image: '\({CI_REGISTRY_IMAGE}:\){VERSION_TAG}'
    container_name: celery-worker
    command: celery -A app worker --loglevel=info
    volumes:
      - /opt/app/.env:/app/.env
      - /opt/app/config_override.py:/app/config_override.py
    depends_on:
      - rabbitmq
    restart: always

  celery-beat:
    image: '\({CI_REGISTRY_IMAGE}:\){VERSION_TAG}'
    container_name: celery-beat
    command: celery -A app beat --loglevel=info
    volumes:
      - /opt/app/.env:/app/.env
      - /opt/app/config_override.py:/app/config_override.py
    depends_on:
      - rabbitmq
    restart: always

In this setup, the .env file is mounted from the host machine using Docker volumes.

This means the application reads environment variables directly from the server instead of relying on CI/CD variables.

This approach is useful when:

• All configurations already exist on the server

• You want quick recovery without rebuilding pipelines

• You want to avoid conflicts between multiple env sources

However, in production, it is recommended to move toward CI/CD variables or a secrets manager for better security and maintainability.