Skip to main content

Command Palette

Search for a command to run...

My GitLab CI Pipeline Failed… Then Worked Without Changing Anything

When My GitLab CI Pipeline Failed Because of a Debian Mirror Issue (And Fixed Itself)

Updated
6 min read
P
Welcome! I’m Prajwal P. I stand at the intersection of technology and efficiency, exploring the dynamic world of DevOps ⚙️. From mastering Cloud infrastructure to orchestrating containers, I am passionate about automating the complex to create the simple. Join me as I document my learning curve, share technical insights, and navigate the ever-evolving landscape of software deployment.

While working as a Junior DevOps Engineer, I recently faced an interesting CI/CD pipeline issue for the first time.

My GitLab pipeline failed during a Docker build with an error that looked like a package problem.

But surprisingly, rerunning the pipeline fixed it instantly.

This experience helped me understand something important about Debian package mirrors and CI environments.

Let’s break down what happened.


The Situation

My GitLab CI pipeline was building a Docker image for a Python application.

The Dockerfile had a standard dependency installation step:

FROM python:3.8-slim-bullseye

RUN apt-get update \
 && apt-get install -y --no-install-recommends \
 build-essential \
 libpq-dev \
 openssh-client \
 && apt-get clean \
 && rm -rf /var/lib/apt/lists/*

This step installs system dependencies required by the application.

Everything looked perfectly normal.

But during the pipeline execution, the build suddenly failed.


The Error I Saw

The GitLab pipeline logs showed this error:

E: Failed to fetch http://deb.debian.org/debian/pool/main/g/gcc-10/g++-10_10.2.1-6_amd64.deb

Error reading from server - read (104: Connection reset by peer)

E: Unable to fetch some archives

The job failed with:

exit code: 100

At first, it looked like:

• a Dockerfile problem
• a broken package
• or dependency conflict

But none of those were actually the issue.


Understanding the Real Cause

The key line was:

Connection reset by peer

This means the server closed the connection while the package was being downloaded.

In my case, the connection to the Debian package mirror was interrupted.


What is a Debian Mirror? (Simple Explanation)

When I first saw the term mirror, I was confused because it sounded like a technical term.

But the concept is actually very simple.

A mirror is just a copy of a software repository stored on another server.

Debian packages are stored in a repository, and instead of hosting everything on one single server, Debian keeps many identical copies of the repository across different servers around the world.

These copies are called mirrors.

All mirrors contain the same packages, but they exist in multiple locations.


A Simple Real-World Analogy

Imagine a popular bookstore.

If there is only one bookstore in the world, everyone would have to go there to buy books.

That store would become overloaded.

So instead, the company opens many branches of the same bookstore in different cities.

Each branch contains the same books, but people can go to the closest store.

Debian mirrors work exactly the same way.

Instead of downloading packages from one central server, your system downloads them from one of many mirror servers.


Example of Debian Mirrors

When we run:

apt-get update

The system connects to a repository like this:

http://deb.debian.org/debian

But this is not a single physical server.

It redirects your request to one of many mirrors such as:

mirror.us.debian.org
mirror.sg.debian.org
mirror.debian.org
mirror.in.debian.org

All these mirrors contain the same packages, but they help distribute traffic across the internet.


Why Mirrors Exist

Mirrors exist to improve:

1. Download Speed

Users download packages from the closest mirror server, reducing latency.

2. Reliability

If one server goes down, another mirror can serve the same packages.

3. Load Balancing

Millions of developers use Debian repositories daily. Mirrors distribute the traffic across multiple servers.

4. Global Access

Developers around the world can download packages from nearby servers.


Why Mirror Issues Sometimes Happen

Although mirrors improve reliability, they can still experience temporary issues.

For example:

• network interruptions
• server restarts
• overloaded mirror servers
• temporary connection resets
• bandwidth throttling

When this happens, package downloads may fail with errors like:

Failed to fetch package
Connection reset by peer
Unable to fetch archives

This is exactly what happened in my pipeline.


The Interesting Part

Instead of changing anything in the code or Dockerfile, I simply reran the pipeline.

And the build worked perfectly.

That confirmed the issue was not related to the Dockerfile or CI configuration, but rather a temporary network or mirror issue.


How to Troubleshoot This Type of Error

When a CI pipeline fails during package installation, follow a structured troubleshooting process.


Step 1 — Identify Where the Pipeline Failed

First, locate the exact step that failed.

In my pipeline it failed during:

apt-get install

This immediately indicates the failure is related to package installation.


Step 2 — Look for Network-Related Keywords

Search the logs for keywords such as:

Failed to fetch
Connection reset
Unable to fetch archives
Temporary failure resolving

These usually indicate network or mirror related issues.


Step 3 — Look at the Exit Code

APT usually fails with:

exit code: 100

This indicates a package installation failure, not a Docker or GitLab runner issue.


Step 4 — Check Which Package Failed

The logs often show the exact package that failed.

Example:

g++-10_10.2.1-6_amd64.deb

This confirms that the failure occurred during package download, not during compilation.


Step 5 — Rerun the Pipeline

If rerunning the pipeline succeeds without changes, it strongly suggests:

• a temporary mirror issue
• a network interruption

This is exactly what happened in my case.


How to Prevent This Issue

One simple improvement is using the --fix-missing flag.

Example:

RUN apt-get update \
 && apt-get install -y --no-install-recommends \
 build-essential \
 libpq-dev \
 openssh-client \
 --fix-missing \
 && apt-get clean \
 && rm -rf /var/lib/apt/lists/*

This allows apt to retry missing downloads instead of failing immediately.


Another Preventive Option

Some teams add retry logic:

RUN apt-get update || apt-get update

This ensures the update command retries if the first attempt fails.


A Small Optimization Tip

If you install:

build-essential

you do not need to install gcc separately, because it is already included.


Key Lessons from This Issue

This experience taught me a few useful DevOps lessons:

  1. Not all pipeline failures are caused by code.

  2. External services like package repositories can fail temporarily.

  3. Reading logs carefully helps identify the real issue.

  4. Making pipelines resilient improves reliability.

Sometimes the simplest solution is:

Rerun the pipeline and observe the behavior.


Final Thoughts

This was the first time I encountered a Debian mirror-related issue in CI/CD.

Although the pipeline failed initially, it helped me better understand:

• how package repositories work
• why mirrors exist
• how network issues affect CI pipelines

These real-world debugging experiences gradually build stronger DevOps troubleshooting skills.

And sometimes the biggest lesson is simple:

Not every failure is caused by your code. Sometimes the infrastructure just needs another try. 🚀


More from this blog

Terraform on AWS

29 posts

Stop clicking in the AWS console. Start coding your infrastructure.