GitLab CI Pipeline Failed Before Build — The Hidden Runner Permission Issue
A real-world GitLab Runner debugging story every DevOps engineer should know.
TL;DR
If your GitLab CI pipeline fails during the Fetching changes stage with an error like:
error: insufficient permission for adding an object to repository database .git/objects
fatal: failed to write object
fatal: unpack-objects failed
The most common cause is incorrect file ownership inside the GitLab Runner workspace.
You can usually fix it by resetting permissions:
sudo chown -R gitlab-runner:gitlab-runner /home/gitlab-runner/builds
Introduction
While working on a CI/CD pipeline recently, I encountered a strange issue where the GitLab pipeline failed even before the build stage started.
At first, it looked like a normal Git error. However, after investigating further, I discovered that the problem was caused by file permission conflicts inside the GitLab Runner workspace.
As someone early in my DevOps journey, debugging this issue helped me better understand how GitLab Runner manages repository workspaces and permissions.
In this article, I’ll explain:
what caused the issue
how I diagnosed it
how to fix it
how to prevent it in production environments
Understanding How GitLab Runner Works
When a GitLab pipeline starts, the runner performs several steps automatically.
Prepare the execution environment
Fetch the repository from GitLab
Checkout the requested branch or commit
Execute the CI/CD job scripts
GitLab Runner stores repository builds in a workspace directory like:
/home/gitlab-runner/builds/<runner-id>/<namespace>/<project>
Inside this directory, GitLab clones the repository and stores its metadata inside the .git directory.
If permissions inside this workspace are incorrect, Git operations may fail during the fetch stage.
The Error
The pipeline failed during the Fetching changes stage with the following error:
error: insufficient permission for adding an object to repository database .git/objects
fatal: failed to write object
fatal: unpack-objects failed
ERROR: Job failed: exit status 1
This error occurs before any CI job starts, meaning the problem happens while GitLab Runner tries to download or update the repository.
Root Cause Analysis
GitLab Runner jobs usually execute using the following user:
gitlab-runner
However, if someone runs commands manually with sudo inside the runner workspace, some files may become owned by the root user.
Examples of commands that can cause this:
sudo git pull
sudo vim file
sudo cp file
sudo docker exec container git pull
When this happens, the repository ends up containing files owned by different users.
Later, when GitLab Runner attempts to fetch new commits, it cannot modify those files.
This leads to the permission error during the fetch stage.
How I Discovered the Issue
To diagnose the problem, I logged into the GitLab Runner server and inspected the runner workspace.
Step 1 — Locate the Runner Workspace
GitLab Runner stores builds inside:
/home/gitlab-runner/builds/
Inside this directory, each project has its own workspace.
To inspect the repository directory and verify permissions:
ls -la /home/gitlab-runner/builds/<runner-id>/<namespace>/<project>
This command shows:
file ownership
permissions
directory structure
The goal here was to check if any files were owned by root instead of gitlab-runner.
Step 2 — Inspect the Git Metadata Directory
Since the error mentioned .git/objects, the next step was to inspect the .git directory.
ls -la /home/gitlab-runner/builds/<runner-id>/<namespace>/<project>/.git
The .git directory contains important repository metadata such as:
commit references
branch information
logs
configuration
Git object database
Example output might look like:
-rw-rw-r-- gitlab-runner gitlab-runner HEAD
-rw-r--r-- root root ORIG_HEAD
drwxrwxr-x gitlab-runner gitlab-runner objects
Most files are owned by gitlab-runner, but one file may be owned by root.
This indicates that a command was executed using sudo privileges inside the repository.
Step 3 — Verify the Object Database
Next, inspect the Git object database directory:
ls -la /home/gitlab-runner/builds/<runner-id>/<namespace>/<project>/.git/objects
This directory stores Git objects such as:
commits
trees
blobs
Git must be able to write to this directory when fetching new commits.
If permissions are incorrect, GitLab Runner cannot update the repository.
The Fix
The simplest fix is to restore the correct ownership of the runner workspace.
sudo chown -R gitlab-runner:gitlab-runner /home/gitlab-runner/builds
Then restart GitLab Runner:
sudo systemctl restart gitlab-runner
After this, the pipeline should run successfully.
Alternative Fix — Clean the Runner Workspace
Sometimes the repository workspace becomes corrupted.
In such cases, removing the entire workspace is safer.
sudo rm -rf /home/gitlab-runner/builds/*
When the pipeline runs again, GitLab Runner will clone the repository fresh.
Things to Check Before Fixing
Before immediately resetting permissions, it’s good practice to verify the environment.
Check file ownership
ls -la /home/gitlab-runner/builds
Check runner status
sudo gitlab-runner status
Check disk space
df -h
Check runner logs
sudo journalctl -u gitlab-runner -f
Production Recommendations
For production environments, consider these improvements.
Prefer Docker executor over Shell executor
Shell executor shares the same environment between jobs.
Docker executor creates isolated environments, which helps prevent permission conflicts.
Use clean Git strategy
In .gitlab-ci.yml, you can enforce fresh repository clones.
variables:
GIT_STRATEGY: clone
Avoid manual changes inside runner workspace
Never manually modify files inside:
/home/gitlab-runner/builds
This directory should only be managed by GitLab Runner.
Improvements for CI/CD Reliability
Some additional improvements that can make pipelines more reliable include:
using ephemeral runners
automating workspace cleanup
monitoring runner health
managing runners using Infrastructure as Code
Tools like Terraform, Ansible, or Kubernetes runners can help maintain consistent runner environments.
DevOps Lesson
CI/CD pipelines depend heavily on the health of the runner environment.
Even a small permission mismatch can break the entire pipeline.
Maintaining clean runner workspaces and avoiding manual modifications is critical for reliable CI/CD systems.
As someone early in my DevOps career, solving issues like this helps build a deeper understanding of how CI/CD systems behave in real environments.
💬 Have You Faced Similar Issues?
CI/CD failures sometimes come from very small environment issues.
Have you ever faced GitLab pipeline failures due to permission problems or runner misconfigurations?
Share your experience in the comments — I’d love to learn how you solved them.
