No Space Left on Device

Got : printf: write error: No space left on device

Check current space

To check the current disk usage on CentOS 7, you can use the following commands:

1. Check disk usage for all mounted filesystems:

   df -h

   The -h flag makes the output human-readable.

1. Check disk usage of specific directories:

   du -sh /path/to/directory

   Replace /path/to/directory with the directory you want to check.

1. Find large files and directories:

   du -ah / | sort -rh | head -n 20

   This command will list the top 20 largest files and directories starting from the root (/).

When dealing with Docker images in a Minikube and GitLab CI/CD setup, managing disk space becomes crucial. Docker images can consume significant space, especially if not managed properly. Here are some best practices for managing Docker images to avoid running out of disk space:

Best Practices for Docker Images

1. Prune Unused Docker Objects Regularly:
   • Docker provides commands to remove unused images, containers, volumes, and networks. Schedule regular cleanups to free up space.
   • Use the following command to remove all unused objects:

     docker system prune -a -f

   • To prune only unused images:

     docker image prune -a -f

1. Limit Image Retention:
   • Configure a retention policy to keep only a certain number of recent images.
   • Use the docker image ls command to list all images, and then remove older images using docker rmi <image_id>.

1. Optimize Dockerfile:
   • Create small and efficient Docker images by following best practices in your Dockerfile.
   • Use multi-stage builds to reduce the final image size.
   • Example of a multi-stage build Dockerfile:

     # First stage: build
     FROM maven:3.6.3-jdk-8 as builder
     WORKDIR /app
     COPY . .
     RUN mvn clean install
     
     # Second stage: run
     FROM openjdk:8-jre-alpine
     WORKDIR /app
     COPY --from=builder /app/target/myapp.jar /app/myapp.jar
     CMD ["java", "-jar", "/app/myapp.jar"]
     
     

1. Use .dockerignore:
   • Use a .dockerignore file to exclude unnecessary files and directories from the build context, reducing the image size.
   • Example .dockerignore file:

     node_modules
     target
     *.log
     
     

1. Tag and Clean Up Images:
   • Properly tag images with version numbers to keep track of them.
   • Remove old images periodically using a script or CI/CD pipeline step.
   • Example cleanup script in a GitLab CI/CD job:

     cleanup:
       stage: cleanup
       script:
         - docker image prune -a -f
       only:
         - schedules
     
     

1. Use a Docker Registry:
   • Push images to a Docker registry (e.g., Docker Hub, GitLab Container Registry) instead of keeping them locally.
   • Pull images from the registry as needed, reducing local storage usage.

Example GitLab CI/CD Pipeline with Docker Cleanup

Here's an example of a GitLab CI/CD pipeline that includes a cleanup job for Docker images:

stages:
  - build
  - test
  - deploy
  - cleanup

variables:
  DOCKER_DRIVER: overlay2

cache:
  paths:
    - .m2/repository
    - node_modules/

before_script:
  - docker info

build:
  stage: build
  script:
    - echo "Building Docker image..."
    - docker build -t myapp:${CI_COMMIT_SHA} .
    - docker tag myapp:${CI_COMMIT_SHA} myapp:latest
  tags:
    - docker

test:
  stage: test
  script:
    - echo "Running tests..."
    - docker run --rm myapp:${CI_COMMIT_SHA} ./run_tests.sh
  tags:
    - docker

deploy:
  stage: deploy
  script:
    - echo "Deploying Docker image..."
    - docker push myapp:${CI_COMMIT_SHA}
    - docker push myapp:latest
  tags:
    - docker
  environment:
    name: production
    url: https://your.production.url

cleanup:
  stage: cleanup
  script:
    - echo "Cleaning up old Docker images..."
    - docker image prune -a -f
  tags:
    - docker
  only:
    - schedules

Monitoring Disk Space

Implement monitoring tools to track disk space usage and set alerts:

1. Install monitoring tools:
   • Tools like Prometheus and Grafana can help monitor disk usage and set up alerts.
   • Install and configure node_exporter for Prometheus to collect disk usage metrics.

1. Set up alerts:
   • Configure alerts in Prometheus to notify you when disk usage exceeds a certain threshold.

By following these best practices, you can effectively manage Docker images, ensuring your Minikube environment and GitLab CI/CD pipelines run smoothly without running out of disk space.

To show the disk usage of Docker images, you can use the docker system df command, which provides a summary of Docker disk usage including images, containers, volumes, and the build cache.

Here’s how you can get detailed information on disk usage for Docker images:

1. Show overall Docker disk usage:

   docker system df
   
   

1. Show detailed disk usage by individual images:

   docker images --format "table {{.Repository}}\t{{.Tag}}\t{{.Size}}"
   
   

Example Output

Overall Docker Disk Usage

Running docker system df will output something like this:

TYPE            TOTAL     ACTIVE    SIZE      RECLAIMABLE
Images          10        5         2.33GB    1.76GB (75%)
Containers      5         3         850MB     850MB (100%)
Local Volumes   3         2         1.2GB     500MB (41%)
Build Cache     0         0         0B        0B

Detailed Docker Images Disk Usage

Running docker images --format "table {{.Repository}}\t{{.Tag}}\t{{.Size}}" will output something like this:

REPOSITORY          TAG                 SIZE
myapp               latest              500MB
myapp               v1.0.0              500MB
nginx               latest              133MB
alpine              latest              5.59MB

Additional Commands

• List all Docker images with sizes:

  docker images --format "table {{.ID}}\t{{.Repository}}\t{{.Tag}}\t{{.Size}}"
  
  

• Inspect a specific image to get more details:

  docker inspect <image_id>
  
  

• List all Docker containers and their sizes:

  docker ps -a --format "table {{.ID}}\t{{.Image}}\t{{.Size}}"
  
  

Regular Cleanup

To avoid excessive disk usage, regularly clean up unused Docker objects. You can use a cron job or schedule a job in your CI/CD pipeline to perform cleanup tasks.

Here's a sample cron job that prunes unused Docker objects weekly:

1. Open the cron job editor:

   crontab -e
   
   

1. Add the following line to schedule the cleanup:

   0 0 * * 0 docker system prune -a -f
   
   

This will run docker system prune -a -f every Sunday at midnight.

By using these commands and best practices, you can effectively monitor and manage the disk usage of Docker images in your Minikube environment.