Create a container image for HPC

Overview

This KB outlines how to do the following:

  • Create a Docker image on your laptop
  • Export the image as a file and transfer it to a HPC server
  • Convert the image to an Apptainer (formerly Singularity) image
  • Run a container

 

Requirements

 

Create a Python file with demo code

cat << 'EOF' > fibonacci.py
import os, pickle, sys, time

CHECKPOINT_FILE = "fibonacci.pickle"

class Fibonacci:

  def __init__(self, n):
    self.checkpoint_i = 1
    self.n = n
    self.a = 0
    self.b = 1

  def run(self):
    print(f"Fibonacci: {self.n} (starting at calc {self.checkpoint_i})")
    self.checkpoint_time = time.time()
    if self.n == 0:
      return 0
    elif self.n == 1:
      return self.b
    else:
      for i in range(self.checkpoint_i, self.n):
        self.c = self.a + self.b
        self.a = self.b
        self.b = self.c
        if time.time() - self.checkpoint_time > 10:
          print(f"Checkpoint: {i}")
          self.checkpoint_i = i + 1
          with open(CHECKPOINT_FILE, 'wb') as f:
            pickle.dump(self, f)
          self.checkpoint_time = time.time()
      return self.b

# Handle calling Fibonacci from a script
if __name__ == "__main__":
  # allow for working with large numbers
  sys.set_int_max_str_digits(0)
  if os.path.exists(CHECKPOINT_FILE):
    with open(CHECKPOINT_FILE, 'rb') as f:
      fib = pickle.load(f)
  else:
    fib = Fibonacci(int(sys.argv[1]))
  # start/continue calculating
  result = fib.run()
  print(f"Result: {result}")
  # clean-up checkpoint file if exists
  if os.path.exists(CHECKPOINT_FILE):
    os.remove(CHECKPOINT_FILE)
EOF


Run the Python file to test it

python ./fibonacci.py 30


Create a Dockerfile

cat << 'EOF' > Dockerfile
FROM python:3.10-alpine

COPY fibonacci.py /src/
EOF


Build the image locally

If not on a Mac with a M1 chip

docker build -t fibonacci:latest .

If on a Mac with a M1 chip, build the image for x86_64 architecture

docker buildx build --platform linux/amd64 -t fibonacci:latest .


Run a container locally

docker run -it --rm fibonacci:latest python /src/fibonacci.py 30


Export the image

docker save fibonacci:latest -o fibonacci.tar


Transfer the image to Discovery

scp fibonacci.tar netid@discovery:


Build an Apptainer image from the Docker image

ssh netid@discovery

apptainer build fibonacci docker-archive://fibonacci.tar


Run an Apptainer container

apptainer exec fibonacci python /src/fibonacci.py 40


Create Slurm job script

cat << 'EOF' > fibonacci-apptainer.sh
#!/bin/bash
#SBATCH --job-name=fibonacci-apptainer
#SBATCH --error=%x-%J.err
#SBATCH --output=%x-%J.out
#SBATCH --requeue
#SBATCH --time=00:01:00
#SBATCH --mem=1G
##SBATCH --account=free
##SBATCH --partition=standard

scontrol show job $SLURM_JOBID > fibonacci-info.$SLURM_JOBID

stdbuf -i0 -o0 -e0 apptainer exec fibonacci python -u /src/fibonacci.py 3000001
EOF


Submit the job to Discovery

sbatch fibonacci-apptainer.sh

NOTE: You can use the squeue command to see the status of your job.

Print Article