Body
Overview
This KB outlines how to do the following:
- Create a Docker image on your laptop
- Export the image as a file and transfer it to a HPC server
- Convert the image to an Apptainer (formerly Singularity) image
- Run a container
Requirements
Create a Python file with demo code
cat << 'EOF' > fibonacci.py
import os, pickle, sys, time
CHECKPOINT_FILE = "fibonacci.pickle"
class Fibonacci:
def __init__(self, n):
self.checkpoint_i = 1
self.n = n
self.a = 0
self.b = 1
def run(self):
print(f"Fibonacci: {self.n} (starting at calc {self.checkpoint_i})")
self.checkpoint_time = time.time()
if self.n == 0:
return 0
elif self.n == 1:
return self.b
else:
for i in range(self.checkpoint_i, self.n):
self.c = self.a + self.b
self.a = self.b
self.b = self.c
if time.time() - self.checkpoint_time > 10:
print(f"Checkpoint: {i}")
self.checkpoint_i = i + 1
with open(CHECKPOINT_FILE, 'wb') as f:
pickle.dump(self, f)
self.checkpoint_time = time.time()
return self.b
# Handle calling Fibonacci from a script
if __name__ == "__main__":
# allow for working with large numbers
sys.set_int_max_str_digits(0)
if os.path.exists(CHECKPOINT_FILE):
with open(CHECKPOINT_FILE, 'rb') as f:
fib = pickle.load(f)
else:
fib = Fibonacci(int(sys.argv[1]))
# start/continue calculating
result = fib.run()
print(f"Result: {result}")
# clean-up checkpoint file if exists
if os.path.exists(CHECKPOINT_FILE):
os.remove(CHECKPOINT_FILE)
EOF
Run the Python file to test it
python ./fibonacci.py 30
Create a Dockerfile
cat << 'EOF' > Dockerfile
FROM python:3.10-alpine
COPY fibonacci.py /src/
EOF
Build the image locally
If not on a Mac with a M1 chip
docker build -t fibonacci:latest .
If on a Mac with a M1 chip, build the image for x86_64 architecture
docker buildx build --platform linux/amd64 -t fibonacci:latest .
Run a container locally
docker run -it --rm fibonacci:latest python /src/fibonacci.py 30
Export the image
docker save fibonacci:latest -o fibonacci.tar
Transfer the image to Discovery
scp fibonacci.tar netid@discovery:
Build an Apptainer image from the Docker image
ssh netid@discovery
apptainer build fibonacci docker-archive://fibonacci.tar
Run an Apptainer container
apptainer exec fibonacci python /src/fibonacci.py 40
Create Slurm job script
cat << 'EOF' > fibonacci-apptainer.sh
#!/bin/bash
#SBATCH --job-name=fibonacci-apptainer
#SBATCH --error=%x-%J.err
#SBATCH --output=%x-%J.out
#SBATCH --requeue
#SBATCH --time=00:01:00
#SBATCH --mem=1G
##SBATCH --account=free
##SBATCH --partition=standard
scontrol show job $SLURM_JOBID > fibonacci-info.$SLURM_JOBID
stdbuf -i0 -o0 -e0 apptainer exec fibonacci python -u /src/fibonacci.py 3000001
EOF
Submit the job to Discovery
sbatch fibonacci-apptainer.sh
NOTE: You can use the squeue command to see the status of your job.