Jupyter on a compute node

Tags jupyter

The following steps demo:

  1. Create a job submission script to run a Jupyter notebook on the Discovery cluster
  2. Submit the job to the scheduler
  3. Create a SSH tunnel to Discovery and browse to the notebook

Create a file named jupyter.sh with the following contents:

#!/bin/bash -l
#SBATCH --job-name=jupyter_notebook
#SBATCH --time=10:00:00
#SBATCH --output=jupyter_notebook_%j.txt
#SBATCH --error=jupyter_notebook_%j.err

# get tunneling info
XDG_RUNTIME_DIR=""
node=$(hostname -s)
user=$(whoami)
cluster="discovery"

# print tunneling instructions jupyter-log
echo -e "
# Note: below 8888 is used to signify the port.
#       However, it may be another number if 8888 is in use.
#       Check jupyter_notebook_%j.err to find the port.

# Command to create SSH tunnel:
ssh -N -f -L 8888:${node}:8888 ${user}@${cluster}.dartmouth.edu

# Use a browser on your local machine to go to:
http://localhost:8888/
"

conda activate <my env with jupyter installed>
jupyter-notebook --no-browser --ip=${node}

# keep it alive
sleep 36000

Next setup a password:

$ jupyter notebook --generate-config
$ jupyter notebook password

<Enter a password and confirmation password>

Once complete, submit the job:

$ sbatch jupyter.sh

When the job starts, take a look at the jupyter_notebook_<job_id>.txt file to see the SSH tunnel command needed

$ cat jupyter_notebook_<job_id>.txt

Next, exit discovery or open a new terminal window, and then run the commands from the output above. Example commands are given below.

# Command to create ssh tunnel:
ssh -N -f -L 8888:j04:8888 netid@discovery.dartmouth.edu

Use a browser on your local machine to go to:
http://localhost:8888/
Print Article