Running Jupyter Notebooks on SLURM via singularity

Build an singularity image

On your local machine build an sigularity image by providing a .def file. Check https://sylabs.io/guides/latest/user-guide/quick_start.html#build-images-from-scratch for more information.

• We use a docker container as base for anaconda. See https://hub.docker.com/r/continuumio/anaconda3/ for more information

Now povide a jupyter.def file:

BootStrap: docker
From: continuumio/anaconda3

%runscript
  echo "Starting notebook..."
  echo "Open browser to localhost:8888"
  exec /opt/conda/bin/jupyter notebook --notebook-dir=/opt/notebooks --ip='*' --port=8888 --no-browser --allow-root

%post
  # Install jupyter notebook
  /opt/conda/bin/conda install jupyter -y --quiet 
  mkdir /opt/notebooks
  apt-get autoremove -y
  apt-get clean

Now create the singularity image:

sudo singularity build jupyter.sif jupyter.def

Transfer the image to julia:

scp jupyter.sif <user>@julia.uni-wuerzburg.de:~

Prepare the jupyter environment

On julia prepare the jupyter environment:

Create a notebooks directory for the jupyter notebook files

mkdir notebooks

Create a singularity SLURM job file

Let's test our rstudio-server first for one hour in the test partition. We can do this e.g. with the following jupyter.sh jobfile

#!/bin/bash

#SBATCH -J RStudioServer
#SBATCH -c 2
#SBATCH -p test
#SBATCH --mail-type=ALL
#SBATCH --mail-user=<your-email-address>
#SBATCH -t 1:00:00

hostname
singularity run --bind notebooks:/opt/notebooks jupyter.sif

This will

• bind mount your $HOME/notebooks directory to /opt/notebooks inside the image

Of course you can also mount other directories to your image via the ---bind option

Test the job file

Submit the jobfile:

sbatch jupyter.sh
Submitted batch job 50982917

You will receive an email when your job starts. In this case our job id is 50982917 and we can check the output file:

cat slurm-50982917.out 
cntest1

Our jupyter server now runs on cntest1

Build an ssh tunnel

First check if the necessary port on julia is not already in use by another user. So check on julia:

ss -tulpen | grep 8888

If you don't get any output the port is available to use. If not, check for another port like 8889 etc

From your local machine connect to julia and forward port 8888 to your local machine:

ssh -L 8888:localhost:8888 <user>@julia.uni-wuerzburg.de

If port 8888 on julia is already in use but 8889 is free, change the command to:

ssh -L 8888:localhost:8889 <user>@julia.uni-wuerzburg.de

Forward the port 8888 of the machine rstudio is running on to julia:

ssh -Nf -L 8888:localhost:8888 cntest1

If port 8888 on julia is already in use but 8889 is free, change the command to:

ssh -Nf -L 8889:localhost:8888 cntest1

In order to login, you need to provide an access token. This access token will be automatically generated on startup. Check your slurm-<jobid>.out file for the current token:

cat slurm-50982917.out 
...
[I 11:54:49.497 NotebookApp] Serving notebooks from local directory: /opt/notebooks
[I 11:54:49.498 NotebookApp] Jupyter Notebook 6.4.5 is running at:
[I 11:54:49.498 NotebookApp] http://cntest1:8888/?token=1bf9027884d9c94f3672e1285d7c882346e9f0712700be0d
[I 11:54:49.499 NotebookApp]  or http://127.0.0.1:8888/?token=1bf9027884d9c94f3672e1285d7c882346e9f0712700be0d
[I 11:54:49.499 NotebookApp] Use Control-C to stop this server and shut down all kernels (twice to skip confirmation).
[C 11:54:49.509 NotebookApp] 

    To access the notebook, open this file in a browser:
        file:///home/alp41ri/.local/share/jupyter/runtime/nbserver-21178-open.html
    Or copy and paste one of these URLs:
        http://cntest1:8888/?token=1bf9027884d9c94f3672e1285d7c882346e9f0712700be0d
     or http://127.0.0.1:8888/?token=1bf9027884d9c94f3672e1285d7c882346e9f0712700be0d

Now you can click on http://127.0.0.1:8888/?token=1bf9027884d9c94f3672e1285d7c882346e9f0712700be0d and see if the tunnel is working

Cleanup

After your job is done don't forget to kill old remaining ssh tunnel processes on julia. The PIDs to kill are in the second row. In this case 392

ps fux 
...
user    392  0.0  0.0  17476  2924 ?        Ss   17:54   0:00 ssh -Nf -L 8787:localhost:8787 cntest1

kill 392

Tipp:

If you have a lot of old, stalled ssh-sessions, you can clean them up via:

ps fux | grep "ssh -N" | grep -v grep | awk '{print $2}' | xargs kill