DEPRECATED

Check Julia 2 documentation instead!

Running RStudio on SLURM via singularity

Build an singularity image

On your local machine build an sigularity image by providing a .def file. Check https://sylabs.io/guides/latest/user-guide/quick_start.html#build-images-from-scratch for more information.

Now povide a rstudio.def file:

Bootstrap: docker
From: ubuntu:20.04

%post
  apt-get -y update
  apt-get -y install r-base gdebi-core wget
  wget https://download2.rstudio.org/server/bionic/amd64/rstudio-server-2021.09.2-382-amd64.deb
  yes | gdebi rstudio-server-2021.09.2-382-amd64.deb

%environment
  export LC_ALL=C
  export PATH=/usr/games:$PATH

%runscript

Now create the singularity image:

sudo singularity build rstudio.sif rstudio.def

Transfer the image to julia:

scp rstudio.sif <user>@julia.uni-wuerzburg.de:~

Prepare the RStudio-Server environment

On julia prepare the rstudio-server environment:

Create a rstudio directory for the settings and necessary runtime folders

mkdir -p rstudio/run rstudio/var-lib-rstudio-server

Tell rstudio-server to use a local database for management by providing a database.conf file:

printf 'provider=sqlite\ndirectory=/var/lib/rstudio-server\n' > rstudio/database.conf

This file should look like this:

provider=sqlite
directory=/var/lib/rstudio-server

Create a singularity SLURM job file

Let's test our rstudio-server first for one hour in the test partition. We can do this e.g. with the following rstudio.sh jobfile

#!/bin/bash

#SBATCH -J RStudioServer
#SBATCH -c 2
#SBATCH -p test
#SBATCH --mail-type=ALL
#SBATCH --mail-user=<your-email-address>
#SBATCH -t 1:00:00

hostname
singularity exec --bind $HOME/rstudio/run:/run,$HOME/rstudio/var-lib-rstudio-server:/var/lib/rstudio-server,$HOME/rstudio/database.conf:/etc/rstudio/database.conf rstudio.sif /usr/lib/rstudio-server/bin/rserver --server-user $USER

This will

  • bind mount your $HOME/rstudio/run directory to /run inside the image
  • bind mount $HOME/rstudio/var-lib-rstudio-server to /var/lib/rstudio-server in the image
  • Provide a /etc/rstudio/database.conf for rstudio via bind mont of $HOME/rstudio/database.conf
  • Singularity will also exec /usr/lib/rstudio-server/bin/rserver in the provided rstudio.sif as your $USER

Of course you can also mount other directories to your image via the ---bind option

Test the job file

Submit the jobfile:

sbatch rstudio.sh
Submitted batch job 50978897

You will receive an email when your job starts. In this case our job id is 50978897 and we can check the output file:

cat slurm-50978897.out 
cntest1

Our rstudio server now runs on cntest1

Build an ssh tunnel

First check if the necessary port on julia is not already in use by another user. So check on julia:

ss -tulpen | grep 8787

If you don't get any output the port is available to use. If not, check for another port like 8788 etc

From your local machine connect to julia and forward port 8787 to your local machine: 

ssh -L 8787:localhost:8787 <user>@julia.uni-wuerzburg.de

If port 8787 on julia is already in use but 8788 is free, change the command to:

ssh -L 8787:localhost:8788 <user>@julia.uni-wuerzburg.de

On julia forward the port 8787 of the machine rstudio is running to julia:

ssh -Nf -L 8787:localhost:8787 cntest1

If port 8787 on julia is already in use but 8788 is free, change the command to:

ssh -Nf -L 8788:localhost:8787 cntest1

Check if the tunnel is working by opening http://localhost:8787/ in your local browser.

Cleanup

After your job is done don't forget to kill old remaining ssh tunnel processes on julia. The PIDs to kill are in the second row. In this case 392

ps fux
...
user    392  0.0  0.0  17476  2924 ?        Ss   17:54   0:00 ssh -Nf -L 8787:localhost:8787 cntest1

kill 392

Tipp:

If you have a lot of old, stalled ssh-sessions, you can clean them up via:

ps fux | grep "ssh -N" | grep -v grep | awk '{print $2}' | xargs kill