JHPCE tips

Running Shiny apps on JHPCE

This little tutorial shows how to run a shiny app server on a JHPCE node in a terminal (no X11 required) and use your local browser to access that shiny app locally through a ssh tunnel. Besides a web browser, a requirement for the local computer (your laptop or desktop PC) is to have a terminal with a working ssh client or any another ssh client software that is able to start a ssh tunnel (on Windows one can use putty for that but there are many other ways to get a ssh client or tunnel working on Windows).

The JHPCE environment assumed here is one with a conda_R module loaded and the shiny package installed (I noticed that conda_R/4.0.x already has the shiny package installed in its system library). After getting a qrsh session, run the shiny app with the option host="0.0.0.0". Optionally, a different port can be selected with the port= option.

For example, if qrsh assigned node compute-086, take note of this compute node name for later, and start a shiny app like this (I'm using the runExample() function here which takes the same parameters as runApp()):

 > library('shiny')
 > runExample("03_reactivity", host="0.0.0.0")

Listening on http://0.0.0.0:5792

Take note of the port number shown in the Listening on http://0.0.0.0:.... message (5792 in this example).

If you started your ssh and qrsh session with X11 display forwarding (unlikely), a remote browser window may popup - just close that, it is not pleasant to work on a remote browser through X11 forwarding (too laggy). If X11 display forwarding is not active an error message like this may appear:

Failed to open connection to "session" message bus: Unable to autolaunch a dbus-daemon without a $DISPLAY for X11
Running without a11y support!
Error: no DISPLAY environment variable specified

This message can be safely ignored -- the shiny app is still running actually, and we'll tunnel it through ssh to your local browser instead, which obviously is much more responsive than a remote browser.

In order to accomplish that, in a local terminal (on your laptop or desktop PC) start a ssh tunnel through the transfer node with the following ssh command (replace computer-086 with the compute node name you got for your qrsh session where you started the shiny app, and replace 5792 with the port shown in the Listening on http://0.0.0.0:.... message; also obviously replace user with your JHPCE login name):

 ssh -N -L 5792:compute-086.cm.cluster:5792 user@jhpce-transfer01.jhsph.edu

If the tunnel started successfully then on your computer point your web browser to the address: http://localhost:5792/ (again, replace 5792 in this example with whatever port was shown in the Listening on http://0.0.0.0:.... message when the shiny app was started)

Running JupyterLab for R and Python on JHPCE

In the following instructions, the term "local" refers to the computer you are physically in front of, the desktop/laptop where you can run a web browser comfortably and where you can also run a terminal with a ssh client or otherwise start a ssh tunnel, while the term "remote" as employed below refers to the JHPCE cluster machines (compute, login or transfer nodes).

Initial setup (only needed once)

In a qrsh session, load your conda_R module (e.g. module load conda_R/4.1.x), jupyterlab should be installed like this:

 pip install --user jupyterlab

Make sure ~/.local/bin is also added to your PATH environment variable, e.g. by adding the line export PATH=~/.local/bin:$PATH in your .bashrc after the line loading the conda_R module.

Then install the R kernel (engine) in order to run R code in JupyterLab notebooks. Start an interactive R session and run the following R commands:

devtools::install_github("IRkernel/IRkernel")
IRkernel::installspec()

Note: If devtools is not installed already, use install.packages("devtools") first.

Working in a JupyterLab session on JHPCE

After the above setup was completed, the following steps should be taken every time, in the given sequence, in order to start a jupyterlab session on a compute node and then access it through your local browser (assuming you have set up JHPCE access using SSH keys).

1. Start Jupyter Lab "server" on the compute node

In a qrsh session on a compute node, launch the jupyter lab server like this:

 jupyter lab --no-browser --port=15432 --ip=$(hostname)

(make sure ~/.local/bin is in your PATH otherwise the command will not be found). Port 15432 is a completely arbitrary number in this example, just use a high port number there >1000. If the port is already in use (e.g. if another user of the same node already uses that same port) the program will automatically try the next port (15433 in this case).

In the output of the successful execution of the above command, make note of the line that mentions the compute node name, e.g.:

   http://compute-096.cm.cluster:15432/?token=.....
or http://127.0.0.1:15432/?token=.....

The command prompt will not come back - the jupyter lab server is currently running in that session and can be shut down with Ctrl+C as the message instructs. Copy the line starting with http://127.0.0.1:15432/?token= in the clipboard (the whole line, including the whole token value)

2. Start the SSH tunnel

On your local computer, in a terminal start a SSH tunnel to that specific compute node like this (make sure to use the same compute node identifier you got at step 1, compute-096 in this example):

 ssh -N -L 15432:compute-096.cm.cluster:15432 user@jhpce-transfer01.jhsph.edu

(replace user with your login name on the cluster, of course). The tunnel can go through the JHPCE login or the transfer node, but I suggest using the transfer node since there could be a lot of data being transferred through the tunnel and the login node might complain, and in general the login node could be slower and more loaded with users.

There might be a warning/error message that can be ignored if it looks like this (as it refers to IPv6 and we do not care about):

bind [::1]:15432: Cannot assign requested address

The command prompt will not come back, that terminal should be kept like that and Ctrl+C can be used to break the SSH tunnel.

3. Connect to the JupyterLab session

On your local computer, point your browser to the 127.0.0.1 url that was shown in the qrsh session earlier when jupyter lab was started - paste in the address field the url that you copied before from the qrsh terminal:

http://127.0.0.1:15432/?token=..... (the full token value obtained at step 1 should be there!)

This should bring up the Jupyter Lab interface in your local browser, where you can now create, save, load and run Python and R notebooks running your code directly within the JHPCE environment.