[Dask-MPI] How to connect to dashboard?

[mtsokol]

Hi!

I've recently switched from dask-jobqueue to dask-mpi in my project as it fits better my domain (explained in #7467, thanks!) and now I have some troubles connecting to the dask dashboard.

With dask-jobqueue I could easily access it with port forwarding but now project's script is executed within sbatch:

#!/bin/bash -l

#SBATCH -J ...
#SBATCH -p plgrid
#SBATCH -A ...
#SBATCH --ntasks-per-node=24
#SBATCH --mem=48G
#SBATCH -t 01:00:00
#SBATCH --nodes=10

module load plgrid/libs/python-mpi4py/3.0.1-python-3.6
mpirun -np 240 python3 -m implementation.experiments

So the dashboard is running on one of the nodes for which I'm of course not allowed to access via ssh.

Is it possible to access dask dashboard in dask-mpi on such HPC cluster?

Thank you for any help!

[andersy005]

@mtsokol, Glad to hear that #7467 accomplishes what you were looking for. To access the dashboard, you will need to do the following

1. In your implementation.experiments, add the following:

from dask_mpi import initialize
initialize()

from dask.distributed import Client
from distributed.scheduler import logger
import socket

client = Client()  # Connect this local process to remote workers

host = client.run_on_scheduler(socket.gethostname)
port = client.scheduler_info()['services']['dashboard']
login_node_address = "supercomputer.university.edu" # Change this to the address/domain of your login node

logger.info(f"ssh -N -L {port}:{host}:{port} {login_node_address}")

# The rest of your code 

2. Inspect the output file of your batch job. You should see something along these lines

ssh -N -L PORT_NUMBER:node03:PORT_NUMBER supercomputer.university.edu

Run that command (from your local machine) to create the SSH tunnel, and then point your browser to localhost:PORT_NUMBER.

I hope this helps.

[mtsokol]

And again thank you for help and guidance!

Now it works as expected.

I've looked at https://mpi.dask.org/en/latest/ docs site and I haven't found this part - is it dask-mpi specific?

I can submit a PR with this short instruction if you consider it a worthwile contribution.

[andersy005]

I can submit a PR with this short instruction if you consider it a worthwile contribution.

Yes, please! A PR to dask-mpi repo on how to access the dashboard for a cluster created via dask-mpi would be very useful.