🐛[bug] Jupyter lab / Tensorboard stuck at Waiting for ... #8198
Labels
Comments
|
I was able to reproduce this problem on my local machine which returned the same problem.
|
|
hello, the most common cause for this symptom is a problem with master container connecting to the task (notebook/tensorboard) container to proxy the incoming request. this is often caused by firewalls or other networking setup issues.
it's peculiar you see the issue on three different OS'es, and you both see the issue. are you working with one shared deployment? or have you each deployed determined separately on all of these OS'es? can you tell me more on how did you deploy determined / which guide did you follow? (is it if you happen to share a corporate firewall / proxy setup, I'd recommend trying to temporarily disabling it to see if it helps. |
|
For me I tried it on my private machine: I have no proxy activated. I started the determined cluster using and also started a agent by running the docker-container When I start "Jupyter lab" using the UI - I see the container getting started and all pip-libs are being downloaded but it hangs itself when "Running" is printed on the screen as shown by @jokokojote I'm root on my machine too. |
|
Hello, I tested it on different machines with different set ups to isolate respectively understand the problem. At first, I indeed tried it on an ubuntu machine inside a cooperate network and run determined using the master, agent and db docker containers diretly (and passed proxy environment variables to the containers). The core functionalities like experiment initlization, (GPU) training, tuning, etc. worked like charm - jupyter and tensorboard did not, yielding the same logs I added in the issue description. Indeed firewall or proxy settings could be the issue here, even though I do not understand why the agent itself worked and no errors were shown in the logs fo tensorboard and jupyter. Since jupyter and tensorboard it did not work on this machine I tried it on my cooperate laptop (Mac) but outside of the cooperate network and set up determined just with Then I asked @KevinHubert-Dev to try it at home with a private machine and private network and he got the same results like he described. |
|
It is highly unusual to see this happen on so many different setups. I'll need your help debugging it. When you start a notebook, there'd be a "registering service" log line in the master logs, e.g. you can |
|
I did what you suggested on my corporate laptop in my private network: Start up with: Master logs: Curl inside master gets timeout: Jupyter container logs: Docker containers running: Agent logs: |
do you have any insight why this does not work? |
|
Verbose mode did not yield anymore information using curl: I am not a docker expert, so maybe this is not relevant, but I was wondering why in your example Containers running after trying to run jupyter: Inspecting the docker networks showed that only db and master container are in the determined_default network, I don't know if this is intended. Agent is in host network mode: Jupyter container is in bridge mode: |
|
I was able to repro the issue with as a temporary workaround, I can suggest installing master and agent using linux packages or homebrew which should address that problem by not having master wrapped in docker. |
|
@jokokojote did you do your last test on macos? or on ubuntu? |
|
Last test was on macOS. Ob ubuntu I started it with: After running |
|
so the ubuntu setup has the proxy configuration. this often causes problems. you'd need to setup otherwise, master and agent has this config, but the spawned containers don't. |
I met the problem almost the same. if master is running on an individual server would it possible to access the registered address that 172.18.0.1? That address is an docker accessable, not LAN wide. |
Sorry, nothing comes to mind. If complex bridge networking is causing issues, you can try switching to host mode networking. Setting up local k8s clusters is also much easier nowadays, so that's another path to consider if you don't want to maintain a raw docker setup. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Describe the bug
I am not sure if this is a bug or I missed some basic config step, but I checked the docs multiple times and did not find any information about this:
Jupyter lab and tensorboard are stuck at "Waiting for ..." after the docker was run successfully w/o any errors shown in the logs.
Tried with 0.26.1, 0.26.0, 0.25.1 and 0.21.2 on MacOS, Ubuntu and Windows.
TensorBoard 0.26.1 logs:
Jupyter 0.26.1 logs:
Jupyter 0.21.2 logs:
Reproduction Steps
det deploy local cluster-up --no-gpu2.a. Open the UI: Tasks -> launch Jupyter
OR
2.b.1 Run an experiment e.g. gan_mnist_pytorch with
det experiment create const.yaml .2.b.2 Open the UI, open the experiment, open tensorboard
Expected Behavior
UI for Jupiter lab / tensorboard should open after some (short) waiting time (or a meaningful error message should show up at least).
Screenshot
Environment
Additional Context
No response
The text was updated successfully, but these errors were encountered: