-
I started a local dask cluster $ dask-scheduler
Scheduler started at 127.0.0.1:8786
$ dask-worker 127.0.0.1:8786
$ dask-worker 127.0.0.1:8786 and ran my project and I have noticed that if I change code in one of my project submodules after running for the first time dask would still run the old code. is this the intended behavior? Admittedly the example below is a little lengthy so I apologize but it will demonstrate the point. project tree
setup.pyfrom setuptools import setup, find_packages
setup(
name="myproject",
version="1.0",
packages=find_packages(where="src"),
package_dir={"": "src"},
entry_points={"console_scripts": ["myproject = myproject.main:main"]},
) main.pyfrom dask.distributed import Client
import dask
from myproject.dask_toy import waiting_method
def main():
client = Client("127.0.0.1:8786") # change this after launching the scheduler
# client.restart()
test = [dask.delayed(waiting_method)(1), dask.delayed(waiting_method)(2)]
dask.compute(*test)
if __name__ == "__main__":
main() dask_toy.pyfrom time import sleep
import logging
logging.basicConfig(format='%(asctime)s %(message)s',filename='example.log', filemode='w', level=logging.INFO)
def waiting_method(x):
logging.info(f"start sleep {x}")
sleep(5)
logging.info(f"stop sleep {x}")
return "SUCCESS" console:start python virtual env and activate it $ cd ~/test
$ pip install dask[distributed]
$ pip install -e . start dask scheduler and 2 workers within the python environment in 3 different consoles and update def waiting_method(x):
logging.info(f"start sleep {x}")
sleep(15) #<------------------------- change to 15
logging.info(f"stop sleep {x}")
return "SUCCESS" save and and rerun: result on example.log:
if we want to update the code, we need a hard restart in from dask.distributed import Client
import dask
from myproject.dask_toy import waiting_method
def main():
client = Client("127.0.0.1:8786") # change this after launching the scheduler
client.restart() #<-------------------- ADD THIS LINE
test = [dask.delayed(waiting_method)(1), dask.delayed(waiting_method)(2)]
dask.compute(*test) |
Beta Was this translation helpful? Give feedback.
Replies: 2 comments 1 reply
-
Not sure if this is helpful, but have you considered |
Beta Was this translation helpful? Give feedback.
-
See issue #8516 |
Beta Was this translation helpful? Give feedback.
See issue #8516