Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Multiprocessing won't inherit configuration from API setup #1606

Open
NeverBehave opened this issue Apr 20, 2023 · 2 comments
Open

Multiprocessing won't inherit configuration from API setup #1606

NeverBehave opened this issue Apr 20, 2023 · 2 comments
Labels
bug Something isn't working needs triage

Comments

@NeverBehave
Copy link

Describe the bug

When using concurrency=multiprocessing, setting from API setup in main process won't be picked up by children.

To Reproduce

I don't have a minimal PoC yet but I would like to write down what I have and edit/update if necessary

Let's say we have the following setup:

cov = coverage.Coverage(
            source=[os.path.join("/root", "src/python")],
            data_file="/root/coverage/.coverage",
            data_suffix=True,
            auto_data=True,
            concurrency=["thread", "multiprocessing",],
            config_file=CONFIG_FILE, # CONFIG_FILE is an empty file
)

and we spawn some process using forkserver

import multiprocessing

multiprocessing.set_start_method("forkserver", force=True)
multiprocessing.set_forkserver_preload(["project.preload"])

def test():
  print("hello!")

 t = multiprocessing.Process(
            target=hello,
            args=(),
            name="some hello",
)
t.start()

When running test, child process will write coverage file into src/python and parent will correctly put it into coverage/

Expected behavior

Children should use the same coveragepy configuration as parent. Or warn user when using multiprocessing mode, configuration set via API won't be picked up.

Additional context

I believe it is not initialized because 1) it is not setup:

cov = Coverage(data_suffix=True, auto_data=True)

and 2) in patch_multiprocessing, only rcfile is passing thru:
os.environ["COVERAGE_RCFILE"] = os.path.abspath(rcfile)

I am trying to use API to control the behavior of coverage instead of configuration file as much as possible since it would be more manageable with other python pipeline and CI tasks. If API is not intended to be use in this case it would be great to have some information listed, and if I miss it could anyone kindly point me to it? :) I do find some information here but it seems unrelated to my issue here: #1320

@NeverBehave NeverBehave added bug Something isn't working needs triage labels Apr 20, 2023
@nedbat
Copy link
Owner

nedbat commented Apr 20, 2023

The docs have this:

.. note::

    The subprocess only sees options in the configuration file.  Options set on
    the command line will not be used in the subprocesses.

I guess it could also mention that configuration via the API won't be used in the subprocess also.

I'm not sure what kind of warning you'd like to see happen? Surely it shouldn't issue a warning just because you used multiprocessing?

@NeverBehave
Copy link
Author

NeverBehave commented Apr 20, 2023

The docs have this:

.. note::



    The subprocess only sees options in the configuration file.  Options set on

    the command line will not be used in the subprocesses.

I guess it could also mention that configuration via the API won't be used in the subprocess also.

I'm not sure what kind of warning you'd like to see happen? Surely it shouldn't issue a warning just because you used multiprocessing?

Ah thanks for the update! I think it would be great to have the document with warning about this behavior in API option explanation, and sub process one as well.

For warning in message, I don't have better ideas for now. It is kinda tricky situation, since options in API will be override by configuration file.

I would suggest when user specify concurrency=multiprocessing, either in config file or api, and any option api is used, we could provide a suppressable message about this behavior. This is merely a suggestion tho, documentation update should be good enough.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working needs triage
Projects
None yet
Development

No branches or pull requests

2 participants