Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Multitasking (Multi-Processing) [minor] #28

Merged
merged 108 commits into from
Apr 28, 2023
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
108 commits
Select commit Hold shift + click to select a range
f4300e0
move `Queue` instantiation up the call stack
ric-evans Apr 21, 2023
d23083d
return number of processed tasks
ric-evans Apr 21, 2023
8f62805
remove gcp support
ric-evans Apr 25, 2023
88c5565
bump GHA package versions; run mypy w/ 3.11
ric-evans Apr 25, 2023
0e46068
<bot> update requirements-all.txt
invalid-email-address Apr 25, 2023
40d336f
<bot> update requirements-nats.txt
invalid-email-address Apr 25, 2023
b826087
<bot> update requirements-pulsar.txt
invalid-email-address Apr 25, 2023
f0ef65b
<bot> update requirements-rabbitmq.txt
invalid-email-address Apr 25, 2023
085682a
<bot> update requirements-test.txt
invalid-email-address Apr 25, 2023
e5fbef9
<bot> update requirements.txt
invalid-email-address Apr 25, 2023
4c01718
include message sending in `process_msg()`
ric-evans Apr 25, 2023
2fa4549
fix `pub` typehint
ric-evans Apr 25, 2023
0ebf255
index on multi-proc: 2fa4549 fix `pub` typehint
ric-evans Apr 25, 2023
b74d197
implement concurrent tasks using `asyncio.create_task`
ric-evans Apr 25, 2023
e933a8e
implement concurrent tasks using `asyncio.wait`
ric-evans Apr 25, 2023
4f1db53
ack
ric-evans Apr 25, 2023
e5d921c
nack
ric-evans Apr 25, 2023
36aa44e
fix "wait for remaining tasks" logic
ric-evans Apr 25, 2023
e25fc5a
add `_ack_nack_tasks()`
ric-evans Apr 25, 2023
5e43478
give `_ack_nack_finished_tasks()` more responsibility
ric-evans Apr 25, 2023
4386acb
re-implement the `timeout_wait_for_first_message` logic
ric-evans Apr 25, 2023
1d390b1
re-implement error handling
ric-evans Apr 26, 2023
bd669be
<bot> update requirements-all.txt
invalid-email-address Apr 26, 2023
b8d24c4
<bot> update requirements-nats.txt
invalid-email-address Apr 26, 2023
158c740
<bot> update requirements-pulsar.txt
invalid-email-address Apr 26, 2023
6092656
<bot> update requirements-rabbitmq.txt
invalid-email-address Apr 26, 2023
aad887b
<bot> update requirements-test.txt
invalid-email-address Apr 26, 2023
512e1aa
<bot> update requirements.txt
invalid-email-address Apr 26, 2023
86dda08
log failed tasks
ric-evans Apr 26, 2023
84ac1de
speed up CI
ric-evans Apr 26, 2023
c7ae4b2
log failed tasks pt-2
ric-evans Apr 26, 2023
b8a8e1e
log failed tasks pt-3 (update tests)
ric-evans Apr 26, 2023
8419f33
log failed tasks pt-4 (update tests)
ric-evans Apr 26, 2023
0921153
command meta args `{{OUTFILE}}` & `{{INFILE}}`; make in/out files unique
ric-evans Apr 26, 2023
1a01c00
update examples
ric-evans Apr 26, 2023
6f765ef
fix `{{OUTFILE}}` & `{{INFILE}}` injection
ric-evans Apr 26, 2023
6b3dffa
update CL args
ric-evans Apr 26, 2023
78b8297
string syntax fix
ric-evans Apr 26, 2023
b81d229
remove ftype args defaults
ric-evans Apr 26, 2023
f55f917
update CL arg help text
ric-evans Apr 26, 2023
b8eb62e
add `test_500__multitasking()`
ric-evans Apr 26, 2023
f986d94
fix patching
ric-evans Apr 26, 2023
c657865
logging
ric-evans Apr 26, 2023
5976c2c
add `multitasking`/`--multitasking`
ric-evans Apr 26, 2023
2cb03b9
tests: use var
ric-evans Apr 26, 2023
255d94a
fix type
ric-evans Apr 26, 2023
e29850b
logging
ric-evans Apr 26, 2023
1d8a684
more logging
ric-evans Apr 26, 2023
300555c
differentiate between "task" and "message"
ric-evans Apr 26, 2023
88a5cbc
try using `asyncio.create_subprocess_shell()`
ric-evans Apr 26, 2023
cc6ead2
fix args
ric-evans Apr 27, 2023
6fe3c08
use `if proc.returncode != 0`
ric-evans Apr 27, 2023
129756b
(debug)
ric-evans Apr 27, 2023
4f46383
(debug 2)
ric-evans Apr 27, 2023
3a63372
add `await proc.wait()`
ric-evans Apr 27, 2023
aa6e833
re-implement `asyncio.wait_for()`
ric-evans Apr 27, 2023
b5429b1
dump subprocess streams to stdout
ric-evans Apr 27, 2023
975c623
re-implement `asyncio.wait_for()` pt-2
ric-evans Apr 27, 2023
8a6fb73
flake8
ric-evans Apr 27, 2023
0c88f3f
dump subprocess streams to stdout pt-2
ric-evans Apr 27, 2023
2de5d1e
stream stdout/stderr
ric-evans Apr 27, 2023
b293e64
use tasks
ric-evans Apr 27, 2023
9c479af
record last line of stderr
ric-evans Apr 27, 2023
e834371
cleanup
ric-evans Apr 27, 2023
b1b4a92
(debug)
ric-evans Apr 27, 2023
3a25cd9
record last line of stderr pt-2
ric-evans Apr 27, 2023
907f762
record last line of stderr pt-3
ric-evans Apr 27, 2023
3d5c0c7
add `SubprocessError` class
ric-evans Apr 27, 2023
8aa7b0b
add `SubprocessError` class pt-2
ric-evans Apr 27, 2023
2d6a0cd
call it `TaskSubprocessError`
ric-evans Apr 27, 2023
3ef538e
add timeout test
ric-evans Apr 27, 2023
95740f5
rework timeout
ric-evans Apr 27, 2023
d05a72f
change error to `FileNotFoundError`
ric-evans Apr 27, 2023
f07d292
logging
ric-evans Apr 27, 2023
40bbb64
logging
ric-evans Apr 27, 2023
2929afc
logging
ric-evans Apr 27, 2023
2437ba1
be more generous with timeout error test values
ric-evans Apr 27, 2023
7b31448
use `asyncio.wait()`'s timeout
ric-evans Apr 27, 2023
75aa49a
fix
ric-evans Apr 27, 2023
e095e58
subproc stream to files in debug directory (optional)
ric-evans Apr 27, 2023
96325b6
update tests
ric-evans Apr 27, 2023
d95ec21
only create pipe if using it
ric-evans Apr 27, 2023
ed7fc81
mypy
ric-evans Apr 27, 2023
12dab63
flake8
ric-evans Apr 27, 2023
4e4fa69
add `test_510__multitasking_exceptions()`
ric-evans Apr 27, 2023
612e5d5
typo
ric-evans Apr 28, 2023
12ed416
updates tests for error-cases & debug dir
ric-evans Apr 28, 2023
5b6144b
updates tests for error-cases & debug dir pt-2
ric-evans Apr 28, 2023
7d1bbde
updates tests for error-cases & debug dir pt-3
ric-evans Apr 28, 2023
81ecbe6
fix file-writing (wb)
ric-evans Apr 28, 2023
79b21b1
updates tests for error-cases & debug dir pt-4
ric-evans Apr 28, 2023
5ed3aab
if using debug dir, include outfile if it was written, error or not
ric-evans Apr 28, 2023
7680844
fix error case in test 401
ric-evans Apr 28, 2023
9bad290
flake8
ric-evans Apr 28, 2023
9c728c5
use `open()` explicitly instead of `PIPE`
ric-evans Apr 28, 2023
b1a424c
fix stdout/stderr filenames
ric-evans Apr 28, 2023
c6b2efd
fix stdout/stderr filepaths
ric-evans Apr 28, 2023
f388dcb
use `asyncio.wait_for()` since were using one task
ric-evans Apr 28, 2023
3e81024
use `with()`'s for stdout/stderr file writing
ric-evans Apr 28, 2023
b43bde8
allow user to pass `str` instead of `FileType`, ex: ".txt"
ric-evans Apr 28, 2023
954190e
remove `_stream_to_file()`
ric-evans Apr 28, 2023
c71cf1b
use the message id along with timestamp for task-id
ric-evans Apr 28, 2023
a5615f8
Revert "use the message id along with timestamp for task-id"
ric-evans Apr 28, 2023
7403f57
take 2: use uuid for task-id
ric-evans Apr 28, 2023
c358500
Merge remote-tracking branch 'origin/main' into multi-proc
ric-evans Apr 28, 2023
d5f2d2a
post-merge cleanup
ric-evans Apr 28, 2023
5a886f9
mypy
ric-evans Apr 28, 2023
fe06bc4
remove TODO
ric-evans Apr 28, 2023
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Jump to
Jump to file
Failed to load files.
Diff view
Diff view
7 changes: 5 additions & 2 deletions ewms_pilot/__init__.py
Original file line number Diff line number Diff line change
@@ -1,9 +1,12 @@
"""Init."""


from .pilot import consume_and_reply, main
from .pilot import FileType, consume_and_reply, main

__all__ = ["consume_and_reply"]
__all__ = [
"consume_and_reply",
"FileType",
]

# version is a human-readable version number.
__version__ = "0.8.0"
Expand Down
1 change: 1 addition & 0 deletions ewms_pilot/config.py
Original file line number Diff line number Diff line change
Expand Up @@ -33,6 +33,7 @@ class EnvConfig:
# meta
EWMS_PILOT_TASK_TIMEOUT: Optional[int] = None
EWMS_PILOT_QUARANTINE_TIME: int = 0 # seconds
EWMS_PILOT_CONCURRENT_TASKS: int = 1

def __post_init__(self) -> None:
if timeout := os.getenv("EWMS_PILOT_SUBPROC_TIMEOUT"):
Expand Down