Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Multitasking (Multi-Processing) [minor] #28

Merged
merged 108 commits into from
Apr 28, 2023
Merged

Multitasking (Multi-Processing) [minor] #28

merged 108 commits into from
Apr 28, 2023

Commits on Apr 21, 2023

  1. move Queue instantiation up the call stack

    @ric-evans
    ric-evans committed Apr 21, 2023
    Configuration menu
    Copy the full SHA
    f4300e0 View commit details
    Browse the repository at this point in the history

  2. return number of processed tasks

    @ric-evans
    ric-evans committed Apr 21, 2023
    Configuration menu
    Copy the full SHA
    d23083d View commit details
    Browse the repository at this point in the history

Commits on Apr 25, 2023

  1. remove gcp support

    @ric-evans
    ric-evans committed Apr 25, 2023
    Configuration menu
    Copy the full SHA
    8f62805 View commit details
    Browse the repository at this point in the history

  2. bump GHA package versions; run mypy w/ 3.11

    @ric-evans
    ric-evans committed Apr 25, 2023
    Configuration menu
    Copy the full SHA
    88c5565 View commit details
    Browse the repository at this point in the history

  3. <bot> update requirements-all.txt

    github-actions committed Apr 25, 2023
    Configuration menu
    Copy the full SHA
    0e46068 View commit details
    Browse the repository at this point in the history

  4. <bot> update requirements-nats.txt

    github-actions committed Apr 25, 2023
    Configuration menu
    Copy the full SHA
    40d336f View commit details
    Browse the repository at this point in the history

  5. <bot> update requirements-pulsar.txt

    github-actions committed Apr 25, 2023
    Configuration menu
    Copy the full SHA
    b826087 View commit details
    Browse the repository at this point in the history

  6. <bot> update requirements-rabbitmq.txt

    github-actions committed Apr 25, 2023
    Configuration menu
    Copy the full SHA
    f0ef65b View commit details
    Browse the repository at this point in the history

  7. <bot> update requirements-test.txt

    github-actions committed Apr 25, 2023
    Configuration menu
    Copy the full SHA
    085682a View commit details
    Browse the repository at this point in the history

  8. <bot> update requirements.txt

    github-actions committed Apr 25, 2023
    Configuration menu
    Copy the full SHA
    e5fbef9 View commit details
    Browse the repository at this point in the history

  9. include message sending in process_msg()

    @ric-evans
    ric-evans committed Apr 25, 2023
    Configuration menu
    Copy the full SHA
    4c01718 View commit details
    Browse the repository at this point in the history

  10. fix pub typehint

    @ric-evans
    ric-evans committed Apr 25, 2023
    Configuration menu
    Copy the full SHA
    2fa4549 View commit details
    Browse the repository at this point in the history

  11. index on multi-proc: 2fa4549 fix pub typehint

    @ric-evans
    ric-evans committed Apr 25, 2023
    Configuration menu
    Copy the full SHA
    0ebf255 View commit details
    Browse the repository at this point in the history

  12. implement concurrent tasks using asyncio.create_task

    @ric-evans
    ric-evans committed Apr 25, 2023
    Configuration menu
    Copy the full SHA
    b74d197 View commit details
    Browse the repository at this point in the history

  13. implement concurrent tasks using asyncio.wait

    @ric-evans
    ric-evans committed Apr 25, 2023
    Configuration menu
    Copy the full SHA
    e933a8e View commit details
    Browse the repository at this point in the history

  14. ack

    @ric-evans
    ric-evans committed Apr 25, 2023
    Configuration menu
    Copy the full SHA
    4f1db53 View commit details
    Browse the repository at this point in the history

  15. nack

    @ric-evans
    ric-evans committed Apr 25, 2023
    Configuration menu
    Copy the full SHA
    e5d921c View commit details
    Browse the repository at this point in the history

  16. fix "wait for remaining tasks" logic

    @ric-evans
    ric-evans committed Apr 25, 2023
    Configuration menu
    Copy the full SHA
    36aa44e View commit details
    Browse the repository at this point in the history

  17. add _ack_nack_tasks()

    @ric-evans
    ric-evans committed Apr 25, 2023
    Configuration menu
    Copy the full SHA
    e25fc5a View commit details
    Browse the repository at this point in the history

  18. give _ack_nack_finished_tasks() more responsibility

    @ric-evans
    ric-evans committed Apr 25, 2023
    Configuration menu
    Copy the full SHA
    5e43478 View commit details
    Browse the repository at this point in the history

  19. re-implement the timeout_wait_for_first_message logic

    @ric-evans
    ric-evans committed Apr 25, 2023
    Configuration menu
    Copy the full SHA
    4386acb View commit details
    Browse the repository at this point in the history

Commits on Apr 26, 2023

  1. re-implement error handling

    @ric-evans
    ric-evans committed Apr 26, 2023
    Configuration menu
    Copy the full SHA
    1d390b1 View commit details
    Browse the repository at this point in the history

  2. <bot> update requirements-all.txt

    github-actions committed Apr 26, 2023
    Configuration menu
    Copy the full SHA
    bd669be View commit details
    Browse the repository at this point in the history

  3. <bot> update requirements-nats.txt

    github-actions committed Apr 26, 2023
    Configuration menu
    Copy the full SHA
    b8d24c4 View commit details
    Browse the repository at this point in the history

  4. <bot> update requirements-pulsar.txt

    github-actions committed Apr 26, 2023
    Configuration menu
    Copy the full SHA
    158c740 View commit details
    Browse the repository at this point in the history

  5. <bot> update requirements-rabbitmq.txt

    github-actions committed Apr 26, 2023
    Configuration menu
    Copy the full SHA
    6092656 View commit details
    Browse the repository at this point in the history

  6. <bot> update requirements-test.txt

    github-actions committed Apr 26, 2023
    Configuration menu
    Copy the full SHA
    aad887b View commit details
    Browse the repository at this point in the history

  7. <bot> update requirements.txt

    github-actions committed Apr 26, 2023
    Configuration menu
    Copy the full SHA
    512e1aa View commit details
    Browse the repository at this point in the history

  8. log failed tasks

    @ric-evans
    ric-evans committed Apr 26, 2023
    Configuration menu
    Copy the full SHA
    86dda08 View commit details
    Browse the repository at this point in the history

  9. speed up CI

    @ric-evans
    ric-evans committed Apr 26, 2023
    Configuration menu
    Copy the full SHA
    84ac1de View commit details
    Browse the repository at this point in the history

  10. log failed tasks pt-2

    @ric-evans
    ric-evans committed Apr 26, 2023
    Configuration menu
    Copy the full SHA
    c7ae4b2 View commit details
    Browse the repository at this point in the history

  11. log failed tasks pt-3 (update tests)

    @ric-evans
    ric-evans committed Apr 26, 2023
    Configuration menu
    Copy the full SHA
    b8a8e1e View commit details
    Browse the repository at this point in the history

  12. log failed tasks pt-4 (update tests)

    @ric-evans
    ric-evans committed Apr 26, 2023
    Configuration menu
    Copy the full SHA
    8419f33 View commit details
    Browse the repository at this point in the history

  13. command meta args {{OUTFILE}} & {{INFILE}}; make in/out files unique

    @ric-evans
    ric-evans committed Apr 26, 2023
    Configuration menu
    Copy the full SHA
    0921153 View commit details
    Browse the repository at this point in the history

  14. update examples

    @ric-evans
    ric-evans committed Apr 26, 2023
    Configuration menu
    Copy the full SHA
    1a01c00 View commit details
    Browse the repository at this point in the history

  15. fix {{OUTFILE}} & {{INFILE}} injection

    @ric-evans
    ric-evans committed Apr 26, 2023
    Configuration menu
    Copy the full SHA
    6f765ef View commit details
    Browse the repository at this point in the history

  16. update CL args

    @ric-evans
    ric-evans committed Apr 26, 2023
    Configuration menu
    Copy the full SHA
    6b3dffa View commit details
    Browse the repository at this point in the history

  17. string syntax fix

    @ric-evans
    ric-evans committed Apr 26, 2023
    Configuration menu
    Copy the full SHA
    78b8297 View commit details
    Browse the repository at this point in the history

  18. remove ftype args defaults

    @ric-evans
    ric-evans committed Apr 26, 2023
    Configuration menu
    Copy the full SHA
    b81d229 View commit details
    Browse the repository at this point in the history

  19. update CL arg help text

    @ric-evans
    ric-evans committed Apr 26, 2023
    Configuration menu
    Copy the full SHA
    f55f917 View commit details
    Browse the repository at this point in the history

  20. add test_500__multitasking()

    @ric-evans
    ric-evans committed Apr 26, 2023
    Configuration menu
    Copy the full SHA
    b8eb62e View commit details
    Browse the repository at this point in the history

  21. fix patching

    @ric-evans
    ric-evans committed Apr 26, 2023
    Configuration menu
    Copy the full SHA
    f986d94 View commit details
    Browse the repository at this point in the history

  22. logging

    @ric-evans
    ric-evans committed Apr 26, 2023
    Configuration menu
    Copy the full SHA
    c657865 View commit details
    Browse the repository at this point in the history

  23. add multitasking/--multitasking

    @ric-evans
    ric-evans committed Apr 26, 2023
    Configuration menu
    Copy the full SHA
    5976c2c View commit details
    Browse the repository at this point in the history

  24. tests: use var

    @ric-evans
    ric-evans committed Apr 26, 2023
    Configuration menu
    Copy the full SHA
    2cb03b9 View commit details
    Browse the repository at this point in the history

  25. fix type

    @ric-evans
    ric-evans committed Apr 26, 2023
    Configuration menu
    Copy the full SHA
    255d94a View commit details
    Browse the repository at this point in the history

  26. logging

    @ric-evans
    ric-evans committed Apr 26, 2023
    Configuration menu
    Copy the full SHA
    e29850b View commit details
    Browse the repository at this point in the history

  27. more logging

    @ric-evans
    ric-evans committed Apr 26, 2023
    Configuration menu
    Copy the full SHA
    1d8a684 View commit details
    Browse the repository at this point in the history

  28. differentiate between "task" and "message"

    @ric-evans
    ric-evans committed Apr 26, 2023
    Configuration menu
    Copy the full SHA
    300555c View commit details
    Browse the repository at this point in the history

  29. try using asyncio.create_subprocess_shell()

    @ric-evans
    ric-evans committed Apr 26, 2023
    Configuration menu
    Copy the full SHA
    88a5cbc View commit details
    Browse the repository at this point in the history

Commits on Apr 27, 2023

  1. fix args

    @ric-evans
    ric-evans committed Apr 27, 2023
    Configuration menu
    Copy the full SHA
    cc6ead2 View commit details
    Browse the repository at this point in the history

  2. use if proc.returncode != 0

    @ric-evans
    ric-evans committed Apr 27, 2023
    Configuration menu
    Copy the full SHA
    6fe3c08 View commit details
    Browse the repository at this point in the history

  3. (debug)

    @ric-evans
    ric-evans committed Apr 27, 2023
    Configuration menu
    Copy the full SHA
    129756b View commit details
    Browse the repository at this point in the history

  4. (debug 2)

    @ric-evans
    ric-evans committed Apr 27, 2023
    Configuration menu
    Copy the full SHA
    4f46383 View commit details
    Browse the repository at this point in the history

  5. add await proc.wait()

    @ric-evans
    ric-evans committed Apr 27, 2023
    Configuration menu
    Copy the full SHA
    3a63372 View commit details
    Browse the repository at this point in the history

  6. re-implement asyncio.wait_for()

    @ric-evans
    ric-evans committed Apr 27, 2023
    Configuration menu
    Copy the full SHA
    aa6e833 View commit details
    Browse the repository at this point in the history

  7. dump subprocess streams to stdout

    @ric-evans
    ric-evans committed Apr 27, 2023
    Configuration menu
    Copy the full SHA
    b5429b1 View commit details
    Browse the repository at this point in the history

  8. re-implement asyncio.wait_for() pt-2

    @ric-evans
    ric-evans committed Apr 27, 2023
    Configuration menu
    Copy the full SHA
    975c623 View commit details
    Browse the repository at this point in the history

  9. flake8

    @ric-evans
    ric-evans committed Apr 27, 2023
    Configuration menu
    Copy the full SHA
    8a6fb73 View commit details
    Browse the repository at this point in the history

  10. dump subprocess streams to stdout pt-2

    @ric-evans
    ric-evans committed Apr 27, 2023
    Configuration menu
    Copy the full SHA
    0c88f3f View commit details
    Browse the repository at this point in the history

  11. stream stdout/stderr

    @ric-evans
    ric-evans committed Apr 27, 2023
    Configuration menu
    Copy the full SHA
    2de5d1e View commit details
    Browse the repository at this point in the history

  12. use tasks

    @ric-evans
    ric-evans committed Apr 27, 2023
    Configuration menu
    Copy the full SHA
    b293e64 View commit details
    Browse the repository at this point in the history

  13. record last line of stderr

    @ric-evans
    ric-evans committed Apr 27, 2023
    Configuration menu
    Copy the full SHA
    9c479af View commit details
    Browse the repository at this point in the history

  14. cleanup

    @ric-evans
    ric-evans committed Apr 27, 2023
    Configuration menu
    Copy the full SHA
    e834371 View commit details
    Browse the repository at this point in the history

  15. (debug)

    @ric-evans
    ric-evans committed Apr 27, 2023
    Configuration menu
    Copy the full SHA
    b1b4a92 View commit details
    Browse the repository at this point in the history

  16. record last line of stderr pt-2

    @ric-evans
    ric-evans committed Apr 27, 2023
    Configuration menu
    Copy the full SHA
    3a25cd9 View commit details
    Browse the repository at this point in the history

  17. record last line of stderr pt-3

    @ric-evans
    ric-evans committed Apr 27, 2023
    Configuration menu
    Copy the full SHA
    907f762 View commit details
    Browse the repository at this point in the history

  18. add SubprocessError class

    @ric-evans
    ric-evans committed Apr 27, 2023
    Configuration menu
    Copy the full SHA
    3d5c0c7 View commit details
    Browse the repository at this point in the history

  19. add SubprocessError class pt-2

    @ric-evans
    ric-evans committed Apr 27, 2023
    Configuration menu
    Copy the full SHA
    8aa7b0b View commit details
    Browse the repository at this point in the history

  20. call it TaskSubprocessError

    @ric-evans
    ric-evans committed Apr 27, 2023
    Configuration menu
    Copy the full SHA
    2d6a0cd View commit details
    Browse the repository at this point in the history

  21. add timeout test

    @ric-evans
    ric-evans committed Apr 27, 2023
    Configuration menu
    Copy the full SHA
    3ef538e View commit details
    Browse the repository at this point in the history

  22. rework timeout

    @ric-evans
    ric-evans committed Apr 27, 2023
    Configuration menu
    Copy the full SHA
    95740f5 View commit details
    Browse the repository at this point in the history

  23. change error to FileNotFoundError

    @ric-evans
    ric-evans committed Apr 27, 2023
    Configuration menu
    Copy the full SHA
    d05a72f View commit details
    Browse the repository at this point in the history

  24. logging

    @ric-evans
    ric-evans committed Apr 27, 2023
    Configuration menu
    Copy the full SHA
    f07d292 View commit details
    Browse the repository at this point in the history

  25. logging

    @ric-evans
    ric-evans committed Apr 27, 2023
    Configuration menu
    Copy the full SHA
    40bbb64 View commit details
    Browse the repository at this point in the history

  26. logging

    @ric-evans
    ric-evans committed Apr 27, 2023
    Configuration menu
    Copy the full SHA
    2929afc View commit details
    Browse the repository at this point in the history

  27. be more generous with timeout error test values

    @ric-evans
    ric-evans committed Apr 27, 2023
    Configuration menu
    Copy the full SHA
    2437ba1 View commit details
    Browse the repository at this point in the history

  28. use asyncio.wait()'s timeout

    @ric-evans
    ric-evans committed Apr 27, 2023
    Configuration menu
    Copy the full SHA
    7b31448 View commit details
    Browse the repository at this point in the history

  29. fix

    @ric-evans
    ric-evans committed Apr 27, 2023
    Configuration menu
    Copy the full SHA
    75aa49a View commit details
    Browse the repository at this point in the history

  30. subproc stream to files in debug directory (optional)

    @ric-evans
    ric-evans committed Apr 27, 2023
    Configuration menu
    Copy the full SHA
    e095e58 View commit details
    Browse the repository at this point in the history

  31. update tests

    @ric-evans
    ric-evans committed Apr 27, 2023
    Configuration menu
    Copy the full SHA
    96325b6 View commit details
    Browse the repository at this point in the history

  32. only create pipe if using it

    @ric-evans
    ric-evans committed Apr 27, 2023
    Configuration menu
    Copy the full SHA
    d95ec21 View commit details
    Browse the repository at this point in the history

  33. mypy

    @ric-evans
    ric-evans committed Apr 27, 2023
    Configuration menu
    Copy the full SHA
    ed7fc81 View commit details
    Browse the repository at this point in the history

  34. flake8

    @ric-evans
    ric-evans committed Apr 27, 2023
    Configuration menu
    Copy the full SHA
    12dab63 View commit details
    Browse the repository at this point in the history

  35. add test_510__multitasking_exceptions()

    @ric-evans
    ric-evans committed Apr 27, 2023
    Configuration menu
    Copy the full SHA
    4e4fa69 View commit details
    Browse the repository at this point in the history

Commits on Apr 28, 2023

  1. typo

    @ric-evans
    ric-evans committed Apr 28, 2023
    Configuration menu
    Copy the full SHA
    612e5d5 View commit details
    Browse the repository at this point in the history

  2. updates tests for error-cases & debug dir

    @ric-evans
    ric-evans committed Apr 28, 2023
    Configuration menu
    Copy the full SHA
    12ed416 View commit details
    Browse the repository at this point in the history

  3. updates tests for error-cases & debug dir pt-2

    @ric-evans
    ric-evans committed Apr 28, 2023
    Configuration menu
    Copy the full SHA
    5b6144b View commit details
    Browse the repository at this point in the history

  4. updates tests for error-cases & debug dir pt-3

    @ric-evans
    ric-evans committed Apr 28, 2023
    Configuration menu
    Copy the full SHA
    7d1bbde View commit details
    Browse the repository at this point in the history

  5. fix file-writing (wb)

    @ric-evans
    ric-evans committed Apr 28, 2023
    Configuration menu
    Copy the full SHA
    81ecbe6 View commit details
    Browse the repository at this point in the history

  6. updates tests for error-cases & debug dir pt-4

    @ric-evans
    ric-evans committed Apr 28, 2023
    Configuration menu
    Copy the full SHA
    79b21b1 View commit details
    Browse the repository at this point in the history

  7. if using debug dir, include outfile if it was written, error or not

    @ric-evans
    ric-evans committed Apr 28, 2023
    Configuration menu
    Copy the full SHA
    5ed3aab View commit details
    Browse the repository at this point in the history

  8. fix error case in test 401

    @ric-evans
    ric-evans committed Apr 28, 2023
    Configuration menu
    Copy the full SHA
    7680844 View commit details
    Browse the repository at this point in the history

  9. flake8

    @ric-evans
    ric-evans committed Apr 28, 2023
    Configuration menu
    Copy the full SHA
    9bad290 View commit details
    Browse the repository at this point in the history

  10. use open() explicitly instead of PIPE

    @ric-evans
    ric-evans committed Apr 28, 2023
    Configuration menu
    Copy the full SHA
    9c728c5 View commit details
    Browse the repository at this point in the history

  11. fix stdout/stderr filenames

    @ric-evans
    ric-evans committed Apr 28, 2023
    Configuration menu
    Copy the full SHA
    b1a424c View commit details
    Browse the repository at this point in the history

  12. fix stdout/stderr filepaths

    @ric-evans
    ric-evans committed Apr 28, 2023
    Configuration menu
    Copy the full SHA
    c6b2efd View commit details
    Browse the repository at this point in the history

  13. use asyncio.wait_for() since were using one task

    @ric-evans
    ric-evans committed Apr 28, 2023
    Configuration menu
    Copy the full SHA
    f388dcb View commit details
    Browse the repository at this point in the history

  14. use with()'s for stdout/stderr file writing

    @ric-evans
    ric-evans committed Apr 28, 2023
    Configuration menu
    Copy the full SHA
    3e81024 View commit details
    Browse the repository at this point in the history

  15. allow user to pass str instead of FileType, ex: ".txt"

    @ric-evans
    ric-evans committed Apr 28, 2023
    Configuration menu
    Copy the full SHA
    b43bde8 View commit details
    Browse the repository at this point in the history

  16. remove _stream_to_file()

    @ric-evans
    ric-evans committed Apr 28, 2023
    Configuration menu
    Copy the full SHA
    954190e View commit details
    Browse the repository at this point in the history

  17. use the message id along with timestamp for task-id

    @ric-evans
    ric-evans committed Apr 28, 2023
    Configuration menu
    Copy the full SHA
    c71cf1b View commit details
    Browse the repository at this point in the history

  18. Revert "use the message id along with timestamp for task-id" 

    This reverts commit c71cf1b.
    @ric-evans
    ric-evans committed Apr 28, 2023
    Configuration menu
    Copy the full SHA
    a5615f8 View commit details
    Browse the repository at this point in the history

  19. take 2: use uuid for task-id

    @ric-evans
    ric-evans committed Apr 28, 2023
    Configuration menu
    Copy the full SHA
    7403f57 View commit details
    Browse the repository at this point in the history

  20. Merge remote-tracking branch 'origin/main' into multi-proc

    @ric-evans
    ric-evans committed Apr 28, 2023
    Configuration menu
    Copy the full SHA
    c358500 View commit details
    Browse the repository at this point in the history

  21. post-merge cleanup

    @ric-evans
    ric-evans committed Apr 28, 2023
    Configuration menu
    Copy the full SHA
    d5f2d2a View commit details
    Browse the repository at this point in the history

  22. mypy

    @ric-evans
    ric-evans committed Apr 28, 2023
    Configuration menu
    Copy the full SHA
    5a886f9 View commit details
    Browse the repository at this point in the history

  23. remove TODO

    @ric-evans
    ric-evans committed Apr 28, 2023
    Configuration menu
    Copy the full SHA
    fe06bc4 View commit details
    Browse the repository at this point in the history