Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Multithreaded Python API and Pylot Example #18

Merged
merged 40 commits into from
Apr 29, 2022
Merged

Multithreaded Python API and Pylot Example #18

merged 40 commits into from
Apr 29, 2022

Conversation

haixuanTao
Copy link
Collaborator

@haixuanTao haixuanTao commented Mar 28, 2022

This PR adds:

  • a couple of examples, with long running CPU/GPU task such as an object detection function.
  • transform the previous single-threaded server into a multithread server.
  • Add a bench to see the time taken by the Python binding as we have seen some huge time sink within it.
  • Refactor some of the previous code,
  • Make a Docker to test Python example

To run the docker container

# Within dora root folder
nvidia-docker build --tag dora .
nvidia-docker run -itd --name dora -p 20022:22 dora /bin/bash
nvidia-docker exec -itd dora /home/erdos/workspace/pylot/scripts/run_simulator.sh
nvidia-docker cp ~/.ssh/id_rsa.pub dora:/home/erdos/.ssh/authorized_keys
nvidia-docker exec -i -t dora sudo chown erdos /home/erdos/.ssh/authorized_keys
nvidia-docker exec -i -t dora sudo service ssh start
ssh -p 20022 -X erdos@localhost
  • Within the docker container, to run the object detection simulation that use ~6Gib of GPU memory:
cd /home/erdos/workspace/dora-rs
export PYTHONPATH=$PYTHONPATH:$(pwd)/examples/pylot
cargo run start-python carla_source_operator send &
cargo run start-python efficient_detection run image &
cargo run start-python sink_eval_plot plot destination
  • Within the docker container, to run just a visualisation of the simulation:
cd /home/erdos/workspace/dora-rs
export PYTHONPATH=$PYTHONPATH:$(pwd)/examples/pylot
cargo run start-python carla_source_operator send &
SOURCE=image cargo run start-python sink_eval_plot plot image

src/zenoh_client/mod.rs Outdated Show resolved Hide resolved
src/zenoh_client/mod.rs Outdated Show resolved Hide resolved
@haixuanTao haixuanTao changed the title Multithreaded Multithreaded Python API and Pylot Example Apr 13, 2022
Copy link
Collaborator

@phil-opp phil-opp left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good to me!

@haixuanTao haixuanTao merged commit b3ab72b into main Apr 29, 2022
@haixuanTao haixuanTao deleted the multithreaded branch April 29, 2022 13:56
haixuanTao added a commit that referenced this pull request May 27, 2022
* Refactoring for multithreading

* Refactoring code in order to use MemoryView

* Resolving multi output by casting Python output type

* Adding Python example runner

* Adding Rayon ThreadPool for CPU bound multithreading

* Adding benches

* Small Refactoring of Python Binding

* Adding documentation to Pylot Demo

* Removing cloning states data using RwLock

* Refactoring Servers to pass messages through tokio channels

* Removing unwrap when possible

* Splitting Zenoh function into separate module

* Refactoring Zenoh into a struct

* Adding several Python fix

* Fix eyre issue

* Adding docker for ease of build

* Fixing docker problem

* Reduce the frequency of source

* Adding better Python Operator

* Improving carla visualisation capabilities

* Enabling better visualisation

* adding object trajectory

* Improving planning

* Refactoring Python

* Adding control operator

* Improving planning operator

* Better Control Operator

* Fixing Planning Errors linked to applying Speed Factor

* Fixing Docker Image Build issues

* Adding a timestamp to messages

* Fixing PID mutlithread errors

* Drop Push Send after Pull period

* Limiting the latency

* Adding InfluxDB

* Fixing Influxdb Naming and quota

* Adding positional data

* Making launching container command faster

* Removing Dora-Pylot

* Refactor Error Handling

* Refactoring Error dubgging function

Co-authored-by: haixuanTao <hai-xuan.tao@student.ecp.fr>
haixuanTao added a commit that referenced this pull request May 27, 2022
* Refactoring for multithreading

* Refactoring code in order to use MemoryView

* Resolving multi output by casting Python output type

* Adding Python example runner

* Adding Rayon ThreadPool for CPU bound multithreading

* Adding benches

* Small Refactoring of Python Binding

* Adding documentation to Pylot Demo

* Removing cloning states data using RwLock

* Refactoring Servers to pass messages through tokio channels

* Removing unwrap when possible

* Splitting Zenoh function into separate module

* Refactoring Zenoh into a struct

* Adding several Python fix

* Fix eyre issue

* Adding docker for ease of build

* Fixing docker problem

* Reduce the frequency of source

* Adding better Python Operator

* Improving carla visualisation capabilities

* Enabling better visualisation

* adding object trajectory

* Improving planning

* Refactoring Python

* Adding control operator

* Improving planning operator

* Better Control Operator

* Fixing Planning Errors linked to applying Speed Factor

* Fixing Docker Image Build issues

* Adding a timestamp to messages

* Fixing PID mutlithread errors

* Drop Push Send after Pull period

* Limiting the latency

* Adding InfluxDB

* Fixing Influxdb Naming and quota

* Adding positional data

* Making launching container command faster

* Removing Dora-Pylot

* Refactor Error Handling

* Refactoring Error dubgging function

Co-authored-by: haixuanTao <hai-xuan.tao@student.ecp.fr>
haixuanTao added a commit that referenced this pull request Jun 8, 2022
* Refactoring for multithreading

* Refactoring code in order to use MemoryView

* Resolving multi output by casting Python output type

* Adding Python example runner

* Adding Rayon ThreadPool for CPU bound multithreading

* Adding benches

* Small Refactoring of Python Binding

* Adding documentation to Pylot Demo

* Removing cloning states data using RwLock

* Refactoring Servers to pass messages through tokio channels

* Removing unwrap when possible

* Splitting Zenoh function into separate module

* Refactoring Zenoh into a struct

* Adding several Python fix

* Fix eyre issue

* Adding docker for ease of build

* Fixing docker problem

* Reduce the frequency of source

* Adding better Python Operator

* Improving carla visualisation capabilities

* Enabling better visualisation

* adding object trajectory

* Improving planning

* Refactoring Python

* Adding control operator

* Improving planning operator

* Better Control Operator

* Fixing Planning Errors linked to applying Speed Factor

* Fixing Docker Image Build issues

* Adding a timestamp to messages

* Fixing PID mutlithread errors

* Drop Push Send after Pull period

* Limiting the latency

* Adding InfluxDB

* Fixing Influxdb Naming and quota

* Adding positional data

* Making launching container command faster

* Removing Dora-Pylot

* Refactor Error Handling

* Refactoring Error dubgging function

Co-authored-by: haixuanTao <hai-xuan.tao@student.ecp.fr>
haixuanTao added a commit that referenced this pull request Jun 10, 2022
* Refactoring for multithreading

* Refactoring code in order to use MemoryView

* Resolving multi output by casting Python output type

* Adding Python example runner

* Adding Rayon ThreadPool for CPU bound multithreading

* Adding benches

* Small Refactoring of Python Binding

* Adding documentation to Pylot Demo

* Removing cloning states data using RwLock

* Refactoring Servers to pass messages through tokio channels

* Removing unwrap when possible

* Splitting Zenoh function into separate module

* Refactoring Zenoh into a struct

* Adding several Python fix

* Fix eyre issue

* Adding docker for ease of build

* Fixing docker problem

* Reduce the frequency of source

* Adding better Python Operator

* Improving carla visualisation capabilities

* Enabling better visualisation

* adding object trajectory

* Improving planning

* Refactoring Python

* Adding control operator

* Improving planning operator

* Better Control Operator

* Fixing Planning Errors linked to applying Speed Factor

* Fixing Docker Image Build issues

* Adding a timestamp to messages

* Fixing PID mutlithread errors

* Drop Push Send after Pull period

* Limiting the latency

* Adding InfluxDB

* Fixing Influxdb Naming and quota

* Adding positional data

* Making launching container command faster

* Removing Dora-Pylot

* Refactor Error Handling

* Refactoring Error dubgging function

Co-authored-by: haixuanTao <hai-xuan.tao@student.ecp.fr>
haixuanTao added a commit that referenced this pull request Jun 15, 2022
* Multithreaded Python API and Pylot Example (#18)

* Refactoring for multithreading

* Refactoring code in order to use MemoryView

* Resolving multi output by casting Python output type

* Adding Python example runner

* Adding Rayon ThreadPool for CPU bound multithreading

* Adding benches

* Small Refactoring of Python Binding

* Adding documentation to Pylot Demo

* Removing cloning states data using RwLock

* Refactoring Servers to pass messages through tokio channels

* Removing unwrap when possible

* Splitting Zenoh function into separate module

* Refactoring Zenoh into a struct

* Adding several Python fix

* Fix eyre issue

* Adding docker for ease of build

* Fixing docker problem

* Reduce the frequency of source

* Adding better Python Operator

* Improving carla visualisation capabilities

* Enabling better visualisation

* adding object trajectory

* Improving planning

* Refactoring Python

* Adding control operator

* Improving planning operator

* Better Control Operator

* Fixing Planning Errors linked to applying Speed Factor

* Fixing Docker Image Build issues

* Adding a timestamp to messages

* Fixing PID mutlithread errors

* Drop Push Send after Pull period

* Limiting the latency

* Adding InfluxDB

* Fixing Influxdb Naming and quota

* Adding positional data

* Making launching container command faster

* Removing Dora-Pylot

* Refactor Error Handling

* Refactoring Error dubgging function

Co-authored-by: haixuanTao <hai-xuan.tao@student.ecp.fr>

* adding capnp metadata into messages

* Allowing Context to propagate throughout node

* Adding better tracing

* Refactoring opentelemetry mod

* Adding a degree parameter in messages

* Adding depth for better tracing

* Put a feature flag on tracing

* Removing unnecessary copy of the messages

* Small refactoring of messages

* Refactoring of context logging

* Adding opentelemetry metrics of system

* Adding process telemetry

* Commenting the build script

* Rename feature tracing

* Add documentation to module

* Adding message documentation

* Remove build script

* skip capnp generated file

* Reformating

* Reformating loop

* Testing example

* Removing zenoh dependencies in Python

* Improving example python api

* removing rayon that is appearing twice due to merge

* Simplifying python binding

* Create a separate crate for messages

* Moving `metrics` and `tracing` into separate crate

* Moving `python` into a separate crate and removing `main` crate

* Refactoring newly created crate

* Remove `depth` from the message `metadata`

* Add `capnp` installation within the CI

* Add `capnp` installation into CI `clippy` step

Co-authored-by: haixuanTao <hai-xuan.tao@student.ecp.fr>
phil-opp pushed a commit that referenced this pull request Aug 25, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

2 participants