ManiSkill2 - Fast Visual RL robotics cleanrl baselines

[StoneT2000]

Opening an issue on adding ManiSkill2 cleanrl baselines. The primary reason is to have good baselines and leverage the fast GPU-accelerated visual rendering of ManiSkill2 for tackling robotics envs. The plan is to include a PPO and SAC implementation (following their continuous variants implemented for mujoco with modifications to support parallel envs for SAC)

• I've read the CONTRIBUTION guide (required).
• I have ensured pre-commit run --all-files passes (required).
• I have contacted @vwxyzjn to obtain access to the openrlbenchmark W&B team (required).
• I have tracked applicable experiments in openrlbenchmark/cleanrl with --capture-video flag toggled on (required).
• I have updated the documentation and previewed the changes via mkdocs serve.
  • I have explained note-worthy implementation details.
  • I have explained the logged metrics.
  • I have added links to the original paper and related papers (if applicable).
  • I have added links to the PR related to the algorithm.
  • I have created a table comparing my results against those from reputable sources (i.e., the original paper or other reference implementation).
  • I have added the learning curves (in PNG format with width=500 and height=300).
  • I have added links to the tracked experiments.
• I have updated the tests accordingly (if applicable).

[vwxyzjn]

ManiSkill2 looks awesome. We would love to have a CleanRL variant for it. I have updated the contribution process here https://docs.cleanrl.dev/contribution to help get started.

[StoneT2000]

Quick update: will make a PR once ManiSkill2 has upgraded officially to gymnasium, which can be followed here: https://github.com/haosulab/ManiSkill2/pull/76