Skip to content

Latest commit

 

History

History
125 lines (108 loc) · 5.3 KB

DETAILS.md

File metadata and controls

125 lines (108 loc) · 5.3 KB
Raw

Details

The Habitat-Sim backend module is implemented in C++ and leverages the magnum graphics middleware library to support cross-platform deployment on a broad variety of hardware configurations. The architecture of the main abstraction classes is shown below. The design of this module ensures a few key properties:

  • Memory-efficient management of 3D environment resources (triangle mesh geometry, textures, shaders) ensuring shared resources are cached and re-used
  • Flexible, structured representation of 3D environments using SceneGraphs, allowing for programmatic manipulation of object state, and combination of objects from different environments
  • High-efficiency rendering engine with multi-attachment render passes for reduced overhead when multiple sensors are active
  • Arbitrary numbers of Agents and corresponding Sensors that can be linked to a 3D environment by attachment to a SceneGraph.

Architecture of Habitat-Sim main classes

The Simulator delegates management of all resources related to 3D environments to a ResourceManager that is responsible for loading and caching 3D environment data from a variety of on-disk formats. These resources are used within SceneGraphs at the level of individual SceneNodes that represent distinct objects or regions in a particular Scene. Agents and their Sensors are instantiated by being attached to SceneNodes in a particular SceneGraph.

Example rendered sensor observations

Performance

The table below reports performance statistics for a test scene from the Matterport3D dataset (id 17DRP5sb8fy) on a Xeon E5-2690 v4 CPU and Nvidia Titan Xp. Single-thread performance reaches several thousand frames per second, while multi-process operation with several independent simulation backends can reach more than 10,000 frames per second on a single GPU!

1 proc 3 procs 5 procs
Sensors / Resolution 128 256 512 128 256 512 128 256 512
RGB 4093 1987 848 10638 3428 2068 10592 3574 2629
RGB + depth 2050 1042 423 5024 1715 1042 5223 1774 1348
RGB + depth + semantics* 709 596 394 1312 1219 979 1521 1429 1291

Previous simulation platforms that have operated on similar datasets typically produce on the order of a couple hundred frames per second. For example Gibson reports up to about 150 fps with 8 processes, and MINOS reports up to about 167 fps with 4 threads.

*Note: The semantic sensor in MP3D houses currently requires the use of additional house 3D meshes with orders of magnitude more geometric complexity leading to reduced performance. We expect this to be addressed in future versions leading to speeds comparable to RGB + depth; stay tuned.

Rendering to GPU Tensors

We support transferring rendering results directly to a PyTorch tensor via CUDA-GL Interop. This feature is built by when Habitat-Sim is compiled with CUDA, i.e. built with --with-cuda. To enable it, set the gpu2gpu_transfer flag of the sensor specification(s) to True

This is implemented in a way that is reasonably agnostic to the exact GPU-Tensor library being used, but we currently have only implemented support for PyTorch.

Experimental: Emscripten, WebGL, and Web Apps

Build hsim_bindings.wasm, our experimental Emscripten-compiled webassembly binary for use in WebGL html/Javascript apps. See the available Javascript bindings at src/esp/bindings_js/bindings_js.cpp. Check out our bindings.html demo app:

  1. Download the test scenes and extract locally to habitat-sim creating habitat-sim/data.

  2. Download and install emscripten (you need version 1.38.48).

    In the emsdk repository:

    git pull
./emsdk install 1.38.48

  3. Activate your emsdk environment

    ./emsdk activate 1.38.48
source ./emsdk_env.sh

  4. Use Node v11.9.0

    Install nvm from here: https://github.com/nvm-sh/nvm#installing-and-updating

    nvm install v11.9.0
nvm use v11.9.0

  5. Build using ./build_js.sh [--bullet]

  6. Run webserver

    python -m http.server 8000 --bind 127.0.0.1

  7. Open http://127.0.0.1:8000/build_js/esp/bindings_js/bindings.html

You can build hsim_bindings.wasm without the demo web apps like so:

  • ./build_js.sh --no-web-apps [--bullet]