[doc] GPU-based GUI API design #68
Conversation
|
✔️ Deploy Preview for docs-taichi-graphics ready! 🔨 Explore the source changes: da24e82 🔍 Inspect the deploy log: https://app.netlify.com/sites/docs-taichi-graphics/deploys/60ee8743d8f6880007eafa3f 😎 Browse the preview: https://deploy-preview-68--docs-taichi-graphics.netlify.app |
|
|
||
| ```python | ||
| gui = ti.ui.GUI('Window Title', (640, 360)) | ||
| ``` |
This comment was marked as outdated.
This comment was marked as outdated.
Sorry, something went wrong.
There was a problem hiding this comment.
I think you're looking at an older version of this file?
There was a problem hiding this comment.
True, therefore some of the comments are folded. Some other parts marked "outdated" by github still applies but the position in the original document has changed or the object calling on has changed, but the idea is the same. Sorry about this, I made these comments a while ago and did not notice they are not published
| for frame in range(10000): | ||
| render(img) | ||
| gui.set_image(img) | ||
| gui.show(f'{frame:06d}.png') |
There was a problem hiding this comment.
It would seems be cleaner to have frame capture in separate API. Then potentially depending on when the frame capture function is called, we may or may not have things like GUI or text captured.
|
|
||
| ## Paint on a window | ||
|
|
||
| `gui.set_image(pixels)` sets an image to display on the window. |
There was a problem hiding this comment.
A more precise name for this function would be present or blit, where present refers to displaying an image, and blit refers to painting a bitmap onto another surface (a display or another image)
| - `ti.Vector.field(3, shape=(x, y))` `(r, g, b)` channels on each | ||
| component (see [vector](../../api/vector.md#vector-fields) for details) | ||
|
|
||
| - `ti.Vector.field(2, shape=(x, y))` `(r, g)` channels on each component |
There was a problem hiding this comment.
Maybe allow RGBA images as well. In most modern graphics APIs 3 channel buffers are emulated with 4 channel one anyways (due to alignment issues)
|
|
||
| mouse_x, mouse_y = gui.get_cursor_pos() | ||
|
|
||
| `gui.fps_limit` sets the FPS limit for a window. For example, to cap FPS at 24, simply use `gui.fps_limit = 24`. This helps reduce the overload on your hardware especially when you're using OpenGL on your integrated GPU which could make desktop slow to response. |
There was a problem hiding this comment.
Consider adding VSYNC in addition to FPS limit to have the image be presented smoothly. (Non-display framerate limits can be hard to enforce / inconsistent)
|
|
||
| ## GUI Widgets | ||
|
|
||
| Sometimes it's more intuitive to use widgets like slider or button to control the program variables instead of using chaotic keyboard bindings. Taichi GUI provides a set of widgets for that reason: |
This comment was marked as outdated.
This comment was marked as outdated.
Sorry, something went wrong.
|
|
||
| `ti.imwrite(img, filename)` can export a `np.ndarray` or Taichi field (`ti.Matrix.field`, `ti.Vector.field`, or `ti.field`) to a specified location `filename`. | ||
|
|
||
| Same as `ti.GUI.show(filename)`, the format of the exported image is determined by **the suffix of** `filename` as well. Now `ti.imwrite` supports exporting images to `png`, `img` and `jpg` and we recommend using `png`. |
There was a problem hiding this comment.
Wishlist: having support for *.HDR, *.exr, *.DDS, etc. These data types are more suited for rendering applications where high dynamic range image or floating point image is required.
There was a problem hiding this comment.
Also seems redundant with the frame capture feature. Maybe add an API to export the frame data back into a taichi field, and use this image IO feature to save it do disk
|
|
||
| ## Clearing and showing a window | ||
| ```python | ||
| window.clear(color) |
This comment was marked as outdated.
This comment was marked as outdated.
Sorry, something went wrong.
There was a problem hiding this comment.
I agree. That was kinda silly. But please look at the newest files..
| ```python | ||
| window.clear(color) | ||
| ... | ||
| window.show() |
There was a problem hiding this comment.
Maybe have windows constantly show? And make show / hide toggle the window visibility state instead of having to call "show" every frame.
|
|
||
|
|
||
| ## 3D Scene | ||
|
|
There was a problem hiding this comment.
A scene graph system may be the best, making it extensible, relatively easy to use, while being performance & GPU friendly.
| # Renderer can render a scene / object | |
| renderer = ti.ui.Renderer() | |
| # Scenes are made up from objects / nodes, each node has a transform matrix and a list of children. The root node is called scene. | |
| scene = ti.ui.Scene() | |
| # Camera | |
| camera = scene.addCamera() | |
| camera.transform = ti.ui.ComposeTransform(...) # Many ways to set transform (e.g. translation / rotation / scale) | |
| camera.fov = math.rad(70) # The FOV of camera | |
| camera.near = ... | |
| camera.far = ... # These should have a reasonable default value | |
| # Primitives | |
| trimesh0 = scene.addTriangleMesh(vertices, numVertices, firstVerex=0) # vertices should (ideally) be a 1D ti.dense | |
| trimesh0.setColor() # This can be a color value, or another ti.dense for per-vertex color | |
| trimesh0.setTexture("textureFile", uv) # UV should be a ti.dense for per-vertex UV. We should also consider supporting using ti.dense images as texture. | |
| trimesh0.transform = ti.ui.ComposeTransform(...) # mesh is a scene node | |
| pointcloud0 = scene.addPointCloud() | |
| trimesh1 = trimesh0.addTriangleMesh(...) # Trimesh 0 contains a child mesh, creating a hierarchical transform | |
| # Lights | |
| pointlight0 = scene.addPointLight() | |
| distantLight0 = scene.addDistantLight() | |
| domeLight0 = scene.addDomeLight() # This can be uniform color or a skybox | |
| # Rendering | |
| # Renders the scene from camera to a target (e.g. window) | |
| renderer.render(target, scene, camera) # target can be the window, another texture, or potentially ti.dense | |
This is extendable, and should be able to run efficiently on GPU. The potential is also there to tightly integrate real-time rendering into Taichi applications with dense field interactions. The UI backend should talk with the taichi backend to transfer the images / fields implicitly. In the beginning iterations we can force the user use the same renderer / taichi runtime or a few supported combinations. (e.g. gl-gl, vulkan-vulkan, cuda-gl, cpu-vulkan)
There was a problem hiding this comment.
It would be ideal to create a seperate module just to manage API crossing. It is always possible to have shared host-device memory (cpu & gpu visible or even coherent). It is also possible to do supported interops such as Vulkan / GL image sharing or GL / cuda
There was a problem hiding this comment.
Having a scene graph would be really nice. However, I think this somewhat defeats the goal of having a simple and easy-to-use API.
It would be ideal to create a seperate module just to manage API crossing.
Yes. This will be necessary if GGUI is to replace the legacy ti.GUI completely. Currently we are aiming to support Vulkan(GUI) <-> cuda(Taichi) only. (@k-ye When/if should I worry about other backends?)
| canvas.clear(...) | ||
| canvas.triangles(...) | ||
|
|
||
| scene.clear() |
There was a problem hiding this comment.
Constantly rebuilding the scene will be costly
There was a problem hiding this comment.
Not necessarily. The implementation is free to do all kinds of caching.
We wish to keep function calls such as canvas.triangles(a,b,c) inside the render loop in order to make it obvious to the users that the newest values of the taichi fields a, b, and c, which might have just been modified this frame, will be used for rendering. This keeps the semantics of the API simple.
There was a problem hiding this comment.
Does the clear() function here means the clearing of the color & depth buffer of the canvas or does it mean clearing of the primitives defined from last frame (essentially a NewFrame() call)?
There was a problem hiding this comment.
canvas.clear(): clearing the color & depth buffer of the canvas.
scene.clear(): clearing the geometries in the scene.
| ```python | ||
| canvas = window.get_canvas() | ||
| ``` | ||
| this retrieves a `Canvas` object that covers the entire window. |
There was a problem hiding this comment.
If you squint at canvas a bit, it is basically a 2d renderer that draws on a 2d texture / image. So maybe canvas should be created from ti.gui? As for the resolution, it seems canvas is always rendering vector objects, it might be fine to assume the canvas coordinate space is (0..1, 0..1)
There was a problem hiding this comment.
So maybe canvas should be created from ti.gui?
I had a discussion with @k-ye about this. The reasoning is that we do not want the users to have to write canvas.render(...) and then write something like window.render(canvas).
As for the resolution, it seems canvas is always rendering vector objects, it might be fine to assume the canvas coordinate space is (0..1, 0..1)
Yes that is the current design. When users call canvas.triangles(...) etc. the ti vertex coordinates and other stuff should be in the [0,1]*[0,1] space.
There was a problem hiding this comment.
You would still need to call window.render(canvas) anyways right? Otherwise how do the user decide whether the canvas is drawn on top of everything else or behind things like UI?
There was a problem hiding this comment.
The plan is to have a simple model: UI widgets are always on top of the canvas.
|
@bobcao3 Thank you very much for your comments! But I think we might be aiming at slightly different things here. I believe the current task is to implement a GUI that (a) is simple to use even for not-so-experienced programmers and (b) does not suffer from the severe performance issues of the old |
I believe that no matter what this API needs to be extendable and possible to be performant. Once you offer a API to the user you can't limit what they can come up with, and it is very hard to create a performant immediate mode GUI / data visualization system. I think scene graph can be easy enough to use (if you don't create a hierarchy it is as simple as the existing API, just that it's not immediate) |
|
Thanks @bobcao3! Let's just stick with the current API design, which is mostly consistent with how |
|
If I did not call scene.clear upon next frame, will the geometries from
last frame persist? Which means we will need to keep a copy every frame
…On Wed, Jul 14, 2021, 6:41 PM Dunfan Lu ***@***.***> wrote:
***@***.**** commented on this pull request.
------------------------------
In website/docs/lang/articles/misc/new_gui.md
<#68 (comment)>
:
> +import taichi as ti
+
+window = ti.ui.Window("Amazing Window",res)
+canvas = window.get_canvas()
+scene = ti.ui.Scene()
+
+
+while window.running:
+ events = window.get_event()
+ if ev.action == ti.ui.ACTION_PRESSED and ev.key == ti.ui.KEY_SHIFT:
+ ...
+
+ canvas.clear(...)
+ canvas.triangles(...)
+
+ scene.clear()
canvas.clear(): clearing the color & depth buffer of the canvas.
scene.clear(): clear the geometries in the scene.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
<#68 (comment)>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/ACY7Q5EYOOIUG3PHKMT2BELTXY4ENANCNFSM475Y7LOQ>
.
|
Yes. The geometries persist if I'm not actually sure if this is the most intuitive semantics. We do really want to make the API for drawing 3d geometries immediate, so that users can simply write However, if we were to go full immediate mode, the api would become something like which does avoid Suggestions? |
| ```python | ||
| canvas.clear(color) | ||
| canvas.triangles(a,b,c,color) | ||
| canvas.triangles_indexed(positions,indices,color) |
There was a problem hiding this comment.
Great work @AmesingFlank ! I wonder could we also use triangles as the function name. You could determine whether drawing triangles with indices by evaluating the function parameters.
There was a problem hiding this comment.
Yes. That could work.
|
Begin & End has been the choice for immediate mode API for all sorts of
systems. A full immediate mode API will have some performance issues, you
can reference legacy GL performance
…On Wed, Jul 14, 2021, 7:41 PM Dunfan Lu ***@***.***> wrote:
If I did not call scene.clear upon next frame, will the geometries from
last frame persist? Which means we will need to keep a copy every frame
… <#m_-3178495068830036824_>
On Wed, Jul 14, 2021, 6:41 PM Dunfan Lu *@*.***> wrote: ***@***.***
<https://github.com/AmesingFlank>* commented on this pull request.
------------------------------ In
website/docs/lang/articles/misc/new_gui.md <#68 (comment)
<#68 (comment)>>
: > +import taichi as ti + +window = ti.ui.Window("Amazing Window",res)
+canvas = window.get_canvas() +scene = ti.ui.Scene() + + +while
window.running: + events = window.get_event() + if ev.action ==
ti.ui.ACTION_PRESSED and ev.key == ti.ui.KEY_SHIFT: + ... + +
canvas.clear(...) + canvas.triangles(...) + + scene.clear() canvas.clear():
clearing the color & depth buffer of the canvas. scene.clear(): clear the
geometries in the scene. — You are receiving this because you were
mentioned. Reply to this email directly, view it on GitHub <#68 (comment)
<#68 (comment)>>,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/ACY7Q5EYOOIUG3PHKMT2BELTXY4ENANCNFSM475Y7LOQ
.
Yes. The geometries persist if clear() is not called.
I'm not actually sure if this is the most intuitive semantics. We do
really want to make the API for drawing 3d geometries immediate, so that
users can simply write ''scece.mesh(vertices,indices,...)" in the render
loop. Having a scene.clear() kind of goes against the immediate mode
flavor.
However, if we were to go full immediate mode, the api would become
something like
while 1:
canvas = window.begin_canvas()
scene = canvas.begin_scene()
scene.mesh(...)
which does avoid scene.clear() but is even more cumbersome.
Suggestions?
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
<#68 (comment)>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/ACY7Q5HVBE3RYJ6CTU65M5LTXZDETANCNFSM475Y7LOQ>
.
|
|
I would suggest instead of using a,b,c separately use a single buffer / 3xN
image
…On Wed, Jul 14, 2021, 8:28 PM Dunfan Lu ***@***.***> wrote:
***@***.**** commented on this pull request.
------------------------------
In website/docs/lang/articles/misc/new_gui.md
<#68 (comment)>
:
> +
+## 2D Canvas
+
+### Creating a canvas
+
+```python
+canvas = window.get_canvas()
+```
+this retrieves a `Canvas` object that covers the entire window.
+
+### Drawing on the canvas
+
+```python
+canvas.clear(color)
+canvas.triangles(a,b,c,color)
+canvas.triangles_indexed(positions,indices,color)
Yes. That could work.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
<#68 (comment)>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/ACY7Q5BNKIGYDJ6R7UQKGSTTXZIWXANCNFSM475Y7LOQ>
.
|
|
For robust APIs, does taichi-dev/taichi_three helps? I used pre-allocated vbo because taichi doesn't support dynamic field allocation to prevent copy overhead, in other words only copy when necessary, keep data around on gpu. For the `need deep copy` every frame problem, you could use the 'id' method, like `scene.update_riangle(0, a, b, c)` only updates the vbo of primtive No. 0. And if not called the triangle will be resistant in gpu memory and thus more efficient.
无法顺畅的大口呼吸，是活着的最好证明
…---Original---
From: "Dunfan ***@***.***>
Date: Thu, Jul 15, 2021 11:28 AM
To: ***@***.***>;
Cc: ***@***.***>;
Subject: Re: [taichi-dev/docs.taichi.graphics] [doc] GPU-based GUI API design (#68)
@AmesingFlank commented on this pull request.
In website/docs/lang/articles/misc/new_gui.md:
> + +## 2D Canvas + +### Creating a canvas + +```python +canvas = window.get_canvas() +``` +this retrieves a `Canvas` object that covers the entire window. + +### Drawing on the canvas + +```python +canvas.clear(color) +canvas.triangles(a,b,c,color) +canvas.triangles_indexed(positions,indices,color)
Yes. That could work.
—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub, or unsubscribe.
|
|
In an immediate mode API we wouldn't really have a concept of VBO
…On Wed, Jul 14, 2021, 9:19 PM 彭于斌 ***@***.***> wrote:
For robust APIs, does taichi-dev/taichi_three helps? I used pre-allocated
vbo because taichi doesn't support dynamic field allocation to prevent copy
overhead, in other words only copy when necessary, keep data around on gpu.
For the `need deep copy` every frame problem, you could use the 'id'
method, like `scene.update_riangle(0, a, b, c)` only updates the vbo of
primtive No. 0. And if not called the triangle will be resistant in gpu
memory and thus more efficient.
无法顺畅的大口呼吸，是活着的最好证明
---Original---
From: "Dunfan ***@***.***>
Date: Thu, Jul 15, 2021 11:28 AM
To: ***@***.***>;
Cc: ***@***.***>;
Subject: Re: [taichi-dev/docs.taichi.graphics] [doc] GPU-based GUI API
design (#68)
@AmesingFlank commented on this pull request.
In website/docs/lang/articles/misc/new_gui.md:
> + +## 2D Canvas + +### Creating a canvas + +```python +canvas =
window.get_canvas() +``` +this retrieves a `Canvas` object that covers the
entire window. + +### Drawing on the canvas + +```python
+canvas.clear(color) +canvas.triangles(a,b,c,color)
+canvas.triangles_indexed(positions,indices,color)
Yes. That could work.
—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub, or unsubscribe.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
<#68 (comment)>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/ACY7Q5CXXJ3SDVA46BSUI63TXZOV7ANCNFSM475Y7LOQ>
.
|
This PR is intended to allow for discussions surrounding the APIs to be implemented in the new GUI system of taichi. Please do not merge it yet.
To view the markdown file included in this PR in a more friendly format, please visit this link
This is a very early draft. Any comments will be much appreciated.