Support Multiple render target (MRT)

[Zyko0]

Operating System

• Windows
• macOS
• Linux
• FreeBSD
• OpenBSD
• Android
• iOS
• Nintendo Switch
• PlayStation 5
• Xbox
• Web Browsers

What feature would you like to be added?

I believe it could be a great addition if ebitengine could support MRT.

At the moment we can write to a single dst image, and pass multiple src image (to a DrawTriangles/DrawTrianglesShader) function, writing to multiple dst images with the same function call (and with the same internal draw call) would be nice for some specific usecases.

On software side, something like:

DrawTrianglesShadersMRT(dst []*ebiten.Image, vertices []ebiten.Vertex, indices []uint16, opts *DrawTrianglesShaderOptions)

On kage side:

func Fragment(dst vec4, src vec2, color vec4) (vec4, vec4, vec4) {
    // Heavy calculations on common (maths, geometry, etc..)
    common := HeavyCalculations(dst, src, someUniforms)
    // 3 destinations textures
    mask0 := Mask0(common)
    colorOut := ColorOut(common)
    dataOut := GetData(common)
    
    return mask0, colorOut, dataOut
}

Why is this needed?

I've had many usecases where 80% of the heavy computations made within a shader invocation are needed for multiple destination images.
Having the possibility to re-use the same vertices, the same shader draw call, the same 80% initial work that is common to all destination images could give new possibilities.

So far, in order to do so, we need:

• To write multiple shaders with repeated code, or a single one with uniforms branches
• To make distinct calls for each of the destination images (and each of these shaders/new uniforms => no batching), even though we use the same vertices and most of the same shader code (and results of the same calculations) => this obviously forces the re-invocation of a fragment shader for the same pixels 2-3 times, and the computations that come with those

In terms of usecases:

• One of the most common use nowadays, is for 3D pipelines where you process some geometry or any kind of maths for meshes / triangles, and you want to write to multiple offscreens at once (https://learnopengl.com/Advanced-Lighting/Deferred-Shading):
  • Diffuse (albedo), Normal, Depth, Specular, UVs, etc..
• Any scenario that would normally require multiple passes over the same geometry!

If this can be supported, it would certainly unlock new rendering potential for Ebitengine, even for 2D/2.5D workflows I believe.
Some existing game rendering pipelines could be optimized on the user side, or improved with new effects (for free almost?) and in general would give a new (advanced) way of designing an (richer) ebitengine application.

Potential hints:

• Metal: multiple texture slices, multiple viewports
• DirectX doc1, doc2

[hajimehoshi]

Wouldn't we need a depth buffer or a stencil buffer first, perhaps?

[Zyko0]

Yes, maybe!
Also, there's a case with depth buffer + MRT where you might want to override a custom depth value (https://registry.khronos.org/OpenGL-Refpages/gl4/html/gl_FragDepth.xhtml)

I'm thinking if we want to write to some destination textures, and sometimes discard() writing to some others, based on some runtime conditions (e.g: one want to overwrite the pixel in the depth buffer only if some conditions are met) => this would complicate the MRT feature a bit

Depth buffering would be great indeed, but I have no idea how we would like to support it (especially since the usual depth buffer is a floating-point texture?)

edit:

Wouldn't we need a depth buffer or a stencil buffer first, perhaps?

But either feature (MRT or new buffer types) doesn't require the other, and can still add value individually

[hajimehoshi]

Depth buffering would be great indeed, but I have no idea how we would like to support it (especially since the usual depth buffer is a floating-point texture?)

As Ebitengine is a 2D game engine, supporting a depth buffer sounds a little odd. I'm not familiar with it so this might be perhaps useful even for 2D game engine, but I am not sure.

[Zyko0]

So it's technically easy to support at the graphic driver level: #2953 is just a minimal working example (for MRT at least) for OpenGL and directx11 (both tested on windows only),

And I think the state of this issue is a API design issue (probably more internal than public), that still needs investigation and discussion on whether it's something we'd like to support (and if so, how), since:

• It makes sense to render to multiple targets (and it's only doable) when the different destinations are separate textures (unmanaged) => A fragment is bound to a destination location, so it can write at the same location on multiple textures, but not at different locations from a single texture
• Based on the previous point, it is possible that this can't be generalized and therefore involves having a different "draw path" for triangles using MRT => Then it could add a maintenance cost for a not-so-required feature for a 2D engine

[hajimehoshi]

It is possible to make destination images separate from atlases dynamically (and actually Ebitengine does so when necessary), but this would degrade performance, right?

[Zyko0]

It is possible to make destination images separate from atlases dynamically (and actually Ebitengine does so when necessary)

I think this would defeat the purpose a little yes, but I wasn't aware of that actually!

Unless, it is stated somehow that:
Images passed to this method will be made unmanaged if they are not already, which might prevent them to be batched with different commands

I didn't consider it, but it's true, that in this case it shouldn't even matter to the user (the fact that an image is made unmanaged), and be accepted since the usage of this function would be a bit special by nature.
The risk, of being concerned (as a user) by losing the batching-capability of an image used as part of an MRT pipeline should be quite low.

but this would degrade performance, right?

I mentioned "made unmanaged" in order to cover for the performances part, assuming that: once it is made unmanaged by ebitengine, it will never be moved again to an atlas or merged with other atlases.
In that case, the cost would only happen once, so it should be okay!

However, if you meant that they can be moved for the sole purpose of ensuring that a draw call can be performed, but that they can be moved back to atlases, then it's not good (we would like this operation to happen once at most).

edit: This would solve the primary issue (and most important one), but then it should also be stated (+panic()) that ebiten.SubImages are not accepted => which is probably okay too!

[hajimehoshi]

I didn't consider it, but it's true, that in this case it shouldn't even matter to the user (the fact that an image is made unmanaged), and be accepted since the usage of this function would be a bit special by nature.
The risk, of being concerned (as a user) by losing the batching-capability of an image used as part of an MRT pipeline should be quite low.

I'm not sure I understand what you mean. I assume the destination textures for MRT are used as multiple source textures for one shader draw call, then even if the textures are separate, this should be efficient. Is this correct?

I mentioned "made unmanaged" in order to cover for the performances part, assuming that: once it is made unmanaged by ebitengine, it will never be moved again to an atlas or merged with other atlases.
However, if you meant that they can be moved for the sole purpose of ensuring that a draw call can be performed, but that they can be moved back to atlases, then it's not good (we would like this operation to happen once at most).

If an image is unmanaged (NewImageOptions.Unmanaged), right, the image never goes to an atlas. If an image is managed, the image might go to atlas again in some conditions (e.g. the image is used as a source for a while, and the image is not used as destination)

[Zyko0]

I'm not sure I understand what you mean. I assume the destination textures for MRT are used as multiple source textures for one shader draw call, then even if the textures are separate, this should be efficient. Is this correct?

Yes! (faster than batched triangles multiplied by N regions on a single texture, since it would be a single region here and just N writes from the same shader call)

If an image is unmanaged (NewImageOptions.Unmanaged), right, the image never goes to an atlas. If an image is managed, the image might go to atlas again in some conditions (e.g. the image is used as a source for a while, and the image is not used as destination)

Okay yeah then it's acceptable I think, I understand what you mean.
Setting it as unmanaged for more control over the performances should be a user tweak then!

• How to handle passing subimages as destinations? I suggest we reject those 👀