Viewport Compositor #99210

New Issue

Omar Emara · 2022-06-27T16:27:13+02:00

Omar Emara commented

2022-06-27 16:27:13 +02:00

This is a task that tracks the current state of the Realtime Compositor project and lists some of its to-dos. For user-level overview and demos, see the blog post on the matter. For feedback and discussions, use the devtalk topic for the project.

Description

The aim of this project is to develop a new GPU accelerated compositor that is both realtime and interactive. As a first step, this new compositor will be used to power the Viewport Compositor, which is a new feature that applies the compositor directly in the viewport. In the long term, the realtime compositor will be unified with the existing compositor, effectively accelerating the existing compositing workflow as well. This task is only concerned about the realtime compositor in the context of the viewport compositor.

Milestones

Add a usable viewport compositor with basic input, processing, transform, and output nodes.
Support basic filter nodes like blur, filter, lens distortion, glare, and other frequently used nodes.
Support the rest of the nodes.
Support multi-pass compositing.

To Do

The following to-dos are not part of the first milestone and are thus not blockers for its merge to master.

Features
- - Single-axis domain repetition. #105642
- - Bicubic interpolation. #105533
- - Allow access to multiple EXR layers and passes. #105547
- - Handling of XYZW vector passes without losing 4th channel.
Portability
- - MacOS support. #96261
Quality
- - Specialized domain realization shaders to avoid fuzziness introduced by interpolation.
- - Full float instead of half float precision for final renders and passes that need it.
Documentation
- - Document the viewport compositor, its supported nodes, and its compatibility with the existing compositor.
Nodes
- See the following section.

Supported Nodes

Input
- - Render Layers.
- - Image.
- - Movie Clip.
- - RGB.
- - Value.
- - Time Curve.
- - Scene Time.
- - Bokeh Image.
- - Track Position.
- - Texture.
- - Mask.
Output
- - Composite.
- - Viewer.
- - Split Viewer.
- - Levels.
- - Output File.
Color
- - Mix RGB.
- - Alpha Over.
- - Invert.
- - RGB Curves.
- - Hue Saturation Value.
- - Color Balance.
- - Hue Correct.
- - Bright Contrast.
- - Gamma.
- - Exposure.
- - Color Correction.
- - Posterize.
- - Tone Map.
- - Z Combine.
Converter
- - Math.
- - Color Ramp.
- - Set Alpha.
- - Alpha Convert.
- - RGB To BW.
- - Separate Color.
- - Combine Color.
- - Separate XYZ.
- - Combine XYZ.
- - Switch View.
- - ID Mask.
- - Convert Color Space.
Filter
- - Bilateral Blur.
- - Despeckle.
- - Filter.
- - Directional Blur.
- - Dilate Erode.
- - Pixelate.
- - Bokeh Blur.
- - Blur.
- - Glare.
- - Anti-Aliasing.
- - Denoise.
- - Sun Beams.
- - Inpaint.
- - Defocus.
- - Vector Blur.
Vector
- - Normal.
- - Map Value.
- - Map Range.
- - Vector Curves.
- - Normalize.
Matte
- - Channel Key.
- - Color Spill.
- - Box Mask.
- - Ellipse Mask.
- - Luminance Key.
- - Difference Key.
- - Distance Key.
- - Chroma Key.
- - Color Key.
- - ~~Legacy Cryptomatte~~.
- - Keying.
- - Double Edge Mask.
- - Keying Screen.
- - Cryptomatte.
Distort
- - Lens Distortion.
- - Translate.
- - Rotate.
- - Flip.
- - Crop.
- - Transform.
- - Scale.
- - Displace.
- - Map UV.
- - Corner Pin.
- - Stabilize.
- - Plane Track Deform.
- - Movie Distortion.

This is a task that tracks the current state of the Realtime Compositor project and lists some of its to-dos. For user-level overview and demos, see the [blog post](https:*code.blender.org/2022/07/real-time-compositor/) on the matter. For feedback and discussions, use the [devtalk topic](https:*devtalk.blender.org/t/real-time-compositor-feedback-and-discussion/) for the project. ## Description The aim of this project is to develop a new GPU accelerated compositor that is both realtime and interactive. As a first step, this new compositor will be used to power the *Viewport Compositor*, which is a new feature that applies the compositor directly in the viewport. In the long term, the realtime compositor will be unified with the existing compositor, effectively accelerating the existing compositing workflow as well. This task is only concerned about the realtime compositor in the context of the viewport compositor. ## Milestones - [x] Add a usable viewport compositor with basic input, processing, transform, and output nodes. - [x] Support basic filter nodes like blur, filter, lens distortion, glare, and other frequently used nodes. - [x] Support the rest of the nodes. - [ ] Support multi-pass compositing. ## To Do The following to-dos are not part of the first milestone and are thus not blockers for its merge to master. - Features - - [x] Single-axis domain repetition. #105642 - - [x] Bicubic interpolation. #105533 - - [x] Allow access to multiple EXR layers and passes. #105547 - - [x] Handling of XYZW vector passes without losing 4th channel. - Portability - - [x] MacOS support. #96261 - Quality - - [x] Specialized domain realization shaders to avoid fuzziness introduced by interpolation. - - [x] Full float instead of half float precision for final renders and passes that need it. - Documentation - - [x] Document the viewport compositor, its supported nodes, and its compatibility with the existing compositor. - Nodes - See the following section. ## Supported Nodes - Input - - [x] Render Layers. - - [x] Image. - - [x] Movie Clip. - - [x] RGB. - - [x] Value. - - [x] Time Curve. - - [x] Scene Time. - - [x] Bokeh Image. - - [x] Track Position. - - [x] Texture. - - [x] Mask. - Output - - [x] Composite. - - [x] Viewer. - - [x] Split Viewer. - - [x] Levels. - - [x] Output File. - Color - - [x] Mix RGB. - - [x] Alpha Over. - - [x] Invert. - - [x] RGB Curves. - - [x] Hue Saturation Value. - - [x] Color Balance. - - [x] Hue Correct. - - [x] Bright Contrast. - - [x] Gamma. - - [x] Exposure. - - [x] Color Correction. - - [x] Posterize. - - [x] Tone Map. - - [x] Z Combine. - Converter - - [x] Math. - - [x] Color Ramp. - - [x] Set Alpha. - - [x] Alpha Convert. - - [x] RGB To BW. - - [x] Separate Color. - - [x] Combine Color. - - [x] Separate XYZ. - - [x] Combine XYZ. - - [x] Switch View. - - [x] ID Mask. - - [x] Convert Color Space. - Filter - - [x] Bilateral Blur. - - [x] Despeckle. - - [x] Filter. - - [x] Directional Blur. - - [x] Dilate Erode. - - [x] Pixelate. - - [x] Bokeh Blur. - - [x] Blur. - - [x] Glare. - - [x] Anti-Aliasing. - - [x] Denoise. - - [x] Sun Beams. - - [x] Inpaint. - - [x] Defocus. - - [x] Vector Blur. - Vector - - [x] Normal. - - [x] Map Value. - - [x] Map Range. - - [x] Vector Curves. - - [x] Normalize. - Matte - - [x] Channel Key. - - [x] Color Spill. - - [x] Box Mask. - - [x] Ellipse Mask. - - [x] Luminance Key. - - [x] Difference Key. - - [x] Distance Key. - - [x] Chroma Key. - - [x] Color Key. - - [x] ~~Legacy Cryptomatte~~. - - [x] Keying. - - [x] Double Edge Mask. - - [x] Keying Screen. - - [x] Cryptomatte. - Distort - - [x] Lens Distortion. - - [x] Translate. - - [x] Rotate. - - [x] Flip. - - [x] Crop. - - [x] Transform. - - [x] Scale. - - [x] Displace. - - [x] Map UV. - - [x] Corner Pin. - - [x] Stabilize. - - [x] Plane Track Deform. - - [x] Movie Distortion.

❤️ 11 🚀 4

Omar Emara self-assigned this 2022-06-27 16:27:13 +02:00

Jacob Merrill commented

2022-07-06 23:31:22 +02:00

I recommend adding a way to store the result of a buffer offscreen, to be used next frame too*
(motion blur and a few other techniques use it)

ie 'Output nodes'

I recommend adding a way to store the result of a buffer offscreen, to be used next frame too* (motion blur and a few other techniques use it) ie 'Output nodes'

Pierre Schiller commented

2022-07-07 07:23:30 +02:00

Is it possible to implement recognizing Cryptomatte ID's?
That would be a huge plus, to apply specific nodes (filters) to a material or a mesh in the scene. The user can even re-map textures and materials
on the fly.

Is it possible to implement recognizing Cryptomatte ID's? That would be a huge plus, to apply specific nodes (filters) to a material or a mesh in the scene. The user can even re-map textures and materials on the fly.

👍 3

Omar Emara commented

2022-07-07 13:01:31 +02:00

I created a devtalk topic for feedback and discussions regarding this project, let us use that topic instead.

https://devtalk.blender.org/t/real-time-compositor-feedback-and-discussion/

I created a devtalk topic for feedback and discussions regarding this project, let us use that topic instead. https://devtalk.blender.org/t/real-time-compositor-feedback-and-discussion/

Vyacheslav Kobozev commented

2022-07-09 08:00:31 +02:00

Very cool project. Please implement UV/position node for compositing. That will allow to distort image easier.
Now I use two blend textures as X and Y.

Very cool project. Please implement UV/position node for compositing. That will allow to distort image easier. Now I use two blend textures as X and Y.

Guido commented

2022-07-10 03:24:51 +02:00

Just for info !

Omar Emara (OmarSquircleArt) :

MacOS versions which don’t support OpenGL 4.3 are expected not to work.

The real-time compositor is built over the GPU module, which uses whatever back-end that is preferred by the underlying system. So when Apple finishes Metal support in Blender, it will automatically work on Metal

Just for info ! Omar Emara (OmarSquircleArt) : > MacOS versions which don’t support OpenGL 4.3 are expected not to work. > The real-time compositor is built over the GPU module, which uses whatever back-end that is preferred by the underlying system. So when Apple finishes Metal support in Blender, it will automatically work on Metal

Matt commented

2022-07-10 17:05:34 +02:00

In #99210#1388196, @Nurb2Kea wrote:
Just for info !

Omar Emara (OmarSquircleArt) :

MacOS versions which don’t support OpenGL 4.3 are expected not to work.

The real-time compositor is built over the GPU module, which uses whatever back-end that is preferred by the underlying system. So when Apple finishes Metal support in Blender, it will automatically work on Metal

Yeah, I'm curious when Metal support would be "done", or support this new feature. Since M1 Pro chip/other versions crash instantly when going into Render View Mode.

> In #99210#1388196, @Nurb2Kea wrote: > Just for info ! > > Omar Emara (OmarSquircleArt) : > >> MacOS versions which don’t support OpenGL 4.3 are expected not to work. > > > >> The real-time compositor is built over the GPU module, which uses whatever back-end that is preferred by the underlying system. So when Apple finishes Metal support in Blender, it will automatically work on Metal Yeah, I'm curious when Metal support would be "done", or support this new feature. Since M1 Pro chip/other versions crash instantly when going into Render View Mode.

Omar Emara commented

2022-07-11 12:08:02 +02:00

@mattli911 The state and progress of the Metal backend can be tracked in #96261 (Metal Viewport ).

dursun ümit commented

2022-08-01 01:37:56 +02:00

please add glare filter, thank you :)

Alex Mehler commented

2022-08-24 00:20:44 +02:00

image node doesnt scale with the realtime viewport. means it will always overlay too small or too large and in wrong position. It is fixed in size while the viewport is not fixed in size.

Evan Wilson commented

2022-08-24 07:33:21 +02:00

@Ulf3000 See:
https://devtalk.blender.org/t/real-time-compositor-feedback-and-discussion/25018/90

and

https://devtalk.blender.org/t/real-time-compositor-feedback-and-discussion/25018/117

@Ulf3000 See: https://devtalk.blender.org/t/real-time-compositor-feedback-and-discussion/25018/90 and https://devtalk.blender.org/t/real-time-compositor-feedback-and-discussion/25018/117

Sergey commented

2022-08-30 20:52:54 +02:00

I really look forward to multi-pass. no way without it ))

Matt commented

2022-09-03 16:46:24 +02:00

Should I make a real bug, if enabling compositor on Mac, can instant crash Blender? Or should we still expect that/is not supported yet.

If I enable it in the Preferences, then enable in the UI dropdown. I said "Use Nodes" in the compositor Workspace, then I tried to go to back to Default Workspace and it closes.

Should I make a real bug, if enabling compositor on Mac, can instant crash Blender? Or should we still expect that/is not supported yet. If I enable it in the Preferences, then enable in the UI dropdown. I said "Use Nodes" in the compositor Workspace, then I tried to go to back to Default Workspace and it closes. ![CleanShot 2022-09-03 at 07.47.19.png](https://archive.blender.org/developer/F13455455/CleanShot_2022-09-03_at_07.47.19.png)

Jacob Merrill commented

2022-09-03 18:46:07 +02:00

In #99210#1412027, @mattli911 wrote:
Should I make a real bug, if enabling compositor on Mac, can instant crash Blender? Or should we still expect that/is not supported yet.

If I enable it in the Preferences, then enable in the UI dropdown. I said "Use Nodes" in the compositor Workspace, then I tried to go to back to Default Workspace and it closes.

I am not 100% sure but, I think this will require the metal viewport *

> In #99210#1412027, @mattli911 wrote: > Should I make a real bug, if enabling compositor on Mac, can instant crash Blender? Or should we still expect that/is not supported yet. > > If I enable it in the Preferences, then enable in the UI dropdown. I said "Use Nodes" in the compositor Workspace, then I tried to go to back to Default Workspace and it closes. > > ![CleanShot 2022-09-03 at 07.47.19.png](https://archive.blender.org/developer/F13455455/CleanShot_2022-09-03_at_07.47.19.png) I am not 100% sure but, I think this will require the metal viewport *

Jacob Merrill commented

2022-09-03 18:46:35 +02:00

ah yes*
see

   Portability
     MacOS support. Needs T96261: Metal Viewport .

ah yes* see ``` Portability MacOS support. Needs T96261: Metal Viewport . ```

fwfeet commented

2022-09-07 20:33:09 +02:00

Why is vector math not listed here? 🤔

Jacob Merrill commented

2022-09-07 22:21:23 +02:00

In #99210#1413637, @costa wrote:
Why is vector math not listed here? 🤔

100% agree

using normal input +dot can be used for all sorts of hacks.

> In #99210#1413637, @costa wrote: > Why is vector math not listed here? 🤔 100% agree using normal input +dot can be used for all sorts of hacks.

Vyacheslav Kobozev commented

2022-09-07 23:54:53 +02:00

In #99210#1413673, @JacobMerrill-1 wrote:

In #99210#1413637, @costa wrote:
Why is vector math not listed here? 🤔

100% agree

using normal input +dot can be used for all sorts of hacks.

As Omar said, and I am agree, the first there should be all that already exist.
And yes, there is a plan for vector math for compositor

> In #99210#1413673, @JacobMerrill-1 wrote: >> In #99210#1413637, @costa wrote: >> Why is vector math not listed here? 🤔 > > 100% agree > > using normal input +dot can be used for all sorts of hacks. As Omar said, and I am agree, the first there should be all that already exist. And yes, there is a plan for vector math for compositor

Feng commented

2022-09-14 04:24:52 +02:00

Is it possible for freestyle to join the Realtime Compositor?

dursun ümit commented

2022-10-04 10:27:26 +02:00

please glare filter please, we need a lot of ☺

hoxabel commented

2022-10-07 22:20:39 +02:00

Will the new Light groups option be available in the Real-time Compositor?

dursun ümit commented

2022-10-07 23:40:33 +02:00

In #99210#1429360, @hoxabel wrote:
Will the new Light groups option be available in the Real-time Compositor?

I agree...

> In #99210#1429360, @hoxabel wrote: > Will the new Light groups option be available in the Real-time Compositor? I agree...

Jacob Merrill commented

2022-10-08 20:43:49 +02:00

Z depth pass is not working atm or AOV
depth_blur_diff_outline_2.blend

/test file
fast outline using a bit of math works at render time but not in viewport.

old upbge 0.2.5x had a commit called 'multiple render targets' or 'render attachments'

maybe we can cherry pick some of it?

Z depth pass is not working atm or AOV [depth_blur_diff_outline_2.blend](https://archive.blender.org/developer/F13639211/depth_blur_diff_outline_2.blend) /test file fast outline using a bit of math works at render time but not in viewport. ![image.png](https://archive.blender.org/developer/F13639217/image.png) old upbge 0.2.5x had a commit called 'multiple render targets' or 'render attachments' maybe we can cherry pick some of it?

michael campbell commented

2022-10-14 05:53:56 +02:00

Hi, any plans to have this working with viewport render animation?

Hi, any plans to have this working with viewport render animation? ![image.png](https://archive.blender.org/developer/F13676012/image.png)

❤️ 1

Alex Mehler commented

2022-10-23 23:25:10 +02:00

since many versions blender always crashes when i duplicate an object and the viewport compositor is turned on .. boom crash. the nodes im using are only filter:soften and hue/sat/val.. nothing else

Vyacheslav Kobozev commented

2022-10-24 01:38:04 +02:00

In #99210#1436394, @Ulf3000 wrote:
since many versions blender always crashes when i duplicate an object and the viewport compositor is turned on .. boom crash. the nodes im using are only filter:soften and hue/sat/val.. nothing else

make tests, if it is crashes always, or with certain filters
make report, attach your file, screencast video, crashlogs (start blender with blender_debug_log.cmd before)

All these will increase chance of fix very much

> In #99210#1436394, @Ulf3000 wrote: > since many versions blender always crashes when i duplicate an object and the viewport compositor is turned on .. boom crash. the nodes im using are only filter:soften and hue/sat/val.. nothing else 1) make tests, if it is crashes always, or with certain filters 2) make report, attach your file, screencast video, crashlogs (start blender with blender_debug_log.cmd before) All these will increase chance of fix very much

Marc commented

2022-11-08 22:13:35 +01:00

This is amazing, now I just need the convert color space node

Steffen Hartmann commented

2022-11-15 20:43:24 +01:00

Can the viewport compositior be enabled in solid mode? Is this planned?
Would be awesome.

Can the viewport compositior be enabled in solid mode? Is this planned? Would be awesome.

Vyacheslav Kobozev commented

2022-11-29 13:07:49 +01:00

In #99210#1446465, @Harti wrote:
Can the viewport compositior be enabled in solid mode? Is this planned?
Would be awesome.

I suppose it. because Workbench is render engine too.

@OmarEmaraDev was Texture support disabled forever? Heavy reason?
A bit strange, because RTC already supports procedurally generated textures.

> In #99210#1446465, @Harti wrote: > Can the viewport compositior be enabled in solid mode? Is this planned? > Would be awesome. I suppose it. because Workbench is render engine too. @OmarEmaraDev was Texture support disabled forever? Heavy reason? A bit strange, because RTC already supports procedurally generated textures.

dursun ümit commented

2022-12-19 12:53:50 +01:00

thank you Omer Emara.

Adam Bach commented

2023-01-16 12:02:37 +01:00

since tracking position node is working for realtime compositor it wouldn't be a right time to implement PLANE TRACK DIFORM and CORNER PIN nodes?
@omar-1 Emara
Now the most waited nodes are:

DISPLACE node useful for vfx effects creation
KEYING node is the most used for green screen removal.
and Mask node

since tracking position node is working for realtime compositor it wouldn't be a right time to implement PLANE TRACK DIFORM and CORNER PIN nodes? @omar-1 Emara Now the most waited nodes are: - DISPLACE node useful for vfx effects creation - KEYING node is the most used for green screen removal. - and Mask node

blender-admin commented

2023-01-30 11:00:26 +01:00

This issue was referenced by 57552f52b2

This issue was referenced by 57552f52b2987adfd1525c3e4109565e3641ef5d

Philipp Oeser removed the

Download

What's New

Blender Studio

Manual

Developers Blog

Documentation

Benchmark

Blender Conference

Development Fund

One-time Donations

Viewport Compositor #99210

Description

Milestones

To Do

Supported Nodes