SharedResouces
Sharing Resources across WebGLRendering Contexts
A proposal for sharing WebGL resources across WebGLRenderingContexts.
Goals:
- Allow sharing WebGL resources across multiple contexts, including contexts in workers.
- Prevent undefined behavior
- Prevent wrong behavior that might just happen to work.
Background:
The OpenGL ES spec defines that you can share a resource (texture, buffer, shader, program, renderbuffer) with 2 or more GL contexts but with some caveats. To guarantee you'll see a change made in one context in other context requires calling glFinish on the context that made the change and call glBind on the context that wants to see the change. In other words.
int tex; eglMakeCurrent(context1) glGenTextures(1, &tex); glBindTexture(GL_TEXTURE_2D, tex); eglMakeCurrent(context2); glBindTexture(GL_TEXTURE_CUBE_MAP, tex); // ambiguous! ASSERT(glGetError() == GL_INVALID_OPERATION);
At this point what kind of texture is 'tex'? In a single context the first glBindTexture would make 'tex' a 2D texture and the second call to glBindTexture would fail with INVALID_OPERATION. With 2 contexts through there is no guarantee which glBindTexture will run first.
The correct usage according to the OpenGL spec.
int tex; eglMakeCurrent(context1) glGenTextures(1, &tex); glBindTexture(GL_TEXTURE_2D, tex); glFinish(); eglMakeCurrent(context2); glBindTexture(GL_TEXTURE_CUBE_MAP, tex); ASSERT(glGetError() == GL_INVALID_OPERATION);
Now it will work correctly
Similarly
eglMakeCurrent(context1)
glBindTexture(GL_TEXTURE_2D, tex);
eglMakeCurrent(context2);
glBindTexture(GL_TEXTURE_2D, tex);
...
eglMakeCurrent(context1)
glTexParameter(GL_TEXTURE_2D, GL_TEXTURE_WRAP_S,
GL_CLAMP_TO_EDGE);
eglMakeCurrent(context2)
glDrawArrays(...) // using tex.
There is no guarantee that tex parameter setting will be visible in the call to glDrawArrays. The correct code, according to the spec is
eglMakeCurrent(context1)
glBindTexture(GL_TEXTURE_2D, tex);
eglMakeCurrent(context1)
glTexParameter(GL_TEXTURE_2D, GL_TEXTURE_WRAP_S,
GL_CLAMP_TO_EDGE);
glFinish(); // must call glFinish
eglMakeCurrent(context2)
glBindTexture(GL_TEXTURE_2D, tex); // must call glBind
glDrawArrays(...) // using tex.
Sync objects or Fences can be used instead of glFinish for speed but once the sync/fence clears a bind is still required to be guaranteed to see the results. Not calling glFinish and/or glBind does not guarantee you won't see the results which means that users may do neither and their app might just happen to work on some platforms and mysteriously have glitches, rendering corruption, gl errors or program failure on others. While native app developers are used to dealing with these issues this is not something we want to pass on to the web.
Use Cases:
Uploading textures asynchronously:
Currently when calling texImage2D, browsers may need to re-decode the image. The image they decoded for displaying may use pre-multiplied alpha or maybe have color correction applied. The options to WebGL allow the user to select non-pre-multipiled alpha and no color correction. In those cases the browser is forced to synchronously re-decode the image. For applications that are streaming data while displaying other data this causes a 'glitch' or 'jank' in the UI. Moving the uploading of a texture to a worker would move that blockage to the worker and fix that 'jank.
Compiling shader programs asynchronously.
Currently compiling and linking shaders is semi-synchronous. Calling compileShader or linkProgram can be asynchronous but all but the most trivial of programs must query their uniform locations and that process must block until both the compiles and link have finished. Moving this process to a worker would let that happen asynchronously and remove 'jank'
Vertex Manipulation in Shaders
Some apps do signification vertex data manipulation whether for loading (unpacking the data) or things like CPU skinning and morph targets. This proposal makes it possible to move that manipulation to a worker freeing up the main thread for other work and/or avoid 'jank'
Javascript Texture Readback
Some applications require regular readback of textures to perform operations such as hit detection or visibility detection (think sparse virtual textures/megatextures). Being able to do the readback on a worker without stalling the main thread would greatly aid in reducing jank.
Proposal:
In WebGL require 'acquiring' and 'releasing' shared resources. Resources can be acquired for READ_ONLY or EXCLUSIVE access. 'acquiring' a resource for EXCLUSIVE access means that no other context can use the resource until it is 'released'. 'releasing' a resource that was acquired for EXCLUSIVE access does an implicit glFinish. More performant implementations insert a sync object. In performant implementations, acquiring a resource would insert a wait for the sync object previously inserted on the release. Multiple contexts may acquire a resource for READ_ONLY access. You may not acquire a resource for EXCLUSIVE access until it is currently not acquired.
All resources start as acquired for EXCLUSIVE access on creation in the context they are created in.
Resources that are acquired must be bound again before used or an INVALID_OPERATION is generated
Once a resource is released it can no longer be used without acquiring it again. In other words.
var tex = gl.createTexture();
gl.bindTexture(gl.TEXTURE_2D, tex);
gl.releaseSharedResource(tex);
gl.bindTexture(gl.TEXTURE_2D, tex); // ERROR!!! This has been
// released and must be
// re-acquired to be used.
The WebGLRenderingContext gains a new 'shareGroup' property of type WebGLShareGroup. That can be passed as a 'shareGroup' context creation parameter to getContext as well as the constructor for WebGLRenderingContexts (see WebGL in Workers proposal). Example:
var gl1 = someCanvas.getContext('webgl');
var gl2 = someOtherCanvas.getContext('webgl', {
shareGroup: gl1.shareGroup
});
These 2 contexts may now share resources.
The WebGLObject class hierarchy becomes
+---------------+
| WebGLObject |
+-+-----------+-+
| |
+--------------+---+ +---+---------------+
| WebGLFramebuffer | | WebGLSharedObject |
+------------------+ +--+----------------+
|
| +-------------------+
+--+ WebGLBuffer |
| +-------------------+
|
| +-------------------+
+--+ WebGLProgram |
| +-------------------+
|
| +-------------------+
+--+ WebGLRenderbuffer |
| +-------------------+
|
| +-------------------+
+--+ WebGLShader |
| +-------------------+
|
| +-------------------+
+--+ WebGLTexture |
+-------------------+
WebGLShareGroup and WebGLSharedObject become copyable objects which means you can send them to another thread (a worker) in postMessage and they are copied (not transferred).
As an example, since I only know WebKit code, the actual C++ would have to change. WebGLSharedObject would become WebGLSharedObjectImpl and WebGLSharedObject would just be a thread safe ref counted reference to a WebGLSharedObjectImpl. Copying a WebGLSharedObject effectively just makes an a new reference.
Suggested IDL:
interface WebGL_shared_resources {
const GLenum READ_ONLY = 0x0001;
const GLenum EXCLUSIVE = 0x0004;
WebGLShareGroup shareGroup;
long acquireSharedResource(
WebGLSharedObject resource,
GLenum mode,
AcquireSharedResourcesCallback callback);
void releaseSharedResources(
WebGLSharedObject resource);
void cancelAcquireSharedResource(long id);
}
callback AcquireSharedResourcesCallback = void ();
interface WebGLShareGroup {
}
interface WebGLSharedObject : WebGLObject {
}
interface WebGLBuffer : WebGLSharedObject {
}
interface WebGLProgram : WebGLSharedObject {
}
interface WebGLRenderbuffer : WebGLSharedObject {
}
interface WebGLShader : WebGLSharedObject {
}
interface WebGLTexture : WebGLSharedObject {
}
Rational:
Q: Why have a WebGLShareGroup object' Why not just use WebGLRenderingContext as a creation parameter to WebGLRenderingContext and Canvas.getContext?
A: To enable sharing across threads (workers). A WebGLShareGroup object is a copyable object so passing one to a worker using postMessage gives the worker something it can use to create a context that is sharing WebGL resources with a context in the main thread.
Q: Why require the 'shareGroup' context creation parameter' Why not just share all resources?
A: Maybe we don't need a shareGroup. If you don't give another WebGLRenderingContext access to JS references to any WebGL resources then it can't access your resources anyway so maybe this extra step is not needed?
Resolution: Use a shareGroup (vote at WG)
Q: How should acquireSharedResource handle cases where the resource hasn't been released?
A:
- It should block: Blocking in the main thread would stall it which freezes the UI. Blocking in workers might be okay but why make the two environments different?
- It should throw an exception or return success/failure: Apps would need to figure out their own way to synchronize access. This brings back the ambiguity in that on some implementations acquire might succeed out of random luck and not on others so an app might randomly run. The user might think I released last frame so I know this frame I can acquire and therefore not surround the code in a try/catch. It works fine on their machine but fails when some other tab affects the timing or when the browser updates to be faster or when the user plugs in a 120hz monitor or on some other machine with different timing.
- It should have a callback: This seems like it would make it less likely that apps can succeed by luck.
Resolution: 3. The acquire method has a callback.
Q: Can a 2nd WebGLRenderingContext use a WebGL resource with out acquiring it?
A: No. Example
Bad
var gl1 = someCanvas.getContext('webgl');
var gl2 = someOtherCanvas.getContext('webgl', {
shareGroup: gl1.shareGroup
});
var tex = gl1.createTexture();
gl2.bindTexture(gl.TEXTURE_2D, tex); // ERROR!! tex does not
// belong to gl2
Good
var gl1 = someCanvas.getContext('webgl');
var sg1 = someCanvas.getExtension("WEBGL_shared_resources");
var gl2 = someOtherCanvas.getContext('webgl', {
shareGroup: gl1.shareGroup
});
var sg2 = someOtherCanvas.getExtension("WEBGL_shared_resources");
var tex = gl1.createTexture();
sg1.releaseSharedResource(tex);
sg2.acquireSharedResource(tex, gl.READ_ONLY, function() {
gl2.bindTexture(gl.TEXTURE_2D, tex);
});
Q: Should you be required to unbind a resource you release?
A:
1) No requirement to unbind.
This has the implication that every call to every method that affects a resource must check if the object currently bound needs to be acquired and is acquired. In other words
gl1.bindTexture(gl.TEXTURE_2D, tex); gl1.releaseSharedResource(tex); gl1.texImage2D(gl.TEXTURE_2D, ..); // MUST FAIL needs check gl1.texParameter2D(gl.TEXTURE_2D, ..); // MUST FAIL needs check gl1.copyTexImage2D(gl.TEXTURE_2D, ..); // MUST FAIL needs check gl1.texSubImage2D(gl.TEXTURE_2D, ..) ; // MUST FAIL needs check
2) Must unbind, manual unbind
This puts a burden on the user to unbind but it means the implementation does not have to check in as many places if an object needs to be acquired and also is acquired. It only needs to check at release time that the object has been unbound and at bind time that the object has been acquired. In other words
gl1.bindTexture(gl.TEXTURE_2D, tex);
gl1.releaseSharedResource(tex); // ERROR can not release.
// Have not unbound.
3) Automatic unbind
We could have release automatically unbind. The implementation could either track where a resource is bound or else check all the binding points. For buffers that's attribs and Vertex Array Objects, For textures it's texture units and framebuffer attachments, For renderbuffers it's framebuffer attachments. For shaders its program attachments. For programs it's the current program.
Resolution: #1
- 2 and #3 would make certain uses very hard. Unbinding includes un-attaching renderbuffers and textures from framebuffers, buffers and vertex array objects and shaders from programs which would then require the use rebind them all.
Instead implementations must track 2 things
a) is this resource currently acquired by this context. If not generate INVALID_FRAMEBUFFER_OPERATION for draw/read commands and INVALID_OPERATION all other commands.
b) has this resource been bound since the last time it was acquired. If not generate INVALID_FRAMEBUFFER_OPERATION for draw/read commands and INVALID_OPERATION all other commands.
Binding includes calling bindXXX as well as re-attaching (attachShader, framebufferRenderbuffer, framebufferTexture2D).
Binding only has to happen once per resource. In other words, if a texture is used in multiple texture units or attached to multiple framebuffers binding it to any single texture unit or any framebuffer is enough to mark that resources as 'has been bound since last acquire'.
Q: Should acquireSharedResource be acquireSharedResources?
A: The current design using a callback, it might be useful to be able to acquire multiple resources at once rather than have to make a chain of callbacks for each individual resource when you need multiple resource.
Resolution: Leave it acquireSharedResource.
Q: Why not just 'acquire' and 'release' instead of 'acquireSharedResource'/'releaseSharedResource'
A: There may be other things that need to be acquired in the future. Choosing a more specific name means less possibility for conflict later.
Q: What should the behavior be if the user attempts to alter a resource acquired for reading only?
A: 1) Every operation that alters a resource must check to ensure the resource has been exclusively acquired. As an example:
sg.acquireSharedResource(tex, gl.READ_ONLY, function(){
gl.bindTexture(gl.TEXTURE_2D, tex);
p = gl.getTexParameter(gl.TEXTURE_2D, ...; // no check needed
gl.texImage2D(gl.TEXTURE_2D, ...; // MUST FAIL, needs check
gl.texSubImage2D(gl.TEXTURE_2D, ...; // MUST FAIL, needs check
gl.drawArrays(...); // MUST FAIL if texture is in the current
// framebuffer as an attachment.
gl.readPixels(...); // no check needed.
});
2) Resources must always be acquired with read and write privileges. This would simplify the internal logic but would make certain use cases more difficult, such as reading a static texture from multiple contexts.
Resolution: #1 Failing the check generates INVALID_FRAMEBUFFER_OPERATION for draw commands and INVALID_OPERATION all other commands.
Q: What happens if one context deletes a resource another context is attempting to acquire?
A: Nothing special. The acquire will succeed when the context that currently has the resource releases it. The context that acquires the resource can use the WebGLSharedObject (buffer, texture, etc...) and will get the normal WebGL errors associated with using a deleted resource.
Q: Can you attach a texture/resource that you have not acquired to a framebuffer?
A: No. An attachment can remain attached while it's release but that framebuffer is not renderable or readable until all of it's attachments are acquired. Example:
var tex = gl.createTexture(); var fb = gl.createFramebuffer(); gl.bindFramebuffer(gl.FRAMEBUFFER, fb); gl.bindTexture(gl.TEXTURE_2D, tex); gl.framebufferTexture2D(gl.FRAMEBUFFER, gl.COLOR_ATTACHMENT0, gl.TEXTURE_2D, tex, 0); sg.releaseSharedResource(tex); ASSERT(gl.checkFramebufferStatus() != GL_FRAMEBUFFER_COMPLETE); // framebuffers with unacquired attachments are not complete gl.drawArrays(....); // must generate GL_INVALID_FRAMEBUFFER_OPERATION gl.readPixels(...); // must generate GL_INVALID_FRAMEBUFFER_OPERATION
Q: How should access be defined?
A:
1) READ and READ_WRITE
Where READ is non-exclusive and READ_WRITE is exclusive
2) READ_BIT and WRITE_BIT
Where you can 'OR' the bits together. Asking for WRITE_BIT makes the request exclusive. Asking for WRITE_BIT and not READ_BIT means you can only do things that affect the state of an object, not things that can read from the object. In other words you can't render with a texture you didn't acquire for read
3) READ_ONLY and EXCLUSIVE
Resolution: #3 There are only 2 modes. READ_ONLY and EXCLUSIVE. The constants are defined such that if it's important READ and WRITE as bits can be added later.
Q: Should this be exposed as an extension or on the WebGLRenderingContext?
A:
1) It should be an extension, at least on WebGL 1.0. What makes this new feature any different from other WebGL features?
2) It should go on the main context. If that's the case why don't all extensions go on the main context?
Resolution: #1 Some browser vendors have expressed it will be a while before they can implement this therefore they'd prefer it was an extension.
Q: What happens if you try to acquire a resource you already have acquired?
A:
- It should fail with an exception
- It should fail with an GL error
- It should succeed by calling the callback during the event loop when the acquire succeeds.
- It should succeed by calling the callback immediately if possible otherwise during the event loop when the acquire succeeds.
Resolution: #1 or #2. Anything else leads to all kinds of issues.
Q: Should there be a blocking version of acquire that is only available in workers?
A: Blocking calls are not allowed in the main thread because the browser itself can not run while the main thread is blocked. Blocking calls are allowed in workers.
1) Have a blocking version of acquire
2) Don't have a blocking version of acquire.
Resolution: #2 Blocking a worker seems like a bad idea as that worker can not process any other messages until unblocked. If it turns out to be important this can always be added later.
Q: What happens when a context is deleted
A: WebGLRenderingContexts are not explicitly deleted but they may be implicitly deleted by garbage collection if there are no references. When a WebGLRenderingContext is deleted all resources it has acquired are released automatically.
Q: Which mode, READ_ONLY or EXCLUSIVE is used when calling checkFramebufferStatus?
A:' Ideally checkFramebufferStatus will return FRAMEBUFFER_INCOMPLETE_ATTACHMENT if an attachment is not acquired but... if you're only going to read from a framebuffer (calling readPixels) you should only need READ_ONLY permission where as if you are going to draw to the framebuffer (clear/drawArrays/drawElements) you need EXCLUSIVE permission. Since checkFramebufferStatus doesn't know your intent it can't give you the correct answer.
Solutions
1) Require EXCLUSIVE permission for framebuffer attachments
That's not really a solution since one of the required use cases is multiple workers reading from the same texture.
2) checkFramebufferStatus assumes READ_ONLY permission only
That means you might still get a INVALID_FRAMEBUFFER_OPERATION if you acquired attachments for READ_ONLY access and then try to draw.
3) expose the DRAW_FRAMEBUFFER and READ_FRAMEBUFFER bind targets
On systems that support multi-sampling there are 2 framebuffer bind targets. DRAW_FRAMEBUFFER and READ_FRAMEBUFFER. DRAW is where drawing happens. READ is where reading happens. Unfortunately since multi-sampling isn't supported everywhere allowing you to bind separate framebuffers to DRAW_FRAMEBUFFER and READ_FRAMEBUFFER won't work.
4) allow DRAW_FRAMEBUFFER and READ_FRAMEBUFFER as arguments to checkFramebufferStatus
In this case you'd still only be able to bind to gl.FRAMEBUFFER but you can call checkFramebufferStatus with either DRAW_FRAMEBUFFER or READ_FRAMEBUFFER or FRAMEBUFFER. The OpenGL spec defines FRAMEBUFFER as meaning both READ_FRAMEBUFFER and DRAW_FRAMEBUFFER so this proposal would just work and be future compatible.
In other words.
gl.checkFramebufferStatus(gl.FRAMEBUFFER) // checks for EXCLUSIVE access gl.checkFramebufferStatus(gl.DRAW_FRAMEBUFFER) // checks for EXCLUSIVE access gl.checkFramebufferStatus(gl.READ_FRAMEBUFFER) // checks for READ_ONLY access
Resolution: #4
Issues
Callbacks:
The decision to use callbacks instead of polling is controversial.
One position is polling should not be allowed as it will encourage bad and or wrong programming. Examples:
Spin waiting Bad!
while(!acquire(tex));
Interval based spin waiting: You do a raf in the main thread and exit. You want to know as soon as possible when a resource is acquired so you do this
function render() {
drawScene()...
wakeMeUpEarlyIfIAcquireBeforeNextRequestAnimationFrame();
requestAnimationFrame(render);
}
function
wakeMeUpEarlyIfIAcquireBeforeNextRequestAnimationFrame() {
var check = function() {
if (acquire(tex)) {
clearInteraval(id);
prepScene();
}
}
// execute this check as often as possible
var id = setInterval(check, 0);
}
It also encourages subtle race conditions
// worker.js
self.addEventListener('message', function(e) {
// No check needed as we assume we wouldn't have gotten
// a message unless we first released on the main thread
var buf = e.data;
gl.acquireSharedResource(buf, gl.READ_WRITE);
gl.bindBuffer(gl.ARRAY_BUFFER, buf);
gl.bufferData(gl.ARRAY_BUFFER, getDyanmicData(),
gl.DYNAMIC_DRAW);
gl.releaseSharedResource(buf);
});
// main
var worker = new Worker('worker.js');
var index = 0
var buffers = [gl.createBuffer(), gl.createBuffer()];
function render() {
// ask worker for new data in one buffer
worker.postMessage(buffers[index]);
// render with the other buffer
index = 1 - index;
var buf = buffers[index];
// Don't check because we believe the worker will be ready
gl.acquireSharedResource(buf, gl.READ);
gl.bindBuffer(gl.ARRAY_BUFFER, buf);
gl.drawArrays(...);
gl.releaseSharedResource(buf);
requestAnimationFrame(render);
}
render();
The code above will appear to work until such time as the worker takes more than 1 frame to complete (user launched another app, user opened another tab, another tab or app started doing something heavy). This kind of subtle race condition will be very hard to find and might never appear on the developer's machine, only the user's
Another position is that polling in JavaScript is just plain wrong for the browser. Browsers for better or worse are at the mercy of JavaScript. They can't do any processing while JavaScript is running since JavaScript may be changing the state of the DOM. A polling API that is meant to poll for the completion of some event is antithetical to that issue. Back to the same example as above, this code
while(!acquire(tex));
freezes the browser until the texture in acquired. If it is never acquired the browser never unfreezes (or puts up an alert letting you kill the page).
Even if that situation never comes up, exposing an API a polling api encourages spin waiting like this.
function acquireForSure(resource) {
while(!acquire(resource));
return;
}
acquireForSure(tex1);
gl.bindTexture(gl.TEXTURE_2D, tex1);
gl.drawArrays(...); // uses tex1
acquireForSure(tex2);
gl.bindTexture(gl.TEXTURE_2D, tex2);
gl.drawArrays(...); // uses tex2
acquireForSure(tex3);
gl.bindTexture(gl.TEXTURE_2D, tex3);
gl.drawArrays(...); // uses tex3
Which is very bad for the browser's UX. Providing only callbacks make this kind of code impossible.
Why a new API? Why not just expose OpenGL as is?
Using shared resources in OpenGL is rife with race conditions. Those race conditions are different on every driver and different on the same machine under different circumstances. I seems best to not pass these problems on to web developers.
It's also been a goal of WebGL to as much as possible provide a consistent API. Don't let the differences in drivers leak down into WebGL. One example of this is WebGL's boxing of uniform locations. We wanted to prevent developers from assuming the locations and from doing invalid location math that happened to work on their dev machine but not in the wild.
Some drivers only require a glFlush and a glBind. Others only require the glFlush. Yet others require the spec compliant glFinish, glBind. Still others require a spec non-compliant glFinish, Unbind, and glBind.
Wrapping all of these issues in an acquire/release api lets the browser deal with these cross platform problems and provide a consistent experience for developers.
Why not solve this by requiring WebGL objects to be transferred by transfer of ownership in postMessage?
You'd transfer the object from one thread to another and it would be neutered for the sender.
(1) This wouldn't help 2 contexts in the same thread. You'd need a new method to transfer the object from one context to another in the same thread
(2) There would be no way to have multiple contexts read the same resource. You'd only be able to support exclusive access.
Is this implementable everywhere?
To implement this efficiently requires either some implementation of fences or sync objects. Most desktop drivers have the needed features. For those systems that don't have the needed feature it is suggested to fall back to the slower glFinish/glBind or glFlush/glBind if those work on a particular driver.
Implementation Suggestions:
Emit warning if an acquire takes too long.
Threaded programming is hard. Consider printing a warning to the JavaScript console if a an attempt to acquire a resource takes too long as it probably suggests errors in bookkeeping.