Workweek graphics toolkit

Sevo workweek 2014-11-03 Graphics Toolkit

• gw: Lots of trouble getting CI reliable under XOrg and xvfb. Trying to have true headless with offscreen Mesa [http://www.mesa3d.org/osmesa.html]. Easiest path was through glutin (pure rust replacement for glfw). Handle creating a window, OpenGL context: Mac, Linux, Android. Does input event handling, too. Getting headless up and running / working. Hoping this removes GLFW and gives us reliable rendering. I have push to glutin. We can do a bunch of other cool things with offscreen Mesa: https://github.com/servo/servo/issues/3789

• kmc: Doing this offscreen lets us make Servo portable, via rendering to a buffer that you blit.

• pcwalton: Much easier than gecko's 2... no, 4 layers. (D3D9/D3D10/Basic/CoreAnimation...).

• kmc: 5 compositors?

• pcwalton: Yes. Have to add all layers features to 5 backends.

• gw: Been deprecating the rust-opengles repository. Now, we have repos for EGL CGL and GLES. The GLES ones are generated through the GL generator macro (from bjz) The CGL ones are manually generated, as are EGL, but they will be ported. Gleam has wrappers for most of this content.

• larsberg: Deprecate GLUT, too?

• gw: Not yet, but I need to talk with mbrubeck about it.

• larsberg: I'm in favor of removing the C library dependencies.

• gw: Yes. I will chat with mbrubeck about removing the other parts of it.

• zwarich: How do we know if this is working? Maybe: 1) a single toolkit for all platforms; 2) stable headless rendering.

• gw: Those are the two big ones.

• kmc: Including GPU painting with that as well? Or is it separate? I don't think we test it with OS Mesa.

• zwarich: Part of the rasterization discussion. But if you care that you claim about , you're not using OS Mesa. You can't ship it - it's not quality software.

• pcwalton: We could write a basic library on it.

• zwarich: Would be hard to match performance and power.

• mbrubeck: On the toolkit side, everything we've looked at is from a game-dev style world where there's a full-screen windowed application. It's very different if you start talking about wanting to embed (e.g., using GTK, Cocoa, or Java's Android libraries) where there's just a part that is a web library.

• pcwalton: That's where we should be exposing CEF. Applications want a stable API with a stable ABI and a model where they manage the event loop, deliver events to it (via callbacks) and have some sort of API to render to the screen. CEF provides it well, handling the annoying job of ABI compatibility that we just implement. It does mean we are stuck with their ABI, though.

• mbrubeck: CEF should be our main focus if we want people embedding it. The glutin or glfw/glut/etc. is the alternate frontend for when you want to write a full application.

• pcwalton: CEF has two modes: offscreen (which is not the default); also one that does the window management itself (the default, used by Adobe Brackets). For the latter one, you need window management facility, which maybe glutin will deliver. But it's not used in all cases. If we want to support it. Also useful for the windowed API for CEF. Regarding Android, it should just use CEF.

• mbrubeck: What?

• pcwalton: There's no CEF API yet there...

• mbrubeck: But the hard part on Android is the JNI stuff calling in anyway.

• pcwalton: The C can call into the JNI. We don't have to call from JNI into Rust or C++. It's the Java calling into the C API.

• kmc: We talked in the past about dropping Servo into Fennec Chrome for Android...

• pcwalton: I think we'd do it that way. Have to either duplicate CEF or could have a port that exposes the Gecko API.

• kmc: is that C++?

• pcwalton: No, it's just C. The advantage of that is that it's less work on the Fennec frontend. The disadvantage is that it's an ad-hoc API.

• mbrubeck: The Fennec is just a bunch of C functions and java code all linking together.

• pcwalton: It was just a hodge-podge designed in a hotel room with no plan. CEF has binary compatibility and is basically well-designed.

• kmc: When you talked about poking private Android APIs, that's stuff Fennec already does?

• pcwalton: Yes. One big disadvantage of the CEF offscreen API is that it does not support GPU rendering - it's just a buffer of CPU memory. This support is not good enough for writing a modern browser. Now, you get a callback OnPaint, which gives you an invalid region and an ARGB buffer. We'd add a new OnPaintAccelerate that doesn't give you the region and instead ask you to set up the GL state and call CEF back to paint in the region.

• kmc: To avoid sending textures between GL contexts?

• pcwalton: All of the cross-process stuff is managed by CEF internally. All of the process spawning and stuff is managed by CEF. Little bit of a lie because it calls your OS main with a bunch of different arguments because Servo should be able to spawn what it needs to. I think we should just support that mode and not worry about cross-process stuff at all.

• kmc: We need to share them externally on X, don't we?

• mrobinson: Yes.

• pcwalton: There will have to be some sort of callback where the CEF embedder allows you to create and destroy the surface. Maybe GetXConnection?

• kmc: It's already a thing in Servo somewhere.

• mattspencer: If you're creating a forward-focusing browser, why do you care about X?

• pcwalton: Not trying to be super-awesome on X, but it's nice for debugging.

• kmc: It's neat that you can use Servo as a desktop linux browser today.

• pcwalton: Don't want to give up Linux and X.

• mattspencer: Trying to kill X completely.

• kmc: It's our source of crashes.

• zwarich: Even Gecko shipped a release with software compositing.

• mattspencer: Just don't make arch. decisions around X.

• kmc: And fixing bugs on a linux driver is hard.

• zwarich: I've noticed we've gone from one thing to +CEF and then a CEF with a bunch of modes. Are we just going to have one port of Servo (CEF-based, CEF+glutin, etc.) or will there be a bunch of different Servo ports?

• pcwalton: We've been adopting a WebKit port model. There's a significant question as to whether that's necessary.

• kmc: They have the compositor in the port, too.

• pcwalton: Yes. It's not clear their port model provides us anything over CEF. In theory CEF provides windowing, a viewer mode (replacing GLFW), an embeddable mode where you send it events - though we will have to extend it. That's pretty much everything.

• kmc: CEF and Fennec recovers android

• zwarich: Make a fennec wrapper around CEF so we don't even worry about it in Servo?

• pcwalton: That's the question: Fennec to CEF or Servo to Fennec? We do lose the Servo executable if we go with CEF.

• kmc: I imagine Rust programs won't want to use a C API. So do we have a Rust wrapper for CEF? It seems like we want a Rust API and CEF on top of that.

• pcwalton: You do get ABI compatibility with the C API, which you won't get from Rust.

• zwarich: Anyone embedding Rust in a GUI are already using bindings to a lot of things that aren't in Rust.

• kmc: Yes, it's just weird not to have a Rust API to Servo.

• zwarich: And because we will always be on the unstable Rust, it insulates clients from that requirement.

• pcwalton: My hesitation is (based on Gecko) being worried about making decisions based on CEF and finding out later that it's not used much. XUL and XPCOM were cool in the 90s, but now... ick. CEF is not because Chromium is cool + popular but that a stable ABI and good C embedding API gives real technical value. And it's been the source of a lot of WebKit's success. no reason to invent our own API here.

• kmc: Blink isn't embeddable anymore, right?

• pcwalton: Chromium Content API that is somewhat being used; a few people are trying to use it. But Google says it's both API and ABI unstable.

• larsberg: zmike looked into it and said the author reported that those APIs were not meant to be stable yet.

• pcwalton: WebKit2 is another alternative, but CEF has much more mindshare and handles more graphics variations (e.g., give it a buffer and it renders into it). The offscreen model. It makes it the responsibility of the port to define what you render into.

• zwarich: Still doesn't deal with surfaces, so WebKit2 is not unified.

• pcwalton: Can create a layer manager in your widget code, along with all your GL state. Then you just provide some draw_window callbacks to support it.

• kmc: Strategically, can we try to be the C embeddable browser of choice?

• pcwalton: Definitely an opportunity.

• kmc: Can ship a binary .so for multiple platforms.

• pcwalton: Have to be conservative. e.g., WebKit can't move away from conservative GC...

• zwarich: And we can't use conservative GC because we're moving to a new SpiderMonkey.

• pcwalton: Since CEF is multiprocess (as of CEF3), it may be a good thing. While we can, having some soft ABI compatibility is reasonable.

• mbrubeck: We're not going to be in trouble yet with back-compat...

• pcwalton: Yes. Most important thing to me is having it all be in C without using SWIG.

• tetsuharu: Will you have Windows support?

• pcwalton: Maybe :-)

• kmc: There's cross-platform, and then there's cross-platform+Windows. e.g., are we going to support Direct3D?

• pcwalton: Chromium and Gecko diverged on this question. Gecko uses the native graphics toolkit on whatever platform it's on - so, D3D on Window (9 or 10, depending on what's supported). Also supports OpenGL on Mac & Linux. And Android, also OpenGL.

• kmc: Is D3D9 XP only?

• pcwalton: Yes. We couldn't compile using Rust for WinXP anyway - no condition variables.

• kmc: cool.

• pcwalton: CoreGraphics on Mac

• kmc: Via Skia?

• pcwalton: Not used yet in Gecko. It also Cairo XRender for 2d vector stuff, which bit us on performance later. Chrome has OpenGL everywhere, with ANGLE to translate to D3D, and Skia to translate.

• kmc: SkiaGL on Windows?

• pcwalton: No, software. And that's part of why they don't get as much GL acceleration. That said, my information is a little out of date.

• zwarich: Will this change reduce the set of things we need to support. So, glutin + CEF is what we use everywhere?

• pcwalton: Yes. How this ties in to what Tetsuharu asked, is that sometimes the native APIs give you wins you can't otherwise have (we won some benchmarks b/c of that).

• kmc: Difference between OpenGL & DirectX is vector graphics that can still be accelerated.

• pcwalton: SkiaGL, too, right?

• kmc: You said they win because D3D was better than Skia?

• mattspencer: Isn't Skia being moved away from to Ganesh (sp?). Should be enabled in Android, too.

• pcwalton: Desktop?

• mattspencer: They're trying to do it on all platforms, but mobile-first, since they expect Android and ChromeOS devices. The desktop just doesn't matter too much.

• zwarich: Difference between SkiaGL and Ganesh? Referred to interchangeably?

• mattspencer: Skia has a number of different software reendering backends.

• zwarich: So it might already be what we would be using if we upgraded Skia?

• misc: 2d often done using 3d hardware on the GPU

• zwarich: Typically text is rendered in CPU and a mosaic of glyphs composited on the GPU.

• zwarich: On D3D and Windows anyway, we should be thinking about more deferred-mode APIs (ATI's Mantle, Metal, etc. that are unfortunately not standardized yet). Most Web browser stuff can be implemented in deferred mode. If we wanted a cross-platform D3D+OpenGL abstraction, you'd probably want to do that instead of an OpenGL extension.

• pcwalton: Can try to support all of the native graphics or can try to go OpenGL everywhere, with the advantages & disadvantages you'd expect. Not clear to me what the right answer here is. Some people (bas) are big D3D boosters; some are in favor of OpenGL. Been using OpenGL only for now.

• mrobinson: If Windows is an afterthought, does it make sense to be writing the native API?

• kmc: Just deferred into the far future. But if Servo

• larsberg: Toolkit stuff - how do we do B2G?

• pcwalton: Maybe we should drop the ports stuff and do CEF entirely?

• zwarich: CEF + native surface integration!

• kmc: Yes, probably our own addons.

• gw: When we chatted about that, the main concern is that CEF is pretty big. Even if only for debugging, it's painful to have to go through all of those layers and just have a surface and render to it.

• pcwalton: How big is the CEF C API?

• larsberg: Pretty big. String allocation and lots of other stuff.

• kmc: And the Rust API would be unstable.

• zwarich: We do have more wiggle room in that we have zero external users of Servo today.

• pcwalton: I count 148 symbols in libcef.

• mbrubeck: People who are writing Rust code are not worried about later days.

• ms2ger: I do like having the servo executable around!

• pcwalton: We'd have the CEFSimple window shell. It also gives us the benefit of supporting everything in the web stack simply. Of course, that API is not useful for things like putting widgets in an actual browser with a URL bar, etc. It's good for replacing the target/servo binary that we have today, though.

• zwarich: So, do we do a WebKit-like ports model or some sort of CEF descendant thing?

• pcwalton: I don't think we get much today.

• larsberg: Except not having to download CEF

• zwarich: If we're just implementing the C API, we could just have something in-tree.

• larsberg: I don't want to regress the speed of the Servo rust build process.

• pcwalton: Agreed!

• mbrubeck: Write a rust CEFSimple and cargoize it for us to use

• pcwalton: +1

• kmc: This ties into the green threading task discussion later, since embedders' threads will not be green threads.

• zwarich: Since we will have custom concurrency abstractions, we will need Servo to be on separate threads.

• pcwalton: CEF reserves the right to create as many threads as it wants.

• zwarich: Doing the WebKit ports model is hard to maintain. Especially once you have a different implementation of some core thing for multiple platforms. For the next year or two, we're not going to get a lot of benefit out of this.

• pcwalton: We will have platform/whatever directories.

• zwarich: For rasterization we can discuss it separately (hopefully).

• pcwalton: Yes. GL vs. D3D is separate. CEF doesn't support acceleration, so we can do what we want.

• kmc: And they wouldn't add it because Chromium doesn't use D3D?

• pcwalton: Yes.

• kmc: So APIs might diverge a bit.

• pcwalton: That's why I've been adding some APIs to my version of CEF.

• kmc: Can you send textures between D3D and OpenGL?

• pcwalton: Only in some special verisons of ANGLE.

• kmc: We'd need it for WebGL if we have D3D compositing.

• zwarich: That's quite the future...

• pcwalton: It does something, though I don't know what it is.

• zwarich: Gecko also worries about D3D9.

• kmc: I wouldn't be surprised if there's a CPU/GPU round-trip...

• pcwalton: That would kill the Humble Bundle.

• zwarich: ANGLE is small. Could hack in whatever we need.

• pcwalton: Agreed.