Mixing DOM with WebGL.

[trusktr]

Here's a demo mixing some DOM content with WebGL content using Three.js.

What I'd like to do is the same thing, but in VR. However, this obviously won't work with WebVR because that is purely WebGL.

Unfortunately, the only way to achieve "mixed mode" with DOM and WebGL is probably using the ame technique as yours: split the page into two parts, and stick a phone in a headset, but it won't work with Oculus or similar, unless we make some serious hacks to it that probably won't ship into the app stores.

But, none-the-less, for those stick-in-headset people, it'd be nice to achieve this.

So maybe we can do exactly the same thing as in my "mixed mode" demo, and just render two of them, one for each eye.

But, things will become complicated. What I want to do, for example, is render an Ace Editor HTML-based code editor (or similar) for both eyes.

We'd have to consider:

• How do we focus the cursor on the editor?
• How do we modify both instances of the editors for each eye?
• etc

We can probably do it by using a MutationObserver and mapping all changes from one eye to the other eye, and only one eye has the real instance, while the other one is always just a copy of the real one. This will obviously get somewhat heavy reflecting DOM mutations from one eye to the other.

---

Any thoughts on how to achieve "VR mixed mode"?

[trusktr]

By the way, I'm abstracting this "mixed mode" in my project.

The threejs branch currently allows one to do stuff like this:

        <script>
            infamous.html.useDefaultNames() // defines default element names
        </script>
        <i-scene experimental-webgl="true" id="scene" backgroundColor="255 0 0" backgroundOpacity="0.8">

            <i-mesh
                id="first"
                size="2 2 2"
                rotation="30 30 30"
                has="sphere-geometry phong-material"
                color="255 0 255">

                <p> This is regular DOM content </p>
                <p>TODO throw helpful error when component not used on i-mesh</p>

                <i-mesh id="second"
                    size="1 1 1"
                    has="sphere-geometry phong-material"
                    rotation="30 30 30"
                    color="100 100 100"
                    position="1.5 1.5 0" >
                </i-mesh>

            </i-mesh>

            <i-node id="light" position="0 0 1.6">

                <i-point-light
                    color="255 255 255"
                    id="llight"
                    position="2 0 0">

                    <i-mesh id="light-helper"
                        has="sphere-geometry basic-material"
                        size="0.05 0.05 0.05"
                        color="255 255 255">
                    </i-mesh>

                </i-point-light>

            </i-node>

        </i-scene>
        <script>
            const n = document.querySelector('i-mesh')
            n.rotation = (x, y, z) => [x+1, y+1, z]

            const light = document.querySelector('#light')
            light.rotation = (x, y, z, time) => [x, 180 * Math.sin(time * 0.001), z]
        </script>

At the moment, the spaces are matching: DOM content appears in the top-left as usual, while Three meshes appear in the middle of the viewport.

I'm currently working on the last piece which is mapping Three coordinates to DOM coordinates. This is opposite of my "mixed mode" demo, where the Three CSS3DRenderer maps DOM coordinates to Three coordinates. In that demo, the DOM is aligned with Three content, in the middle of the viewport. I'd like to do opposite: alight Three content with DOM content starting at the top left of the viewport.

[AScriver]

Just took a peek at the project you linked to. Absolutely incredible.