Refactor Notes

[KTRosenberg]

Refactor Notes

Smaller Notes

/wasm directory

• For testing WebAssembly integration. (support other languages including C and C++ to take advantage of useful libraries)
• For the class, should probably delete references to this. It's useful for more advanced things. C has typed structures that can be used to fill buffers, which is nice.
• wasm has changed since this was created, so it could probably be done better.
• used in /js/wasm.js, wasm.js likely loaded somewhere.

misc. Delete These

• /js/util/mtt (and any places this is loaded)

misc. Update

• /js/third-party/gl-matrix (Should update to the latest version. It's useful for internal things because it plays nicely with the GPU apis if you want a quick solution. It's used in places. Not to be exposed to students making their own linear algebra graphics)

/render directory

• this version of the project mixed-in a WebXR samples repo, but that repo likes examples that are deeply nested and a bit challenging to follow/integrate. It was useful at the time, but I would strongly recommend pulling-out only the necessary pieces from the third-party code and flatten them / combine them with one renderer path. The most useful piece of the third-party code has skyboxes built-in, as I recall. Otherwise, there's inheritance (extends Node) and jumping around that makes things hard to reason about sometimes. That's unnecessary and arbuably a lot of friction for a simple renderer. Anything labeled with Immersive Web Community Group is the third-party code.

clay.js general

• if you see something like orthogonalVectorComponents

  XR-Course/js/render/core/clay.js

  Line 4572 in 412e9bf

  let orthogonalVectorComponents = (x, y, z) => {

  — the "Components" part is just saying that this function is a version of another function that accepts an array rather than the components e.g. (x, y, z). I created some of these variants because it was useful for flexibility, and I believe creation of arrays appeared a lot in the profiler (but I do not recall). I think it's generally useful to have these variants so you don't have to create an array to pass-in data. This is another one:

  XR-Course/js/render/core/clay.js

  Line 4623 in 412e9bf

  let vertexArrayComponents = (dst, dstOff, p0, p1, p2, n0, n1, n2, uv0=0, uv1=0, r=1, g=1, b=1, wt0=1, wt1=0, wt2=0, wt3=0, wt4=0, wt5=0) => {

  . I'd keep these or provide both. No need to create an array if you already have the data as separate components.

• XR-Course/js/render/core/clay.js

  Line 4665 in 412e9bf

  clayExtensions.init(this, gl, canvas);

  some alternative way to create billboard particles that avoids interfering with other code. Also an example of trying to add extensions to clay in general. I believe this one supported a different data layout than the default and was useful for some examples. (Used for the butterflies and flower I think.)

• XR-Course/js/render/core/clay.js

  Line 4717 in 412e9bf

  let r3 = Math.sqrt(1/3);

  through the bottom. I was trying to support directional and point lights and there's a simple API for it here. I think it ought to exist if the shaders support them.

clay.js textures

• There was a bug in which textures were used before their data were loaded, but to allow display of objects requiring textures before the texture was loaded, I used a basic enum state machine to mark
  whether a texture was loaded.

  XR-Course/js/render/core/clay.js

  Line 515 in 412e9bf

  else {

• XR-Course/js/render/core/clay.js

  Line 570 in 412e9bf

  textureRecord.resourceIdx = (textureRecord.resourceIdx + 1) % textureRecord.resource.length;

  I was also trying to figure out how to reduce the bottlenecks in texture binding. Here, I believe I modulo cycle through a bunch of different textures so when uploading per-frame, it's a different texture being modified than the one from the previous frame. This hopefully tells the GPU driver not to stall on a previous draw before writing to the texture. I don't think this was the largest bottleneck, but this idea of multi-buffering GPU resources in-theory should avoid stalling the GPU.

• XR-Course/js/render/core/clay.js

  Line 510 in 412e9bf

  gl.bindTexture (gl.TEXTURE_2D, textureRecord.resource[i]);

  I recommend supporting a NEAREST texture mode specifically for textures you want to look like retro sprites and pixel art. LINEAR doesn't get the right look.

Overall

• I think the biggest bottleneck here is that there's a weird tight coupling with the Immersive Web Group code, which makes it hard to understand the control flow for the renderer.
• Secondly, GPU resources (textures, buffers, etc.) are used raw for the most part here, which makes it challenging to change the surrounding API. As I've described in side-channel chats, I think the best approach would be to create wrapper objects with IDs and metadata (such as IDs pointing either to triangle buffers or whatever it is you choose to represent other resources). Then the implementation can easily be changed as new features are added or APIs change, or as more kinds of objects are added with different fields, but which could use the same structure. See WebGPU's style: https://webgpufundamentals.org/webgpu/lessons/webgpu-fundamentals.html The objects are defined as nested JS objects with clear names for fields, so they're readable, and you can use or ignore optional fields. For example, using the raw gl.buffer is inflexible to multiple cases that could otherwise use the same wrapper object. You might want a mesh that refers to unique triangle data, or you might want the mesh to refer to another GPU data buffer that contains shared data for instanced draws (like cubes, no need to create multiple triangle buffers with the same cube). Or maybe you want a giant triangle soup and meshes should contain the triangle buffer ID (the actual buffer could be stored in some external data structure), a start offset, and a length. I would suggest that objects in the system also be these nested structures pointing to GPU resource wrappers like mesh, material, etc. Materials can have texture IDs. (I really vouch for using IDs to refer to resources so it's easy to change what resources are referred to and to have multiple objects refer to each other. It's a pretty standard practice to have these IDs or "handles."). The mesh wrapper object can also have different usage flags. e.g. did the mesh change and do the data need to be reuploaded? Is this an instanced object? Are the data static and not-in-need of upload? The idea is that if you wrap the resources, the underlying implementation can change. If you use them directly, you kind of couple with the API, which means more refactors.
• It might actually be a good idea to do a soft rewrite with really simple code. Optimize for simple readable API, flexibility in the underlying implementation, and flatness. Test and improve performance when uploading/changing buffer and texture bindings. Right now, a lot of data are being reuploaded to the same slots, which might cause stalls. Explore using Uniform Buffer Objects and offsets to avoid stalls. Explore the wrapper + ID idea to start migrating towards a more flexible underlying system (and this will be convenient when switching to WebGPU or supporting that in-parallel)

// ideas
let renderableObject = {
    “mesh”: {ID: …, etc.},
    “material”: {
        “texture”: {…},
        “color”: …
    }
};

let mesh = {
    “data” : <triangle data>,
    “resource” : <gl.buffer or wgpu.buffer>,
    …
};

let texture = {
    "texture" : gl.createTexture(),
    "idx" : gl.TEXTURE0 + index,
    "src" : <str name of texture>
};

// creating IDed resources (buffer, texture)
// map numerical IDs to the resource instance
idToResourceTypeX = new Map();
// store human-readable name of the object for debugging
nameToResourceTypeX = new Map();


myMesh = meshMake({ ... object for params to keep API flexible});

//
let nextMeshID = 0
const INVALID_MESH_ID = 0
// reuse IDs when resource destroyed
let meshIDFreeList = []
const meshMake = (args) => {
    let newID = 0;
    if (meshIDFreeList.length > 0) {
        newID = meshIDFreeList.pop();
    } else {
        newID = ++nextMeshID;
    }
    ... // do other things like update the maps... create the resources...the wrapper object
}

// repeat for textures and other kinds of buffers

// define uniform layout via JS object:
let uniformLayout = {
    uniforms : [{
        offset : ...,
        type: ...,
        name: ...,
    },
    {
        ...
     }]
};

// because you can have multiple layouts...
// maybe use this layout object behind the scenes to parse the data and build both webgl and webgpu
// shaders... or to update the data conveniently using JS notation. It depends on how direct you
// want users to be with the underlying API
createShader(uniformLayout)
// use the uniformLayout to update a Uniform Buffer Object with this data at an offset
updateUniforms(uniformLayout, bufferID, offset);
// bind this offset into the buffer representing the data for an object at that offset.
// using a buffer storing data for multiple objects, and binding at an offset then drawing per object
// should in-principle avoid GPU stalls since you're not overwriting data being used by previous draw calls
bindUniforms(bufferID, offset);