BlockifyVR (Virtual Reality)

[Brackets-Coder]

BlockifyVR

This is a Virtual Reality extension I've been working on since January 27th of 2023.

The end goal is to have:

• Little to no performance overhead with the extension
• Cross-platform compatibility for all major 6DOF headsets A-frame supports
• An easy-to-use system that's coherent and efficient
• Works well with any 3D engine
• Bug-free enough for optimal user experience without frequent disruptions

Here are some things to note:

• It includes a local copy of the A-frame JS library, version 1.7.0, MIT License. This is for VR Management, rendering, and ease of development as I'd have to write the entire thing from scratch using WebXR, WebGL, and the Gamepad API otherwise. Setting up the VR boilerplate and stereoscopy and other things would be a pain without it.
• Gallery thumbnail, documentation, and sample projects are not currently available and will be worked on as I get nearer towards release
• The rendering uses a plane fixed to the camera which directly mirrors the scratch stage with an unshaded texture each frame. This is relatively performant and works well, but there could be better solutions to look into. The plane is placed 1 meter away from the camera and A-frame uses stereoscopic rendering to reduce eye strain. Note that the plane automatically scales to the size of the camera (display) while attempting to maintain the ratio of the stage. I use this instead of drawing directly to the canvas because it won't let me and I would have to re-write the renderer that's already tailored for VR

[hammouda101010]

that looks cool

[hammouda101010]

that looks cool

It looks cool on the flat screen right now, but it's not so great when you try it. You'd probably get motion sickness. I'm still working on it.

sadly i dont think that's possible to stop this.

[Xeltalliv]

Does this show the same scratch 3D perspective to both eyes?

[Brackets-Coder]

Does this show the same scratch 3D perspective to both eyes?

A-frame has stereoscopic rendering built in and I've enabled it in the rendering settings, so no. This fixes some issues with visualizing depth, so the eye offset allows your brain to process the image with more clarity. Even if the matrices aren't used in the renderer, it should properly use the correct eye offset. Why do you ask, just out of curiosity? If you'd like I could add a block that disables or enables this feature, in case the Scratcher wants to build that framework themselves, which I'd doubt.

Keep in mind this is still a work in progress and most of the code is many months old so it's not the most efficient way to do VR.

[Xeltalliv]

Why do you ask, just out of curiosity?

Yes. At one point I tried to extend my AR extension to support AR headsets and VR as well, but the multiple eye rendering has been the main thing holding me back. I was just curious to know how you solved it.

[Brackets-Coder]

Why do you ask, just out of curiosity?

Yes. At one point I tried to extend my AR extension to support AR headsets and VR as well, but the multiple eye rendering has been the main thing holding me back. I was just curious to know how you solved it.

I believe that there can be two XRViewerPose matrices for each eye if available, according to Mozilla Docs. You might want to look into that again. It'd be great if Augmented Reality could support AR headsets such as the Oculus Quest 3 in addition to just mobile touchscreen devices. There should be a views array that can do this. An example can be seen here. Just a suggestion though. I simply think having AR headsets would be a nice update.

[Xeltalliv]

I'll take a look when I have time. Thought it will probably be hard because the only kind of VR I have access to is phone-based 3DOF no controller VR.

Also, do you mind giving some links, so that I can faster figure out where to look?

[Brackets-Coder]

I'll take a look when I have time. Thought it will probably be hard because the only kinf of VR I have access to is phone-based 3DOF no controller VR.

Also, do you mind giving some links, so that I can faster figure out where to look?

As for VR testing, you wouldn't really need a VR headset. I've used Meta Quest's Immersive Web Emulator extension for testing when I didn't care to load up the entire headset.

[Drago-Cuven]

ah yes, I remember seeing a trailer for this on scratch. I wanted to help, but now I can't

[Drago-Cuven]

nvm I think I can
i'm Martinelplayz on scratch btw

[Xeltalliv]

I'll take a look when I have time.

I took a look and have no idea. It's all code specific to AFRAME. I don't know AFRAME. AFRAME is too big and complicated for me to try to debug, especially if the only tool in my disposal is WebXR API Emulator browser extension.

[Thebloxers998]

Nic3

[Thebloxers998]

Nice*

[siskka7]

uld i try the extension?

Let's say no. to

then how to use it ???

[Thebloxers998]

Hey, have u tried it on a vr

[Thebloxers998]

If it's not buggy anymore then you should try to get it to turbowarp

[Thebloxers998]

If it's not buggy anymore then you should try to get it to turbowarp

You're going to have to be patient. Please refer to this comment to see the planned release schedule.

It's still significantly buggy and it will be a few months before I submit this for review, and possibly many more after that before it's on the gallery.

Hopefully it gets added

[CubesterYT]

I'll look at it soon

[CubesterYT]

My bad, I fell asleep yesterday once I got home, I'll look it over today if I don't crash out

[CubesterYT]

First step of review while I'm at work

[CubesterYT]

Refrain from using HTML elements inside your documentation, there's Markdown alternatives.

[Brackets-Coder]

Okay, so what kind of markdown alternative do you suggest for these hover-over definitions?

[Brackets-Coder]

I can't seem to find a similar alternative, but chatGPT suggests doing footnotes

[Xeltalliv]

@CubesterYT There are no Markdown alternatives and Simple3D documentation has also been using those for a long time for the table of contents.

[Brackets-Coder]

@CubesterYT There are no Markdown alternatives and Simple3D documentation has also been using those for a long time for the table of contents.

Yes, the table of contents links are fine, but @CubesterYT was referring to the I used for the 6DOF hover definition

I also thought about doing

Press this to see more!

something like this with a summary in longer, more technical backend descriptions that might be helpful to more advanced users, which isn't supported in Markdown.

[Brackets-Coder]

@CubesterYT the GitHub slash commands preview has the details block as an option:

<details><summary>Details</summary>
<p>



</p>
</details> 

Surely we can make exceptions for this and links?

[Brackets-Coder]

Nothing is fixing this issue 😭 but I did set the texture filter to nearest so it looks crisper now :)

[Brackets-Coder]

I'ma take a brain break for a week or so and come back in a bit

[Brackets-Coder]

Alright I've added a private commit that intercepts console logs and JS errors, then reports them back in Scratch blocks allowing me to "see" the console on the Quest. If there's no errors I'll be very upset and this will just have to be merged with a known visual glitch that can't seem to be fixed: it works perfectly on desktop and the scaling logic is sound....

But I'm expecting some kind of issue, if there's no errors reported then we have a problem, Houston

[Brackets-Coder]

Ya'll I fixed it just gotta clean up some things before I commit

[Brackets-Coder]

One last bug to fix for sure:

• I fixed display scaling, now the size is consistent no matter what the resolution of the stage is. Changing resolution only changes visual quality/sharpness as intended, instead of changing the visual size like it did previously which messed with the perspective perception of the brain (e.g., things have proper scale now)
• The display is black upon entering VR until after pressing the oculus button - this is likely a DOM Update/canvas or texture invalid/Initialization/Timing issue and shouldn't take forever to fix but it's still bothering me

[Brackets-Coder]

I'm going to try a few things to fix the last bug, mainly just deferring the texture assignment with requestAnimationFrame to attempt to prevent invalidation before the frame buffer is ready...

Meanwhile, @CubesterYT I have a few questions before we do official reviews:

1. This extension is fairly large and 98% of it is the library. I didn't want to remove it and use the native APIs directly because of ease of readability and extensibility in later updates. Is 1.2 MB acceptable or should I do what I did with Ammo Physics (3D Physics Extension) #1974 and make a custom build that removes the extras I don't need?
2. Over the past few months I've been attempting to optimize some of the horrible 2-year-old block code...
   return document.getElementById(device).object3D.position[position]; this one line is a lot more readable than the big if-else mess I had before but I've got a few concerns about performance. how bad is it if the DOM is accessed frequently? I was planning on just storing references in a constant object but I never got around to it. I'll probably add finalizing touches after I get this last known issue fixed

[Brackets-Coder]

Does this show the same scratch 3D perspective to both eyes?

A-frame has stereoscopic rendering built in and I've enabled it in the rendering settings, so no. This fixes some issues with visualizing depth, so the eye offset allows your brain to process the image with more clarity. Even if the matrices aren't used in the renderer, it should properly use the correct eye offset. Why do you ask, just out of curiosity? If you'd like I could add a block that disables or enables this feature, in case the Scratcher wants to build that framework themselves, which I'd doubt.

Keep in mind this is still a work in progress and most of the code is many months old so it's not the most efficient way to do VR.

I've determined that this is incorrect and outdated information from a previous period of ignorance about how A-frame works

@Xeltalliv the answer is yesn't. Yes and no. It's fairly complicated: A-frame has no way of knowing what kind of 3D engine the scratch project uses, so there's no true stereoscopy; the scratch stage only renders one view. However, the entire scratch stage is mirrored into A-frame at real time, and the A-frame renderer uses stereoscopy. This makes the flat texture on the flat plane have a very convincing illusion of depth, but it's "fake" stereoscopy.

I was considering adding capabilities to the existing blocks to enable/disable A-frame's stereoscopy settings and return the projection/transformation matrices for the left and right eyes instead of the generalized camera, but quickly realized it would be unfeasible considering the ambiguity of the situation—how would it handle headsets with one display per eye (as with some modern headsets) rather than the traditional one-display two-lenses approach? How would a-frame use each project's unique stereoscopy implementation to render the correct view? Would it always be split down the middle, or would there be an alternate implementation? Simply plugging the stereoscopy directly into the project and having the project render two views might theoretically result in each eye having two views, which clearly wouldn't work, so I decided against it.

Either way, I'm convinced most users would find the built-in stereoscopy solution compatible with any of their projects, easy to use, and convincing enough for most scenarios. While it doesn't handle complex and realistic depth scenarios (like putting your index finger up close to your nose and focusing on a faraway object so that your index finger appears to be seen double), in my testing it's enough that everything appears as expected and there's nothing jarring that would throw users off.

[Brackets-Coder]

Progress update and A Request

I've been working on various optimizations and bug fixes throughout the entire extension, along with preparing it for the final release. Over the past few months I've rearranged the extension structure to more closely follow A-frame's best practices, allowing me to fix a breaking issue and update past version 1.5.0 - it's now on the latest, which provides improvements for my workflow. Additionally, Meta/Oculus seems to have fixed an issue in their browser where the controllers wouldn't connect immediately upon entering VR, but only after pressing the system button. I don't know how other browsers on other platforms behave but overall this is an improvement.

Yesterday I was working on a change to the design philosophy of display scaling. I added a "resolution scale" block (it only accepts values from 0.1-2.0) and refactored the system in the component to set the size of the scratch stage to the size of the renderer times the resolution scale, instead of dynamically deciding what to do based on what size is already present. This makes the scaling much more predictable, easier to understand, compatible with Augmented Reality, and behaves more as the users might expect—and the resolution scale block allows users to control the balance of quality and performance.

@Xeltalliv and @CubesterYT However, there's two tiny issues that need addressing:

1. In implementing the new scaling system as described above, runtime.setStageSize() sets the runtime size of the stage, not the actual canvas size of the stage. This makes going into fullscreen much higher resolution than what is expected, so performance drops significantly and attempting to draw textures larger than the allocated size will result in an overflow error and stop the rendering until the session is restarted. This issue makes some scenarios in the extension unusable. I attempted directly mutating Scratch.vm.renderer.gl.canvas width and height, but that didn't seem to update anything (or it got overriden) Is there a way to set the actual canvas size of the scratch stage at runtime?
2. The display appears black when initializing a VR experience, but after pressing the Oculus button, it scales and behaves as expected (aside from the issue outlined above), correctly managing resolution scale. I'm assuming this is a readiness/timing issue with the scratch stage or the a-frame rendering canvas (e.g., using it before it's ready), but it could also be the material (through I've ruled that out with a forced recompile). ChatGPT is telling me that pressing the oculus button triggers a framebuffer update which makes the canvas ready again, so everything after that event works as expected. I tried calling

        runtime.redrawRequested = true;
        runtime.frameLoop.stepCallback(); // I think this one kind of works because it syncs the framerate with A-frame in the tick function
        runtime.renderer.draw()
        runtime._step();

and wrapping the entire texture logic in a requestAnimationFrame to defer it until it's ready, but none of those work. Is there a way to force an update of the scratch stage to make it ready for drawing?

[Brackets-Coder]

Sorry for writing an essay