So, while explaining my father what my 53k Quest 3 is good for (other than gaming), we discussed the example of an experience where you can set paintings on your room walls using mixed reality. And this excited me so much that the next day I built it for him 😉.

Init : Research

I was so damn excited that before sleeping while in bed I searched YT for resources to help me. I basically needed just 2 things : -

1. A passthrough room-wide experience just like the 'First Encounters'. (Ref 1)

2. A way to use my controller's ray to spawn paintings at the hit point of ray. (Ref 2)

Step 1: MR Setup

The Valem tutorial uses Oculus Integration (OI) which is now deprecated. So, I thought I should use my new Udemy MR course that uses the latest Meta XR SDKs. And also seemed to be faster and easier to develop.

So, I set that up but couldn't find some of the crucial prefabs that were used in Valem's. I tried to find the samples on Oculus Developer website and Github where they claimed to have those but not useful. Ultimately I had to go with OI only.

And finally, my MR setup was up and ready for development. Thus, I first followed one of his videos and made a simple ball shooter where balls collided with walls, ceiling, fans, lights, and furniture, including the books kept on my bed!

Well that was an inspirational breakthrough!

Step 2: Controller ray interaction

The main challenge in this project was to get the hit point of the quest controller and instantiate the painting model there. But ref 2 used a smarter or say maybe the ideal way of throwing manual raycasts. And so that made it all pretty easy and straightforward.

Now I was able to spawn paintings everywhere my raycast hit, all over my room. The next thing I tried was to spawn paintings only on the walls. And this was achieved using something called - semantic classification.

But it had one major flaw about which I was always worried - the paintings were not perfectly aligned with the wall, and those behind me were with their front to the wall - the classic VR alignment issue. But this was enough as a prototype to impress my father and flaunt the power of my headset.

Upcoming...

• Hand Tracking for two handed frame pose detection to spawn paintings

• Grab Transformer to set painting scale and position it anywhere in the room

• Spatial Anchors to convert it to a VR Room Decor app with different decoration items such as plants, furniture, etc.

See you in the next one 👋😉

References

1. Valem for MR scene understanding

2. For controller raycast logic