top of page

Multisensory Storytelling in VR

Week 1: Unreal Beginner Crash Course

This week I started by watching the first half of this UE5 crash course tutorial. Here are a few screen shots of what I've worked on so far along with the various tools I'm learning about.

  • creating and editing materials

  • Using the Lumen lighting system

  • downloading Quixel Megascans and importing them into a project

  • creating a cinematic animation

  • adding environmental lights to a level

  • using landscape sculpting tools

  • using Landmass tools along with landmass layers

  • Installing new plugins

  • Using experimental water tools

  • Creating an ocean

  • Creating a lake

  • Creating a river

  • Hopefully this will work in VR soon

Week 2:

This week I finished up the UE5 crash course tutorial and also got to play around with our 3D scans from NYCAP 3D in Mixamo. Here are some of the highlights:

  • Playing with Fracture Tools

  • Using Landscape Tools

  • Using Foliage Tools

  • Using Procedural Foliage Tools

  • Taking an In-Scene Screenshot

Week 3: Playing with a 3D scan in Mixamo

Week 4: Notes On Pearl

Pearl was one of the first experiences in immersive storytelling in VR that I’ve seen and I was pretty enamored with it on first viewing. It’s a short experience, less than 6 minutes, however it felt much longer than. Which is a good thing considering the story jumps years at a time through the experience of our protagonists (a girl and her father) growing up and getting older together. The experience consists of a series of vignettes detailing small moments of their lives, first of the father raising his daughter, then of her coming-of-age and becoming independent.

The environment for this story remains the same throughout each vignette, you are in the passenger seat (both literally and figuratively) inside the family car. It’s not often you think about memories being created in a vehicle, since we don’t typically spend a lot of time in one. Though this particular vehicle carries a lot of weight for this particular story because it serves multiple functions early-on in the story. Not only is it a pragmatic device for transportation, but the car also serves as a home, a safe space, a space for play and creativity. It’s also a device that is a popular metaphor for agency, independence and freedom.

I appreciated how much detail was in each and every pixel in the experience. Which is important because there’s nothing to do other than sit back, listen to the music and look around the interior of the car. Not every scene takes place in the car, some scenes require you to look through various windows to get a sense of what’s happening. This I felt was a particularly nice touch and made the environment feel more real. I love having to look through a rainy or foggy window to see what’s happening, as if I’m an actual passenger in the car, or even the car itself.

Each vignette happens so fast that it’s a “blink or you’ll miss it” kind of experience. At first I was a bit annoyed at how fast time was moving but I believe that feeling to be a core concept to the overall story, that life moves by fast and memories are created just as fast. You don’t always get to pick what’s remembered or for how long. The song playing throughout was so catchy, I watched it a few more times to see if I missed anything.

Overall it was a great story, that felt complete and chock full of details that added depth and breadth to its major beats. There were a lot of details in the car itself as it changed over the years that I wish I could take a moment and look at. I thought it might be nice to have the option after the video to visit the various scenes as a still frame where I could have a bit more time to just look around and even open the glove box up and see what kind of paraphernalia would be shoved in there.

Pearl can be viewed in one of three ways, a VR experience, a 360 video, or as a 2D short film. Because there isn’t a lot of interactivity, I’m not sure Pearl needs to be viewed in VR or 360 video. It’s certainly accessible to anyone in multiple formats, which I appreciate and is important especially if you want as many people as possible to see it. The VR experience puts more focus on the third protagonist of the story (the car), and it actually makes you feel like you are the car. When the car was towed away after breaking down, I actually felt upset that I was missing out on the action - I had become the car at that point and was invested in being an integral part of these lives that ride around inside of me. That sounds weird, I know - but it’s true. Having a fixed camera takes that personal experience out of Pearl and keeps focus on the central characters, removing your agency from the equation. All in all, the VR experience (and to a lesser extent the 360 video experience on YouTube) adds a bit more immersion to the story, but it certainly isn’t a necessity and doesn’t make Pearl any more enjoyable or delightful than it already is.

Week 4: Thoughts On Notes on Blindness

Notes on Blindness was an interesting experience in trying to visualize what it's like to be blind. What stuck out most to me was how well the spatial audio aligned with the visuals of each element of sound, both in the environment and in the animations of the visuals. The footsteps would visually vibrate in sync with the rhythm of the audio which made the visual component of the experience very successful. The 3D shaders that were used made for a very light, ephemeral and monochromatic look for most of the experience. This added unity to each and every visual element in the scenes while removing color from your senses - which thematically ties back to the story.

I appreciated the moments of interactivity where you could look at a specific object, which would trigger the next sequence. It gave you the opportunity to sit with whichever environment you were in for however long you wanted to and take in the ambient sounds and atmosphere. It also meant that you had to be present and aware that you were required to rely on your own sight to control a character who was blind. Because of this you are constantly reminded of your privilege of having access to all of your senses.

The sequence that was the most impactful to me, was when it was raining and we were trying to leave the house. The rain caused such a deafening of all other senses that our character requires to navigate the world, that he felt he was trapped in a void with no way to move forward. That for him was truly a moment of panic and helplessness. During this moment, I was only able to see a pair of footsteps and some small semblance of a landscape while listening to the rain surrounding me. As you look towards the footsteps ahead and begin to move forward, because of the lack of visible environment, I actually started feeling a bit queasy and was experiencing mild vertigo. This feeling of vertigo put me as close as I probably can be during this experience to understanding how he felt at that moment, it was quite powerful. While I don't think the creators of Notes on Blindness intended for the viewer to have a physical response to this scene, it definitely added to the overall experience for me in a very positive and profound way.

Overall, Notes on Blindness is an exquisite use of spatial audio to tell a compelling story, and so far one of the best experiences I've had where the use of audio is the central mechanic. My only minor quip with it was the contrast with the bright blue 3D models and the black background caused a bit of eye strain by the end of it. I found myself wanting to close my eyes at various points to give them a break. But again, only a very minor complaint of an otherwise beautiful experience in VR.

Week 5: Notes on Gloomy Eyes - Scale and Perspective

Gloomy eyes is the first experience I've done in VR that played with scale in a really interesting way, something that would be quite difficult to experience the same way in 2D. I was struck with just how impressive the visuals were knowing that some of those fireflies are literally the size of a pixel on my headset, yet it all rendered beautiful and was incredibly full of detail. I can imagine how masterfully done the lighting was, because most of this experience happens in complete darkness, except for the particular scene you're watching, which is only illuminated by objects in the scene - or very rarely small portions of the sky.

Most VR experiences put you at roughly human size/height, Gloomy Eyes decided it wanted you to be this massive spectator that looks down upon the world, sort of like playing with a doll house. This perspective I thought fit in nicely with the Tim Burton-esque style of the characters and setting.

Even as a simple spectator, Gloomy Eyes manages to pull you into its world by literally surrounding you with it. Each vignette of the overall story transitions from one part of your VR play area to the other, making you rotate your body to watch the next scene unfold. This is all done by dragging your eyes to the next sequence using some sort of object in the previous one (fireflies, roller coasters, a moving sled, etc.). It reminded me of a stage play in that way. The environment was entirely black, except for the spotlight shining down on the main characters in whatever section of the stage you were meant to look at. It's a very effective use of space and manages to uproot any sense of grounding location or time of the story which makes the whole experience feel very ephemeral - which definitely goes with the whole theme of the story itself.

Overall it was a very beautiful story and experience. I will say the resolution on my Quest 2 certainly didn't feel good enough to see some of the more nuanced facial expressions on some of the characters and I imagine I might have missed some depth there because of it. I'm sure the animations were incredibly well done and detailed but I did struggle to see some of that due to the overall scale of the environment and actors.

I do wish we had more VR experiences that played with scale in a similar fashion. I think there's a lot to be explored in creating non-Euclidian environments that utilize transitions of scale, and not just of space.

Week 5: Notes on Playthings: VR Music Vacation

Playthings certainly emanates vacation vibes out of every pixel. It's an incredible lighthearted sandbox experience where you get to use various tools (both realistic and silly) to smack around various foods and play them like a musical instrument. Playthings is a surreal adventure that allows you to spend as much time playing whichever food object you prefer, whether it's a hamburger drum kit or a spiral hotdog marimba.

Overall the experience was very short, I spent maybe 10 minutes playing around and as I'm not terribly musically inclined I felt a bit lost in what I should be doing. There's very little direction, which I think is a positive, however I felt there could be some more effort in guiding you to experiment with various sounds and configuration of the food. At the very end of the experience you are free to travel to each island and continue playing with more food, so why not have a more guided experience in the forefront of the game?

I thought the use of portals to teleport to the various other islands was really nice, I wish more VR experiences used this technique, however as I was playing it on a Quest 2, I found that my apartment space wasn't conducive to the size of the floor required in the game, so it was a bit tricky teleporting myself to and from the various locations. I think using an optional controller based teleportation system would have been a nice addition to the experience.

Notably the most significant part of the game is the sound. I found that the sounds to be incredibly pleasant and musical. The gummy bears for example, had multiple sounds attached to each one, which added a lot of depth to it. I liked that you could physically move the food around with your controllers and it would spring back which made it even more fun and interactive to play.

This game has a lot of what Martin Jonasson & Petri Purho would call "Juice", which just makes for a better overall experience.

Week 6: My Metahuman

Free to download:

Download ZIP • 195.96MB

Week 8: Notes on In the Eyes of The Animal + Giant

I've never experienced anything quite like In the Eyes of The Animal. The particle system and the dynamics with which it changes size and speed felt really natural and fluid in VR. It's incredible how atmosphere and audio can really guide your visual experience. Despite the world being rendered out of round pixels, I felt fully integrated into the environment I was placed in.

I loved being a dragonfly, and even more so being a frog. I thought syncing up the firefly wing movement with the human breath cycle was a brilliant and effective way to connect you more intimately with the animal. The vibration involved in the frog vocalization was particularly satisfying. I will say the subpac never fails to scare me, so once there was an established rhythm to the vibrations and it was a bit more predictable, I was able to settle into the experience a bit better. I was extremely impressed with the owl section where you could control the direction of your movement in space, which would allow you to focus your vision and see the environment more clearly.

Overall it was a great experience, I wish it were longer honestly. I think I could spend at least 10 minutes as each animal flitting about or just sitting there watching fireflies.

I can only imagine how scent would have added to the experience, I would have loved to try this out in an actual forest or with a scent machine - or even that helmet with the forest floor glued to the front.

Contrasted with In the Eyes of the Animal, Giant was really disturbing to me. A lot of that comes from the fact that I had all of the context behind the story. Being able to feel the ground shaking and the and the sound waves of the bombs going off does a great job of injecting real fear while you're trying to focus on the story. It's hard not to put yourself inside of these characters' minds when you're plunged directly into their environment. I appreciated the subtle details littered throughout the basement to ground you into the world and the experience itself gave you plenty of time to look around and explore the space for yourself. The story was told in an effective way that didn't hit you over the head with it's narrative, and you were allowed to slowly but steadily come into the realization of where you were and what was happening.

The actors did a great job considering the challenges of filming with green screens, though I'm wondering how things would have felt if the characters were rendered in CG instead of composited into the game. Giant reminded me a fair bit of the style of old FMV games. I imagine it might have felt too "gamey" if the characters were CGI. Seeing actual people was just as effective, maybe even more so.

There was a point in the narrative that struck me as interesting, when the father said that they should have left and apologized. That was a very quick and short statement that I felt was pretty significant to the story, could their actions have avoided the outcome? Why did they make that decision if there was an alternative choice?

Anyway, I left Giant with more questions and almost crying so I would say the experience was very effective.

Notes on Tree and Showing Tree

I really love showing tree. The content of the work is so very close to my heart and definitely fits within my personal interested for what I want my own work to feel like to others experiencing it. I think the execution of the concept is really tight and succinct. I love the sound design, the forest sounds so rich and alive, the animals were well animated and interactive. It feels great getting to be a home to many animals and organisms as you grow tall and big.

I showed tree to 6 people, including my partner and some friends. I asked them which multi-sensory element they felt was the most impactful to them that elevates the experience. Most people were really impressed with the scent part of the experience and some of them even said that it reminded them of a specific forest they had visited, either recently or in their childhood. My partner particularly appreciated the ceremonial aspect to planting the seed in the pot and receiving one at the end. One person mentioned the heat at the end as being particularly impactful as the fire spread up the trunk of the tree.

I had fun chatting with people afterwards and we ended up sparking some conversations about other VR experiences they had tried with multi-sensory elements. The consensus is we need more of these kinds of VR experiences and that it's difficult to find out about them on those rare occasions. These types of experiences are particularly great for those experiencing VR for the first time because it provides a higher level of immersion and is a peak example of just what the technology can do.

Notes on Tampopo

Wow this movie was wild, I still don't really know what to think about it. I felt the sound design was really excellent, it added a lot of hilarity and silliness to all of the scenes where people are slurping food. I'm pretty sure all the food that was eaten in this film generated a huge amount of noise. It reminded me of my studies in Shanghai when I learned from a Chayishi (tea specialist) that the best way to drink tea is to slurp it. That way the oxygen mixes with the tea, which enhances both the smell and taste. I'm sure that principle also applies to ramen.

I thought the restaurant scenes were very interesting. But I loved the scene where a woman was teaching people how the west eats spaghetti, as it highlights common misconceptions around food between various cultures. That reminded me of this old HSBC commercial. It's a bit of a stereotype but it is interesting to think about how much the act of eating a meal is tied to one's culture, including many aspects that we don't even think about.

HSBC commercial (2007)

Tampopo Spaghetti Scene

My old roommate in college used to work for Ichiran, a ramen shop in the city known for providing solo booths to eat ramen by yourself. My experience with ramen has always been a solo activity. I was either making instant ramen late at night when pulling all-nighters or grabbing ramen at Udon West next to my art school in Saint Marks. To me Ichiran embodies what it was like eating ramen in Japan where the shops are so small that you can really only fit 5 or 6 people in at a time. The goal when you're eating ramen is really just to get in, eat, and get out. There were some stalls I visited that was standing room only. You were basically there for 15 minutes before you paid your bill and walked off, which is just perfect for sobering up after a late night of drinking while heading home. The other food in Japan was so good that we really only had ramen a handful of times and again it was mostly during situations where you just need to fill yourself with carbs. It really is the ultimate comfort food experience.

Overall this movie made me quite nostalgic and I genuinely learned more about how ramen is made. Ramen Misoya is a great ramen shop in the east village that tie their broths to different regions in Japan. The menu talks about which styles and which ingredients are popular in certain towns so you're essentially tasting the food of that particular region of Japan.

That oyster scene though - that was something.

3D Scan to Unreal

I decided to scan our couch in our apartment. It's not a great scan, but the texture doesn't look too bad. We call this couch the god couch because it's the most comfortable piece of furniture we've ever owned. It's the kind of couch that cradles you to sleep and night and is oftentimes more comfortable to sleep on than our bed. Whenever we have a rough night sleeping, the couch never fails to provide a restful nights sleep. We spend a significant amount of time on this couch, it's the center point of our apartment space. It's where we eat, relax, and even work.


bottom of page