After an incredible first experience at MIT Reality Hack, and particularly after reading Jared’s recent reflection on sixteen years of developing XR at Microsoft, I felt compelled to share a reflection of my own impressions and experiences with XR (cross reality) as a creative technologist entering the space (to hopefully reread sixteen years from now and marvel at how far we’ve come!).
MIT Reality Hack 2023, Cambridge MA
At the hack, my team worked with the HTC Vive Elite XR to create an AR experience where people could add individual memories that transformed a collective memory sculpture based on evolving collective sentiment. While we learned a lot and are continuing to work on the project, it was very important to us to create something that highlighted and made beautiful people’s individual emotional experiences as well as the collective.
Collective Memory Sculpture, Media Lab
In thinking about AR (Augmented Reality), it is useful to revisit the most fundamental question of the space: what type of reality are we augmenting? Or, what parts of reality are we augmenting?
In seeking an answer to this question, I find it helpful to think of very controlled realities like games. I’ve been playing Fortnite recently and in its newest fourth chapter, Fortnite introduced a system that prompts the player, throughout their gameplay, to enable certain augments. Players can even roll for a different set of augments if they are unsatisfied with the initial batch of options.
In Fortnite, an augment grants the player a unique buff that gives them an edge throughout the rest of the match. My personal favorite augments include First Assault which makes it such that the first bullet in the magazine of the player’s assault rifle deals bonus damage, Aerialist which allows the player to deploy their glider at will throughout the entire game, and More Parkour which enables heightened energy regeneration after mantling or hurdling.
Augments, Fortnite Chapter 4
These augments suit my gameplay because I often die quickly, so extra damage on my first bullet is usually the difference between continued gaming or endless spectating. Similarly, Aerialist and More Parkour encourage and reward me for focusing on movement as a fundamental defense mechanism throughout the game. These boosts will ideally make me a more powerful player in the long run.
In game-speak, "buff" means to make more powerful. So in the context of building in XR: what is the unique buff that a certain software or experience provides? In using [x] technology, what aspects of our lives are we making more powerful?
In Jared’s insightful article reflecting on the hyper-growth of XR during the pandemic, he shares how the focus of many experiences were attempted recreations of work-life as we knew it, or as the pre-WeWork era has known it. This was a response to the sudden need to recreate in-person experiences remotely. But by reconstructing the office, in the newest most nimble of playgrounds, new tech was simply giving second life to the beliefs and structures that created the original office: bureaucracy, surveillance, and the guise of productivity.
Frankly, this is not an augment I would enable in my own gameplay.
In the same vein, Facebook’s metaverse was largely a meme because it failed to construct a reality neither similar enough nor different enough from our own. Avatar Zuckerberg introduced the world to the metaverse from a low-poly office that included a big desk, chairs, and large windows. It is striking to think that before there were legs in this infinite world of possibility, there was a desk to sit behind. Why go through the trouble, and headache (literally), of donning a headset only to sit behind another desk?
Metaverse, Facebook
Jared shared this cartoon, which accurately sums it up.
Metaverse Hype Cycle credit Tom Fishburne, see more at marketoonist.com
It is important to take a moment to reflect on what a new form of reality really means. Generally speaking, there are two predominant definitions of reality:
"the world or the state of things as they actually exist, as opposed to an idealistic or notional idea of them" and "the state or quality of having existence or substance"
Existence is pretty much defined by the many; something exists as long as many others also confirm its existence. There are scientific proofs, yes, but for the parts of life that are most human (hope, love, meaning) there are no proofs and yet these concepts are accepted as real because most of us have personal experience with them.
If a tree falls in a forest and no one is around to hear it, does it make a sound? If a desk exists in the metaverse, and no one works from it, is it still a tool for work? If we build a playground in the metaverse, but no one knows how to play, is it still a playground?
In the context of XR, this reminds me of the inherent importance of multi-user experiences. For example, it is because XR experiences are accessible to many of us, individually, as well as collectively and concurrently, that AR/VR is worthy of the R.
As for the second definition of reality, we are called to think about substance. Substance is something effectual, meaningful, and real.
'Meaning' Generated by DALL-E
What makes anything in our lives substantive? Substantive experiences give our lives meaning. From movies to books, to relationships and activities, these are only substantive in the sense that they challenge our principles, encourage us to grow our worldview, and force us to evolve the ways we think and feel. Do the XR experiences you’ve known achieve this?
While I’ve had my fair share of trying on beauty products with AR filters only to feel bad about my own IRL appearance, endless scrolling through apps on my VR homepage only to have a sore neck and sore eyes and the same qualitative experience as scrolling on my phone, and chatting with people in VR chat spaces only to endure meaningless small talk and vague sexual harassment (*exhales*), I’ve also had some genuinely amazing experiences.
When I played Super Hot on the Oculus Quest in the summer of 2020, I was totally captivated by the elastic space-time, so much so that I accidentally punched my partner whose body aligned perfectly with the mesh of a foe, as I forgot he was within the boundary! That’s literally how much I suspended my disbelief. It was so cool to have my motion control the speed of the game; it forced me to pause and move with very precise intention. In playing this game, my experience of movement within my human body was heightened.
Super Hot Still, Super Hot Team
I will never forget my first time using Tilt Brush in VR and feeling a deep sense of awe at the default workspace which was a simple pastel gradient all around me and what felt like as far as the eye could see. Using my whole body to paint within such a beautiful, serene space was a huge part of why I began my digital art journey.
The first artwork I made in VR
As a last example, I have a fun memory of entering a VR cinema space where there were lengthy previews. Only a few people were seated in the theatre, while the rest of the crowd was mulling about. At this time, I didn’t know how to turn on chat features but really wanted to interact with others; this was why I was in a VR theatre! After standing very awkwardly in front of a fellow cinephile, I realized I could shake my popcorn out of its bucket. They did the same, and shortly after we realized we could dump the whole buttery thing out on each other’s heads! Now I can’t think of a better reason to suspend the laws of physics.
As we move forward into this post-pandemic chapter where people yearn for human connection more than ever but also desire to give our collective imagination the freedom it’s always sought, it is imperative that we build, make, and create with our principles close at heart. We are, after all, building new realities that will inform the evolution of our brains and our children’s brains and even, dare I say, sculpt the human spirit as we bravely construct this trans-human world.