Introduction

In recent years, technology has advanced rapidly and terms such as augmented reality (AR) and virtual reality (VR) have begun to be heard more frequently. While VR creates completely virtual worlds, AR enriches the real world with virtual elements. This thesis focuses on extended reality (XR), the combination of AR and VR, and explores how digital objects can improve user experiences.

Skeuomorphism makes digital elements look like real-world objects, giving users familiarity with new technologies. For example, the first cars were designed to resemble horse-drawn carriages, and the recycle bin icon on computers looks like an actual trash can. Such designs make digital interfaces more intuitive and user-friendly.

Thanks to XR technologies such as Apple Vision Pro and Meta Quest, interaction design is moving away from 2D flat screens and into more immersive 3D experiences. However, XR technology is still new and difficult to get used to for many people. At this point, skeuomorphism can help users adapt to XR environments more easily and make these environments more intuitive and attractive.

Research Question

This thesis tries to answer how can skeuomorphism be adapted to create more intuitive and immersive interfaces in XR environments. 

Furthermore, it will also try to find the answers for how can metaphors be extracted from the digital interfaces and be reapplied as skeuomorphic interface elements and how can XR environments benefit from skeuomorphic designs in developing clear and intuitive conceptual models.

An Extended Reality Overview with Affordances

The technology makes our lives easier, but also each new device has more abilities, which makes the interaction with the devices harder. (Norman, 2013)

Technology is moving at breakneck speed, bringing us new gadgets that enhance how we interact with the digital world. As Norman (2013) noted, each new device opens up different ways for us to engage with technology. We’ve come a long way from using simple monochrome icons on pixelated screens to now having advanced wearables. Virtual reality (VR), for instance, has been around since the 1960s (Coburn et al., 2017). Early VR systems were bulky, hard to use, and prohibitively expensive. But now, the costs have dropped, and VR is gaining traction in various fields, thanks to devices like the Apple Vision Pro and Meta Quest. These advancements have propelled VR, augmented reality (AR), and mixed reality into the spotlight. Oculus founder Palmer Luckey even predicts that XR devices might become the most crucial computing platform of the future (Orland, 2014).

XR, which includes both VR and AR, is fascinating because of how it blends or replaces reality. VR aims to substitute real experiences with virtual ones, completely immersing the user in a digital environment. In contrast, AR overlays digital information onto the real world, recognizing and enhancing the physical space around the user (Coburn et al., 2017). The ultimate AR device would make virtual elements seem so real that users wouldn’t even notice the device itself. This creates a spectrum from augmented to fully virtual environments, often referred to as Extended Reality (XR) (Extended Reality XR | Immersive VR | Qualcomm, n.d.).

The question of what constitutes reality is not new. Nietzsche, for example, argued that our senses—what we see and hear—are merely brain constructs (Nietzsche, 1873/1992, as mentioned in Gozzi, 1999). Aristotle identified five main senses: sight, hearing, touch, smell, and taste, but modern research suggests we have many more, possibly up to seventeen (as mentioned in Macpherson, 2011). While we can’t yet replicate all these senses digitally, significant strides have been made, especially with the primary senses. Stereo headphones can effectively simulate auditory experiences, while high-resolution screens with head-tracking sensors create convincing visual environments (Coburn et al., 2017). More complex senses, like touch and smell, are harder to mimic, though there are promising developments. For example, BMW has worked on haptic feedback systems (Horn, 2016), and researchers at Beihang University have developed devices to replicate smells (Basu, 2023).

XR is set to revolutionize our interaction with digital environments, surpassing the limitations of two-dimensional screens on traditional devices like computers and smartphones. These older devices rely on pointers and signifiers for navigation (Norman, 2013), but XR aims to replicate real-world interactions, showing just how limited our current interfaces are (Kolb & Oswald, 2014).

XR Devices

The way we experience visual immersion has evolved significantly. It began with panoramic images like those of the Battle of Borodino, among the first to make viewers feel part of the scene. Later, stereoscopic photographs introduced a sense of three-dimensional depth. In 1960, Morton Heiling's Sensorama, the first head-mounted display, created the illusion of still images, though it wasn't yet interactive. These early innovations paved the way for today's advanced XR technology.

Recently, the first wave of XR devices for consumers emerged. Companies like Samsung and Google leveraged smartphones to create XR devices, while Oculus focused on standalone devices with built-in computing power and custom sensors for more sophisticated interactions.

Today, two major players dominate the XR market: Apple's Vision Pro and Meta's Quest series. The Vision Pro offers a premium, polished experience for a niche audience, whereas Meta’s Quest devices are more affordable with some trade-offs in quality. Although I couldn't test Apple’s XR device, I extensively used Meta Quest 2 and Meta Quest 3.

A standout feature of the Vision Pro is eye-tracking, which is absent in the Quest 3. With Vision Pro, looking at an icon acts like hovering a mouse cursor, and clicking is achieved by pinching your thumb and index finger. The device relies on hand gestures and eye movements for interaction. Another unique feature is a physical crown that lets you adjust the visibility of the real world. Turning the crown can replace your surroundings with a virtual background, providing a transition between VR and AR, similar to active noise cancellation in high-end headphones.

When it comes to interface design, many XR devices still rely on floating flat screens that keep interactions largely two-dimensional. Despite their advanced capabilities, like scanning interior spaces, these features often don’t show up in the user interface (UI) as interactive elements. This means the full potential of these devices isn't being utilized for immersive interactions. 

The UI design for both devices features floating windows, whether the environment is a passthrough view of the real world or a virtual overlay. As discussed earlier, the flat screen interfaces and interaction methods result from technological limitations of the time, preventing natural mapping (Norman, 2013).  However, considering these devices can scan environments and afford interacting naturally with 3D objects, this capability is not really used in the interface design. Interface elements usually appear as floating windows or volumes, but the design remains two-dimensional, with interactions typically involving eye tracking from a distance or trying to point with a hand. Many application interfaces look and feel similar to those on an iPad.

Norman (2013) emphasized that natural mappings in real-world objects reduce the need for explicit memorization, a concept mirrored in XR interfaces. There's potential for reevaluating these two-dimensional interactions in favor of more immersive, three-dimensional experiences influenced by skeuomorphic design.

What is Skeuomorphism?

Skeuomorphism, as defined by the Interaction Design Foundation, refers to designing graphical interfaces that mimic real-world objects in appearance and interaction (What Is Skeuomorphism?, n.d.). This approach leverages familiar metaphors to make new technologies more intuitive, easing the learning curve. While often associated with interface design, skeuomorphism has historical roots. For instance, when ancient Greeks transitioned from wooden temples to stone, they replicated wooden architectural elements in stone (Vickers & Gill, 1996, as cited in Chu, 2023). In more recent history, early cars were designed to look like horse carriages, and the first televisions resembled regular furniture (Norman, 2013).

Skeuomorphism has been very important in the world of digital interfaces for years. It all began in 1981 with the Xerox Star, which introduced the desktop metaphor, making computers feel more familiar to users. Apple improved the idea in 1985 with the Macintosh Finder, transforming everyday office items into digital icons like files and folders (The First GUIs, n.d.). Then came the iPhone in 2007, which was designed to make digital elements look and feel like their real-world counterparts, thanks to the new multi-touch screen. For instance, the calculator app looked just like a physical calculator and even gave audio and visual feedback when a button is pressed. This design approach got more improved with the iPad in 2010, where digital books looked like they were on real shelves, and swiping to turn a page felt like flipping through an actual book, thanks to the new large touchscreen it featured.

Why Skeuomorphism?

Norman (2013) explains that skeuomorphism involves incorporating familiar elements into new technologies to make them more approachable. This approach helps ease users' fears and biases towards new tech. The Medical College of Wisconsin defines cognitive load as the amount of information our working memory can handle (Cognitive Load Theory, 2022). Technologies like smartphones and tablets can overstimulate users, increasing the cognitive load. 

Research shows that while flat interfaces are seen as timeless and simple, skeuomorphic interfaces are rated higher in identity, interest, and familiarity (Li et al., 2014, as cited in Urbano et al., 2022). Prensky (2001, as cited in Kolb & Oswald, 2014) coined the term 'digital natives' for those born into the digital age. Studies by Backhaus et al. (2018, as cited in Urbano et al., 2022) and Urbano et al. (2022) indicate that older adults prefer and perform better with skeuomorphic interfaces, though younger users also show improved results with them.

Given that XR technology represents a significant leap and is new for many users, skeuomorphism can help ease the transition, reduce cognitive load, and make the technology more user-friendly for both older adults and digital natives.

Is it Skeuomorphism?

Earlier, it was mentioned how the best XR experience simulates all human senses seamlessly and how skeuomorphism—making digital objects resemble real ones—helps people adjust to new tech and reduces cognitive load.

True skeuomorphism isn't just about how things look or sound; it's also about how we interact with them. Take Apple's Books app for the iPad, for instance. Many XR apps feature digital objects modeled after real-world items, but they don't always capture the full essence of skeuomorphism because they miss out on natural interactions. XR devices have the potential to change that by creating more intuitive and realistic interactions. To see what true skeuomorphism looks like, I tested different AR and VR apps to explore how well they replicate real-world interactions.

Starting with the app TRIBE, it simulates a DJ set with interactable elements like buttons that click and scratch disks that mimic real ones. However, it lacks hand-tracking, which limits its ability to fully replicate the natural interactions you'd get with a real DJ setup. In contrast, the Djay app on Apple Vision Pro uses hand tracking to enhance its skeuomorphic design, providing a more authentic DJ experience.

Another example is the VR game 'Hand Physics Lab' tested on the Meta Quest 2. One level features a keyboard where your hand movements are tracked, allowing you to press keys just like on a real keyboard, complete with responsive sound effects depending on the pressure applied. This immersive interaction feels natural and is easier to manage compared to a more traditional flat-design keyboard in the game.

In a different level of the same game, players encounter a safe with a dial that must be turned to unlock. Here, the game excels in mimicking the real-life experience of manipulating a safe, from the tactile feedback to the sound and visual fidelity, creating an incredibly lifelike interaction.

Lastly, the augmented reality game 'Cryptic Cabinet' scans your real environment and allows you to interact with digital objects placed within it. This app uses Lidar sensors to enhance the realism, scanning not just the room but also furniture to provide occlusion, which greatly adds to the immersion and realism. The game blends digital elements with your real surroundings, providing a different, more comfortable experience than typical VR. The objects look and feel tangible, and interacting with them feels natural. Remarkably, even after two hours of play, the game does not cause discomfort or cognitive overload.

These examples show how XR applications use skeuomorphism to bridge the gap between digital and real-world interactions, enhancing user experience by making digital environments feel as natural as possible.

How Should Skeuomorphism be Implemented?

So far the research has explored how skeuomorphism benefits various user groups, particularly in XR environments, by replicating real-world objects in virtual settings. While it can enhance user familiarity and comfort, simply duplicating real-world elements into digital interfaces can sometimes clutter them. Thoughtful skeuomorphic design, as seen in the iPad's Books app, not only replicates physical books but also introduces features impossible in the analog world, such as text highlighting and massive storage capacity. This shows how digital environments can transcend traditional limitations.

Skeuomorphism needs to balance familiar design with the unique capabilities of digital platforms to keep interfaces functional and engaging. XR technologies extend beyond visual and audio, allowing objects in virtual realities to defy physical constraints like gravity and space. This opens up creative possibilities that challenge our conventional understanding and enhance user interaction by leveraging the digital realm's boundless potential.

Consequently, skeuomorphism in XR should be integrated with balance, recognizing the inherent qualities of the digital world while simultaneously utilizing known metaphors to improve user comfort and intuition. This careful consideration makes sure skeuomorphism not only corresponds to the physical world but also enriches the user experience by making the most of the vast potential of the digital platform.

Home and Desktop Environment Metaphors

The term „metaphor“ comes from the Greek word metapherein, meaning ‘to transfer’ (Merriam-Webster, 2024). Metaphors connect two separate ideas by applying the characteristics of one to another, playing a crucial role in how we think and communicate (Erickson, 1995; Cila et al., 2014). In design, metaphors help bridge distinct concepts and solve design problems. For instance, in user interfaces like the Macintosh file system, the terms ‘file’ and ‘folder’ are metaphors that help users visualize the digital actions behind the scenes.

It’s easy to confuse metaphors with skeuomorphism, but they serve different purposes. Skeuomorphism focuses on design that mimics the look, feel, and interaction of real-world objects to enhance usability. Metaphors, on the other hand, explain how something works conceptually. For example, calling a digital receptacle a ‘trash bin’ utilizes a metaphor to explain its function, which is deleting files. If that bin is visually detailed to resemble a real trash bin, complete with textures and sounds, that’s skeuomorphism.

Unlike metaphors, idioms must be learned. A prime example is „drag and drop“ in digital interfaces; there's no direct metaphor for this action, but once users learn it, it greatly simplifies their interaction with digital UIs.

Back in 1981, the Xerox Star introduced the desktop metaphor, revolutionizing how users interacted with digital content by organizing it into files and folders much like a real desk. This metaphor was initially tailored for professionals transitioning from physical to digital documents, making the digital environment more intuitive. As computers expanded from office to home use, companies like Microsoft adapted to this change with products like Microsoft Bob, which incorporated home environment metaphors into digital interfaces to attract a broader audience.

Bob, released by Microsoft in 1995, was a standalone app for Windows designed with cartoonish, skeuomorphic elements to mimic a home environment. Users were greeted with an opening door leading to a virtual interior, where household items acted as gateways to various apps like a calendar, address book, clock, and educational tools like GeoSafari. Each app featured a character to help navigate the interface. Despite the immersive design, the limited affordances of the time meant interactions were confined to a two-dimensional screen. This necessitated the use of new idioms and analogies to convey familiar metaphors. According to Norman (2013), once a technology is widely adopted, changing it becomes difficult due to the 'legacy problem'. The same thing happened with Bob, since at that time, it had already been a while since the computers were introduced, people had already been accustomed to idioms and metaphors, and there were no need to introduce new ones that would challenge the legacy ones. However, even though Bob eventually died and became a history, some of the metaphors can still be seen in our current devices. Widgets, such as a clock or a photo frame, can be placed on the 'home'screen, and be customized very similar to a house. 

 The XR devices are very new, and no idiom has actually become an 'idiom' yet. This creates a huge opportunity to invent new ways to interact with the devices. Since current XR devices are more focused toward general audience and will be used mainly at homes, it makes sense to have a home environment and bringing digital metaphors by reevaluating them with skeuomorphism and affordances of XR devices. In order to achieve this, which metaphors to redesign must be selected.

Workshop

There are billions of apps in current devices, and it is simply impossible to include all of them in the research. However, according to Statista, most activities done by smartphone users are firstly communicating with people, secondly mailing, followed by online banking, listening to music, and watching videos. Considering the time and the focus of the project, only these five activities are selected for further research.

 After selecting the activities, a workshop is conducted with 15 people. The workshop was held fully online, and it consisted of two rounds for each question. In the first round, people are given an activity and were asked to draw the thing they imagine when they think about that particular activity. The word 'thing' here was important to force people imagine physical objects. Also, the attendees were asked not to draw any device which already has a two-dimensional home or a desktop metaphor UI, such as phones and computers. 

In the second round, the attendees were asked to imagine themselves interact with the 'thing' they drew, then write their interaction step by step. This helped people not only visualize an image, a metaphor, but to imagine how they actually remember themselves interacting with it.

The results of the workshop have shown that even though there were some examples that the attendees imagined the same objects and similar interactions for certain activities, there were no single answers to associate with one metaphor. Consequently, the idea that the understanding of metaphors and mental models differs from person to person is strengthened by the findings.

Considering the evolution of the home environment metaphors and the workshop results, it was seen that there is not a single solution for everyone, and this versatility aspect should be carried to the project as well. Secondly, contemporary digital metaphors, idioms, and mental models should be reevaluated due to the drastic new affordances provided by modern external reality devices.

Interaction System

As mentioned by Norman (2013), the information required to use a particular object should be available in the world by natural mapping, and by having the natural mappings, memorizing becomes unnecessary, which is also one of the main purposes of skeuomorphism as well. Natural mappings and effortless human-virtual object interaction can only be possible by using only the hands and no controllers. 

It should also be noted that haptic feedback when users interact with the objects is also an essential thing, however, this doesn’t only have to be through the vibrations, but can also be through specific visual and audio feedback. 

Secondly, as mentioned before, there are some possibilities in the XR environments that are not normally possible in the real world and one of them is the fact that the XR device can calculate the position of the virtual object and the user’s hands, and measure the distance in between. This opens up whole new possibilities, and even shutters the understanding of interface.

According to Merriam Webster (Definition of INTERFACE, 2024), an interface is “a surface forming a common boundary of two bodies, spaces, or phases.” This means that the idea of the interface is traditionally very two-dimensional, even if the interaction point is very articulated and three-dimensional. To achieve the natural mappings and instinctive interactions, and also to leverage the affordances provided by the XR devices, a more three-dimensional and proximity-based interaction system should be introduced.

Interspace: Interspace is the name of the interaction system used in this project. It is more proximity-based, and three-dimensional compared to the traditional idea of an interface.

Various sketches were also done to explore the idea deeper and experiment what it affords.

To start the interspace design, some interactions extracted from the workshop results, such as pull, push, and rotate, were reinterpreted in the digital environment. Considering the absence of haptic feedback in XR, and the skeuomorphic intent to recreate the feeling of using a physical object from everyday life, a new feedback mechanism was created using different exhortation methods. Accordingly, each signifier has an interspace surrounding it. When the user's hand enters this interspace, the signifiers adapt to the incoming interaction by changing shape in line with the user's intention using the hand pose. For example, when the user brings their hand closer to a pull bar, the pull bar understands that the user is approaching, and provides visual feedback by physically approaching the position of the hand. This visual feedback is complemented by auditory feedback at the moment of interaction, thus compensating for the absence of haptic feedback. At the same time, feedback is given to the user's intention by changing shape according to the user's hand pose. For instance, if the user approaches a rotate knob used to adjust the volume using only two fingers, the rotate knob becomes smaller compared to the user's fingers and a faster, less precise sound change occurs. However, if the user approaches the same rotate knob using their whole hand, this is interpreted as a more precise and slower interaction, and likewise, the rotate knob grows according to the position of the user's fingers and provides the necessary feedback. Using multiple purposes for the same signifier, it is intended to decrease the UI clutter and confusion.

Later, these ideas were converted into AR prototypes, to better visualize how it works.

Home Interspace

After the interaction system is created, the research continued with the interspace system use case examples and object proposal. I created the objects using the workshop results, and for each question from the workshop, three corresponding objects are created considering the time and the scope of the thesis. However, it should also be mentioned that many more objects can be created with the interaction system, and these are only some of the examples.

Similar to prototypes, the objects are created in XR, since being inside the medium during the design process allowed to have a better understanding of the interior space and the object properties. After the object creation and some prototype trials, a room mapping was created using the Quest 3 device, and the virtual objects are placed inside the interior space.

The „Home Interspace“ project explores skeuomorphism in XR environments, replicating real-world materials and forms in virtual objects. By maintaining interactions similar to their real counterparts and utilizing the flexibility of interspace, the project minimizes visual clutter. The goal is to create a familiar feeling for users, allowing them to understand and use objects intuitively without learning new idioms or mental models, thereby reducing cognitive load.

The project also reinterprets positive aspects of current digital devices and interfaces, such as unlimited storage, in a more traditional-skeuomorphic manner. For example, users can store and browse unlimited music using skeuomorphic elements. They can drag and drop music by physically interacting with virtual cassettes or CDs, with instinctive visual cues like 'ghost' objects guiding them. Similar interactions apply to other objects, like browsing contacts by rotating a knob and attaching them to a phone model, ensuring a seamless and intuitive user experience.

Followingly, interior renders were created using architectural software Sketchup and Enscape, later these interior renders were enhanced using photo-manipulation software Adobe Photoshop. The intention of creating these interior renders was to show the virtual-real amalgamation, and how these objects can actually become a part of the interior spaces and daily lives.

Final Testing

A short final testing was conducted to assess the project's success in achieving its goals and delivering desired outcomes, involving two rounds with five participants, both experienced and inexperienced with XR devices. In the first round, participants performed a task using the Quest 3's native flat interface. In the second round, they experimented with the Skeuomorphic Home Interspace, interacting with the objects as they wished. All testing occurred within the same house.

Later, the participants filled out a form consisting of 5 questions related to the project and their experiences. The feedback was overwhelmingly positive. Participants showed a noticeable mood shift when switching from the flat interface to the skeuomorphic home interspace. In the flat interface, they were frustrated and struggled to navigate, with some wanting to remove the device. However, in the skeuomorphic home interspace, they were more engaged and began exploring and interacting with objects intuitively, even without prior instruction. Despite some objects being unfamiliar in real life, participants found it easy to understand and use them.

Conclusion

The main research question was how skeuomorphism can be adapted to create more intuitive and immersive interfaces in XR environments. The study began by examining the devices and their affordances in the second chapter, 'An Extended Reality Overview,' and continued in the third chapter, 'Skeuomorphism,' by explaining the concept and its implementation in new devices and environments.

The thesis emphasized that skeuomorphism is not a rigid set of rules but a method for extracting and reapplying everyday metaphors using new affordances. Daily metaphors were studied in chapter four, 'Metaphors, Analogies, and Idioms.' Chapter five, 'A Skeuomorphic Home Interspace,' introduced a new interaction system inspired by real-world objects, reimagining these interactions within the capabilities of contemporary XR devices. These new interactions were used to create virtual objects with colors, textures, and conceptual models inspired by their real-world counterparts.

The project uses advanced technologies like colored pass-through augmented reality, hand tracking, and room mapping. These features impressed the final testers and garnered positive feedback. However, being a design project, it faced technical and time limitations that prevented the full implementation of the interspace concept. To truly showcase the project's potential, a more advanced Unity prototype and additional testing are necessary.

In summary, the study underscores the need to enhance interactions and interfaces in XR devices. As XR technology evolves, it offers more opportunities for natural interactions. Incorporating skeuomorphism can revolutionize user interaction and interface design, as evidenced by our workshop findings and final tests. By using real-world metaphors, skeuomorphism helps users transition from flat, two-dimensional interfaces to immersive, three-dimensional experiences. Even those new to XR found it easy to navigate and interact with digital objects in the Skeuomorphic Home Interspace.

Ultimately, skeuomorphism not only improves usability but also sets the stage for more immersive and user-friendly XR experiences, leveraging familiar metaphors within the capabilities of modern XR technology.

Thesis PDF