Report on the 2nd expert panel of the 2019 NEC Future Creation Forum
Shared experience in the mirror world:
A future of resonance through experience design
NEC has hosted the NEC Future Creation Forum since 2017. This project that seeks to envision the ideal future has brought together diverse experts from Japan and abroad for discussions. Last year’s forum concluded that divisions at various levels of society are at the heart of many of the challenges we face. As a means of overcoming these divisions, NEC has proposed a future concept of a “Will Resonant Society.” The project continues to pursue its aim of building a society where people can create diverse value as their dreams and ambitions resonate with others.
This year, the forum will host four expert panels on the topics of relationship, experience, value & trust, and learning/unlearning, aimed at deepening the discussion with an eye towards implementing ideas in society.
Click here for the report of the 1st expert panel on the topic of relationship.
The four participants in the second expert panel, titled “Redefining experience from the mirror world,” were game creator Tetsuya Mizuguchi, CEO of Enhance, Inc. and a specially-appointed professor at Keio University Graduate School; architect Keisuke Toyoda, a noiz partner & gluon partner; NEC Fellow Katsumi Emura; and WIRED Japan Editor-in-Chief Michiaki Matsushima, who served as the moderator.
Searching for a new form of experience
Technological advances in virtual and augmented reality (VR/AR), as well as the Internet of Things (IoT) and artificial intelligence (AI), will enable new experiences. These changes in our experiences will not only transform entertainment and creativity through VR games or AR content, but will likely alter our very lives and society as a whole. In particular, as 5G wireless networks spread and a “mirror world” in which the real and digital worlds perfectly overlap becomes reality, the very nature of experience is likely to change dramatically.
The mirror world was first discussed by Kevin Kelly, the founding executive editor of WIRED magazine, in a lecture at the NEC Future Creation Forum held as part of the C&C User Forum & iEXPO 2018. The term describes a vision in which our present reality and a digital world in which everything is digitally described overlap to form a new world. It is believed that the mirror world will become one of the next big things as technology continues to evolve.
Two questions were posed to the panel: “what new experiences will be possible in 2050?” and “how can we realize resonating minds in 2050?” In the year 2050—the time horizon of the NEC Future Creation Forum—what will be the value of experience? Will it be possible to create experiences that induce the resonance of minds that is key to realizing the future envisioned by this project? The discussion among the four experts touched on several hundred years of historical experience, spanning various fields including games, architecture, and cities.
The impact of sharing experiences
In thinking about the first question, “what new experiences will be possible in 2050,” we cannot overlook changes in both the real world and digital realms. Until now we’ve tended to think about the real and digital worlds as existing in opposition, but X Reality (XR) technology, most notably VR and AR, as well as the spread of high-speed telecommunications technology such as 5G means that the gap between the two is growing ever narrower. In this situation, Toyoda proposes a new method for describing the world that he calls “common ground” that will connect the digital and real worlds.
“Common ground will be a platform that allows humans and digital agents such as avatars to interact on equal footing. We might assume that avatar-like digital agents are aware of the real world, but we still have not assembled sufficient data to make that possible. That is why there needs to be a more general-purpose platform wider than individual services and manufacturers.”
Without the creation of a new platform like common ground, it will be difficult to have higher-dimensional experiences in the real and digital worlds. Mizuguchi also points out that looking back on the past, we are now in the midst of a major paradigm shift.
“We are in the midst of a shift from an era of information design to an era of experience design. Since the invention of movable type, information design has accelerated and conveyed information through the use of low-resolution two-dimensional mediums such as letters, pictures, and photographs, but in the future, it will be possible to do higher-dimensional design. Sharing and publishing experiences, and achieving more multimodal expressions will be possible.”
Mizuguchi’s idea of sharing experiences resonates with the “experience net” technology proposed at last year’s NEC Future Creation Forum. Emura recalled thinking “by creating an environment that enables a near-realistic simulated experience, children will be able to think about what they really want to do.” The kind of experience net described by Emura could offer new educational methods in which multidimensional information is absorbed through the sharing of experiences.
Mizuguchi points out that “the boundaries of architecture are likely to expand considerably.” Indeed, if all kinds of expression become multidimensional and the digital and real worlds merge together, the nature of architecture and cities will also change. In response to Mizuguchi's point, Toyoda remarked that “future cities will become complex higher-dimensional entities that we will not be able to draw,” arguing that future architects will need to adopt a perspective beyond three static dimensions.
“If architecture becomes multidimensional, the kind of data necessary to create buildings will also change. For example, when creating a building specifically for VR/AR, someone like Mizuguchi-san will need to be involved in the design. Architectural data of floors, walls, and materials has many potential uses, so managing this data could help accelerate the implementation of new services.”
Designing atmosphere and ambiguity as part of a “moving sculpture”
As architecture becomes multidimensional, it will also highlight qualities and sensations that traditional engineering and physics are unable to manipulate. As Emura says, “when you go to a temple or church, there is a sensation that something is different about the space. But that is not something that is part of the design.” Indeed, historic buildings, particularly religious institutions, possess an aura that is hard to fully understand. Toyoda argues that it is important to accept such ambiguous things in their ambiguous state.
Mizuguchi suggests that going forward the field of architecture should address these atmospheric and ambiguous elements that were beyond its scope in the past. Mizuguchi, who has released various games, installations and other works that focus on “synesthesia” over the years, has long been familiar through his work creating ambiguity and atmosphere. When asked by Toyoda how he designs experiences, Mizuguchi likened the process to making “moving sculptures”.
“From an original multimodal concept, we discuss as we output an idea in various forms, and it feels like we are making a moving sculpture. Sometimes we also change due to feedback from the sculpture. Perhaps it’s a sort of ‘co-evolution,’” Mizuguchi remarks.
If we create sound and visuals separately, it would not be possible to experience the sculpture-like production process that Mizuguchi talks about. Toyoda points out that “sharing could lead to new value.” Indeed, there is value created by sharing a certain task, time, or experience. At the same time, that is also something multidimensional that reflects feedback from experiences in many ways.
Thinking about such difficult-to-describe sensations and atmosphere means rethinking the value of things that we can recognize or quantify and those that we cannot. The information scholar Dominic Chen also happened to argue in the first session that modern society tends to quantify everything, but that quantification can also bridge divisions by making it possible to share pain.
Nonetheless, Emura notes that “what we feel cannot be reduced to numbers,” and that it is also important to recognize the value of things that cannot be quantified or recognized. Toyoda underscored the “need to develop hybrid methods that can combine the potential of experiences that can be consciously manipulated through technology such as XR, with the sense of history and place that only emerges on a subconscious level.”
Efficiency alone cannot make us happy
How will the emergence of experiences involving multimodal and high-dimensional information change our society? At this point, the discussion moved on to the second question, “how can we realize resonating minds in 2050?”
How can we use experiences that are extended through various technologies to expand resonance from “I” to “we” and prevent the divisions that prevail in society? Toyoda suggests that the appearance of avatar-like digital agents would mean that human relationships will become more diverse, but at the same time, that physicality and materiality are becoming more important.
“The value of physicality and materiality may be reevaluated in a different way than today. In some cases, it may be better to meet directly, but in other cases, it could be better to use XR. We will have to foster the ability to choose the right options.”
However, Toyoda argues that we should not use this technology only to improve efficiency. Mizuguchi adds that “simply increasing efficiency will result in experiences no different from today. We must create a new kind of richness beyond efficiency, something that we might call ‘post-convenience.’” Emura responds, “As discussed at the previous session, there is a question of whether we can produce value in a form other than efficiency”.
Additionally, Mizuguchi points out that “humans are constantly seeking something more than just efficiency.” Many people take pleasure in films and literature because they try to incorporate the stories into themselves, and listen to music as a constant source of stimulation. Information and efficiency alone cannot make us happy. That is why physical experiences that span multiple senses, such as fragrance and movement, are more deeply memorable.
Matsushima says, “In cities of the future, we might exchange physical sensations, emotions, and even experiences just like messages, but it would not be good if such a society became closed. We must create cities that can also develop new relationships as resonance instead of just empathy.” Indeed, the issue of efficiency is also closely related to urban design. Toyoda reflects that “our present-day toolbox is too limited.”
“Digital technology should allow us to work with abstract things that cannot be drawn by ordinary means, but there are a limited number of tools that we can use. When tools are limited it means that the correct answer and system of evaluation are pre-determined. This can make us feel stuck. When designing cities, we will need to go outside the traditional toolbox.”
The potential for trust to emerge from shared experience
New technology may make new cities possible, but it is also true that technology cannot be so easily implemented in cities. Overseas, tech giants such as Google and Alibaba are working together with cities to pursue proof-of-concept experiments, but many projects are inviting backlash from citizens. Toyoda argues that this makes the 2025 Japan International Exhibition (2025 Osaka/Kansai Expo) a golden opportunity.
“The Osaka/Kansai Expo is a great opportunity to create a virtual city for half a year and conduct various social experiments. There will be no social risks because the Expo will be dismantled after its finished, nor any opposition from residents. The chance to gather data and know-how should prove to be very valuable.”
This is why Toyoda argues that “we need to create a completely different structure than before.” Mizuguchi shares Toyoda’s perspective, adding that “the 2025 Osaka/Kansai Expo will not be dealing exclusively with tangible things, but for those who visit the venue, it would be nice to have a rich experience combining both tangible and intangible elements.”
Matsushima reminds us of the importance of trust, saying that “even while conducting social experiments as part of the 2025 Osaka/Kansai Expo, in order for the city itself to change, we must think about we can build trust among citizens in the technologies that enable new experiences.” Emura responds to Matsushima that “it seems that trust is connected to the size of the city. We have to think about how to recover something like a trust-based commons.”
Trust cannot be established in only one direction. That's why a platform like common ground that is built up by everyone rather than a few tech giants will become more crucial. Toyota also points out the bidirectional nature of the word “resonance.”
“Empathy alone remains one-directional, even when based on high-dimensional information. However, resonance has a way of influencing in both directions simultaneously. Going beyond 5G in the future, all kinds of information could instantly resonate, making shared experience possible.”
Toyoda helped to wrap up the discussion with his observation that humility may be required in order to develop activities in a wider area in the future. “By being humble, we can accept that there are still a lot of areas that we don’t understand and tools that are yet to be explored. There’s a pervasive sense that there is no more room to grow, but if we have the humility to explore the margins, society could become more dynamic.”
The second expert panel raised several important keywords such as common ground, multimodality, post-convenience, and shared experience, while linking back to aspects of the discussion of the first panel. The four participants suggested the new possibilities of experience in 2050, and at the same time, made clear that updating expressions, business, and relationships for the emerging era of experience is connected to the realization of a “Will Resonant Society.”
The next expert panel will pick up threads of this discussion when it focuses on the question, “What are value and trust in the mirror world?” The NEC Future Creation Forum seems to expand its scope each time, and its destination remains uncertain—perhaps the forum itself is beginning to trigger resonance.