Tackling Technical and Societal Issues of Extended Reality Through MetaHKUST

HKUST is setting out to help shape the digital world of immersive learning that frontier augmented and virtual reality technologies are opening up, as well as alert people to the societal issues that such realms will need to address in their development.

Imagine taking a course in a lecture theater filled with avatars, including the speaker’s and your own. Imagine a class where the professor greets the 100 students present, 50 in person and 50 apparently there but actually located at a campus around 100km away. These scenarios may seem futuristic. However, such total virtuality¹ and mixed reality² environments for education are almost here, with initial trials underway as part of MetaHKUST, the University’s go-ahead extended reality³ initiative.

And this is only the beginning for the engineering and emerging technologies’ venture. By the end of this year, it is expected that the avatars populating HKUST’s 3D virtual classroom will possess the facial features and mannerisms of the students and lecturers they represent, with real-time reactions that echo their human counterparts. The classroom’s look’n’feel will be enhanced and scaled up to accommodate 200 or more users. Within three years, it is planned that HKUST and its recently opened HKUST(GZ) campus in Nansha, Guangzhou, will each be operating a physical classroom, equipped with state-of-the-art sensors and motion-capture cameras for the more technically challenging mixed reality learning environment, as well as sharing use of the virtual reality classroom.

These arrangements will enable students from both campuses to meet each other and learn together in real time without actually traveling between the two locations. It will allow HKUST students, guest learners and lecturers from other institutions to attend remotely. Broadening out from classes, there will also be a digital twin up and running for each campus, with the expectation that MetaHKUST will add convenience and greater efficiency to multiple aspects of University life. Examples range from visiting the Finance Office and accessing advisory services to receiving blockchain-secured diplomas or transcripts in the form of non-fungible tokens (NFTs).

Implementing MetaHKUST turns the “Unified HKUST, Complementary Campuses” vision behind the two campuses’ operations into a working model. Moreover, it becomes possible to identify the real research challenges of building a large-scale teaching and learning meta environment, according to Prof. Pan HUI, one of the international trailblazers of research and development in augmented and virtual realities and a core mover behind the initiative. As one of the first higher education institutions to build and explore the challenges of such a scalable endeavor beyond gaming, it also offers a way for HKUST researchers and educators to play a significant part in shaping the overall metaverse, now at its formative stage, “the very, very beginning”, Prof. Hui said.

Metaverse: a fascinating technical and societal quest

What is the metaverse? The Encyclopedia Britannica definition is a “proposed network of immersive online worlds” where users interact with each other and consume services and goods through virtual or augmented reality technologies, with the issue of how the different worlds and communities will seamlessly interconnect to enable users to move across the platforms (interoperability) yet to be resolved.

What these worlds will comprise, their design, and standards are also undetermined on a collective basis. For Zuckerberg followers, the metaverse is a utopian creation where people work, socialize, and play games. Business people see a burgeoning realm of market opportunities expected to reach US$800 billion in 2024, according to a Bloomberg forecast. For Prof. Hui and his research team, it is a fascinating technical and societal quest, with the potential for more accessible education, greater social equality, and better communication across the world, as well as major issues that need to be addressed related to privacy, governance, and data security, among other thorny areas.

Augmented reality – technology of the future

Prof. Hui, who joined HKUST in 2013 as a member of the Computer Science and Engineering at the School of Engineering, is now Chair Professor of Computational Media and Arts, and Director of the Center for Metaverse and Computational Creativity at HKUST’s new Guangzhou campus. At the Clear Water Bay campus, he is Chair Professor of Emerging Interdisciplinary Areas, and long-time Director of HKUST-DT System and Media Laboratory, a joint arrangement with Deutsche Telekom Innovation Laboratories, researching virtual reality and augmented reality systems, social media, big data, and mobile computing.

His own introduction to different kinds of reality started after his undergraduate studies at the University of Hong Kong (HKU) in the 1990s when the internet was the hottest technology of the day for those in electrical and electronic engineering and computer science. A months-long cycling trip to internet-less Tibet in 2002 ahead of starting his HKU MPhil postgraduate studies made him curious about alternative communication networks when the internet is not working or accessible. This led on to his PhD (2008) at the University of Cambridge exploring short-range radio, human mobility, and centered on bridging mobile and social networks. He then worked at Deutsche Telekom Innovation Laboratories in Berlin, Germany, on cutting-edge cloud computing applications before deciding that augmented reality would be the technology of the future, not only for him but everyone else.

Immersive experiences boost learning outcome

In the past 10 years, Prof. Hui has focused on building high-performance, energy-efficient and scalable software for mobile wearable devices and cloud systems; and devising mobile augmented reality algorithms and systems for immersive data visualization and human-data interaction. His published papers now total more than 450, with over 25,000 citations. He also has 32 European and US patents. The concept of MetaHKUST arose at the end of 2021 when students were tiring of the limitations of learning via 2D videoconferencing technologies and extensively used during the pandemic. Rather than a sense of being outside looking in, classrooms using 3D virtual reality or mixed reality offer students a way to feel “present”. These perceptions are an essential part of the learning experience, with global research already showing that immersive experiences in general – such as in-country language learning – can boost learning outcomes, he said.

Wearing a headset and represented by their avatar, the 20-30 students in MetaHKUST’s November 2022 trial were able to enter the University’s virtual reality classroom remotely, take a seat, move around, and go up to the professor and talk to him or her. “As they don’t see their own avatar, psychologically, they feel part of that environment,” Prof. Hui said. “You can just turn your head and see other students. You can move around. Education users want some self-presence. You feel the experience is better. With videoconferencing, I cannot move to see another person.” In a mixed reality classroom where a real person can be projected, the “immersive presence” will be even stronger, he noted.

Participation from multicultural team and multidisciplinary academics

Such work involves a host of cutting-edge technical challenges for his team of postdocs, postgraduates, and undergraduates, from over 10 different countries (he is a firm believer that diversity sparks creativity and new perspectives after his own multicultural experiences at Cambridge and in Berlin, among others). System architecture for mixed reality classrooms, user interactivity and perception challenges, and latency issues are just some of the topics being tackled.

HKUST academics from a range of disciplines are also participating. These currently include Prof. WANG Yang, Vice-President for Institutional Advancement, who is the leader and driver of MetaHKUST development at the Clear Water Bay campus, Prof. Kani CHEN, Industrial Engineering & Decision Analytics and Mathematics, and Prof. Tristan BRAUD, Integrative Systems & Design and Computer Science & Engineering, both in the School of Engineering. At HKUST(GZ), Prof. WU Jingshen, Vice-President for Teaching and Learning, has been pushing forward the mixed reality classroom construction and development and many students from his Red Bird MPhil Program have chosen MetaHKUST as their MPhil research projects. The aim is to expand the project to more faculty and students at both campuses to accelerate advances.

The broader ecosystem of metaverse development

But technical issues are not the only ones covered in a growing number of publications, and postgraduate theses’ topics alongside implementation of the physical and virtual classrooms of MetaHKUST. The initiative is also spurring the exploration of societal questions arising from the extension of virtual and augmented reality technologies into different areas of people’s lives.

A particularly popular contribution is the comprehensive “All One Needs to Know About Metaverse: A Complete Survey on Technological Singularity, Virtual Ecosystem, and Research Agenda” (2021), a paper available to specialists and non-specialists via the Creative Commons. In one of the first publications of its kind, Prof. Hui and his co-authors consider not only the impact of metaverse-related technologies but the broader ecosystem such a development encompasses: the virtual economy and content creation, social acceptability, security and privacy, trust and accountability, and the behavior and potential immortality of people’s avatars. The paper has attracted more than 500 citations and over 111,000 individual user reads, and brought many invitations from different sectors for Prof. Hui to discuss the metaverse.

Among recent speaking engagements, Prof. Hui was invited to join a panel discussion in October 2022 at the 90th INTERPOL General Assembly. INTERPOL’s membership connects police forces across 195 countries. Its annual assembly drew 1,000 security ministers, chiefs of police, and other delegates from around the world in New Delhi, India, including Hong Kong’s police commissioner. “I’ve never felt so safe!” Prof. Hui said. Activities at the assembly included the unveiling of the first metaverse designed for law enforcement, which is being used for training. As a founding member of the INTERPOL expert group on the metaverse, Prof. Hui will be working closely with law enforcement specialists globally to raise concerns and try to make these realms safe.

Academia to push more on ethical and societal aspects

In providing societal input, Prof. Hui sees a special role for academia in evolving the metaverse. “Sometimes the big tech companies may not have such things in mind when they design their metaverse as they are more interested in the commercial aspects. But as academics, we have to push more on the ethical, societal side. While we focus on technological development of the metaverse, we can also identify what other potential issues there are and voice them as early as possible so that society knows about them.

“In the beginning, nobody knew what was going on with online social networks (OSNs), that people could be so manipulated. If this can happen with OSNs, it can happen in the metaverse. But now we have learned from everything that has happened with social networking and people know they need to be more careful.”

The future that awaits us

Raising awareness in this way is particularly important, for Prof. Hui. An IEEE Fellow, International Fellow of the Royal Academy of Engineering, and Member of the Academy of Europe, he views the development of the metaverse as the future that awaits us. He sees people want convenience, business wants efficiency, and both want communication. The three combined will ensure that as virtual and augmented reality technologies improve, more and more content will become available and incorporated into people’s lifestyles. They will be used to hold meetings, when people sit in restaurants, or to call up information overlaid on locations they are visiting.

“Fifteen years ago, we didn't have smartphone. At that time, people wouldn’t believe that you would use your phone to do all the things we use it for today. Now we can’t go out without bringing our phones. Similarly, people will get used to virtual content,” he said.

It is also why Prof. Hui feels it is imperative at this formative time that it is shaped to be a “better metaverse”. “A metaverse, with respect, accessibility and equality, enabling more global collaboration by providing a platform for people from different regions to communicate, study, and learn immersively,” he said. “A metaverse for good.”

¹ Simulated experience with 3D near-eye displays of an immersive virtual world + user interaction.

² Augmented reality (digital overlaid on the physical) + user interaction.

³ Augmented, virtual, and mixed reality.

This story was originally published in the SENG In Focus magazine (Spring 2023 edition, issue 35, P.12-16).