The establishment of a specially designed mixed reality classroom by the School of Engineering (SENG)’s Center for Engineering Education Innovation (E2I) in recent years has accelerated the integration of the metaverse and other hi-tech educational tools into enhancing students’ learning experience. These innovative components allow students to engage in collaborative and immersive online learning while advancing their 3D virtual design and modeling skills, boosting online interactivity in class experience and stimulating their creativity and inspiration.

The state-of-the-art classroom contains online teaching equipment and an indoor positioning system that allows instructors and students to engage in mixed reality teaching and learning in multiple ways. Among the highlights are (1) a virtual reality recording set-up, including an in-house devised motion capture system, whereby the instructor can record an experiment for students to carry out in a “virtual laboratory”; and (2) a large-scale curved screen, which together with the indoor positioning system provides 3D stereoscopic display. By tracking the instructor’s head position and orientation, the system creates the illusion the instructor is immersed in the video on screen.

In spring semester 2023, a common core course on “Engineering Team Design Experience” (CORE 1200) taught by Prof. Ben CHAN Yui-Bun, Director of E2I and Associate Professor of Engineering Education in the Department of Civil and Environmental Engineering, was conducted in this mixed reality classroom. The course aims to train students’ creativity, problem solving and design skills, and attracted a full quota of 40 plus students who came from different Schools and disciplines. For the first time, the Spatial metaverse platform was used in the course where students were tasked to design and build the 2050 HKUST campus and a new HKUST icon in an interactive real-time 3D virtual environment.

Some students imagined that the 2050 campus will be half underwater and half above the sea. Some envisioned a space campus, where researchers can use wormholes to travel between different planets, while others built a HKUST MTR station to solve the transportation problem. Some even predicted the need of providing baby care facilities by AI robots to help students who are already parents, to tackle the city’s growing aging population.

“By allowing all students to co-create content and give feedback to each other in real time on the metaverse, greater learning effectiveness can be achieved. In addition, the metaverse has no space and geographical restrictions, so it is easier for students to realize their ingenious ideas and unleash their creativity,” he said.

Although the University has fully resumed face-to-face classes, Prof. Chan believes that some online pedagogies advocated during the pandemic are still applicable to physical classes to improve students’ engagement and maintain their interest in class.

Prior to COVID-19, the CORE 1200 course involved hands-on experience of designing and building a real engineering artifact. During 2020-22, when learning from home was necessary, students were taught instead how to design and build virtual 3D models. Combining 3D design and modeling with a real-time virtual platform does not only enable students’ work to be shown in real time, hence allowing instructors to better understand their progress and give advice, but also offers a greater space for students to interact, exchange ideas, and express themselves freely, so as to boost their creativity and inspiration.

“Without restrictions on time or locations, the virtual platform allows us students to update our design anytime and anywhere. We can also find inspiration from our classmates’ work – which will keep changing and give the virtual campus a fresh look every once in a while when we go inside,” said Pauline KWOK Sze-Ling (Year 3, BEng in Electronic Engineering), a student of the CORE 1200 common core course in spring 2023.

Another student in the course, Eva XIE Sze-Wing (Year 1, School of Engineering) developed her interest in 3D design and modeling when she took the Cornerstone Engineering Design Project course in fall 2022, an experiential course for first-year engineering students. She said the virtual classroom in CORE 1200 provided unlimited space for her to design and create and allowed her to express her creativity freely.

“CORE 1200 offers us lots of opportunities for co-creation, collaboration and discussion from the beginning to the end, which is quite different from other courses that emphasize more on individual work. Throughout the process, we are able to see our ideas and imagination visualized in real time in the metaverse and can make revisions and adjustments together – which make the learning process highly interactive and fun,” Eva said.

Currently most of the learning experiences that adopt the metaverse are facilitated in a virtual setting by using a VR headset and controller, yet Prof. Chan believes the potential of metaverse in education is far greater than this and huge opportunities are yet to be tapped.

“The metaverse, which allows students across the world to co-create and interact in an immersive environment in real time, will form the trend of future education. In the future, people will spend more and more time in the virtual world and learning on the metaverse will be more personalized. Students can acquire different knowledge according to their own interests,” he said.

In addition to the use of Spatial, in one of his research projects, Prof. Chan uses Unreal Engine (a 3D computer graphics game engine) to develop a VR system to simulate landslides at HKUST. Working in collaboration with the Geotechnical Engineering Office of the Civil Engineering and Development Department, HKSAR Government, the project enables users to experience the natural hazard in an immersive environment, challenge them to think of solutions, such as building protective walls, and then simulate their effects.

With generative AI technology making waves around the world, Prof. Chan has been quick to leverage the tool in his class. These include having his students to use ChatGPT to summarize the meaning of certain terms, and then to teach other students, as well as using Midjourney to generate images that help students to visualize their concept during presentations, so that others can understand it more easily.