The Mixed Reality Future of Teaching and Learning
The pandemic has accelerated the introduction of engineering design in the virtual world and online education development at the School of Engineering, opening up innovative pedagogies to enhance students’ education experience
Learning how to do an experiment via a virtual laboratory. A collaborative project carried out alongside students in another part of the world yet as if they were in the same room. Advancing 3D virtual design and modeling skills early.
COVID-19 restrictions have brought a host of challenges to university teaching and learning, especially for in-person classes and lab training. However, for Prof. Ben CHAN Yui-Bun, the crisis has also brought 18 months of unexpected opportunities and wider scope to advance the School of Engineering’s pedagogies and boost students’ learning experience and enthusiasm for a career as an engineer.
Prof. Chan is Director of the School’s Center for Engineering Education Innovation (E2I), a unit devoted to exploring frontier education to ensure that engineering faculty and students at HKUST are provided with the latest understanding and resources on learning based on leading engineering education research.
He is particularly excited about the Center’s establishment of Hong Kong’s first mixed reality teaching and learning classroom for incorporating virtual reality-based components and immersive online learning, among other options. The adoption of software promoting collaborative online learning and virtual 3D modeling respectively in selected courses has also spurred student interactivity and design skills.
Mixed reality
The HKUST-designed classroom contains online teaching equipment and an indoor positioning system that allows instructors and students to engage in mixed reality teaching and learning in multiple ways. Some 60-80 students can attend on-site in a regular classroom setting or around 50 if groupwork is required. It can also accommodate remote learners and a combination of face-to-face and online attendees. Highlights of the classroom to date include:
i) A virtual reality recording set-up, including an in-house devised motion capture system, whereby the instructor can record an experiment for students to carry out in a “virtual laboratory”. Students, wearing goggles and using a controller, can touch and use the equipment they encounter in the virtual lab (comprising specially built 3D models of each real-world object needed), then conduct the experiment following the instructor’s lead. Both instructor and students are represented by avatars in the virtual environment. In addition to providing an alternative if physical lab access becomes limited, as during the pandemic, other benefits include repeat experimentation opportunities prior to actual lab work, no use of actual materials, and less equipment safety concerns. A pilot has been created for a geotechnical engineering tri-axial test lab module.
ii) A large-scale curved screen, which together with the indoor positioning system provides 3D stereoscopic display. By tracking the instructor’s head position and orientation, the system creates the illusion the instructor is immersed in the video on screen. This eliminates the feeling that online learners are simply watching a video of the class that face-to-face students are being taught, instead delivering an authentic feeling of interaction in real-time between teacher and students in class and those attending remotely, according to Prof. Chan. The one-course-two-classrooms approach has been tested multiple times with universities in the Mainland, South Korea, and Japan.
Online interactivity
The E2I team adopted the Miro online collaborative whiteboard software platform for use in the School’s common core course, Engineering Solutions to Grand Challenges of the 21st Century, which relies heavily on student participation during class. Miro enables students to work together through their mouse icons. In a class of 40 students, there will be 40 mouse icons using the same Miro board, enabling the teacher to monitor what each student is doing and how they are contributing to groupwork. Students can work in teams discussing their assigned task in Zoom breakout rooms, while at the same time pasting relevant materials and sticker notes on the whiteboard. They can also be “allowed” to see how other teams are tackling their particular tasks, creating a highly interactive class experience. In 2021, due to COVID-19, the course, led by Prof. Chan, was taught online, with lecture notes, pre-class exercises, and class activities reconfigured to fit the Miro platform. The result was “way out of our expectation”, Prof. Chan said, with some activities proving more engaging in Miro than in-person.
3D modeling
Blender, a 3D design and modeling software, was used, together with Unity, an interactive real-time content creation platform, in the Cornerstone Engineering Design Project, an experiential course for first-year students. Prior to COVID-19, the course involved designing and building a real engineering artifact. Over 2020-21, when learning from home was necessary, students were taught instead how to design and build virtual 3D models. The realism and unlimited design capability provided by the combined technologies gave the School’s students a new way to express their creativity and provided a head start in the fast-developing area of virtual 3D modeling. Miro was also employed for the creativity training component of the Cornerstone course, enhancing teamwork.
Better teaching, better learning
Prof. Chan is keen to stress that enlisting these hi-tech tools is not a matter of convenience in difficult circumstances, but driven by the objective of enhanced teaching quality and more impactful learning experience for students.
“We are working very hard trying to craft a different experience, which is even better than if it was face-to-face,” he said. To ensure this is the case, the Center engages in a thorough trial process prior to implementation in a course, involving regular meetings with its members to discuss new options, followed by lectures or workshops to test out potential software and to see whether students are happy with the technology. “This is the process we have gone through again and again and again.”
E2I was established in 2010 and since then the Center has tested out hundreds of different software and educational tools, he explained. Educators and students have had a better experience with a few, which are the ones E2I introduces. Center members are usually the pioneers in putting such a teaching innovation into practice, given the learning curve that adopting a new technology involves. This then allows the E2I instructors to provide training to other educators who subsequently become involved.
While Prof. Chan does not see the virtual lab totally replacing reality-based experimentation and testing at the moment, he notes that technology is moving fast. Two or three years ago, wearing a virtual reality headset for two hours would be an issue, given the weight of the goggles and connecting wires. “Even though you are watching the virtual world, you still feel that this is fake because of the equipment you are wearing. Now virtual reality goggles don’t have wires, they are very light, and you could wear them all day without any problem.”
Technical matters, such as variable internet connections depending on users’ location and equipment, malfunctioning hardware, along with familiarizing instructors with each new technology, still create certain limitations. However, Prof. Chan is confident that solutions will be found and that the possibilities that online and virtual reality-based teaching open up will see the development of a new generation of educational opportunities.
With a return to face-to-face classes, what next?
What Prof. Chan is hopeful will happen at this point is an extension of the types of pedagogies used in courses, with an online learning platform or virtual reality experience incorporated into one or two components of the course, alongside face-to-face teaching to enhance the learning experience.
One beneficial legacy of COVID-19 restrictions is that both students and faculty have become more open and comfortable with the idea of learning in a setting other than a traditional classroom. Rather than a lecture or PowerPoint followed by after-class discussion by students, “everyone actually has some kind of expectation that there should be something more interesting,” he pointed out. “Education can provide a better experience for students when we have multiple settings in the same course.”
With the School, along with the rest of HKUST, set to resume regular face-to-face teaching in the Fall 2021 semester, he is keen to ensure that this momentum remains. “We are extremely active because we know we could easily go back to the old ways. If we change the way we perceive education, we have to do it now, when minds are open to new pedagogies and technologies.”
Trend-spotting and trend-setting
To Prof. Chan, one of the key tenets of being an educator, especially in the engineering field, is spotting when a new development is likely to become a central skill in the future, and ensuring students can be taught it so they are prepared in their future lives.
For example, 3D printing, which would enable students to bring their 3D virtual designs into the real world, was prohibitively expensive until recently. Now the cost of such printers has reduced to a level where it has become potentially feasible.
“I always have in mind that we should keep up with the pace of technology,” he said. “If, for example, you see that in the future 3D modeling and 3D printing are among the skills that students will need, we have to teach them now. Even if it is difficult, we have to start ahead of time. That’s why we have university education.”
