Usually, the best way to learn a foreign language is to immerse yourself in the environment where many people speak that language. The more exposure, along with the pressure to communicate with people, will help you quickly grasp and practice new vocabulary. But not everyone has the opportunity to live or study abroad.
In partnership with IBM Research, Rensselaer Polytechnic Institute (RPI) – a Troy University, New York – offers students here a novel option in learning Chinese: a virtual environment 360 degrees, take them instantly to a crowded street in Beijing or a crowded Chinese restaurant. Students will have to bargain with roadside stalls, or order food, and the environment is equipped with various AI capabilities to respond to them in real time. Since this is only a trial class, the first course will take place this summer, only for 6 weeks for evaluation purposes.
This project is inspired by two members of RPI, who often use role-playing games to help students learn Chinese. In the past few years, many studies have found that interactive learning environments can help improve your understanding and retention of language. A study published in 2018 also confirmed that learning Japanese in a virtual 3D environment made students better grasp the vocabulary they encountered in the emulator. Based on these ideas, professors have partnered with IBM Research to discover whether they can create their own benefits for their students.
Besides creating digital slides of street scenes around students, the virtual environment also uses a variety of sensors to ensure dynamic compatibility with students' words and actions. Microphone, worn on participants, will transmit their sound directly into the voice recognition algorithm. Cameras will track their movements and gestures to register each time they point to objects or step to different virtual characters. For example, if a student points to a plate of fish in a restaurant setting and asks what it is, a virtual character can respond to that question with the name and description of the plate. This leading technology also allows each virtual character to offer more complex answers to difficult questions (eg, what is the history of the disk?) By taking advantage of the knowledge available. available on Wikipedia (however, the discussion topic is still limited, mainly around the exercises that students need to complete).
Many technologies within the virtual environment are now available on commercial products in the market, paired together by researchers to create a homogeneous experience. But some technologies are developed exclusively for the project. For example, Mandarin has five tones, causing difficulties for many new learners, but is very important in conveying the meaning of sentences. For example, just saying the word "sell" (卖 sharpening) is a bit lighter so you can mistake to "buy" (买 mǎi). Therefore, researchers have created an algorithm for analyzing tonality in students' pronunciation. It compares those tones with native people's tones and points out differences, then provides live audio and visual feedback in a virtual environment. That allows students to ask a virtual character how to say something and immediately practice new vocabulary.
Hui Su, director of the Immigration and Cognitive Systems Laboratory at IBM Research, said the group is still in the early stages of understanding the effectiveness of the virtual environment. But in a test run at the end of 2017, researchers found it helped increase interaction and interest in students' foreign language learning, and helped them quickly learn new words.
For example, before making a food call exercise at a restaurant, students have not been taught how to pay for the food they call, but by watching classmates and discussing with the characters. Virtual, many students have mastered the vocabulary needed to do so. "What a little surprise "- Su said – "One of the students commented that this is an effective way to teach foreign languages ".
In the first year, the new course will use the virtual environment for half of the time, the other half is the normal learning method, but this may change in the future.
If the class proves to be effective in improving students' learning outcomes, it can serve as a role model for other classes and expand into other languages. But it can also be applied outside the university to serve leaders, train state employees, or conduct any other preparatory activities that can benefit from set up context and role-playing.
The ultimate goal of the project will be to support researchers' long-term mission: understanding how role-playing and cognitive environments can impact learning and collaboration, Su said.