Researchers have created the first artificial intelligence model that predicts how people move their eyes and fingers while typing in order to truly understand how people type on touchscreens. The AI model can simulate how any sentence would be typed by a human on any keyboard design. It makes mistakes, detects and corrects them, and forecasts how people will adapt to a new auto-correction system or keyboard design.
Touchscreen keyboards are notoriously difficult to use. We rely on sight to move our fingers to the right places and check for errors because we can’t feel the keys, a combination of efforts we can’t pull off at the same time. Researchers at Aalto University and the Finnish Center for Artificial Intelligence (FCAI) have created the first artificial intelligence model that predicts how people move their eyes and fingers while typing in order to truly understand how people type on touchscreens.
Artificial intelligence that mimics how people type on a smartphone – including making errors – could lead to better on-screen keyboards. It makes errors, detects and corrects them, and also predicts how people adapt to a new auto-correction system or keyboard design.
The AI model can simulate how any sentence would be typed by a human on any keyboard design. It makes mistakes, detects them (though not always immediately), and corrects them in the same way that humans do. The simulation also predicts how people adapt to changing situations, such as how their writing style changes when they begin using a new auto-correction system or keyboard design.
‘Previously, touchscreen typing was understood primarily through the movement of our fingers. AI-based methods have shed new light on these movements, revealing the importance of deciding when and where to look. ‘We can now make much better predictions about how people type on their phones or tablets,’ says project leader Dr. Jussi Jokinen.
The research, which will be presented at ACM CHI, lays the groundwork for developing better and even personalized text entry solutions. ‘Now that we have a realistic simulation of how humans type on touchscreens, it should be a lot easier to optimize keyboard designs for better typing – fewer errors, faster typing, and, most importantly for me, less frustration,’ Jokinen says.
Artificial intelligence and machine learning are two branches of computer science that are closely related. These two technologies are the most popular when it comes to developing intelligent systems. Despite the fact that these are two related technologies and that people sometimes use them interchangeably, they are still distinct terms in many situations.
In addition to predicting how a generic person would type, the model can account for different types of users, such as those with motor impairments, and could be used to develop typing aids or interfaces for these groups. For those who are not facing any particular challenges, it is possible to deduce what type of keyboard, or auto-correction system would best serve a user based on personal writing styles – for example, by noting the mistakes that occur frequently in texts and emails.
The new method builds on the group’s previous empirical research, which served as the foundation for a cognitive model of how humans type. The researchers then created a generative model that can type on its own. The work was completed as part of a larger Interactive AI project at the Finnish Center for Artificial Intelligence.
The results are supported by reinforcement learning, a classic machine learning method that the researchers extended to simulate people. Reinforcement learning is typically used to teach robots to solve tasks through trial and error; the team discovered a new way to use this method to generate behavior that closely resembles that of humans, including mistakes and corrections.
‘We gave the model the same abilities and limitations that we do as humans.’ When we asked it to type quickly, it figured out how to make the best use of these abilities. ‘The end result is very similar to how humans type, but without having to teach the model with human data,’ says Jokinen.
Comparison to human typing data confirmed that the model’s predictions were correct. In the future, the team hopes to simulate slow and fast typing techniques in order to create useful learning modules for people who want to improve their typing skills, for example.
