In this presentation, we introduce a novel computer vision based attack that automatically discloses inputs on a touch enabled device. Our spying camera, including Google Glass, can take a video of the victim tapping on the touch screen and automatically recognize more than 90% of the tapped passcodes from three meters away, even if our naked eyes cannot see those passcodes or anything on the touch screen. The basic idea is to track the movement of the fingertip and use the fingertip's relative position on the touch screen to recognize the touch input. We carefully analyze the shadow formation around the fingertip, apply the optical flow, deformable part-based model (DPM) object detector, k-means clustering and other computer vision techniques to automatically track the touching fingertip and locate the touched points. Planar homography is then applied to map the estimated touched points to a software keyboard in a reference image. Our work is substantially different from related work on blind recognition of touch inputs. We target passcodes where no language model can be applied to correct estimated touched keys. We are interested in scenarios such as conferences and similar gathering places where a Google Glass, webcam, or smartphone can be used for a stealthy attack. Extensive experiments were performed to demonstrate the impact of this attack. As a countermeasure, we design a context aware Privacy Enhancing Keyboard (PEK) which pops up a randomized keyboard on Android systems for sensitive information such as password inputs and shows a conventional QWERTY keyboard for normal inputs.
Dr. Xinwen Fu is an Associate Professor in the Department of Computer Science, University of Massachusetts Lowell. He received BS (1995) and MS (1998) in Electrical Engineering from Xi'an Jiaotong University, China and University of Science and Technology of China respectively. He obtained his PhD (2005) in Computer Engineering from Texas A&M University. Dr. Fu's current research interests are in network security and privacy, network forensics, and computer forensics.
Qinggang Yue is a PhD student at University of Massachusetts Lowell. He received a master's degree from the Institute of Information Engineering, Chinese Academy of Sciences. His research interests are mobile security and privacy. He loves hiking.
Zhen Ling received the BS degree in computer science from the Nanjing Institute of Technology, China, in 2005, and a PhD degree in computer science from Southeast University, in 2014. He joined the Department of Computer Science at the City University of Hong Kong from 2008 to 2009 as a research associate, and then joined the Department of Computer Science at the University of Victoria from 2011 to 2013 as a visiting scholar. His research interests include network security, privacy, and mobile computing. He loves hacking systems.