Tangible Interaction Design on Multi-touch Display

This dissertation presents technologies and interaction models that combine tangible interaction with multi-touch UI. It describes TUIC, a technology that enables tangible interaction on capacitive multi-touch displays without requiring any hardware modifications. TUIC simulates finger touches on capacitive displays, such as iPad, iPhone, and 3M’s multi-touch displays, using passive materials and active modulation circuits embedded inside tangible objects, and can be used with multi-touch gestures simultaneously. We demonstrate three TUIC approaches on iPads and 3M’s multi-touch displays: passive (2D), active (frequency), and hybrid. In addition, we have extended TUIC to support bidirectional tangible interaction. For object tracking, we can use TUIC tags. For communication, we embed photodiodes into objects, and transmit data to them by programmatically changing the screen’s brightness levels. We also present three novel tangible interaction models for tabletop-size and tablet-size multi-touch devices. First, our multi-display map touring system helps people use figurines to navigate through continuous panorama based on the street view, thus enhance better orientation perception and walkthrough experience. Second, Clip-on Gadgets solve the problem of fingers obscuring the screens and provide haptic feedback by extending the interaction area of multi-touch devices with unpowered physical controllers. Third, Social Toy uses motorized actuators to create ambient social awareness.
Our contributions include providing low-cost and easy-to-build techniques to enable tangible interactions on off-the-shelf multi-touch devices, empowering developers to explore and create diverse TUI applications, and making TUI accessible to end users.