Back to our Insights
Back to our Insights

The Ultimate Guide to Touchless Technology

Educational

Clay AIR

A practical guide to touchless technology: the zero UI, gesture-based interfaces, technological and UX considerations.

Over the past year, consumers have become more concerned than ever with physical interactions, hygiene, and the availability of touchless technology in public spaces. 

According to the McKinsey consumer sentiment research, touchless purchases and physical distancing are among the top two priorities for a consumer when they decide to make a purchase in-store. And they are right to have concern. 

These concerns are valid as research shows that the concentration of bacteria found on a public self-serve kiosk is 2000 times higher than on a toilet seat. 

To address consumer concerns and industry challenges, we have developed this resource to act as a guide to touchless technology as a solution for end-users. 

Key Points

In this guide, we will discuss interaction challenges in a post-COVID world, including a definition of touchless technology, and how best to implement it as a solution. 

Part A will Cover the Trend Toward Zero UI Touchless Technology 

1. Touchscreens and Shared Devices: Hygiene Concerns 

Touchscreens and shared devices are raising hygiene concerns among users. We will look at the challenges commercial outlets face with these devices and explore how zero-touch technology can serve as a solution. 

2. Trending Towards a Zero User Interface

Consumers are pushing for touchfree controls, meaning that devices will shift towards Zero UI. This refers to a user interface that is controlled through voice recognition, gesture recognition and biometrics as contactless interaction models. 

3. Limitations to Voice and Biometrics: When do you Need Gesture-Based Touchless Interactions?

Noisy environments and privacy concerns challenge voice control and biometrics as universal control solutions. Across several use cases, we will explore gesture control technology as a multimodal solution. 

Part II Will Cover Guidelines to Implementing Touchless Interaction Technology 

We will look at use cases and user environment considerations for touchless gesture technology and guidelines to choosing the right touchless technology provider for each scenario. 

4. RGB vs Time-of-Flight cameras

In this section we will look at the main differences, pros and cons of RGB and ToF cameras as the most common cameras integrated into smart displays.   

5. Hand Tracking v. Gesture Recognition: Different Approaches of Touchless Technology 

While it is commonly thought that hand tracking technology is the same as gesture recognition technology, they are substantially different approaches. In this section we discuss the difference between these two types of touchless technologies and implementation strategy. 

6. Hardware and Software Bundles v. a Hardware Agnostic Solution

In this section we will explore the pros and cons of hardware and software bundles.  

Part C wil cover User Experience Considerations

7. UX considerations

This includes considerations for the end-user such as provisional UI feedback, best practices, the inclusion of technology partners and accessibility. We will define KPIs for creating a seamless user experience. 


Part ATouchless Technology: Moving Towards Zero UI

1. Touchscreens and Shared Devices: Hygiene Concerns 

Touchscreens and shared devices such as kiosks, self-service counters and interactive displays are present in our everyday lives. Each of these interactions however, is a concern for consumers in a post-COVID world where hygiene is of utmost importance. 

Touchless technology offers seamless integration as a hygiene conscious solution. 

Let’s look at a few examples of touch-based interactions that can be replaced with touch-free interactions. 

Examples of Touch-Based Interactions

Touchless controls can be implemented into a wide range of touchscreens, shared devices and even non-digital interfaces:


Public Spaces

  • Payment terminals
  • Phones
  • ATMs
  • Wayfinding interfaces
  • Interactive displays for public use

Workplace:

  • Light switches
  • Elevator signals
  • Door handles

Transportation

  • Ticket and self-service kiosks
  • Parking meters
  • Bus fares
  • Baggage claim 

Retail:

  • Wayfinding 
  • Checkout kiosk 
  • In-store virtual try-on experience

Restaurants & Hospitality

  • Check-in 
  • Order 
  • Special requests  

right hand controls an object at distance using gesture recognition touchless technology
Gesture recognition through Nreal AR glasses powered by Clay AIR

Touch-Interaction Hygiene Challenges 

Many existing solutions such as sanitizing interfaces between use, implementing anti-bacterial sprays, offering hand sanitizer and UV radiation are not easily scalable and require costly supervision. 

This poses a challenge for interface and digital display manufacturers and users of those displays such as, retail brands, transportation services companies, medical device operators, and many more industries where touch-based controls have traditionally been used. 

Touchless Technology as a Solution for Hygienic Interactions

The one-stop solution to hygienic interactions includes touchless technology such as: voice control, remote mobile app-based interactions, biometrics and gesture controls as Zero User interfaces, which we will cover in the next section.

2. Trending Towards Zero User Interface

With the breakthroughs in image recognition and natural language processing, powered by advanced computer vision and machine learning, we are heading towards what is called “Zero UI” (user interface). 

a person's hand navigates a self service interface with touchless technology and gesture recognition

Touchless interfaces – Clay AIR

Zero UI: Touch-Free Control Interfaces 

Zero UI is a control interface that enables users to interact with technology through voice, gestures, and biometrics such as facial recognition. 

It’s also a part of “invisible technology,” interfaces that are so integrated into our day-to-day lives that users rarely consider the technology behind the interaction or experience. 

Smart devices, IoT sensors, smart appliances, smart TVs, smart assistants and consumer robotics are predominant examples of devices in which Zero UI is becoming increasingly integrated. Control interfaces include natural interaction modes such as voice or gestures.

Multimodal Zero UI: Interacting with Smart Devices Through Voice, Gestures, or Biometrics

The end-user solution for universal safe and seamless interactions calls for a multimodal Zero UI. This presents options for interaction such as voice and gesture controls, and biometric interactions, often co-existing with touchscreen interfaces.

Presenting users with the right Zero UI solution in each use case will improve service, brand image and accessibility by giving users options that they are comfortable with based on intuition and ability. 

3. Limitations to Voice Control and Biometric Interactions, and Benefits of Touchless Gesture Technology  

Voice controls may not function in noisy environments such as public spaces or construction sites.

In the case of biometrics, such as face recognition, these have limited functionalities in terms of user controls and inputs, and raise data privacy concerns. 

Why should brands and manufacturers consider implementing touchless gesture controls?

  • They work in multimodal user interfaces in various environmental conditions
  • They are not as intrusive as biometric controls
  • Their universal comprehension as interactions works across languages and cultures for a global user base. In applications such as retail centers and airports, this significantly improves the UX at scale. 

When should you consider implementing gesture-based interactions? 

Environment where touchless gesture interactions are suitable Example  Solving for
Loud and noisy spaces construction sites, manufacturing…  Voice controls hampered by noise
Enables precise interactions without taking off gloves/gear
Silent environments In a car with sleeping kids  Indiscreet voice interaction 
Where removing gear is not safe Manufacturing, building sites or medical spaces When employees wear protection glasses or helmets, complicating biometrics, audio or touch interactions
Masks that prevent clear audio 
Where users need accessibility  Diverse users Unreliable voice recognition (accents, elderly, women, …) 

Part B – Guidelines to Implementing  Touchless Interaction Technology 

In this section, we will look at various offerings of hand tracking and gesture recognition technology. This includes guidelines to choosing the right technology.

  • First we will look at which hardware is the most appropriate for the business objective of your company, focusing on two types of sensors: RGB cameras, as they are the most commonly found, and Time of Flight (ToF)
  • We will also look at the difference between hardware and software bundles
  • Lastly we will look at the differences between gesture recognition and hand tracking technologies.

4. Hardware Considerations: Comparing RGB and Time-of-Flight Cameras

Choosing the right technology provider involves careful consideration as touchless technology solutions are dependent on a device’s sensors and camera(s), and therefore rely on the OEM’s (original equipment manufacturer’s) specifications. 

Do you need a specific camera to implement gesture controls? 

Factors such as existing hardware, budget and user experience will determine integration specifications. 

Depending on a device’s onboard camera and sensors, some may require a custom gesture control solution. 

Other technology providers will offer hardware agnostic solutions, able to adapt to hardware configurations, including onboard monochrome cameras. 

In this section, we take a look at the two most common cameras embedded on smart displays: RGB and Time-of-Flight. 

Leveraging RGB to Create Engaging Interactive Experiences

Best suited for… Well suited for self-service kiosks inviting the user for a fun and interactive experience in shopping or entertainment centers. 
How it works RGB cameras capture the same wavelength of light that the human eye is sensitive to red, green, blue. 
Pros Common due to their accessible price, found on smartphones
Allow for consumer-friendly features such as autofocus and blur
High-resolution images
Cons Color sensors are intrinsically slower, as it needs to process 3 times the amount of data that a monochrome sensor processes
Frame rate is slower
More sensitive to light and shade, which makes segmentation more challenging than when using IR sensors

RGB cameras can be complemented by a depth sensor, augmenting the traditional image information with depth data. 

comparison between a RGB image and a Depth Map image, output of a infrared sensor
RGB vs ToF image – Source

Time of Flight: High-End Camera for Precision Use-Cases

Best suited for… Environments where high precision interactions are used
How it works ToF cameras calculate distances between the camera and the object by projecting an artificial light (laser or LED) onto the object and measuring the ‘time’ it takes for the light to travel out and back. 
Pros More compact than other setups like stereo vision
Captures more details than other cameras at a faster pace
Cons Using various ToF at the same time may disturb another user’s experience 
Background light can mislead the depth measurement
More expensive than RGB or monochrome cameras at scale

5. Hand Tracking Versus Gesture Recognition: Different Approaches to Touchless Technology 

Above we discussed implementation considerations of the various types of sensors used to capture gestures. 

Regardless of the hardware used to capture images, touchless gesture technologies can vary: from simple gesture recognition to advanced hand-tracking technology. 

While it is commonly thought that hand tracking technology is the same as gesture recognition technology, they are substantially different approaches. 

Gesture-Based Recognition Technology: A Low Battery-Consumption Option for Basic Touchless Interactions

Most gesture recognition solutions are not machine learning based and rather use computer vision filters to differentiate the background from the object’s shape at the foreground. 

This method can work in a controlled environment, or with the use of an infrared sensor (strong background vs foreground contrast).

Limitations include environments where the background is unstable (various colors, moving background) or when there are various objects on the foreground (two hands). 

Gesture-based technologies overview

Applications Practical for situations where a limited number of gestures are needed for simple interactions. 
Example: a binary request such as: validate or cancel, and previous or next page
Pros Low battery consumption 
Cons Not practical for advanced or dynamic user interactions 
The software may have difficulty recognizing gestures that vary slightly from the predetermined gesture pose. 
Higher risk of false positives
Higher risk of user friction
Devices Recommended for devices that require low battery consumption

Tracking-First Gesture Recognition Technology: Universal and Highly Accurate Gesture Recognition for Seamless Interactions 

Tracking-first gesture recognition technology is significantly different from gesture-based recognition technology. 

Tracking-first gesture recognition technology detects the hand, and then places it in a bounding box. At this point, the recognition technology identifies key points of interest, usually located on the knuckles and palm of a hand. 

Depending on the software provider three to twenty-one points of interest are tracked on a hand continuously over the duration of the interaction. 

Gestures are recognized when the points tracked in space create a certain combination and therefore more precise than gesture-based recognition technology. 

Tracking first technologies overview

Applications High performance and safety critical use cases.
Automotive, training, and simulations where highly accurate interactions are integral. 
Pros Advanced gesture recognition for complex interactions.
Highly accurate, even when gestures differ slightly between users.
Self-learning, adapts to users.
Cons In some cases, higher battery consumption (requires more computational power).
Devices Self service kiosks, check-in booths, touchless check-in, wayfinding kiosks, in-car gesture controls, retail, hospitality and service interaction displays and interfaces. 

6. Hardware and Software Bundles vs a Hardware Agnostic Solution 

A hardware and software bundle may offer a compelling package up front in terms of initial deployment and pricing. 

In the long term, codependencies between the hardware and software and other providers may result in a provider lock-in that is harmful to long term flexibility and growth.

On the other hand, a hardware agnostic solution means that gesture control models can be implemented with a variety of cameras and multimodal interfaces.

It is a more reliable solution to deploy as a cohesive experience across multiple devices. This is far more cost effective in the long run, and highly adaptable. 

Hand tracking using Nreal glasses’ monochrome camera

Part C – User Experience Considerations: From Provider Selection to UI design  

In this section we will consider the user experience in relation to the technology used for touchless gesture controls, inclusivity and accessibility design considerations, and the importance of choice for multimodal interactions. 

7. User experience: user’s effort, accessibility and multi-modal interaction design

Minimize the User’s Effort

Technology partners play a key role in designing user experience. A track record in human-machine interaction design, user experience design guideline standards and best practices for usability are important to evaluate ahead of time to optimize the impact of a solution’s implementation. 

This will reduce points of friction during user interactions, and boost overall rating of the experience. 

UX capabilities to look for when choosing your technology partner include:

  • Guidelines on visual feedback, cues and signals to guide a user through a new interface 
  • Awareness and guidelines on the cultural meanings of hand poses and gestures
  • Inclusive technology and UX

Inclusive and Accessible by Design 

Hand tracking and gesture recognition technology is advancing to accommodate individuals who may be uncomfortable with touch or voice interactions. 

All hand colors and sizes, hands with or without accessories, and hands that belong to eldery or youthful individuals should be equally recognized and understood. 

This ability comes down to the practice and approach of a technology provider. Inclusivity and accessibility is deeply linked with machine learning and the data used. As such, a technology provider must work with a diverse range of data to create a solution that works for a diverse body of users. 

Multimodal Interactions: Providing Consumer Preferences 

As discussed above in multimodal touchless control solutions, voice, and gesture controls are solutions that can complement one another. In specific environments, a certain combination of multimodal touchless solutions will provide the best result. 

Choice also provides the consumer with their preferred means of interactivity, creating a meaningful experience to consumers who feel their needs have been addressed with a personalised solution. 

Summary

In summary, interactions in a post-COVID world are leaning towards Zero UI, accelerated by hygiene considerations. 

Touchless technologies for gesture controls is optimal in a multimodal control interface, and addresses consumer concerns with data protection, privacy, and ease of use. 

When it comes to choosing and implementing gesture-based interactions, many factors are to be taken into consideration: 

  • What is your existing technology stack? Do you need retrofitting?  
  • Is the technology partner compatible with your hardware? 
  • What technology is your partner offering? Is the solution power efficient, accurate and versatile? 
  • Is the technology provider offering advice and support on the user experience design?
  • What about post-implementation improvement?

With any additional questions on these guidelines or if you’d like to get this piece in PDF format, don’t hesitate to contact our team here

Product

ClayControl

Enabling next generation interactivity with digital interfaces through gesture controls.