Human-Machine Interfaces (HMI)

Human-Machine Interfaces (HMI) is used extensively in the cabs of heavy-duty OEM equipment both on and off-road, and allows a person to interact with a machine, like an instrument panel, the joystick, the steering wheel, the brakes, the knobs and buttons of the infotainment system, Etc.

HMI interactions can be a visual signal, an audible signal, how the machine communicates to a person, or how they communicate back. This includes all the different Advanced Driver-Assistance Systems (ADAS) and safety systems that beep, flash, vibrate, or otherwise inform or warn the operator.

Due to these advancements, the cab has gotten busier with more HMIs communicating information to the operator. This has led to sensory overload — where too much information from numerous HMIs creates distractions, leading to a decrease in safety.

As these machines have become more complex, the amount of information generated has increased significantly. For a while, the trend has been to provide as much information as possible to operators — stemming from the idea ‘more information is best.’ However, over the last decade, a widespread recognition that more information is not necessarily better has grown.

The industry has begun to recognize that instead of continuously providing all available information, they must transition to offering systems that intuitively separate and communicate helpful information to the operator. That way, the operator understands what they need to know when they need to know it.

More than ever, our tools are connecting to computers, digital systems, and other devices more to automatically carry out tasks. Yet, for these advanced technological solutions to be successful on heavy-duty mobile machines, they must be intuitive and avoid information that merely distracts users.

Simply because information is available, that doesn’t mean it’s the best idea to provide it to the operator. Too much information from numerous sources could cause information overload and distraction. If there is an information overload, operators can’t focus on what they’re trying to accomplish, or the most critical information to avoid an incident. This is why it’s so vital to place the right information front and center, both visually and audibly.

The issue at hand is there are already too many distractions in the cabs of most mobile equipment on the road, like infotainment systems, GPS navigation, lane-keeping systems — the list goes on. This is why the cab’s HMIs must provide accurate information to the operator only when needed or useful to the operator.

What’s Needed for HMI Success

Regardless of the HMI, its purpose is to provide useful information to aid the operator in their task. This means that the equipment and user must communicate and interact smoothly with minimal distractions when it comes to HMI success.

However, new advances in in-vehicle technologies and rapid deployment of these advanced systems in vehicles already on the road have left questions. Questions that AAA investigated in their “Understanding the Impact of Technology” 2019 study.

The AAA study found that people who regularly use ADAS technologies are more likely to be distracted and overly reliant on the very solutions designed to improve safety and driver-awareness. With nearly 93 percent of all new vehicles equipped with advanced driver-assistance features, like lane-keeping, cruise-control, distance tracking, and GPS navigation functions, drivers are now twice as likely to be distracted.

According to AAA’s study, the most distracting infotainment systems can take a driver’s attention from the road for an average interaction time of 7 – 31.4 seconds, like using GPS navigation or using the entertainment system. That means drivers are 22 – 41% more distracted while using interactive ADAS systems while on the road.

This over-reliance happens once drivers move from the “novelty phase” to the “post-novelty operational phase.” This is because people begin to adapt their behaviors, and over-reliance and too much trust in the systems automation features start to develop.

Researchers at AAA collaborated with Virginia Tech Transportation Institute and found this to be true, “These drivers are more apt to take their eyes and attention away from the road. Research in other industries shows that pilots and nuclear technicians demonstrate similar patterns of over-reliance on automated systems. These behaviors can eventually lead to distraction.”

As noted, over-reliance is another potential issue caused by providing too much information to the operator. As long as operators are expected to play an integral part of operations, they must be kept engaged. For example, if an operator could perform almost all of their job without looking out the window, that would be an inevitable issue. The operator can only almost do the task at hand due to disengagement.

Even at Level 3 autonomy, the vehicle can only operate most of the time without any driver interaction. It can drive on its own, but it’s not fully-autonomous, so it will miss things. This means the driver must be available to take control of any given situation. Drivers have to be ready to take evasive action immediately. 

Although the human component is vital to safely operating equipment up to Level 3, human nature continues to be a weakness. So, not only are humans psychologically hard-wired to either be engaged or disengaged — but HMI displays and ADAS solutions have proven to have a direct causality with over-reliance.

Developing Solutions to Further Advanced Technology Trends

In the past decade, operational and business needs have brought on exciting HMI developments — including quite a few new capabilities. Today, there is a lot of information and interactions crossing between these new HMI capabilities. It’s becoming more common to see evolved forms of HMI in the heavy-duty mobile market, and these modernized interfaces are creating more opportunities for equipment interaction and analysis.

Still, recent research has led the industry to acknowledge they need to consolidate HMIs to minimize distractions. One idea has been to focus on the combination, creating a common space, where a single HMI or system does a myriad of tasks. 

Instead of having several HMIs — like camera monitoring systems (CMS) with a 360-degree view, sounds alerting to lane changes, and another HMI in the corner beeping when the vehicle veers — competing for the operator’s attention, you can consolidate these HMIs. With a single intuitive and consistent HMI telling operators what they need to know, when they need to know it, this would successfully remove the clutter from the dash in the cab and decrease the level of distraction.                                                                                                                                                               Image courtesy of Xite Solutions North America                       There is work being done in the industry to minimize HMIs through common space; they have acknowledged it is essential. One example is standardizing common alerts — where all sounds from any HMI alerting to a single issue are the same. But it must be intuitive and has to help the operator tackle the task at hand.

The progression of HMIs in the cab follows a bell-shaped curve. For a long time, the industry focused on offering more information, going up the curve. Today, there has been a broad recognition that now this information needs to be filtered. Soon we will see the bell curve turn back down as OEMs and HMI manufacturers provide less information by providing the right information at the right time for the operator.

Avoid Sensory Overload via Threat Analysis

In most cases, current technology is a vessel for information transfers and can cause a sensory overload. From the early 1900s to the present, this transfer of information has gone from a trickle to a fire hose — today, more information is thrown at people in a single day than our grandparents got in their entire lifetime. Which poses the question: How do we prioritize all this information and pay attention to what really matters? 

First and foremost, it’s essential to be cautious about overloading anyone with irrelevant information, especially operators driving on roads and near personnel and VRUs. Operators of heavy-duty mobile machinery need to be assisted in their efforts, not given more white-noise. 

When HMI displays alert drivers whenever an object is detected, telling them how far away it is, and its relative motion — that’s too much information. The operator only needs to get high-level alerts, which result from a threat analysis — when the sensor system communicates data on any object in proximal range to equipment. The threat analysis looks at this information and then decides how significant a threat is and how likely a collision is.

This has to do with developing and then providing accurate information via the HMI display, which is crucial because the driver’s information is only as good as the machine’s information. Data accuracy and the process of turning that data into information are solely up to the sensor, and thankfully sensors have been making recent advancements.

That’s one of the many reasons the improvements made in sensor technology are focused on providing more accurate systems, like PreView® Radar. One of the essential steps in creating these systems is evaluating information streams, determining what to tell the operator, and when. This also includes time derivatives and understanding relative motion — all vital pieces to completing a successful threat analysis. 

In a threat analysis, stationary objects identify as low-level threats compared to objects-in-motion. In contrast, an object with a relative motion moving away from equipment elicits a lower threat level than an object moving toward equipment would. 

For example, a parked car is considered a low-level threat, but driving close to VRUs, like worksite personnel or bicyclists, elicits a more severe, high-level threat. Making this differentiation is what the industry wants these systems to be intelligent enough to determine. By compiling every piece of information, it’s easier to determine the threat level and distinguish between them to communicate the most critical threats to identify to the operator.

One of many organizations that recognize false and unnecessary alerts are distractions that can hinder operator acceptance and trust in the system(s), and its use-cases is the ADAC in the European Union. However, when it comes to the requirements for false positives or distractions, some organizations, like the National Highway Traffic Safety Administration (NHTSA), provide minimal guidance. There is a difference between meeting requirements and being the operator’s top pick. Ultimately, a system that meets regulations must also be the best for the operator.

Understanding these requirements, PRECO builds purpose-built systems for specific problems in specific markets, focusing on providing the right information at the right time. For example, PreView Side Defender®II was rated as the No. 1 blind spot assistance system — receiving zero false positives or nuisance alerts in the ADAC testing service’s 2019 report, which recommended Side Defender®II be standardized by manufacturers, even before legal obligation.

Side Defender®II is just one example of how well PRECO understands the industry’s safety needs. PRECO’s systems detect VRUs, trees, traffic lights, and many other stationary and moving objects. Still, because the radar is conducting a threat analysis, the system can prioritize information and minimize unnecessary distractions from false alerts to low-threat objects.

How High-Performance HMIs Work with PRECO Systems

Supporting the mitigation of HMI distractions, PRECO designed all of its sensors to easily support existing HMIs in the cab, like CMS displays, to ease the integration process. To decrease the number of HMIs in the cab, PRECO offers a single audible and visual display for multiple sensors that can support sensors covering up to a 360-degree view of the environment surrounding heavy-duty mobile equipment.

PRECO’s PreView® Radar solutions use CAN-based J1939 messages allowing the systems to support any HMI that supports J1939. These systems can also use AUX out electrical signals to drive simple HMIs. PRECO’s AUX out signals are either on or off, and the J1939 provides a lot of information and threat level analyses for various objects.

Additionally, PRECO simplifies integrating its sensors with existing HMIs, like simple displays and advanced monitor systems. This also includes the option of eliminating the monitor and integrating it with the existing in-cab HMI display.

Combining PRECO’s award-winning active safety technology with existing HMI displays, the PreView system completes a threat analysis, establishes the high-threats, and communicates this information via audible and visual alerts. Thus, intuitively providing a better sense of the environment surrounding equipment to operators by triggering customer-specific, purpose-built alerts.

Heavy-duty mobile equipment with HMI’s need to establish threat analysis processes to prioritize information and avoid nuisance alerts and distractions successfully. It’s then the HMI’s responsibility to communicate the crucial high-threats to the operator at the right time.

With PRECO’s advanced threat analysis, the OEM and aftermarkets have access to tools to help their equipment operators avoid incidents caused by human or machine errors. Now humans and machines can safely coexist with fewer distractions via PRECO’s advanced collision mitigation technology.

Peering into the Future, Here is Where HMI is Headed

HMI now needs to address the new wave of sensor-driven solutions that the market is demanding. The latest technological revolution of connected devices for the Internet of Things (IoT) and OEM mining and construction industries all require stable and precise sensors, such as PRECO’s Radar safety technology. 

It’s hard to predict precisely how the technology will progress in the coming years, but there is no doubt we will see it continue to evolve and improve. In the near-term, ground transportation applications are likely to remain focused on collision avoidance. Continued focus toward fully autonomous vehicles will see more trucks and buses equipped with GPS, LiDAR, Radar, CMS, and other systems to combine their signals to obtain an even more accurate representation of the environment.

Object detection and avoidance systems are becoming more prevalent on heavy-duty mobile machines as a means of not only improving safety but also creating more automated features. These include personalized, purpose-built solutions with advanced software and algorithms that enable situational understanding for machines and operators.

Today, PRECO plays an integral role in advancing safety technology solutions and is well-positioned as the market evolves into semi- and eventually full-autonomy. Eventually, the hope is to achieve fully autonomous vehicle operations. 

As manufacturers continue to develop more automated features and eventually fully autonomous vehicles, radar sensors will become an increasingly larger part of the technology mix. The collision avoidance and object detection capabilities they provide are already enabling the use of ADAS, which are stepping stones along the road to autonomy — the ultimate safety solution.