Human-Machine Interfaces (HMI)

Human-Machine Interfaces (HMI) is used extensively in the cabs of heavy-duty OEM equipment both on and off-road, and allows a person to interact with a machine, like an instrument panel, the joystick, the steering wheel, the brakes, the knobs and buttons of the infotainment system, Etc.

HMI interactions can be a visual signal, an audible signal, how the machine communicates to a person, or how they communicate back. This includes all the different Advanced Driver-Assistance Systems (ADAS) and safety systems that beep, flash, vibrate, or otherwise inform or warn the operator.

Due to these advancements, the cab has gotten busier with more HMIs communicating information to the operator. This has led to sensory overload — where too much information from numerous HMIs creates distractions, leading to a decrease in safety.

As these machines have become more complex, the amount of information generated has increased significantly. For a while, the trend has been to provide as much information as possible to operators — stemming from the idea ‘more information is best.’ However, over the last decade, a widespread recognition that more information is not necessarily better has grown.

The industry has begun to recognize that instead of continuously providing all available information, they must transition to offering systems that intuitively separate and communicate helpful information to the operator. That way, the operator understands what they need to know when they need to know it.

More than ever, our tools are connecting to computers, digital systems, and other devices more to automatically carry out tasks. Yet, for these advanced technological solutions to be successful on heavy-duty mobile machines, they must be intuitive and avoid information that merely distracts users.

Simply because information is available, that doesn’t mean it’s the best idea to provide it to the operator. Too much information from numerous sources could cause information overload and distraction. If there is an information overload, operators can’t focus on what they’re trying to accomplish, or the most critical information to avoid an incident. This is why it’s so vital to place the right information front and center, both visually and audibly.

The issue at hand is there are already too many distractions in the cabs of most mobile equipment on the road, like infotainment systems, GPS navigation, lane-keeping systems — the list goes on. This is why the cab’s HMIs must provide accurate information to the operator only when needed or useful to the operator.

What’s Needed for HMI Success

Regardless of the HMI, its purpose is to provide useful information to aid the operator in their task. This means that the equipment and user must communicate and interact smoothly with minimal distractions when it comes to HMI success.

However, new advances in in-vehicle technologies and rapid deployment of these advanced systems in vehicles already on the road have left questions. Questions that AAA investigated in their “Understanding the Impact of Technology” 2019 study.

The AAA study found that people who regularly use ADAS technologies are more likely to be distracted and overly reliant on the very solutions designed to improve safety and driver-awareness. With nearly 93 percent of all new vehicles equipped with advanced driver-assistance features, like lane-keeping, cruise-control, distance tracking, and GPS navigation functions, drivers are now twice as likely to be distracted.

According to AAA’s study, the most distracting infotainment systems can take a driver’s attention from the road for an average interaction time of 7 – 31.4 seconds, like using GPS navigation or using the entertainment system. That means drivers are 22 – 41% more distracted while using interactive ADAS systems while on the road.

This over-reliance happens once drivers move from the “novelty phase” to the “post-novelty operational phase.” This is because people begin to adapt their behaviors, and over-reliance and too much trust in the systems automation features start to develop.

Researchers at AAA collaborated with Virginia Tech Transportation Institute and found this to be true, “These drivers are more apt to take their eyes and attention away from the road. Research in other industries shows that pilots and nuclear technicians demonstrate similar patterns of over-reliance on automated systems. These behaviors can eventually lead to distraction.”

As noted, over-reliance is another potential issue caused by providing too much information to the operator. As long as operators are expected to play an integral part of operations, they must be kept engaged. For example, if an operator could perform almost all of their job without looking out the window, that would be an inevitable issue. The operator can only almost do the task at hand due to disengagement.

Even at Level 3 autonomy, the vehicle can only operate most of the time without any driver interaction. It can drive on its own, but it’s not fully-autonomous, so it will miss things. This means the driver must be available to take control of any given situation. Drivers have to be ready to take evasive action immediately. 

Although the human component is vital to safely operating equipment up to Level 3, human nature continues to be a weakness. So, not only are humans psychologically hard-wired to either be engaged or disengaged — but HMI displays and ADAS solutions have proven to have a direct causality with over-reliance.

Developing Solutions to Further Advanced Technology Trends

In the past decade, operational and business needs have brought on exciting HMI developments — including quite a few new capabilities. Today, there is a lot of information and interactions crossing between these new HMI capabilities. It’s becoming more common to see evolved forms of HMI in the heavy-duty mobile market, and these modernized interfaces are creating more opportunities for equipment interaction and analysis.

Still, recent research has led the industry to acknowledge they need to consolidate HMIs to minimize distractions. One idea has been to focus on the combination, creating a common space, where a single HMI or system does a myriad of tasks. 

Instead of having several HMIs — like camera monitoring systems (CMS) with a 360-degree view, sounds alerting to lane changes, and another HMI in the corner beeping when the vehicle veers — competing for the operator’s attention, you can consolidate these HMIs. With a single intuitive and consistent HMI telling operators what they need to know, when they need to know it, this would successfully remove the clutter from the dash in the cab and decrease the level of distraction.                                                                                                                                                               Image courtesy of Xite Solutions North America                       There is work being done in the industry to minimize HMIs through common space; they have acknowledged it is essential. One example is standardizing common alerts — where all sounds from any HMI alerting to a single issue are the same. But it must be intuitive and has to help the operator tackle the task at hand.

The progression of HMIs in the cab follows a bell-shaped curve. For a long time, the industry focused on offering more information, going up the curve. Today, there has been a broad recognition that now this information needs to be filtered. Soon we will see the bell curve turn back down as OEMs and HMI manufacturers provide less information by providing the right information at the right time for the operator.

Avoid Sensory Overload via Threat Analysis

In most cases, current technology is a vessel for information transfers and can cause a sensory overload. From the early 1900s to the present, this transfer of information has gone from a trickle to a fire hose — today, more information is thrown at people in a single day than our grandparents got in their entire lifetime. Which poses the question: How do we prioritize all this information and pay attention to what really matters? 

First and foremost, it’s essential to be cautious about overloading anyone with irrelevant information, especially operators driving on roads and near personnel and VRUs. Operators of heavy-duty mobile machinery need to be assisted in their efforts, not given more white-noise. 

When HMI displays alert drivers whenever an object is detected, telling them how far away it is, and its relative motion — that’s too much information. The operator only needs to get high-level alerts, which result from a threat analysis — when the sensor system communicates data on any object in proximal range to equipment. The threat analysis looks at this information and then decides how significant a threat is and how likely a collision is.

This has to do with developing and then providing accurate information via the HMI display, which is crucial because the driver’s information is only as good as the machine’s information. Data accuracy and the process of turning that data into information are solely up to the sensor, and thankfully sensors have been making recent advancements.

That’s one of the many reasons the improvements made in sensor technology are focused on providing more accurate systems, like PreView® Radar. One of the essential steps in creating these systems is evaluating information streams, determining what to tell the operator, and when. This also includes time derivatives and understanding relative motion — all vital pieces to completing a successful threat analysis. 

In a threat analysis, stationary objects identify as low-level threats compared to objects-in-motion. In contrast, an object with a relative motion moving away from equipment elicits a lower threat level than an object moving toward equipment would. 

For example, a parked car is considered a low-level threat, but driving close to VRUs, like worksite personnel or bicyclists, elicits a more severe, high-level threat. Making this differentiation is what the industry wants these systems to be intelligent enough to determine. By compiling every piece of information, it’s easier to determine the threat level and distinguish between them to communicate the most critical threats to identify to the operator.

One of many organizations that recognize false and unnecessary alerts are distractions that can hinder operator acceptance and trust in the system(s), and its use-cases is the ADAC in the European Union. However, when it comes to the requirements for false positives or distractions, some organizations, like the National Highway Traffic Safety Administration (NHTSA), provide minimal guidance. There is a difference between meeting requirements and being the operator’s top pick. Ultimately, a system that meets regulations must also be the best for the operator.

Understanding these requirements, PRECO builds purpose-built systems for specific problems in specific markets, focusing on providing the right information at the right time. For example, PreView Side Defender®II was rated as the No. 1 blind spot assistance system — receiving zero false positives or nuisance alerts in the ADAC testing service’s 2019 report, which recommended Side Defender®II be standardized by manufacturers, even before legal obligation.

Side Defender®II is just one example of how well PRECO understands the industry’s safety needs. PRECO’s systems detect VRUs, trees, traffic lights, and many other stationary and moving objects. Still, because the radar is conducting a threat analysis, the system can prioritize information and minimize unnecessary distractions from false alerts to low-threat objects.

How High-Performance HMIs Work with PRECO Systems

Supporting the mitigation of HMI distractions, PRECO designed all of its sensors to easily support existing HMIs in the cab, like CMS displays, to ease the integration process. To decrease the number of HMIs in the cab, PRECO offers a single audible and visual display for multiple sensors that can support sensors covering up to a 360-degree view of the environment surrounding heavy-duty mobile equipment.

PRECO’s PreView® Radar solutions use CAN-based J1939 messages allowing the systems to support any HMI that supports J1939. These systems can also use AUX out electrical signals to drive simple HMIs. PRECO’s AUX out signals are either on or off, and the J1939 provides a lot of information and threat level analyses for various objects.

Additionally, PRECO simplifies integrating its sensors with existing HMIs, like simple displays and advanced monitor systems. This also includes the option of eliminating the monitor and integrating it with the existing in-cab HMI display.

Combining PRECO’s award-winning active safety technology with existing HMI displays, the PreView system completes a threat analysis, establishes the high-threats, and communicates this information via audible and visual alerts. Thus, intuitively providing a better sense of the environment surrounding equipment to operators by triggering customer-specific, purpose-built alerts.

Heavy-duty mobile equipment with HMI’s need to establish threat analysis processes to prioritize information and avoid nuisance alerts and distractions successfully. It’s then the HMI’s responsibility to communicate the crucial high-threats to the operator at the right time.

With PRECO’s advanced threat analysis, the OEM and aftermarkets have access to tools to help their equipment operators avoid incidents caused by human or machine errors. Now humans and machines can safely coexist with fewer distractions via PRECO’s advanced collision mitigation technology.

Peering into the Future, Here is Where HMI is Headed

HMI now needs to address the new wave of sensor-driven solutions that the market is demanding. The latest technological revolution of connected devices for the Internet of Things (IoT) and OEM mining and construction industries all require stable and precise sensors, such as PRECO’s Radar safety technology. 

It’s hard to predict precisely how the technology will progress in the coming years, but there is no doubt we will see it continue to evolve and improve. In the near-term, ground transportation applications are likely to remain focused on collision avoidance. Continued focus toward fully autonomous vehicles will see more trucks and buses equipped with GPS, LiDAR, Radar, CMS, and other systems to combine their signals to obtain an even more accurate representation of the environment.

Object detection and avoidance systems are becoming more prevalent on heavy-duty mobile machines as a means of not only improving safety but also creating more automated features. These include personalized, purpose-built solutions with advanced software and algorithms that enable situational understanding for machines and operators.

Today, PRECO plays an integral role in advancing safety technology solutions and is well-positioned as the market evolves into semi- and eventually full-autonomy. Eventually, the hope is to achieve fully autonomous vehicle operations. 

As manufacturers continue to develop more automated features and eventually fully autonomous vehicles, radar sensors will become an increasingly larger part of the technology mix. The collision avoidance and object detection capabilities they provide are already enabling the use of ADAS, which are stepping stones along the road to autonomy — the ultimate safety solution.

Differentiating & Detailing PreView Tech

How does Side Defender®II differ from the Sentry® blind spot monitoring system? Are the systems designed to be used in tandem with one another on a vehicle? What applications are each system best suited for?

PreView Side Defender®II and PreView Sentry® were designed as blind spot monitoring systems. Each is equipped to handle different proximity detection needs and scenarios within the heavy-duty industries.

The new Side Defender®II was designed for medium and heavy-duty trucks operating primarily on the road providing Lane Change Assist and Vulnerable Road User (VRU) detection. The Side Defender®II is intended to be installed on the sides of trucks with a fixed detection pattern extending up to 6 m to each side of the sensor and out 3 m (one lane width) from the side of the truck to alert the operator of only moving objects while ignoring stationary objects.

Side Defender®II’s system is oriented towards on-road trucks; when trucks try to merge or change lanes, the systems warns the driver of cars, motorcycles, or bikes (VRUs) in the side blind spot to avoid collisions. Side Defender®II has the added benefit of warning the driver of VRUs in the side blind spot of the truck, again to avoid side collisions.

Sentry is a short to medium range radar sensor designed for heavy-duty object detection and collision mitigation applications supporting up to 360-degrees of blind spot monitoring in medium and heavy-duty equipment. Sentry’s radar was designed with a wide-viewing capability, making it optimal for those with backing, forward, or side blind spot monitoring needs – offering the operator a cocoon of proximity detection and warning of threats.

Depending on the blind spot monitoring needs, the Sentry has a programmable detection pattern that can be configured specific to the intended application. Thus allowing for installation on the back, front, and sides of the equipment to warn the operator of the closest moving or stationary object.

Can you provide an overview of Side Defender®II’s system – how it works and what enhancements have been made?

Side Defender®II is the state-of-the-art side collision avoidance solution explicitly developed for over-the-road fleets in trucking, municipalities, waste, and recycling. Side Defender is an intelligent technology born of 20 years of experience in truck side radar systems. As our fourth iteration of the technology, it is a testament to our dedication to improving our systems by tackling the evolving needs of the industries we serve through the continuous use of our experience.

Side Defender®II is able to ignore stationary objects such as guardrails and parked cars while alerting on moving objects such as bicyclists and moving vehicles in the adjacent lane. The detection pattern for the Side Defender®II when the vehicle is stationary is approximately 5 m fore and aft of the sensor and 3 m from the side of the vehicle. When in motion, the detection pattern expands to 6 m fore and aft of the sensor.

VRU Operation

At speeds of 19mph or less, the Side Defender®II gives a priority to alerting the operator of moving bicyclists in the detection zone. A visual alert is provided on the in-cab display when a moving object is detected while both an audible and visual alert are provided when the turn signal is active and a moving object is detected. Stationary objects such as street signs and parked cars are ignored.

Lane Change Assist Operation

When the vehicle speed is above 19mph, the Side Defender®II gives priority to alerting on moving vehicles in the adjacent lane. Stationary objects such as guard rails or concrete barriers are ignored to minimize ‘nuisance alerts’. This mode is optimized for blind zone collision mitigation during lane changes and merges.

By filtering out extraneous objects, so only legitimate threats are sent to the operator, Side Defender®II improves awareness to avoid collisions.

What technologies are utilized within PreView’s system?

PreView’s proximity detection systems are radar-based. Our RADAR (Radio Detection and Ranging) technology uses FMCW (Frequency Modulate Continuous Waves), providing better object detection resolution and virtually instantaneous object information such as range, velocity, and angle. FMCW technology inherently performs poorly in close range, preventing detection of objects up to the face of the sensor, but our engineers have developed a proprietary method that mitigates this issue.

What enables the system to differentiate between stationary and moving objects? Why is this an important feature?

The importance of differentiating between stationary and moving objects has been a focus of PRECO’s since bringing the PreView family to fruition. When a heavy machine is in use, the operator needs to be aware of their surroundings, especially where they cannot see. This may seem obvious, but until now, there has been a lag in technology to solve this issue effectively.

When operating, being alerted to every object within range, even things you know are there and aren’t something to be worried about, would become quite a nuisance to anyone in that situation. Often, we have seen systems such as these being ultimately turned off, or ignored altogether, permeating an already dangerous situation, which is why, in the on-road system, operators normally only want to be alerted to moving objects, especially those moving towards the vehicle.

Understanding this struggle, we sought to create a solution intelligent enough to understand the difference between a high-level threat and a non-threat. By ignoring stationary objects*, the system can actively alert the operator only when needed, eliminating nuisance alarms, false positives, and reducing the risk of sideswiping during lane changes or hitting a vulnerable road user (VRU).

What enables the system to do this is the integration of speed messaging, CAN-bus integration, and intelligent technology within our systems. Using the information gathered through the CAN-bus, our technology can ignore stationary objects based on the selected detection zone, the speed of the machine/other objects, and proximity detection.

*street lights, guardrails, parked vehicles, benches, and buildings (when they are not detected by the speed monitoring system to be a danger within the range of the machine)

What is new about the enhanced version of this display?

The newest display for the Sentry and the Side Defender®II is a crucial enhancement driven by the industry’s growing interest. The PreView® v2 displays were designed for use with all the PreView CAN based sensors. These displays provide the operator with an audible and visual indication of the direction and proximity of detected objects and support from support from 1 to 8 PreView sensors installed in various positions around the vehicle or equipment. Through LED bars around the perimeter of the display, the D2002 uses colors and flash rate to indicate direction and proximity of detected objects. The audible rate also provides proximity information. The D2002 also has a secondary CAN-bus that acts as a listen only so it can be easily connected to the vehicles CAN-bus to obtain vehicle information such as its speed, turn signal status and reverse status without causing interference.

In addition to the functionality of the D2002, the G2000 utilizes an external GPS antenna to provide the vehicle speed information required by some advanced PreView sensors.

What makes PreView’s story different from other similar types of systems on the market?

PRECO was established in 1947 when we engineered, designed, and manufactured the well-known backup alarm, which can now be found across all industries.

With this background in alarms, cameras, monitors, and mirrors, PRECO began to understand early on what each of these systems was wrong: a large margin for human error. Lights and alarms outside of equipment require people to move out of the way to avoid being injured; the safe operation of mobile machines was dependent on those in the surrounding area; and cameras required operators to be actively engaged and alert, while watching the feed on the monitor and make maneuvers with limited awareness of the surrounding area. Which is why we decided to put the solution back in the cab, and into the hands of the driver.

In creating PreView’s radar technology, our engineers have mitigated these issues. Built to outlast the machines they are made to protect, our robust systems can withstand any environment and all-weather conditions with unfaltering reliability.

In successfully reducing nuisance alerts by differentiating between potential hazards and non-hazards, we believe PreView’s intelligent proximity detection technology is now the only blind spot radar in the market capable of consistently ignoring stationary objects.