Improving safety for all road users is at the heart of everything we do at TuSimple. Creating a safe Level 4 Autonomous Driving System (ADS) for Class 8 Semi Trucks requires world-class testing capabilities

To help achieve this mission, TuSimple created the first-of-its-kind Systems Integration Lab (SysIL) that incorporates a fully functional Semi + ADS into a fully instrumented lab. The SysIL adds a layer of safety and test coverage not possible with simulation, test benches, and road tests alone. TuSimple’s SysIL enables a faster path to a safe and feature-complete ADS.

The Challenge

Performing safety-critical testing on closed tracks is expensive, requires a large cast of people to travel to a test track, is difcult to perform as repeated regression testing, and incurs all the track testing costs each time.

Additionally, there are things you would never do on a truck intended to return to the road. For example, it would be unsafe to cut into high-reliability power and communication harnesses to simulate failure modes, and then expect that truck to return to road operations. These types of testing must be executed in a controlled, system-level environment. Piecewise testing on isolated Hardware in the Loop (HiL) stands lacks the “integration effect” at the system level. Testing where all the subsystems interact is necessary to nd undesirable interactions you would miss on partial subassemblies.

“We hired experts in systems level integration and testing from automotive, aerospace, and defense industries to make this happen. It was a great investment! Every nding improves the safety of our ADS. Every new test becomes part of our validation and certication evidence that we are meeting our Safety Case.” Adrian Thompson, VP of Systems & Safety Engineering.

Why does this matter? Improved safety and quality, reduced schedule and cost.

Improved Safety

“A Systems Integration Lab is the only way to test combinations of features, functions, and failure modes, end-to-end, in a safe environment. We artificially create complex situations, nuanced failure modes, and more sophisticated sensing and automation that cannot be replicated on a track or on a partial Hardware in the Loop test stand.” Tim Jones, Director of Systems Commercialization and Next Gen Programs.

Applied Learnings from Safety-Critical Industries

The SysIL underscores TuSimple’s commitment and leadership in this space as it continues to evolve its Systems Engineering, Safety Engineering, and Independent Verification & Validation rigor.

Other safety-critical industries already know how valuable a Systems Integration Lab is to the continuous integration of new hardware, software, and firmware. A SysIL enables engineers to quickly identify and resolve gaps in requirements, design, integration, and validation efforts.

Safety Critical Benefits of a Systems Integration Lab

Always Fully Instrumented	Never missing things once we decide they are important to measure & control
Full Complement of Hardware	Actual complete hardware, software, and rmware gives best fidelity signal
Inject/Detect Anywhere	Ability to access full system and nonstandard test points opens test options dramatically
“Test as you Fly”	Incorporate as much realism during operational testing as possible to tease out failures in a safe environment
Independent	Assessments of performance/lag/leaks from outside the system under test by inserting stimulations and senses not controlled by the ADS itself
Pause On Fail	Pauses the system for any limit violation to aid in diagnostics (before the next step destroys evidence) and see exact messages and signal levels during fault for troubleshooting
Continuous Access	Never wait for a truck to test a theory or validate a build; many tests will operate automated and overnight
Integrate Where You Instrument	Learn faster & adapt faster on a dedicated system-level rig, where you can update components and see results in detail
Automate! Trend!	Every test that will persist provides fast, consistent, and trended test data; detect when you weren’t even looking!

A Systems Integration Lab is both Complementary and Supplementary

A SysIL is complementary to other testing modalities because it provides additional condence in features and functions tested in unit testing, subsystem and HiL testing, simulation & regeneration testing, analysis, and track & road testing. It is supplementary because the SysIL enables unique tests not possible on piecemeal testing solutions, with deeper instrumentation, better ability to divide and conquer the verication of subsystems, and better holistic data.

Test Capabilities Added by the Systems Integration Lab

Test Capability	What Is It?	Benefits
Man-in-the-Middle testing	Incorporating external computers and communication cards into the middle of Ethernet, CAN, and other communication buses to defeat, delay, or modify messages.	Stress test late message timing effects Check if the system properly detects and reacts to lost message traffic Replace otherwise healthy messages from hardware to simulate faults Verify our system reacts in an appropriate and timely manner Automate communications related regression testing to ensure communications remain robust through all development changes
Power Distribution Failure Testing	Incorporates relays and control automation techniques to simulate hard-to-handle power failures on an autonomous vehicle. Provides a methodology to test loss of power to any safety critical device. Goes beyond theoretical failure analysis or limited disaggregated HiL test benches, providing empirical evidence of coverage and system wide reactions.	Safely access power distribution lines not possible on a road-worthy truck Stress test power for brown-out and black-out conditions Check if the system properly detects and reacts to power failures Gain nuanced control over power feeding devices Automate power related regression testing to ensure the system remains robust through all development changes
Enhanced Data Acquisition	Provides higher delity signal and timing analysis than on-board systems alone.	Safely access signals not possible on a road-worthy truck Collect more data channels, at a higher rate of speed than we could do with an ADS enabled truck alone Integrate independent data acquisition results into automated Safety Case evidence creation
Customized Command and Control of Subsystems	Automated test equipment provides the ability to connect, disconnect, or reroute signals to specialized measurement, recording, and analysis equipment.	Accelerate and enhance testing in a number of ways to stimulate subsections of the system design without having to build separate Hardware in the Loop (HiL) test stands for each area of interest Provide faster testing by only engaging and resetting specic pieces of the design as required
Accelerate Verification of Failure Monitoring	Automate the injection of every known failure message from OEM faults (sensors, brakes, steering, engine, transmission, etc.), TuSimple generated hardware faults, and software generated monitoring faults	Prove that each one would result in the correct Minimum Risk Maneuver (MRM) to safely end a mission
Adversarial Testing	Provides a safe place to test “what if” theories. Want to know how your system responds to a loss of communications from the primary to secondary controller? Pull the plug!	More freely test failure reactions without the risks of track or road Empirically test a hypothesis almost as fast as you can think of failure modes Once you identify a good test, you can permanently automate and instrument it as part of your long term System Validation & Verification plan, repeated on future builds

Improved Quality

The Systems Integration Lab provides TuSimple with the ability to make modications that would not be safe or possible on a truck intended to return to the road. As such, we are able to get more detailed diagnostic information on signals and timing that we wouldn’t otherwise be able to see.

By automating the collection and analysis of SysIL data, we improve the repeatability of our testing and gain clearly comparable test results from build to build. Using Lean Six Sigma approaches, we can build automated detections of in-spec, but out-of-family test results to identify unintended consequences of changes. By building automated reporting into the SysIL, we can quickly grow our body of evidence that the truck is operating safely.

Reduced Schedule

The SysIL reduces the development lifecycle by unearthing problems earlier, which would be harder to detect or remain dormant and unknown in the system. One of the biggest hindrances to schedule is having to rework features after they have been released. Testing on independent Hardware in the Loop (HiL) test stands takes time and lacks the full system perspective for complete results. Testing on the track takes time to schedule and deploy teams, trucks, and supporting resources, and then time to get the test findings incorporated back into the workflow.

Testing on the Systems Integration Lab can be done whenever required, at the full system level, with automated testing and reporting. Faster, more automated testing with broader coverage and sensing capabilities is the best way to accelerate development and close gaps completely.

Reduced Cost

While the cost of harvesting a perfectly built-out ADS-enabled truck for a lab is not trivial, the payback period is very short. TuSimple has been actively migrating any track and road testing related to managing component failures to the Systems Integration Lab. We can test and retest for specific failure detection, logging, and responses in a repeatable, reliable, and cost-effective manner

Once we incorporate a test into the SysIL, it’s ours forever. With fewer personnel required to perform testing, zero travel for people and equipment, and faster system-level regression testing and results to developers, the SysIL pays for itself.

Conclusions

A Systems Integration Lab is an absolutely necessary capability for any company building a safety-critical autonomous vehicle.

A SysIL:

• Provides system wide connections not otherwise possible
• Enhances integration and test coverage feedback
• Allows testing of failure modes not safe to perform on the track or road
• Pays for itself in both improved schedules and lower costs over time

TuSimple has seen the technical findings and benefits of having a SysIL, proven the efficiency value, and is prospering from the additional understanding that comes from a fully instrumented system. We believe maintaining such a Systems Integration Lab capability should be one of the minimum standards for any company intending to operate autonomously on public roads. We are super proud of our team, and we can see the benefits of their work every day.

The post TuSimple’s Dedicated Systems Integration Lab Enables Safer Path to Level 4 Autonomous Semi Trucks appeared first on TuSimple.