There is an industry-wide misconception that this form of rapid iteration
and improved flow applies only to software or small applications and systems.

— Industrial DevOps Whitepaper

Hardware Teams in SAFe

By Cindy VanEpps, 321 Gang Inc.

Why Hardware Teams in SAFe

While agile and SAFe do come from a pedigree of software development, the value of the underlying principles is being proven by teams that engineer hardware-inclusive systems. Rather than wait for other innovators to dominate their markets, these teams and their leaders choose to become educated on agile and SAFe®, embrace the constraints inherent in the systems they build and innovate on their ways of working.  Ultimately, the economic value is embodied in the reduction of late-integration breakage.  In hardware-based systems, this breakage is usually orders of magnitude costlier than in software-only systems. Therefore, if the primary goal is to reduce the risk of late-integration breakage, hardware teams must start by answering this one question: What can we do?

How we engineer, build, manufacture, procure, test and release hardware-inclusive systems, by its very nature, often imposes impractical and costly constraints on the ability to build, test and demonstrate in small batches with fast feedback cycles. Focus on the underlying principles of lean and agile are bringing a major shift in how hardware-inclusive systems are being built and brought to market. This focus on values and principles provide inspiration that leads to improved value delivery for hardware teams.  Industry leaders are opening their minds to what is possible, exploiting innovations unique to hardware and driving trends that are moving hardware-based systems engineering toward better economic outcomes using agile and SAFe.

This article describes five shifts in practices for hardware-inclusive systems, as well as the economics behind them.

• Move from V-model to cadence-based learning cycles
• Leverage proxies for hardware
• Design and architect for agility
• Shift focus from phase-gate process to built-in quality
• Address cultural biases head-on

Move from V-model to cadence-based learning cycles

The V-model of systems engineering has been a long-held best practice that “provides guidance for the planning and realization of projects” [1]. The tradition, inertia, and pervasiveness behind using the V-model often constrain hardware engineers’ moving to agile.  The practice of agile in hardware, therefore, is still relatively in the early adopter phase [2].

While software engineers use feedback cycles to demonstrate a working system, hardware engineers demonstrate new knowledge, for example with simulation results, trade study analysis or updating a model. These are incremental steps that ultimately lead to manufacturing. New knowledge creates the opportunity for early adjustments that provides the shortest and most economical path to a final product.  Fast-feedback cycles both identify errors early, thus reducing risk, and also open doors to exploit unforeseen opportunities.  Because of the material cost of hardware-inclusive systems, the ability to pivot late in a development cycle based on either of these factors is highly constrained.

In software-only systems, the emphasis of small batch sizes and fast-feedback cycles supports the goal of continuous delivery. In hardware-inclusive systems, the value realized by small batch sizes and fast-feedback cycles is more focused on reducing risk through full-system integration and the resulting learning. To achieve this value often requires the use of proxies for some parts of the hardware, which leads us to the next practice.

Leverage proxies for hardware

In agile, the goal is to build and learn as quickly as possible. Proxies, which are lower-cost substitutes for the real “bent metal” hardware, provide a means of integrating and learning earlier in the lifecycle and at a lower cost. Models used in electrical and mechanical Computer-Aided Design (CAD) systems are examples of proxies for hardware. They provide many types of virtual analysis for fast feedback prior to manufacturing. 3D printing also provides a quick, cost-effective means of prototyping for both mechanical and electrical hardware.

Digital twins combine virtual models with data from the operational systems to validate the models and better predict how systems will behave in the future. A digital twin becomes more than just a proxy during development, as it continues to “live” alongside the physical component during production use, even harvesting data from real-time usage to learn and improve the system.

Economically, the organization must weigh the cost of creating proxies against the values afforded by truncating errors as well as exploiting discovered opportunities early in the process. With a focus on economics – the costs and potential value – the right combinations of options for hardware proxies can be explored.  Hardware teams have the responsibility for the final product as well as providing proxies to support continuous integration and learning across the larger system. A team of teams learning together on the value stream ultimately provides better outcomes.

Design and architect for agility

Agile strives to take small engineering changes and test them as soon as possible in an environment that resembles production as closely as possible. This is achieved by creating modular designs with well defined, stable interfaces between components. An interface can be virtual (signal, data) or physical (power, force, light, heat, etc.). Modularity can impact form factor and unit costs but provides many key values, for example:

• In both development and in operation, systems are quicker and cheaper to support and upgrade.
• Components can be validated independently of the entire system by exercising the interface.
• Components can evolve independently from the rest of the system so long as the interface remains stable.

These values must be weighed with potential manufacturing and form factor tradeoffs to make the best product lifecycle decisions.

How parts are connected also impacts agility.  Solder and welds reduce manufacturing costs but impact the cost and time for changes in both the operation and development environments. Designing interface points to accommodate replaceable/upgradeable parts leads to better agility.

Moving functions from hardware towards programmable devices also help agility. Designs that incorporate FPGAs and System-on-Chips (SoC) can be more quickly tested and upgraded than custom ASIC parts.  Indeed, the industry has seen a large drop in the number of new ASIC designs over the past several years as the value of agility becomes more widely practiced.

Economics will dictate how modular, updateable and extensible we want and can make our solutions to be.  Hardware-based systems must find the balance between continuous delivery of value and the cost of design and manufacturing. Leveraging experimentation along with a continual learning culture provides the economic stimulus to become market innovators.

Shift focus from phase-gate process to built-in quality

The long lead time activities for hardware such as design, approvals, procurement, and manufacturing harken back to traditional, legacy software delivery processes a decade ago. Software systems suffered long lead times for standing up test environments, approval processes of abstractions of the system (e.g. requirements and designs) and manual, therefore error-prone deployments.  In hardware-inclusive systems “Quality by Inspection” has been a standard of quality assurance, with extensive checklists and reviews layered into the V-model though the phase-gate process.

Well-known agile practices of collective ownership, transparency, test-first development, integration, and simulation help weave quality into the fiber of how solutions are built. Quality is demonstrated, assessed and adjusted at every fast-feedback cycle.  This enables solution builders to adjust the quality as they go rather than adjusting their tolerance for sub-optimal quality when pressed against a delivery deadline. These practices also apply to hardware.  Many hardware engineers already apply these practices in FPGA and SoC designs since they are software-like. But others are also applying them to electric and mechanical designs by early testing against CAD changes to discover anomalies and interface violations.

The SAFe principles of taking an economic view, applying systems thinking, etc. all the way through organizing around value support the core value of built-in quality.

The cost of poor quality ranges from rework to warranty costs to disastrous consequences, depending on the solution provided. Most organizations invest significant resources to ensure quality, so the economics of shifting to built-in quality practices is often only a transformation cost, and one that will pay for itself easily over time.

Address cultural biases head-on

Engineering is fundamentally a critical-thinking and analytical discipline, as well as a creative discipline.  Traditional skillsets of designers versus manufacturing emerge from the complexity of the requisite knowledge and tasks. The resulting culture is often reluctant to show partially completed work to other critical thinkers for fear of criticism.  This culture fosters the perception of collaboration as micro-management, which can impede the adoption of the collaborative, social, collective ownership model of agile.

Teams of people responsible for providing value to an end-user must learn to share knowledge and admit what is still being learned throughout solution development. “Swarming” a solution, that is, being open to all perspectives to enable learning, working and compromising together provides a powerful force for innovation and value delivery.  Bringing all perspectives onto the value stream, including procurement, manufacturing and operations support is key to the reduction of late-integration breakage.

Transformation to new ways of working, as exampled in the shifts in practices highlighted here, must, therefore, include a focus on the people building the solutions. Their perspectives, buy-in, and ability to shift together collaboratively, are vital to achieving the goal of improved economic outcomes.

Summary

An organization’s ability to apply agile and scaling agile practices is a differentiator in the innovation and quality of the solutions provided. This summary of five key shifts in practices that distinguish solutions provided in hardware-inclusive systems is a starting point. Armed with the underlying principles, innovative leaders will further the means by which value is delivered in a more economical and differentiating way.

Learn More

Last update: 26 March 2020