Media, regulators, and plaintiff lawyers show little sign of letting up on Toyota. Gurus have weighed in on sticky accelerators, electric power steering, and brake software. Most mentioned that Toyota’s rapid growth was one cause, but few concluded that anything is amiss with the Toyota Production System (TPS). Too many others use variations of it with good effect. Significantly, TPS wasn’t a factor in Toyota’s non-performance taking a raking.
This drama is playing differently from what might have occurred a decade ago. The whole auto industry is entering the era of drive-by-wire with integrative complexity. In the coming decade, more industries may find themselves with analogous challenges.
Toyotawas exceedingly slow to open up and communicate. Tone deaf to American politics, and fresh meat on media inquisition griddles, Toyota’s first spokesmen were embarrassingly unfamiliar with complex software issues. Companies with a sorry record, still warm from the grill, would have handled it better. Because of their iconic reputation, overconfidence, and stalling for time, Toyota’s debacle became the industrial version of Tiger Woods’ fall from grace.
Toyota’s first recalls were typical rare event situations. Seldom seen, but drastic defects escape testing as NPF, no problem found, so companies must monitor whether reality in the field has flaws not predicted by testing. But the recall of the Lexus GX 460 for stability system delay was a goof that should have been caught in testing.
For the record, Toyotadropped to 21st place overall in the 2010 J.D. Power initial quality survey, but had three vehicles ranked best in segment. It was near the top during 2009. Toyotashowed well (17th of 20 car companies) in all customer complaints filed with NHTSA in the past decade. Air bags, not Toyota-type problems, were by far the top reason for complaint. But for model years 2005-2010, Toyota’s 1113 complaints of unintended acceleration topped the list in that category, far exceeding Ford, No. 2 on that list, with 387. Alleged fatalities from unintended Toyotaacceleration: 34. Toyotahad issued prior recalls for unintended acceleration. Independent engineering investigations revealed that both floor mats and accelerator linkage design could cause accelerator sticking. No one could rule out software as an additional root cause of unintended acceleration1. Electromagnetic interference can never be completely ruled out on sensors or circuit boards.
Sifting through detailed complaint reports is boring. Most are obviously groundless, and complaints are minute fractions of total vehicles in the field. When does any company conclude that a serious pattern is emerging out of background whine? However, standing publicly mute amid signs of a gathering storm while fighting the problem internally invites Watergate-style inquisitions: Who knew what, when, and what did they do about it — political theater. Culprits juice the news; technical dilemmas and process flaws make dull YouTube videos. Hanging the guilty in effigy sustains a media lynching until crowd frenzy dissipates. Being a marquis name company lengthens the torture time while plaintiff’s lawyers circle ever lower on discovery missions, and reporters scrounge for tidbits off the carcass.
Consensus Critiques
Chastened Toyotaand all Toyotawatchers quickly absorbed a few important lessons. In addition to sluggishness in public communication, many major media accounts alluded to the three that follow:
- Upon becoming “Bank Toyota,” the company could pay huge dividends from excess cash, or use it to expand rapidly. It chose to expand, and to pile up more cash by racing to be the biggest volume car company in the world. Thereafter, most Toyotaobservers noticed signs of slippage. The system is not just techniques; it’s an instinctive way to think, not learned casually, but through long practice while being mentored. The growth rate out stripped mentoring capacity. Setting up global learning centers did not accelerate absorption of The Toyota Way as much as hoped.
- Supplier and employee relationships had been fraying in Japan. With the recall, criticism broke into the open. Toyotahad begun to rely on more temp labor, usually dismissed after two years. Suppliers were pressured to cut cost without the old collaboration on process improvement by which to do it — and with less equal sharing of savings. In short, Toyotawas accused of adopting an “American” business attitude, violating The Toyota Way that had made it famous. Some attributed this to Toyotaa decade ago being listed on major stock exchanges, slowly degrading into just another investor-whipped growth machine. Supplier acrimony spilled over to the United Statesand Europe. A two-continent anti-trust investigation of wiring harness suppliers began checking whether OEMs and suppliers had colluded to divide the global market. Though this appeared not to be triggered by the Toyotacase, the industry feared that a narrow interpretation of bid solicitation would crimp the close collaboration necessary to integrate modern vehicle designs and supply logistics2.
- Toyotanever adequately scaled its system up to global scope. For instance, over the last two decades design responsibility gradually shifted to foreign centers, but too much stayed bottled up in ToyotaCity. Toyota’s systems strongly depend on “peopleware,” personal interaction among people. In Japan, this worked well for decades, and almost as well in transplant locations once the system took hold. But systems for global communication did not mature3.
(Personal aside: I first saw The Toyota Way in print in 2002 at the Toyota Institute. They did not want it published, so it never appeared in Target. But chatting with Toyotaemployees off and on for the next few years revealed that they had barely heard of it, and none had seen it. Propagation through mentoring advanced at the speed of glacial ooze.)
Less Obvious Critiques Customers in Lot-Size-One: Toyotahas long claimed that even one dissatisfied customer is one too many; that customers in lot-sizes-of-one are served by production in lot-sizes of one. A wrongful death legal suit makes that dramatic. Legal logic takes over from operational logic. Part-per-million quality is a statistic. In a lot-size-one court case, that’s merely evidence. Customers are killed in lot-sizes-of one.
If a plaintiff’s lawyer establishes that responsible people in a company were aware of a serious safety flaw and did nothing because it was not cost-effective (dollars applied to statistics), the most business-friendly jury is unlikely to hold for the company. In the 1970’s cases of Ford and the Pinto gas tank, that revelation rendered Ford defenseless.
From what is known, Toyotawas not that callous, but merely delayed public disclosure, stalling until they could find root-cause fixes. Saying that you don’t know a fix for a problem other than warning drivers to be cautious is an embarrassing situation, but no worse than the one in which Toyotafinds itself. In recent years, many other companies have had that predicament, notably pharmaceutical and medical device companies. The more a company strives for perfection, the worse the predicament. Customer expectations rise to the level of infallibility. Observing “process excellence” ceases to be an effective legal defense — or a media defense.
For example, in pharmaceutical cases like Merck and Vioxx, relying on statistics from a clinical trial is no longer good enough even if no plaintiff shows that either procedures or analyses from a clinical trial were flawed. Patients come in lot-size-one, so it is not totally predictable whether a drug will be either effective or have a serious side effect on one individual, different from all those in a trial. Drug performance has to be monitored in the field over its life cycle, with appropriate actions taken. That industry’s business model is slowly shifting. Similar concerns may shift the business model for other industries too.
In this situation, companies and industries must become more transparent. Toyota’s legal defense has been weakened by reluctance to reveal data from crashes recorded in its black boxes, officially called event data recorders, or EDRs. Not much was recorded, and only Toyotacan read the data. Having a system that no investigative authority can read is one black mark. Selectively disclosing crash data is another. Regardless of Toyota’s intent, plaintiff’s lawyers paint this practice as hiding culpability, in violation of Toyotaadmonition to go to the scene, get the facts, and deal with the facts4.
Software and software integration: It’s not 1985 any more. Like many vehicles from leading competitors, Toyota’s recalled vehicles were driveby-wire. Engineering software critics quickly fingered Toyotasoftware, hinting that Toyota’s design expertise in complex software lagged.
Ford and GM piled on Toyotawith offerings for Toyotaowners to switch, but had second thoughts. Drive-bywire vehicle design, slowly penetrating the industry, opens design envelopes for lighter weight, better-performing, less-polluting vehicles. But public confidence in drive-by-wire is at least as big a concern as technology. Shaking public confidence in drive-by-wire safety could set back the vision of cars that almost drive themselves. The design intent is to encase vehicles in an electronic safety cocoon. Electronics experts critical of the auto industry charge that all auto companies have been slow learning the intricacies of software-laden systems. For example, they suspect that electromagnetic shielding may be inadequate to deflect interference from cell phones and other devices, the reason we shut them off in airplanes. Interference can occur in now-you-see-it-now you-don’t spasms that leave no trace for later fault detection, and are hard or impossible to completely rule out5.
All automotive companies have had trouble integrating complex software. Software from suppliers has to mesh into the total system; no small task. In 2005, Mercedes had an embarrassing recall limited to just the braking software for premium E-Class cars6.
With complex software, it’s impossible to know if all the bugs have been scrubbed. The countermeasure is to design system architecture so that failsafe and system redundancy will rescue the package from small bobbles — sometimes called failing gracefully. In particular, one must program automotive software so that if the system receives contradictory signals to accelerate and brake at the same time, the brake signal overrides.
Toyotadid not have this feature, nor did several other car companies. In crisis, Akio Toyoda said that Toyotawould add brake override capability to its software systems. Others quietly said the same. Brake override quickly became an industry standard. Software is the integrative backbone of 21st century vehicles. A typical vehicle has about 10 million lines of code (more than commercial aircraft) functioning on 30-50 different controller boards. Software and sensors have to be integrated. For example, an airbag system has to decide within 30 milliseconds whether a vehicle is in a real crash, or whether it only hit a big bump, and airbag deployment may not be the only action triggered to minimize crash intensity7.
Vehicles more and more resemble computers and have to be regarded that way. The next generation of vehicle designs will multiply the total lines of code by several times. Besides rapid volume expansion, in the past 10 years Toyotaplunged deeper and deeper into complex system design — and may not have sensed how many big rocks lurked under that water. If you’re into lean thinking, one of the first questions is what kind of waste in automotive software systems results in them having more lines of code than aircraft that have flown by wire for decades? Integrating software modules from a number of different suppliers into one overall effective system has become another area of supplier collaboration. Toward A Life Cycle Service Business: Rolling computers have to be serviced like computers. Other products headed in a similar design direction will have to do likewise. In July 2009, Ford probably set the direction of a computerized vehicle industry by cranking up TeamCenter In-Vehicle Systems (IVS), developed by
Siemens8. This system starts a record for each vehicle at line-off, just as if it were a computer. Hardware subsystems are recorded, plus all software and any patches or updates to it. When a vehicle comes into a dealership, wand the vehicle identification number, and this record comes up. Without this, technician diagnosis takes more hunching and guessing. A system update can be downloaded from Dearborndirectly to the vehicle.
Ford will doubtless add more capability to this system. All automotive OEMs must cobble together some system to manage software. They know that better is necessary, so IVS is only a start. Such systems should help early diagnosis of problems in the field, turning most recalls into a regular service routines. (Almost unnoticed, Honda and GM both had high-volume recalls during the Toyotacrisis.) Complexity may force life cycle management before environmental needs mandate vehicle hardware designs that are much closer to cradle-to-cradle cycles.
Business Models: At some point, car companies and their suppliers will have to keep their products in peak operating condition over a life cycle. That will nudge them into a different business model. Customer reaction is apt to control the rate at which this happens. Many of us prefer to own a car rather than be tethered to a car service system, prayerful about its competence. Some of us want to service a vehicle or modify it ourselves, but deciding to mess with one may carry a risk. Will the OEM still honor its warranty? Or worse, in case of an accident, will the insurance company pay?
From an industry view, a big shift in business model is just as dicey — the riskiest kind of innovation a mature company can attempt, confusing to old customers that might switch, but attracting new ones is no sure bet. Working through gut-wrenching shifts in internal cultures and systems tax any leadership. For example, look at what the personal computer did to DEC, IBM, and Xerox. Technology wasn’t a problem. New business models weren’t a problem — in concept. Human drag dragged the companies down. And they were “high-tech.”
Post-Toyota Excellence
System Sustainability and Development: First, no system guarantees immunity from serious error. Overconfidence lulls us into thinking that a failsafe mechanism or protocol is 100 percent perfect. Why worry? Turn attention to something else.
Second, if Toyotacan undermine its own culture of excellence, anybody can, and most companies never shed some contradictory influences when beginning lean. Incentives favoring quantity (growth) over quality push a system to its limits. When penalties for performance lapses are harsh, discovering these limits is unpleasant. Bonus fever at big banks dramatically illustrates this. Reward smart people to rush on, unthinking, and they won’t think ahead.
Third, excellence with rapidly advancing technology is a big step up in learning speed from excellence in replication, addressed by lean manufacturing or TPS. Lean thinking is a basis for structuring a learning system for everyone, mostly embedded in the work itself — and extending to an enterprise if one considers networks of customers and suppliers. But reducing unknown, untried physical phenomena to waste-free operations entails learning much more, much faster. The auto industry illustrates this point. It’s trying to pour new technology into vehicle designs as fast as possible while assuring that each new design generation has the reliability Toyotahas long maintained.
Product and Process Design: For decades, the Toyota New Product Development System created new models with high reliability and few quality issues, using fewer engineers working within much shorter lead times. Much of the design work culled pre-tested, documented packages to integrate the best combination of all available packages to suit the needs of the intended drivers — and within a cost target. (The original RAV4 design was a work of genius with this system, a new platform with a new look, but with a 30 percent parts carry over from previous vehicles.)
This worked well as long as Toyotadesigners did not venture far outside the bounds of the tried and true. Flexibility of people, equipment, and tooling kept Toyota’s cost base below that of competitors. Toyotainnovated equipment and processes. Until the Prius in 1997, Toyotawas not known for being first to bring new product technology to market. Styling might be bland, but customers got a durable vehicle with quality attention to details like fit and finish. Whenever competitors bobbled, Toyotapicked up market share.
But in the past decade, despite all the auto industry’s financial problems, its pace of technical change has picked up. OEMs have been unable to master all the technology going into a vehicle for some time, so suppliers now contribute more significant innovation. Lean factories with good compliance quality practices are basic qualifications to be in the industry. The differentiator is the innovation that a supplier brings to vehicle design. In short, continuous process improvement is demanded; continuous innovation is expected; and integrating this continually changing circus is taking the industry into a different world.
On this new planet, Toyotahas stumbled, but is not out of the running. No other OEM is an acclaimed benchmark either. All are attempting to develop much better all-around operational learning systems. Toyota’s approach suggests that developing human discipline using such a system is as important as developing new software, and some competitors still have not grasped that.
But heavy dependence on software defines this new world. Developing a new engine, for instance, is nearly impossible without integrating a thicket of sensors with software. Simply hanging attachments on a block no longer suffices; both hardware and software must be too tightly integrated for that. To squeeze everything into a package, many components and systems have to be co-designed. Systems interoperability has to be checked by simulation before committing to a physical prototype. In addition, engine system architecture has to anticipate how it will integrate into the total systems for the vehicle. DFX, “design for everything” is here, and design lead-times are shortening. That’s why software engineers are a majority of many new product design teams today.
Now What?
For over a century the auto industry has influenced all manufacturing. The Ford system came from it. The Toyota Production System came from it. At times the industry has lagged and been arrogant; but its high profile and its final products represent almost every kind of manufacturing. In addition, safety and environmental demands on it have often portended what lay ahead for others. Lag very long in any aspect of this business, not just production, and you’re dead.
This industry is in transition, wrestling with its own complexity keeping up with itself. Its core technology is up for grabs: gasoline, diesel, LPG, flex fuel, fuel cell, hybrid, or all electric? Can vehicles triple present fuel economy? Will they do this by making driving nearly robotic while the interiors become entertainment and communications cabins?
So it’s time to move on, recognizing that production is only part of what it takes to be an excellent enterprise. Getting our entire act together to deliver new technology in a flawless package is an overwhelming operational challenge in this new world.
Reference:
1. Edmunds.com survey; summary at: www.edmunds.com/help/about/press/161706/
article.html.
2. “Anti-Trust Raids Could Start an Ugly New Era of Automaker-Supplier Relations,” Supplier Business (online), Mar. 1, 2010.
3. Lindsay Chappell, “Toyota’s U.S.-Japan Disconnect,” Automotive News, March 1, 2010.
4. Danny Robbins, “Toyota Secretive on Black Box Data,” Associated Press, March 8, 2010(one of many similar news items on this topic).
5. “Tin Whiskers May have Caused Toyota Recalls,” Feb. 22, 2010at www.evertiq.com/news/16264#comment (cites Keith Armstrong, called in to advise NHTSA on possible electronic causes of Toyota’s problems).
6. “Dark Days at Daimler” Business Week, Aug. 15, 2005.
7. Robert Charette, “This Car Runs on Code,” IEEE Spectrum, Feb. 2009. (Charette said that a premium car has 100 millions lines of code. Car companies say that it is only 10 million or so today, going up to 100 million by 2015, in another design cycle or two. IEEE blogs have a lot of insider commentary on automotive software and its problems.)
8. Siemens’ customized a system for Ford after sale tracking beyond the commercial PLC demo that you can see at: www.plm.automation.siemens.com/en_us/industries/automotive/index.shtml.
