Analysis: IBM and Pennsylvania share blame in massive IT train wreck

After significant delays and cost overruns, Pennsylvania has terminated a contract with IBM to provide new software for the state's unemployment compensation system.

The state's Secretary of Labor and Industry, Julia Hearthway, made the decision following an $800,000 independent study [pdf download], conducted by Carnegie Mellon University’s Software Engineering Institute, that blames both IBM and the state. At the time of cancellation, the project was 42 months behind schedule, with a $60 million cost overrun, based on an original budget of $106.9 million.

InfoWorld reports that Hearthway cancelled the project because, "the problems we've identified cannot be solved." IBM said it was "surprised by today's announcement," adding, "In complex information technology implementations, there is accountability on both sides for system performance and service delivery."

The department seemed incredulous that the cancellation surprised IBM: "We've been meeting weekly with them and they know full well all the problems. That's ridiculous that they say they're surprised. They must be in denial."

Untangling the mess

Large IT projects virtually always involve highly technical, labor-intensive activities performed by numerous people and multiple companies with competing priorities. Making matters more complicated, these big budget projects tend to create their own center of gravity in which procurement decisions are politicized and managers must coordinate diverse resources for extended periods.

The Carnegie Mellon report makes clear that this project failed because both sides had lapses in fulfilling their own responsibilities. As the state and IBM wrangle blame in legal proceedings that may take years to resolve, it is apparent that both sides share culpability. The report findings focus on three themes: software quality; procurement and implementation; and program management.

Software quality. The report explains that IBM's software development plan followed "industry and company standards and practices," although the company lacked the "discipline to execute." Among other deficiencies, IBM did not perform sufficient testing and quality assurance, leaving the system at risk for technical failure upon deployment.

The report also faults the state for not providing sufficient oversight on IBM in key areas such as testing. For example, the state "prematurely" accepted software into production that had "known defects impacting performance."

Another example of the state mismanaging its obligation to oversee the details of IBM’s work, the department approved requirements, "without fully understanding what they were approving.” The state's inexperience, combined with IBM's "lack of rigor and discipline," created an environment in which the software could possibly fail if released.

Procurement and implementation. The assessment report identified "four major weaknesses” in the department's purchasing, saying the RFP contained:

(1) unprioritized and often ambiguous requirements, (2) lack of detailed and objective source selection criteria, (3) the lack of specification of detailed... operational system attributes, performance measures, associated metrics, and a risk management framework, and (4) the absence of an explicit risk assessment of bidder plans, schedules, and assumptions.

The incomplete and ambiguous procurement specifications created a poor foundation for the entire project. Poorly defined projects are common when an organization lacks the experience to foresee potential pitfalls.

Although the department should have created better specifications, it appears IBM breached its duties and responsibilities by accepting this poorly defined project. Enterprise buyers engage large system integrators like IBM precisely to bring the experience needed to construct the project properly.

This situation calls IBM’s judgment and fitness to execute into question and raises the important question why they accepted this flawed project in the first place. One can speculate that IBM accepted the project despite being aware of the flaws, assuming that subsequent change orders would cover whatever issues might arise after the project started.

In addition to taking on a project defined ambiguously, the report says IBM made three important and unrealistic planning assumptions that created additional risk:

1. There would be no legislative or rule changes affecting the project
2. Unemployment claim demands would remain constant
3. There was no need to analyze the code, business logic, or database of the old system

Because of these wrong assumptions, the department and IBM "had different expectations and understandings about the planning and execution of the program at contract award." In plain English: this project was screwed from the start.

In summary, Pennsylvania wrote a poor requirements document and IBM accepted a flawed project despite the presence of substantial risks that should have been obvious.

Ineffective governance and program management. Throughout the project, neither the department nor IBM established clear metrics and expected outcomes against which they could measure progress.

In addition, both sides failed to provide the resources needed to handle the work required on this large project. Aside from saying the department failed to "provide the level of DLI [Department of Labor and Industry] staffing assumed in the Contractor's [IBM's] proposal," the report calls out the department's lack of resources:

From the start, DLI was not able to provide adequate resources to staff its planned management and governance approach. Further, DLI made no formal delegation of roles, responsibilities, and authority for management of the program. DLI's approach to managing the UCMS [unemployment compensation] program from contract award to early 2011, led to a situation in which no one in DLI was accountable and responsible for the administration of the program. As a result, there was no effective oversight or timely action to make definitive decisions to mitigate the systemic risks that were continually highlighted by the IV & V contractor.

The report also presents a scathing indictment of IBM's resource management on the project, which started in 2006:

The size and churn within the Contractor's workforce has also contributed to program discontinuity. Since the start of the project, 638 different contractor staff members have worked on the project with the majority of the workforce having less than one year on the project and 75 percent having less than two years.

Regarding governance, the report concludes:

Overall project governance and management were insufficient for the complexity of the effort necessary to achieve project and mission goals.

The leadership challenge

Unfortunately, cases such as this happen all the time. High rates of failure, attest to the complexity and challenges associated with running large IT projects. It is worthwhile noting that IBM is also at the center of a massive IT failure in Australia, in which an Au$6.19 million project somehow required Au$1.2 billion to “develop and operate properly.”

These cases raise an important question regarding relationships among enterprise buyers, software vendors, and system integrators. The collaborative nature of enterprise implementations demands that all parties work together to serve the customer’s best interests. Successful projects rely on a give-and-take in which the parties cooperate in good faith.

However, the collaboration breaks down when the parties succumb to their own pressures, limitations, and self-interested agendas. For example:

• The customer may not possess sufficient expertise to put together a solid request for proposal, as apparently happened here. In this case, the department effectively pushed significant project ambiguity onto IBM.
• The system integrator may bid the project low, hoping to get in the door and later create change orders that push up the price. We can only speculate as to IBM’s willingness to accept an ambiguous project, but these things come to mind.
• In some situations, although not discussed in this case, the software vendor’s sales people may suggest a product that does not adequately fit the customer’s needs.

Each of these illustrative points presents a situation in which conflicting goals and agendas can drive IT failure. Because these dynamics are so prevalent, I have written extensively about the IT Devil’s Triangle, which describes conflicting relationships among customer, software vendor, and services provider.

Perhaps the department and IBM could have salvaged the project had each side met its own responsibilities and obligations. Unfortunately, that did not happen.