Sunday, April 25, 2010

Questions about the Crash Study Pilot

At Bikesafer.com, we've been closely following the ongoing motorcycle crash study crisis.  Although the OSU study has been somewhat upstaged by the new MSF/VTTI naturalistic study, we are still interested in why the OSU study is such a basket case.  After being misinformed on various aspects of the finances, from OSU's estimates of up to $12M for a full study and a per-crash cost estimate of $8K to 9K per crash, we put in a Freedom of Information Act request for information on the pilot study, on which OSU seemed to be basing its own cost estimates behind their ever-greedier funding grabs.   These shennanigans presumably helped to persuade MSF to take its funding to VTTI's study, so we think the manner in which the pilot was conducted is critical to understanding why the OSU study disintegrated so spectacularly.   


We received the FOIA documents a couple of weeks ago, and we analyzed its detailed schedules, eight revisions and sparse money amounts.  We have our pilot study timeline analysis here.  The timeline just irons out a confused story containing four separate revisions of the project schedule, and ties things together a little.   It is a work in progress, but we think very accurate, as far as it goes.  We added some commentary here.  The contract was originally issued in September 2005 for a total of $467,103, with the work to be complete by February 8, 2007. In the event, the contract was amended eight times, the price rose to $994,201 and it was not completed until December 2009. This is a 112% cost overrun and a more than 200% time overrun.

Cost and Time Overruns

Odd things we noticed: Revision 5, was signed in April, 2008 when the cost was jacked up to $994K. The final contract amount was set at almost exactly the same amount as the 2006 funds allocated under SAFETEA-LU. As the Fed 2006 funding was presumably a known factor in April 2008, whereas we consider it unlikely that the Feds could have predicted in 2006 a 2008 contract cost down to the nearest couple of hundred bucks, we think it is reasonable to infer that Westat simply set the revision 5 price to match the funds available.

That's quite a trick. The contract payments were split into a fixed fee for Westat, and reimbursement for actual costs, and covered activities out to December 2009. Under FOIA section (2) (b), NHTSA redacted the amounts of the split, so we don't know how it was split. Apparently, Westat made a mistake in the split amounts and had to be changed in June 2008 under contract revision 6. That still required them to accurately estimate expenditure out 18 months, quite a feat for a crew that made a half-million dollar error in the overall project cost and an almost three year time overrun. A case might be made for tailoring expenditure to consume the available funding. 

How the money was spent

One more point that arises from Revision 5. They specified that they had spent $150,440 of the contract funds up to February 29, 2008. As the following step, C.4.5, was still due under revision 5 - April 25, we infer that this $150,440 covered only the steps C.4.1, C.4.2 and C.4.3, which were the to read the OECD methodology, host a kick-off meeting and prepare a work plan. We don't see any other deliveribles from C.4.5 onwards that could have been completed before the revision 5 signing in April 2008. This means that over 32% of the initial contract amount went to reading a few documents, having a meeting and preparing a work plan. As the work plan went through more revisions after 2008, we don't think the original work plan could have been very good. This might be a good place to start looking for the reasons for the huge cost and time overruns.

It seems that Westat was almost miraculously good at predicting how to wring the last few bucks out of this cash cow, and very bad at planning and budgeting a project. Simultaneously. An amazing feat of self-serving.

Is there enough here to get the Westat guys in front of the Transportation Committee to answer some tough questions about the use of our tax money? We would point out that over 13,000 motorcyclists died in crashes during the three year project overrun.  And the cost estimates may have been responsible to some extent for the funding flight. 

The role of Dynamic Science in performing much of the data collection work is not revealed in these documents.  Their competitively-bid contract with Westat is not public knowledge, so all we can say is that they presumably charged at least for eight weeks of training and about three months of data collection for their team of three people, which are documented in the timeline.  This might be about 2500 hours assuming 21 40-hour weeks, but we don't know this.  We can't tell anything about the cost structure of the case-gathering work.  But it is getting a little clearer what questions remain to be asked.
 
The timeline is a work in progress.  There are more external, documentible facts that we are adding to the timeline, and the anticipated technical report, due out from NHTSA by May 1st, will yield more information.  We'll bring it to you.

0 Comments:

Post a Comment

Links to this post:

Create a Link

<< Home