- Bikesafer Home
- About Us
- Crash Crisis
- Crash Study Info
- Study FAQ
- Study History
- Pilot Study
- Refer a Friend
Motorcycle Crash Causation Study: The Story So Far
We followed the money, after doing our cost comparisons.
We concluded that the cost structure of the study was totally out of whack, to the extent that the taxpayer is paying more than three times, per crash studied, what it ought to cost. We think that this might be what MSF saw when they decided to withhold their funds, and, if so, we agree with them.
SAFETEA-LU was passed in 2005 and section 5511 proposed the new Crash Causation study. The study contract was given to the Transportation Center at Oklahoma State University in Stillwater, which had a specialty in transportation infrastructure research. This was as a result of an earmark (pork) courtesy of Sen. Daniel Inhofe. The OSU researchers are quoted as 'not having asked' for the pork.
Meantime, shenanigans continued over the study. This included issues of funding, which included a reduction in funding from the original SAFETEA-LU $2.8 million down to about $2.1 million. Additional funding was eventually pieced together from various sources, as documented in our stakeholders section. The Safetyresearch.net blog entry from February has more.
Additionally, concern for bikers rights issues resulted in restrictions on the study. It won't be allowed to make recommendations for public policy changes, especially in relation to helmets, as discussed in our FAQ.
Although the study is under the management of US DOT's Federal Highway Admin., FHWA, the pilot study and preparatory work has been done by Federal Highway Transport Safety Administration, FHTSA. They contracted out this entire piece to Westat.
The Federal funds allocated to the study include $994K allocated to the 2006 budget and an additional $1.408M in the 2007 year.
The contract covered several activities:
- Training of researchers in data collection and crash reconstruction as required by the OECD guidelines.
- Systems development, including creation of a Microsoft Access database and data entry screens for validating data and providing dropdown help lists for code selection. This is much fancier than the basic Excel (spreadsheet program) and SPSS (a standard statistics package) used in the Thai study and is likely to address issues the Thai researchers reported in entering the correct system codings for crash data items.
- Carol Tan's 2008 report gives milestones of November 2006 for the completion of the Data Collection Instruments and Coding in November 2006, and the completion of the data collection training in April 2007.
- The pilot study crash data collection phase started on 8th December 2008 and ran for three months, as scheduled. We don't know exactly when it finished, but this FHWA document was updated with the total Westat contract amount on or before March 10th, which at the time of writing is the last update for this document. The final price is given as $994,201. The pilot collected data on 53 crashes, which was more than the 29 originally estimated.
- Other activities such as consultancy, QA (Quality Assurance), Westat's profit margin and administration were also charged under the contract.
We don't know how the contract amount was spent, but estimate that about half was spent on the pilot study data collection, and the rest for systems development, training and other headings. Dynamic Science subcontracted the crash data collection work, using three researchers based in Los Angeles, at a team cost probably in excess of $300 per hour. We have no indication that these costs contributed to the excessive per-crash collection costs, we are inclined to think that the data collection seems to have been efficiently done at a good level of productivity.
Freedom of Information Act documents on the Pilot Study Contract.
Bikesafer.com submitted a Freedom of Information Act request and we have been told by NHTSA that the full report will be posted on their website by May 1, 2010. We analyzed the pilot study contract information here. The contract was originally issued in September 2005 for a total of $467,103, with the work to be complete by February 8, 2007. In the event, the contract was amended eight times, the price rose to $994,201 and it was not completed until December 2009. This is a 112% cost overrun and a more than 200% time overrun.
Cost and Time Overruns
Odd things we noticed: Revision 5, was signed in April, 2008 when the cost was jacked up to $994K. The final contract amount was set at almost exactly the same amount as the 2006 funds allocated under SAFETEA-LU. As the Fed 2006 funding was presumably a known factor in April 2008, whereas we consider it unlikely that the Feds could have predicted in 2006 a 2008 contract cost down to the nearest couple of hundred bucks, we think it is reasonable to infer that Westat simply set the revision 5 price to match the funds available.
That's quite a trick. The contract payments were split into a fixed fee for Westat, and reimbursement for actual costs, and covered activities out to December 2009. Under FOIA section (2) (b), NHTSA redacted the amounts of the split, so we don't know how it was split. Apparently, Westat made a mistake in the split amounts and had to be changed in June 2008 under contract revision 6. That still required them to accurately estimate expenditure out 18 months, quite a feat for a crew that made a half-million dollar error in the overall project cost and an almost three year time overrun. A case might be made for tailoring expenditure to consume the available funding.
How the money was spent
One more point that arises from Revision 5. They specified that they had spent $150,440 of the contract funds up to February 29, 2008. As the following step, C.4.5, was still due under revision 5 - April 25, we infer that this $150,440 covered only the steps C.4.1, C.4.2 and C.4.3, which were the to read the OECD methodology, host a kick-off meeting and prepare a work plan. We don't see any other deliverables from C.4.5 onwards that could have been completed before the revision 5 signing in April 2008. This means that over 32% of the initial contract amount went to reading a few documents, having a meeting and preparing a work plan. As the work plan went through more revisions after 2008, we don't think the original work plan could have been very good. This might be a good place to start looking for the reasons for the huge cost and time overruns.
It seems that Westat was almost miraculously good at predicting how to wring the last few bucks out of this cash cow, and very bad at planning and budgeting a project. Simultaneously. An amazing feat of self-serving.
Is there enough here to get the Westat guys in front of the Transportation Committee to answer some tough questions about the use of our tax money? We would point out that over 13,000 motorcyclists died in crashes during the three year project overrun.
The Hurt study cost $501,814. Assuming the university levied 20% for the institution and project overhead, that indicates a per-crash cost of $446. Adjusting for inflation at about 230 percent between 1981 and 2009, that Hurt numbers are worth about $1.15M in today's dollars for the total cost and about $1025 per accident. Hurt has some costs that the current study doesn't - in 1981 the computing was probably done on a mainframe, and the 29 listed project personnel included two analyst programmers, for example. The OECD methodology requires some additional steps, and records about twice the number of data items per crash, but the OECD methodology is based on Hurt's and both studies require the same basic steps. Hurt interviewed 2310 bikers for his control group as opposed to OSU's proposed 600. And the project personnel currently recognized for the OSU study are Dr. Ahmed and three researchers from Dynamic Research, Inc., a much smaller effort than Hurt's 29-person team. There is no obvious reason why the current study should cost eight to nine times what Hurt's did.
Maids was done in Europe and the data was collected in 1999 and 2000. The total study cost was 2.5M Euros. The average exchange rate in 1999 was 0.94 and 1.08 in 2000, so the dollar and euro were roughly at parity during this time. We'll leave the numbers in Euros. Schlepping 20% for the five institutions as before, that would indicate a per-crash variable cost of 2170 Euros per crash sampled. The OECD methodology used in Maids is exactly the same as for the current study, but the Europeans had the added complication of five locations in five different countries with presumed travel and translation costs, whereas the current study is being done in Los Angeles. Maids lists 48 project personnel. It seems like the Europeans also got a much better deal than the one on offer from OSU.
The Thailand study was privately funded by Honda and we don't know the cost, but doubt it would translate well to US conditions.
The Safetyresearch.net blog entry from February of this year quotes OSU researchers giving a number of $7M - that's an uptick of two million or more in eight months. At the February figure, the per-crash cost, assuming 20 percent for OSU overhead, and $994K on the Westat contract and pilot, comes to about $5440 each for the other 847 crashes.
That's a much better deal than the one now on offer. One wonders what might have changed in eight months to cause an inflation of more than a third in per-crash costs.
The Cost Problem
Dr Samir Ahmed, the Principal Investigator at OSU, has been an unhappy camper during this process. He's been putting out escalating total study cost figures - from $7M in February 2009 to northwards of $10M depending on which day you ask. He has complained about 'not having an account' and 'not asking' for the assignment. He seems to have been heavily circumscribed by restrictions on what the study can say. He described the funding available as "generous" and "beyond what was initially thought would be needed for the research" as recently as 2007. Now he's lost the MSF funds and is proposing a lame, truncated study.
He doesn't seem to have spent the four years since SAFETEA-LU studying up on this new academic area for him. When we talked to him in October 2009, he appeared not to have read the important Maids Study, which is a recent use of the OECD methodology. He wasn't able to comment on the error Maids made in their ABS population group sampling, or the fact that both Hurt and Maids asserted that weather has no effect on crash causation, and seemed vague on motorcycle technical issues and the riding experience in general.
Bikesafer.com invites Dr. Ahmed to a live oral examination on the published literature of motorcycle studies, which we will video and publish on U-tube. And then to take a motorcycle around an MSF Basic RiderCourse range, again on video. Name your time, Dr. A., and choose your bike.
In the absence of reliable information, we're not going to get into the data collection costs, which have been estimated at $8000 to $9000 per crash. This is excessive in terms of what these studies have historically cost.
There have been suggestions that there were quality problems in the Maids and Thai studies. We investigated, talked to researchers from both studies and searched high and low for any evidence of systemic quality issues in either study. We found a few details and evidence of initial teething problems with the development of the OECD methodology, but nothing suggesting that a major increase in QA sampling is required, which is one of the possible issues inflating costs. We know we can't prove a negative, but we hope that anyone who is alleging quality issues with these studies will be prepared to provide good evidence to that effect.
We also note that although Westat was the major contractor for the pilot study, that OSU is capable of the administration work, possibly assisted by seconded personnel from the study stakeholders. They should do that for their administrative cut of the study funds. It is inefficient to contract out simple administrative tasks for an additional markup.
We also suggest additional use of seconded technical and consultative resources from the industry and safety organizations, like Maids did.
The issue now rests with the 900 sample size. The initial requirement for 900 to 1200 crashes is 'the least we consider adequate from a statistical point of view' according to Dr Ahmed in 2007.
The cost issues led to the MSF funds being withheld and the study is now gutted to 300 crash samples.
The only way to get a study that is adequate from a statistical point of view is to get costs under control, reinstate the sample size to 900 plus and get the MSF funding back on the table.
The late great Harry Hurt was recently interviewed on another subject, and in general conversation, is quoted in this forum entry as "...he expressed a lot of disappointment in the OSU study based on the limitations placed on it by the funding organization."
OSU has taken on the sacred trust of performing this study. They have an obligation to make sure it is done right. Lives are at stake.