Saturday, November 14, 2009

Crash Causation Study update.

Firstly, I need to correct an error, the Maids and Thai study was done by professional researchers, not grad students as we had assumed incorrectly. We have updated the site accordingly.  We apologize to all concerned and anyone we misled.  We have corrected the information here.   We plan to refine and correct our information in real time and are researching the issue constantly. 

That does mean that the 2.5 million Euro Maids study costs are directly comparable to the current OSU study, which has more than twice the potential funding.

We found that Westat completed the pilot study, crash data collection training, systems development and administration work for $942K. As far as we can tell,  Dynamic Systems did three months of work and collected the data on 53 crashes for their estimated rate of $350 an hour (for the team) and about 3.5 mandays per crash.  Although it might seem like a lot, it is not the cause of the cost overrun.  They bid competitively for the work and collected much more than the estimated number of crashes for the pilot, still completing on time. Although we don't know the exact amount they charged, we think they executed well and probably earned their pay.  

More than half of the estimated $8.5K per crash went to QA work, consultancy and administration costs, best we can tell. 

A large share of this probably went to QA.  The QA work for the pilot  project was overdone.   Instead of the OECD mandated 10 percent QA rate, it was done at the 100% level in the pilot and it is planned to do it at a 70% rate for the entire sample.  We don't know what Westat charged for administration, but OSU should be doing this for their cut of the study funds.  This, plus maybe some consulting and any overtime or additional work charged by Dynamic Science probably accounts for the lion's share of the pilot per-crash costs.

In addition, Maids created technical support committees for their project from personnel seconded by the bike manufacturers, and organizations like MSF, AMA,,  the Head Protection Lab and the state mororcycle safety officers might be willing to provide consultancy support.  This would help minimize any additional technical support needed.

We think that part of the problem was the funds voted for the Westat work in 2006 was this $944K.  If they got the project overhead part of the project - the training, system work and admin - done for maybe a bit over a half million, and had (say) about 450K left to collect the 53 crash studies, with QA, admin and consulting, then they had an incentive to either spend the approx $8.5K per crash or leave cash on the table.  This is not necessarily indicative of the true future data collection costs, especially if the QA issue is corrected and consultancy costs and admin reduced. This was a costs-plus-fee contract and the amound was fixed in the 2006 SAFETEA-LU allocations, so we don't know if Westat had any incentives to do the job economically.

The encouragement to spend a big chunk of the per-crash funding on QA, at ten times the level mandated by the OECD methodology, came from OSU.  They cited quality issues in Maids and the Thai study, which are not warranted.   Besides a few teething problems getting the methodology right for control group recruitment, soon sorted, we can find no-one else that alleges quality issues in either study, and we spoke to several knowledgeable people and study participants on the issue.  These studies were of excellent quality, because they followed the methodology, executed well and had over 900 crashes sampled.  We should consider ourselves fortunate to have studies this good. 

Unfortunately, the current study in its apparent state of disarray does not seem likely to produce anything as good as Maids.  To the detriment of the many bikers who might needlessly die as a result.

We think, if you can get the MSF funds back on the table, there is more than enough funding to produce a good study with the full 900 or more samples.

OSU has been quoted as being reluctant to commit to the right numbers for fear they might be liable for the full study.  But they had no hesitation in taking public funds for so doing. 

We think the House Transportation Committee should look into this problem.  The fact that the original projected 900 to 1200 crash objective has been overridden by the officials charged with its administration is, in our view, sufficient grounds for the committee to look into it, and hold hearings if necessary.

Maybe if someone waved a big stick, the study stakeholders might be encouraged to get around a table, accept responsibility for this sacred trust, and sort out a reasonable project plan that gets the job done.

We have been promoting our petition and speaking to bike groups around the country.  I can say that motorcyclists are getting mad, and supporting the petition, once the facts are laid out.  We are helping several conventional news outlets work on stories on the issue, and talking to bikers.  How many angry bikers are the stakeholders trying to collect? 

And we will continue to find the facts, follow the money, and obtain better information


Sunday, November 8, 2009

Motorcycle Crash Causation Study: Some fixes.

We are heart-scalded.  The crash causation study is approved by DOT, a week later MSF piles in with their announcement withholding their funding.  Resisting the urge to paint the MSF as the bad guys, despite their poor gamesmanship and the fashion in the motorcycle safety industry for MSF to be the root of all evil, we spent the month investigating the problem.
We found a lot of information, documented the whole issue, and we followed the money, identifying four institutions and enterprises who have had income from the study funds.  We also started a petition, which has had very good support from groups like SCRC, ABATE individual members, Wolf Pack and Ride like a Pro.
But we felt bad.   Faced with the fiasco that is the study, and taking on the task of investigating the problem among a web of confidentiality clauses and doublespeak, after identifying where the vast majority of funds went, including a lot of unnecessary spending and preferential contracts, we decided that we would be nattering nabobs of negativity no more.
Today, we are announcing our proposed fix for the study problems.  Basically, it identifies several areas where the study managers strayed from the OECD methodology, points out five specific areas of overspending and makes recommendations to wind in the out-of-control per-crash study costs.
In our view, if these recommendations are implemented and the MSF funds are drawn back into the process, there is more than sufficient funding available for not 900 but in excess of the 1200 crash sample originally envisioned.  1500 crashes would be possible, at good quality.
Study stakeholders, now you can make it so.

At least, rather than a stream of negativity, now you have some concrete proposals to discuss.   Time to move forward on this issue.