Experts aren’t always impartial or very expert.

I am an Expert!

On December 6th in Williams v. Illinois, the U.S. Supreme Court heard oral argument on expert witness issues. As noted at SCOTUS Blog,

Plain English Issue: Whether a court violates a criminal defendant’s rights under the Confrontation Clause [Sixth Amendment] by allowing an expert witness to testify about the results of DNA testing conducted by another analyst who has not appeared as a witness at the trial. (Insert added)

The opinions of expert witnesses, as distinguished from facts within their personal knowledge, can be presented to a jury because experts are considered to have uncommon knowledge in obscure fields where laymen lack the qualifications to tread unassisted. The expert testimony in Williams had been offered, not to establish the reliability of the underlying tests, but to provide the expert’s conclusions based on test results. A problem is that tests sometimes get screwed up and, with only the conclusions drawn from them and no opportunity for counsel to cross examine those who performed the tests, conclusions drawn by the ultimate link in the chain may be no less flawed than the unexamined tests themselves; some test flaws impair the reliability of test results and some don’t. Even though instructed that an expert’s testimony cannot be considered as to the reliability of independently made tests on which he based his conclusions, juries can easily accept his conclusions, and hence implicitly the test results, because the expert accepted them. In those circumstances, there can be little opportunity for a jury to decide otherwise.

The relevant criminal law is complex and the Supreme Court could go either way in Williams v. Illinois. However, that is not what this article is about. Rather, it is about the often unquestioning acceptance of the opinions of our betters, the “experts.” Their opinions are often sought in judicial as well as non-judicial contexts. The media, the government and others often cite their opinions with no examination of the underlying data beyond summaries of whatever the expert may have concluded. An expert having relied upon data, the general assumption is that the data are therefore no less reliable than the expert himself. In a non-judicial context, there is usually no opportunity to cross examine the expert as to the bases of his reliance, his own methodology or anything else. The trend became quite apparent in the field of “climate science,” where only a few perverse “anti-science climate skeptics” were said to disagree with those who established and then perpetuated the mann-made global warming dogma; that dogma long prevailed and, while a bit tarnished, remains powerful. “Garbage in, garbage out” has largely been ignored.

Aside from working for about four years some forty years ago as a young prosecutor, then defense counsel and later special court martial military judge, my experience in criminal proceedings is nil. In later years, my trial experience was in hearings before administrative law judges at the Federal Communications Commission (FCC) where the rules of evidence are less rigorous than in criminal proceedings. Two proceedings involving expert witnesses come to mind as suggesting that reliance on “experts” who rely on work done by others should be tempered.

One proceeding involved an application for a permit requested by a television station in St. Joseph, Missouri to increase its radiated power and to relocate its antenna to a two thousand foot high tower at the middle of a triangle drawn from Topeka to Kansas City to St. Joseph. An at best marginally improved signal to the immediate St. Joseph area would have resulted, along with dramatically increased penetration of the more lucrative Kansas City and Topeka television markets. I represented a television station opposed to grant of the permit. Proceedings are rarely litigated on the basis of what the proponents and opponents actually want and fear; generally, such things are irrelevant and only matters concerning the “public interest,” as defined by the FCC, are considered.

The principal issue before the FCC was whether construction as proposed would deprive a few thousand residents of a small crescent shaped area of their only television signal of Grade B or better quality because they received no other cognizable service and because, after the proposed move to the new taller tower and despite the power increase, it was acknowledged that the calculated Grade B contour of the requesting station would no longer reach them. The principal issue became whether, without regard to engineering calculations, residents of the crescent actually received television service.

I offered expert testimony by a witness with long experience in the television ratings business, which relies heavily on statistically valid data analysis. He had prepared a questionnaire to be used for a telephone survey, the conduct of which he then supervised, of area residents picked by selecting the xth (e.g., the fifteenth) residential listing on every nth (e.g., third) page of the pertinent local telephone directories and then verifying residence in the crescent by finding the given address on a map. If the selected phone number was outside the crescent, the next name in the column was chosen, similarly verified and so on. Based on the results of the survey, he testified that hardly any (he provided numbers but I don’t remember what they were) television station was received there other than the St. Joseph station. While not challenging my expert’s methodology, the other side offered the expert testimony of a professor of marketing at a mid-western college as supporting a different conclusion. A college professor in a statistics-related field, his qualifications as an expert were also accepted.

The professor’s testimony relied on what he characterized as a “random sample” of residents within the crescent and on confidence limits calculated on the assumption that the sample had, in fact, been drawn randomly. He had designed a questionnaire and had recruited graduate students to conduct house-to-house interviews. He had instructed them to record answers to a question about which television stations each selected household received and deemed “acceptable for viewing.” There were three problems, at least two of which ultimately destroyed the purported randomness and hence the reliability of the survey to the point that the results could not be considered.

First, the students had been given no instructions on how to select, randomly, houses within the crescent (e.g., every fifth house on every third block); that’s considered very important by statisticians. Without such instructions, the students were free to select whichever houses they might wish, based on such subjective factors as they wished: barking dog — don’t interview there; tall roof top antenna — interview there because the desired results (many stations may be received) seem most likely; long driveway — too much trouble; ramshackle house or group of houses — the residents might be unfriendly or just have rabbit ears and not provide the desired results so don’t bother. This problem, alone, could have invalidate the survey.

Second, the students had been given no instruction on whether, or if so in what circumstances, to note on the survey forms any unsolicited comments. Many such comments were apparently offered and some students recorded a total of about ninety of them. There was no way to determine how many did not record how many comments. On one survey form, where station X was recorded as “acceptable for viewing,” a comment on the back of the form was “reception sucks.” Another comment was “only when the weather is good.” Many such comments were facially inconsistent with the responses recorded on the fronts of the forms. Third, the professor had neither seen the comments himself nor tabulated the data from the returned questionnaires but had delegated the task to graduate students. He had then relied upon those tabulations in preparing his (written) direct testimony.

On cross examination, the professor acknowledged that he had never previously seen any of the comments. I spent many pleasant hours cross examining him about facially obvious conflicts between the recorded survey responses and the comments noted on the backs of the forms. He was unprepared and had no answers. Because of those comments, and because of the inherently non-random nature of the survey, the judge threw it and the professor’s testimony out. It was very sloppy work by an “expert.” Had I not been able to cross examine him, his conclusions would have been accepted at face value.

Another pertinent example involved the only applicant for a permit to construct an FM radio station in a small community with no existing radio service (a “white area”). Under the technical calculations generally used by the FCC, the small community received no cognizable radio service from any station anywhere and hence the applicant (my client) was entitled to a substantial preference. However, under Technical Note 101 in the FCC’s rules, it might have been possible for those opposing the preference to show that some radio service was in reality available there due to the refraction of distant AM radio signals passing over a sharp mountain ridge. Opponents presented an expert in the field of radio engineering who testified that, based on measurements taken by his firm, a cognizable signal reached the small community. He had the requisite academic degree and had been employed in the field as a radio engineer for several years.

On cross examination, he acknowledged that he had received the requisite data from those who had done field measurements, had fed the data into a computer as his boss had directed and then printed out the computer-generated analysis which formed the sole basis of his direct testimony. He acknowledged that he had not otherwise analyzed the data or the topographical characteristics of the area before or after entering the data in the computer. He was unaware of what factors the computer software relied upon in reaching its conclusions. He was the only expert witness offered. Following cross examination, the judge struck his “expert” testimony and conclusions from the record and did not consider them in arriving at his final decision.

Experts are necessary because people without expertise in technical fields generally lack the competence to draw reasonable conclusions in those fields. Having very little knowledge of radio engineering, I spent about one whole day prior to each of the cross examinations being tutored by my own experts on flaws to look for. Many experts are competent and are well prepared to be questioned, vigorously, about the data underlying, and the bases for, their conclusions; if they aren’t or that can’t be done, little if any reliance should be placed on their conclusions.

In some cases, the data upon which experts rely are not even made available for analysis by other experts who are “non-believers” and sometimes it later turns out that the data themselves were, in fact, unreliable. Perhaps the most recent and notable occurrences have been in the politically and ideologically charged area of “climate science,” where

There are three threads in particular in the leaked documents which have sent a shock wave through informed observers across the world. Perhaps the most obvious, as lucidly put together by Willis Eschenbach (see McIntyre’s blog Climate Audit and Anthony Watt’s blog Watts Up With That ), is the highly disturbing series of emails which show how Dr Jones and his colleagues have for years been discussing the devious tactics whereby they could avoid releasing their data to outsiders under freedom of information laws.

They have come up with every possible excuse for concealing the background data on which their findings and temperature records were based.

This in itself has become a major scandal, not least Dr Jones’s refusal to release the basic data from which the CRU derives its hugely influential temperature record, which culminated last summer in his startling claim that much of the data from all over the world had simply got “lost”. Most incriminating of all are the emails in which scientists are advised to delete large chunks of data, which, when this is done after receipt of a freedom of information request, is a criminal offence.

This does not mean that we should never rely on experts; they are necessary in our world of exponentially expanding technological complexity. I could no more divine the reliability of specific DNA tests or the reliance to be placed on specific DNA analyses without an opportunity to hear questions answered by those who conducted the tests and made the analyses than I could independently measure the distance from my home to Mars. However, to spread a patina of infallibility over such experts, with no questions asked about the factual and other bases for their conclusions, leads far down the steep and slippery slopes of technocracy and toward lazy beliefs in whatever dogmas our high priests and their supporters may, for whatever reasons, currently select for promotion.

About danmillerinpanama

I was graduated from Yale University in 1963 with a B.A. in economics and from the University of Virginia School of law, where I was the notes editor of the Virginia Law Review in 1966. Following four years of active duty with the Army JAG Corps, with two tours in Korea, I entered private practice in Washington, D.C. specializing in communications law. I retired in 1996 to sail with my wife, Jeanie, on our sailboat Namaste to and in the Caribbean. In 2002, we settled in the Republic of Panama and live in a very rural area up in the mountains. I have contributed to Pajamas Media and Pajamas Tatler. In addition to my own blog, Dan Miller in Panama, I an an editor of Warsclerotic and contribute to China Daily Mail when I have something to write about North Korea.
This entry was posted in Addiction, Administrative Agencies, Corruption, Dep't of Information, Facts, FCC, Global Warming, Ideology, JAG, Law, Linguistics, Media-morphing, Military, Military justice, Politics, Society, Supreme Court, United States. Bookmark the permalink.

1 Response to Experts aren’t always impartial or very expert.

  1. Pingback: Opinion Forum » Experts Aren’t Always Impartial or Very Expert

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s