It Takes a Village to Declassify an Error Bar

0
4

One might naively think that an error bar on a scientific measurement should not be classified if the measurement itself is made public. But the United States government (USG) thought otherwise when releasing data on meteors that its system of sensors measured over the past 30 years. Upon further reflection, however, the government approach makes sense since error bars might provide hints about the effectiveness of sensors used for national defense, potentially revealing weaknesses and compromising the security of the nation.

The meteor data were not expected to attract major attention by astronomers. But the situation changed at 6:20 P.M. EDT on March 31, 2019, when, after a radio interview on meteors, I suggested in an e-mail to my undergraduate student at Harvard University, Amir Siraj, that we search through the government database for interstellar meteors—that is, objects that originated outside the solar system prior to their collision with Earth.

While harvesting the data on fireballs and bolides, compiled by a NASA Jet Propulsion Lab center known as CNEOS, we discovered a tentative cousin to ‘Oumuamua—the only previously known interstellar object, which was seen through reflected sunlight as it passed near the Earth on October 19, 2017. The new object was a meter-size meteor detected as it burned up in Earth’s atmosphere near Papua New Guinea on January 8, 2014 at 17:05:34 UTC. It moved so fast that its past trajectory was inferred to be unbound to the sun. We calculated that its excess speed should have been around a whopping 40 kilometers per second outside the solar system.

This was our inference from the reported velocity and position of the meteor at its time of impact. But how confident should we be in this tantalizing conclusion, given that the tabulated USG measurements did not include error bars? In the submitted version of our preprint, we calibrated our statistical confidence by estimating error bars based on documented differences between measurements by USG sensors and independent detection systems for other meteors. We attempted to be as conservative as possible in attributing these discrepancies only to inaccuracies in the USG measurements. But this was not enough to garner widespread approval from colleagues.

As soon as our preprint was posted online on the arXiv, critics were quick to dismiss our conclusions on Twitter, Facebook and e-mail by repeatedly referring to another recent paper that asserted that the “USG sensors data are generally unreliable for orbit calculations.” Most meteor “experts” were skeptical of our inference without having direct evidence that it is actually wrong. In our preprint, we stated that the measured speed needs to be reduced by an unreasonable level of 45 percent in order for the 2014 meteor to be bound to the sun, far more than the expected uncertainty from a USG system of state-of-the-art sensors that was generously funded for national security purposes.

A few weeks after the submission of our paper for publication, we received two negative referee reports. One of the referees said: “The paper’s results hinge entirely upon a single measurement from an unknown instrument with unknown uncertainty…. So we have a deeply flawed claim of an interstellar fireball coupled with an unfounded estimate of frequency. But this allows the authors to go on and make spectacular claims about the spatial density of interstellar objects, the mass loss of such objects from nearby stars and the number of such objects that have hit the Earth in its history. Both the premise and the conclusions of the paper are deeply flawed and I recommend against publication.”

Fortunately, I visited Washington, D.C., around the same time to chair the Board on Physics and Astronomy of the National Academies. At dinner, I sat next to board member Alan Hurd from the National Security Education Center at the Los Alamos National Laboratory. When I mentioned this meteor story to Alan, he was intrigued to help from across the “fence of national security.” A week later, Alan arranged a meeting with Matt Heavner, the data science program manager for global security, intelligence and emerging threats at Los Alamos, and together they devised a plan to ask the relevant federal authorities for the declassification of the measurement error on the 2014 meteor and possibly the entire CNEOS catalog.

The plan devised by Alan and Matt worked perfectly. In a matter of days, Matt met with officials at the Office of Science and Technology Policy in the White House and subsequently spoke with the person who analyzed the 2014 meteor data. As a result, Amir and I were authorized to make the following formal statement in our paper: “The uncertainties on each of the velocity components are better than ±10 percent (Heavner, M., private communication).”

Given the CNEOS sample size of about 200 meteors and assuming standard Gaussian statistics for the declassified uncertainties, we calculated the probability that the 2014 meteor originated from the solar system as being one part in a hundred thousand. In other words, despite the sweeping doubts from experts, the declassified error bar implied that the meteor arrived from interstellar space with a statistical confidence of 99.999 percent. This is highly significant; when I realized that my wife is that special, I married her.

There are two important morals to this story. First and foremost, it is evident that most “experts” tend to adopt a skeptical rather than a neutral prior on an exciting scientific result that did not originate from one of them. The 2014 meteor provides a unique case study since the measurement uncertainties were unknown to certified experts but retrievable from other scientists.

There are a variety of possible reasons for this “expert bias,” including their experience that novel results are rare and that conservatism is a safe approach for avoiding mistakes and maintaining a good reputation, or the jealousy associated with the realization that members of the club of experienced experts were scooped by outsiders on a major discovery that caught the attention of multiple media outlets. Recognizing this “expert bias” should motivate “nonexperts” to compensate for it, by downgrading critical comments and social pressure that are not rooted in evidence and by encouraging creative scientists, especially early in their careers, to publish innovative insights.

The second moral is that we are very fortunate to have constructive colleagues in national laboratories like Hurd and Heavner, who are committed to service rather than their egos as otherwise is common in the culture of academia, and who are willing to promote blue sky science by removing unnecessary barriers to public dissemination of data.

Thanks to Alan and Matt, we now know that an interstellar meteor burned up in the Earth’s atmosphere 3.8 years before ‘Oumuamua was discovered. And we can also appreciate the benefits of having unbiased colleagues who choose not to jump on the bandwagon of dismissal but instead help in revealing a scientific truth. We can only hope that their exemplary practice would encourage gatekeepers of journal publications and academic clubs to be more open-minded and improve the efficiency of our scientific endeavor.


Source link
Originally posted by: Abraham Loeb

LEAVE A REPLY

Please enter your comment!
Please enter your name here