In topological condensed matter physics, where major discoveries could hold big implications for fields like information technology, the reliability of such discoveries could be greatly enhanced by taking several steps, like presenting larger sets of data, say Sergey Frolov and colleagues. Their insights are based in part on four original experiments they did that correspond to either theory predictions or published work. “Overall,” write the authors, “although replication crises are typically perceived to be a problem in fields less quantitative than physics, the overemphasis on smoking- gun claims has the potential to affect the reliability of findings irrespective of field.” Electrons in solids are restricted by their energies to certain intervals or bands that determine the solid’s electronic properties. These bands can be re-ordered into new topologies in non-traditional or re-structured materials, yielding unique electronic properties that could be useful in applications such as quantum computing and data storage. Guided by theory, condensed matter physicists design experiments to identify when a new topological regime has been achieved in a material, looking for a particular “smoking gun” signal that indicates the new topology. Now, Sergey Frolov and colleagues show in four experiments in topological physics that these smoking gun signals can also arise from trivial phenomena that mimic the true signal of a topological change. By fine-tuning their experimental parameters, Frolov et al. suggest, some researchers may be introducing a confirmation bias that points to the “discovery” of a smoking gun. Releasing comprehensive data sets, disclosing the full volume of their study and exploring alternative scenarios for the signal could help mitigate the risk of misleading smoking guns, Frolov et al. conclude.
