By Devin Coldewey From TechCrunch

A new study is making waves in the worlds of tech and psychology by questioning the basis of thousands of papers and analyses with conflicting conclusions on the effect of screen time on well-being. The researchers claim is that the science doesn’t agree because it’s bad science. So is screen time good or bad? It’s not that simple.

The conclusions only make the mildest of claims about screen time, essentially that as defined it has about as much effect on well-being as potato consumption. Instinctively we may feel that not to be true; technology surely has a greater effect than that — but if it does, we haven’t found a way to judge it accurately.

The paper, by Oxford scientists Amy Orben and Andrew Przybylski, amounts to a sort of king-sized meta-analysis of studies that come to some conclusion about the relationship between technology and well-being among young people.

Their concern was that the large data sets and statistical methods employed by researchers looking into the question — for example, thousands and thousands of survey responses interacting with weeks of tracking data for each respondent — allowed for anomalies or false positives to be claimed as significant conclusions. It’s not that people are doing this on purpose necessarily, only that it’s a natural result of the approach many are taking.

“Unfortunately,” write the researchers in the paper, “the large number of participants in these designs means that small effects are easily publishable and, if positive, garner outsized press and policy attention.” (We’re a part of that equation, of course, but speaking for myself at least I try to include a grain of salt with such studies, indeed with this one as well.)

In order to show this, the researchers essentially redid the statistical analysis for several of these large data sets (Orben explains the process here), but instead of only choosing one result to present, they collected all the plausible ones they could find.

For example, imagine a study where the app use of a group of kids was tracked, and they were surveyed regularly on a variety of measures. The resulting (fictitious, I hasten to add) paper might say it found kids who use Instagram for more than two hours a day are three times as likely to suffer depressive episodes or suicidal ideations. What the paper doesn’t say, and which this new analysis could show, is that the bottom quartile is far more likely to suffer from ADHD, or the top five percent reported feeling they had a strong support network.

In the new study, any and all statistically significant results like those I just made up are detected and compared with one another. Maybe a study came out six months later that found the exact opposite in terms of ADHD but alsodidn’t state it as a conclusion.

This figure from the paper shows a few example behaviors that have more or less of an effect on well-being.

Ultimately what the Oxford study found was that there is no consistent good or bad effect, and although a very slight negative effect was noted, it was small enough that factors like having a single parent or needing to wear glasses were far more important.

Yet, and this is important to understand, the study does not conclude that technology has no negative or positive effect; such a broad conclusion would be untenable on its face. The data it rounds up are (as some experts point out with no ill will toward the paper) simply inadequate to the task and technology use is too variable to reduce to a single factor. Its conclusion is that studies so far have in fact been inconclusive and we need to go back to the drawing board.

“The nuanced picture provided by these results is in line with previous psychological and epidemiological research suggesting that the associations between digital screen-time and child outcomes are not as simple as many might think,” the researchers write.

Could, for example, social media use affect self-worth, either positively or negatively? Could be! But the ways that scientists have gone about trying to find out have, it seems, been inadequate.

In the future, the authors suggest, researchers should not only design their experiments more carefully, but be more transparent about their analysis. By committing to document all significant links in the data set they create, whether they fit the narrative or hypothesis or go against it, researchers show that they have not rigged the study from the start. Designing and iterating with this responsibility in mind will produce better studies and perhaps even some real conclusions.

What should parents, teachers, siblings and others take away from this? Not anything about screen time or whether tech is good or bad, certainly. Rather let it be another instance of the frequently learned lesson that science is a work in progress and must be considered very critically before application.

Your kid is an individual, and things like social media and technology affect them differently from other kids; it may very well be that your informed opinion of their character and habits, tempered with that of a teacher or psychologist, is far more accurate than the “latest study.”

Orben and Przybylski’s study, “The association between adolescent well-being and digital technology use,” appears in today’s issue of the journal Nature Human Behaviour.