Quantcast

SC Connecticut News

Wednesday, October 16, 2024

Yale study emphasizes need for larger datasets in brain-behavior machine learning

Webp pdb35xqzt3c00yieveg2mvly0a3u

Peter Salovey President | Yale University

Peter Salovey President | Yale University

Datasets that are too small may lead researchers to overlook relationships between the brain and behavior, a new study finds.

By Mallory Locklear

July 31, 2024

When designing machine learning models, researchers first train the models to recognize data patterns and then test their effectiveness. However, if the datasets used to train and test aren’t sufficiently large, models may appear less capable than they actually are, a new Yale study reports.

For models that identify patterns between the brain and behavior, this could have implications for future research, contribute to the replication crisis affecting psychological research, and hamper understanding of the human brain, researchers say.

The findings were published July 31 in the journal Nature Human Behavior.

Researchers increasingly use machine learning models to uncover patterns linking brain structure or function to cognitive attributes like attention or symptoms of depression. These links allow researchers to better understand how the brain contributes to these attributes and potentially predict who might be at risk for certain cognitive challenges based on brain imaging alone.

But models are only useful if they’re accurate across the general population, not just among those included in the training data.

Often, researchers split one dataset into a larger portion for training the model and a smaller portion for testing its ability. A growing number of studies have subjected machine learning models to more rigorous tests by evaluating their generalizability on an entirely different dataset made available by other researchers.

“And that’s good,” said Matthew Rosenblatt, lead author of the study and a graduate student in Dustin Scheinost's lab at Yale School of Medicine. “If you can show something works in a totally different dataset, then it’s probably a robust brain-behavior relationship.”

Adding another dataset into the mix comes with complications regarding a study’s “power.” Statistical power is the probability that a research study will detect an effect if one exists. For example, a child’s height is closely related to their age. If adequately powered, that relationship will be observed; if “low-powered,” there’s a higher risk of overlooking it.

There are two important aspects to statistical power: dataset size (sample size) and effect size. The smaller one aspect is, the larger the other needs to be. Strong relationships like age and height can be observed even in small datasets. But subtler relationships require more data collection.

While equations exist to calculate how big a dataset should be for adequate power, there aren’t any easy calculations for determining how large two datasets—one training and one testing—should be.

To understand how training and testing dataset sizes affect study power, researchers used data from six neuroimaging studies and resampled it repeatedly while changing dataset sizes.

“We showed that statistical power requires relatively large sample sizes for both training and external testing datasets,” said Rosenblatt. “When we looked at published studies using this approach — testing models on a second dataset — we found most datasets were too small, underpowering their studies.”

Among published studies examined by researchers, median sizes for training and testing datasets were 129 and 108 participants respectively. For measures with large effect sizes like age, those sizes were sufficient for adequate power. But for medium effect sizes like working memory, such datasets resulted in a 51% chance of failing to detect a relationship; for low effect sizes like attention problems, those odds increased to 91%.

“For these measures with smaller effect sizes, researchers may need datasets of hundreds to thousands of people,” said Rosenblatt.

As more neuroimaging datasets become available, Rosenblatt expects more researchers will opt to test their models on separate datasets.

“That’s a move in the right direction,” said Scheinost. “Especially with reproducibility being the problem it is; validating a model on an external dataset is one solution. But we want people to think about their dataset sizes. Researchers must do what they can with available data but should aim to test externally as more data becomes available.”

ORGANIZATIONS IN THIS STORY

!RECEIVE ALERTS

The next time we write about any of these orgs, we’ll email you a link to the story. You may edit your settings or unsubscribe at any time.
Sign-up

DONATE

Help support the Metric Media Foundation's mission to restore community based news.
Donate

MORE NEWS