Concerned father, statistician develops software to improve skills therapy

UNIVERSITY PARK, Pa. — Mabel Ramos’s favorite song is “Ghostbusters” by Ray Parker Junior. From morning until night, if Mabel is awake, she is listening and dancing to — or asking to listen to — the number one Billboard hit from 1984. Though some parents might be annoyed by listening to a single song repeatedly, her father, Mark Ramos, said he is delighted by his daughter’s ability to dance, communicate and express her enthusiasm.

Mabel, who is five years old, has autism spectrum disorder. Mark, assistant research professor of health policy and administration at Penn State, said that she is reaching developmental milestones that are typical for a two- or three-year-old child. When she was two years old, her development was almost stagnant — she could not speak or sit up by herself. According to Mark, Mabel’s increased development is largely due to the therapies she receives.

When his daughter was in therapy, Mark discovered that — for a specific type of treatment — statistical methods could be used to help measure whether children were meeting developmental milestones. Mark wrote software code that therapists can use to choose thresholds for verifying that children with autism spectrum disorder and other developmental disabilities are mastering skills. The free software and its application were described in a recent article in Behavior Analysis in Practice.

Therapies for autism spectrum disorder

“When you take your child to their two-year pediatric appointment, the doctor hands you a checklist where you indicate the milestones they are reaching,” Mark explained. “When my wife and I went through the list, we realized that Mabel wasn’t developing her communication or movement abilities at all.”

Shortly after that appointment, Mabel began physical therapy, occupational therapy and Applied Behavior Analysis, which is commonly used to help people with autism spectrum disorder build specific skills. When Mark, who is a trained statistician, visited his daughter’s treatment facility, he was fascinated by Mabel’s treatment, especially a component of her therapy called discrete trial training (DTT).

Discrete trial training — learning step by step

In DTT, every task is broken down into its component parts. The participant performs each component repeatedly until they master it, and only then do they move on to the next component.

Most children learn how to wash their hands as one process from approaching the sink to drying their hands, for example, but this may not work for children with autism spectrum disorder.

Taught via DTT, however, hand washing might be broken down into sequential components, including: walk to the sink, turn on the water, wet hands, put soap on hands, rub hands, rub hands under water, turn off water and dry hands. The child would perform a single component of the task a prespecified number of times and reach a certain level of success — like successfully walking to the sink eight out of 10 times — before they would be taught the next component of turning on the water.

“I immediately loved DTT,” Mark said. “For one thing, I could see it was helping Mabel learn. But also, it was very scientific and systematic. DTT generates a lot of data that provides objective measures of progress. As a statistician, that was very exciting for me. But I noticed an issue between the performance thresholds and the level of mastery that the children had actually demonstrated.”

Performance is not mastery

A performance criterion is a specific score — a single data point, according to Mark. For example, it is a measure of whether a child was able to reach 80% on a specific trial of a task. Mastery, on the other hand, is a question of probability — a prediction of how often a child will be able to complete the task at any point in the future.

Mark learned that when children needed to master a skill with 80% success, they were typically expected to complete four out of five or eight out of 10 trials successfully. But as a trained statistician, Mark knew that probability of mastery is not equivalent to performance criteria and that the number of trials used mattered considerably.

“If a child performed 80% on a task, they will not necessarily be 80% successful each time; their actual projected mastery level would be a little lower,” Mark said. “Fortunately, there are basic statistical procedures that can estimate what level should be set as the performance criteria so students can perform a task to a specified level of mastery.”

Free software for therapists

Mark created a freely available software called Measurement of Individualized, Evidence‑Based Learning (MIEBL). In MIEBL, users — ideally, the clinicians who run DTT for children with developmental disabilities — enter the performance criterion and the number of items in the trial.

MIEBL employs Bayesian estimates — probabilities based on predicted performance and updated with observed data. For example, if the performance criterion is set at 80% on 10 items, the average mastery for students reaching this mark will be 77.27%. This means that children who reach 80% on the 10-item trial can only be expected to succeed at the same task 77.27% of the time in the future. So, if 80% mastery really matters, the performance criterion should be set to 90% of 10 items. Children who achieve 90% would be expected to successfully complete the task at least 80% of the time.

Performance criteria are already carefully set in DTT, Mark said. Typically, if a skill is important, but not critical — like color identification — the performance criteria will be set at 80%. For critical skills — like safely crossing the street — the performance criteria will be set at 100%.

According to Mark, the intention of MIEBL is to enable therapists to know exactly what level of mastery they can expect from participants for any given performance criteria and number of trials.

Mark recently shared the software with Mabel’s therapists and said he hopes they — and other therapists — will begin to use it routinely.

The future for Mabel and for Mark’s software

“To be clear, I do not think there are huge gaps in the DTT process,” Mark said. “I simply wanted to create an easy tool to let therapists verify whether children are meeting the targets they think they are hitting. If people use this tool and tweak their standards on certain tasks or simply verify that their current performance criteria are correct, that would be a great outcome.”

Though the project was very different than his typical work as a researcher in the Department of Health Policy and Administration, Mark said he loved working on this project because he believes the approach could eventually help his daughter or other children who need support.

“DTT is helping my daughter grow into the fun, capable, music-loving child she is becoming,” Mark said. 

And what will Mabel be focused on?

“She just loves to dance,” Mark said. “And fortunately, she changes her favorite song every week or two.”