Single Case Experimental Design Conference: Enhancing Replication and Validity
By: M. Christopher Newland and Tim Slocum
ABAI, with valuable support from The Wing Institute, hosted a conference to enhance the use and rigor of single-case experimental designs (SCEDs) in the scientific literature.
The three-day event, held in Minneapolis in late September 2025, brought together 139 researchers, practitioners, reviewers, and journal editors, including 19 students. In this highly interactive conference, they explored current developments in the design of experiments and interventions using methods that are such a fundamental component of our science. With decades of experience with SCEDs, however, some practices have evolved that challenge the quality of results and their reproducibility. This, of course, is similar to the replicability crises that have arisen in other arenas. Confronting these issues head on is the major theme of this conference.
Highly participatory Conference
A typical conference involves a slate of speakers and an audience that sits and listens, but not this one! The goal of the conference was an ambitious one: culture change. The conference organizers, Tim Slocum and Chris Newland, wanted to advance the SCED culture so that authors, reviewers, and practitioners used and reported on these designs with enhanced rigor to improve their replicability and their acceptance in the broader community.
A key step in changing culture is changing how members of that community talk among themselves. Accordingly, the program was divided into topic segments, with each topic covered by two to four speakers. After each topic, participants broke out into small groups that were tasked to discuss the topic, address pre-arranged questions about it, and pose questions for the authors and audience to discuss at panels at the end of the conference. A full half of the conference was devoted to group discussions so audience members were active participants in the entire event, not passive recipients of what was said.
Focused program Comprising experts on SCeds
The program covered a range of topics pertinent to SCEDs, including how they arose and how to conduct, report, consolidate, and review them.
Historical perspective. Mike Perone and Jennifer Ledford launched the conference with a review of how SCEDs arose in the experimental analysis of behavior. This experimental design originated as a powerful alternative to group designs, one that allowed investigators to visualize the operation of fundamental behavioral processes and, later, to demonstrate how these principles can be applied
to address important issues in a variety of settings. Replication was baked into these designs, a feature that guaranteed reproducibility. The designs evolved alongside the propagation of standards and quality-control control measures and as they did, the issue of what types of errors arise became clearly defined. Much is made about Type 1 errors (false positives), but Type 2 errors (missing an effect) can also lead to the failure to identify a phenomenon or an effective intervention. The flexibility and rigor of SCEDs reduces the risk of both error types.
Questionable and Improved Research Practices. Tim Slocum reviewed conclusions arising from a series of workshops he and his colleagues conducted to identify practices that mitigate against the likelihood of replication or that undermine the conclusions that are drawn.These include, for example, insufficient reporting of experimental procedures, data recruitment or selection factors, cherry-picking data or participants, or disconnects between results and discussion. Not one to stop at identifying problems, Dr. Slocum suggested approaches for remedying these problems, including full reporting of procedures, greater care in describing results and discussion, and rigorous peer review. This work, which inspired this SCED conference, was reported in a recent paper in Perspectives on Behavior Science.
Systematic Reviews of SCEDs. James Pustejovsky and Daniel Drevon addressed gaps in the review of SCEDs. There is a long-standing literature on how to combine the results of group designs into comprehensive narratives or quantitative or metanalytic reviews, a synthesis that is assisted when data are normally distributed and effect sizes can be measured in standard-deviation units. Such syntheses are often unavailable to reviewers of SCEDS and, accordingly, approaches to conducting quantitative reviews of SCEDs are less developed. Improved techniques to review SCEDs will enhance their acceptance by other investigators, third-party payers, and those who establish standards for evidence-based practices. After identifying challenges to combining the results from SCEDs, Drs. Pustejovsky and Drevon highlighted good practices in publishing studies and in conducting reviews in order to strengthen the quality of the reviews and the rigor of our science and practice.
Reporting Standards. Wendy Machalicek, Kimberly Vannest, and Anna Ingeborg Petursdottir addressed the importance of reporting the methods and results fully and comprehensively, while avoiding the cherry-picking of results and participants. Too often the rationale for making a conclusion from a chart is poorly developed, leaving readers or other consumers to wonder just how the data supports the conclusions drawn. And finally, the reporting of statistical conclusions must be made thoroughly, and in line with standards existing with group designs, so as to avoid the rate of false positives, which can result in promoting ineffective practices and false negatives, which can prevent a potentially useful practice from being propagated. Comprehensive reporting practices are important elements in facilitating the review, replication, and application of results from SCED-based studies.
Analyzing Single-Case Results. In a double session, John Ferron, Katie Wolf, John Michael Falligant, and Mariola Moeyaert provided a framework for the analysis of various single-case designs. The topics included visual analyses of data, the use of statistical models tailored for the analysis of SCED data, and the application of quantitative models that have arisen from the basic experimental literature. Considerable attention was given to the assumptions underlying the methods and the conditions under which statistical methods do and do not apply. A simplistic incorporation of techniques developed for group designs can lead to erroneous conclusions but analytic techniques that are grounded in the SCED designs that give rise to the data collected will improve the quality and rigor of the analyses and decrease the Type 1 (false positives) error rate.
Mixed Methods Approaches. Angel Fettig and Shawna Harbin described the importance of incorporat-ing qualitive research designs and linking them to the SCEDs used by those at the conference. Properly conducted and combined with traditional SCEDs, qualitative methods can help address questions about improving interventions, identifying why something succeeded, or failed, or help resolve discrepancies seen in home, school, or laboratory settings.
Quality-Control Indicators. Joseph Lambert and Tara Fahmie examined the benefits and limitations of expert consensus panels that develop research design conventions. Quality control standards were developed to guide the design of research prospectively and to standardize retrospective appraisals of the quality of evidence. However, rigidly applied, they become a procrustean bed on which designs customized for specific situations may not fit comfortably. When our goal is the advancement of our understanding of behavior and how to apply this understanding, there is a risk that an over-emphasize of such indicators can, as the authors note, result in the tail wagging the dog.
Facilitating Replication Research. Jason Travers and Matt Tincani addressed the important issue of replication, a critical characteristic of any science. If we have identified an important regularity or behavior principle, then that should be reproducible when the conditions are repeated. Indeed, the degree to which a phenomenon is or is not reproducible is important in understanding its determinants and its application. It is crucial for external and internal validity. The presenters addressed the broader replication crisis and then focused on its appearance in SCED studies. By comparing the published with the gray literature (dissertations, theses, unpublished reports) they offered evidence that null results are underreported, and that can represent an important loss of information for others who seek to reproduce or apply a result.
Presenters and Reviewer Panels. Chris Newland, a conference co-organizer, chaired a panel of all the presenters in which they answered or, at least addressed, questions and comments that arose during break-out sessions. This resulted in a lively discussion and deeper examination of many of the issues that arose in the conference.
He also chaired a panel of journal editors: Terry Falcomata, Daniel Fienup, Suzanne Mitchell, Stephanie Peterson, and Mandy Rispoli who discussed how journals can improve the publication standards for SCEDs. If there is to be any improvement in the rigor of such designs, it must come from the journal editors who serve as the gatekeepers for what gets published. If contingencies matter at all, then these are the folks who set and apply them, and if they work, then the authors of papers will rise to the occasion and produce a literature that is even more secure, and more rigorous than the high-quality papers that we have now.
Conference reception
ABAI is a data-driven enterprise and its mounting of conferences is no exception. All ABAI conferences are evaluated by the attendees and compared with other conferences. The evaluation of this conference was exceptionally high, with 98% of respondents rating the overall conference, the program, and the poster session as “very good” or “excellent.” Narrative comments praised the interactive nature of the conference, with comments like:
- I enjoyed this conference immensely. The structure (talks followed by active, structured workshops) promoted deep thought and networking.
- This was one of the best organized and most engaging conferences I have been to.
- I thought the topics discussed were incredibly interesting, and the presentation/workshop format was great. It gave the opportunity for deep conversation and thinking. It also gave the opportunity to get to know others in the field. I really loved the range of expertise in the room, from doc students to senior researchers who know a lot about this topic. This was one of the most fun and engaging conferences I have been to in a long time (roughly in the last 25 years!). Thank you for the opportunity to be there.
- This was such a wonderful and unique experience! I was so grateful to have been in the audience for this! I loved the table groups arrangements! It was a great way to network with others and step outside my personal and professional comfort zone.
- I was a bit leery of the interactive workgroups coming into this, but I found them to be extremely satisfying.I don’t know if it was by design or not, but the table groupings of senior and junior people was fantastic! [ed comment: it was by design!] Very well thought out and engaging. I wish more conferences were like this.
- So cool to have all of these editors in the same room!
The conference generated enthusiasm about both its topic and its structure. Numerous people approached us commenting that there should be more conferences that were interactive, and more that addressed the important issue of improving the quality of single-case experimental designs
About Author
