Page 113 - AIH-2-1

Artificial Intelligence in Health ML approach for client and program effects in OUD retention

which program characteristics can be adjusted to maximize we utilized analysis of variance (ANOVA) to compare
the benefits of workforce diversity for OUD client retention. mean differences across the four years. This approach
This study is also of relevance to the field of computational allowed us to identify patterns, trends, and variations
science, using machine learning to showcase an application in the data, providing a detailed understanding of how
of a novel approach to understanding heterogeneity. each variable evolved over the study period. To examine
the heterogeneity of the association between workforce
2. Methods diversity and retention in OUD treatment, we used the
We relied on nationally representative data from causal forest method in which weights were incorporated
the National Drug Abuse Treatment System Survey to make the data nationally representative. 28,29
(NDATSS), a dataset containing eight waves of survey data The dataset used in our study was organized at the
from outpatient substance use treatment programs (OTPs) program level, meaning that each record corresponds to
from 1988 to 2017. 30,31 Each wave incorporated a large a single program. Therefore, when we refer to percentages
percentage of programs from the previous wave, except of specific client demographics, we are indicating the
programs excluded due to closure. More details on the proportion of those clients relative to the total number of
NDATSS dataset can be found elsewhere. In this paper, clients within each program.
21
we looked at the last four waves of the NDATSS (110 OTPs
in 2000, 142 in 2005, 184 in 2014, and 190 in 2017). Causal forest is particularly well-suited for this analysis
as it estimates the client and program-specific treatment
2.1. Dependent variable effects of workforce diversity on retention. By doing so, it
highlights how the presence of a diverse workforce might
We used an established measure of retention, the influence retention rates in different programs. In addition,
percentage of clients in treatment for more than 3 months the causal forest method generates variance estimates,
in a treatment program, as the dependent variable. This
measure has been used in other studies. 4,21,32 which allow us to assess whether the observed treatment
effects are statistically significant and different from zero.
2.2. Independent variables This approach not only quantifies the impact of workforce
diversity but also provides a measure of the confidence we
The key independent variable is workforce diversity, which can have in these effects, revealing the conditions under
we define as the percentage of staff self-identified as Black which workforce diversity plays a crucial role in enhancing
or African American. This measure has been used in other OUD treatment retention.
studies. 17,18,21,32 To apply the existing estimation method
for HTE, we dichotomized the treatment variable. Thus, 3. Results
we consider programs with more than 20% Black staff
as having high workforce diversity. This threshold was We found significant differences among variables across
chosen because more than 50% of the programs in our the four different years that we examined. Table 1 presents
sample had less than 20% Black staff. The other relevant the comparative analysis by year. The percentages of clients
independent variables that define the heterogeneity of the in treatment for more than 3 months were significantly
treatment effect on client retention rates include program different across years (P < 0.001). The percentages of
and client characteristics such as percentage of Black Black clients were also significantly different across years
clients, percentage of Latino clients, accreditation by The (P < 0.001), with the percentages of Black clients being
Joint Commission (TJC), ownership status, program type lower in the last two waves (2014 and 2017). More programs
(private-for-profit, private-not-for-profit, public), staff- were from states that expanded Medicaid coverage in 2017
to-client ratio, proportion of staff who have graduate compared with 2014 (P < 0.001). There was an increasing
degrees, percentage of unemployed clients, and whether trend of program age across years (P < 0.001). The results
the program is located in a state that expanded Medicaid also showed that fewer programs were owned by another
coverage. organization in the last two waves (P < 0.001). The staff-
to-client ratio was significantly different across years
2.3. Statistical analysis (p = 0.024). The results also showed that the percentages

We conducted a comprehensive comparative analysis of unemployed clients were higher in the last two waves
of all variables across the four-year period to assess (P < 0.001).
any significant differences or associations. Categorical Results from the causal forest method (Table 2)
variables were examined using the Chi-square tests to showed that 61 OTPs had statistically significant positive
determine if there were statistically significant associations treatment effects for workforce diversity. This means that
between variables over time. For continuous variables, these 61 OTPs would significantly benefit from having

Volume 2 Issue 1 (2025) 107 doi: 10.36922/aih.3750