Lessons Learned

How might key insights from these data analyses inform findings we share back with the field, ultimately informing local practice?

Local Demonstration Project: Humboldt County. The purpose of the Humboldt County analysis is to provide a local example of demonstrating impact. Since 2016, the county pursued competitive grants to support its districts through education (e.g., Multi-Tiered System of Support) and health (e.g., Triage Grant Program, known as Bridges to Success), in addition to its ongoing funding in both sectors (e.g., Local Control Funding Formula and Mental Health Services Act). The series of analyses began by exploring the results of the different initiatives and programs independent of each other and included graphical displays that explore the interrelationships between the programs (e.g., the results on CA Healthy Kids Survey between MTSS and Bridges cohorts). The analyses also examined potential differences across districts that participated in MTSS, engaged in Bridges Referral services, and administered the 2020-21 CHKS Survey. Because these were exploratory analyses, any results have not been causally evaluated, and thus should not be interpreted as the direct effects of these programs. Key findings include:

  • Engaging in the Bridges referral services was associated with an increase of 12.6 percentage points in graduation rates when tested both independently and jointly with the other interventions of interest (e.g., MTSS and CHKS). Engaging in Bridges was also associated with a statistically significant decrease of 10 percentage points in drop-out rates when tested jointly with other interventions. Further, when controlling for MTSS Participation, the model uncovers a strong impact of the Bridges program on dropout rates.

  • Participation in the 2020-21 CHKS Core Survey was associated with a 3.2 percentage point decrease in rates of suspension when tested independently, and a 3.8 percentage point decrease in rates of suspension when tested jointly with other interventions. Student surveying has been shown to be associated with a decrease in suspensions, aligned with our expectations.

  • MTSS Participation was associated with a statistically significant decrease of 0.7 percentage points in dropout rates when tested independently; this effect disappears when tested jointly with Bridges.

  • Indications were stronger for non-academic parameters than for academic parameters.


Vision of Success:


Based on the statewide synthesis, spotlights, and group discussion, effective investments would manifest in the following ways:


All LEAs in receipt of grant funds would be required to design implementation for impact and collect program data on metrics for evaluation purposes. LEAs will identify and address SEL needs using the following methods:

  • Incorporate SEL surveys/screeners daily, weekly, monthly, or annually, to meet local needs

  • Ensure participation rates for valid and reliable analyses

  • Measure fidelity of integrated supports to ensure effective implementation (e.g., FIA)

  • Collect program data to inform local and regional initiatives (braiding and streamlining at local level)

  • Report program data for the statewide evaluation of the program to inform future funding/ sustainability needs

  • Engage in regional infrastructure to support effective implementation (e.g., CA Statewide System of Support-Geo Leads)

  • Report out on improved outcomes for school climate/culture, student mental health/wellbeing, conditions of learning, and academic outcomes

  • Provide universal supports through the system of support beyond infrastructure for effective implementation (consolidated dashboard and screeners linked to relevant data)


Recommendations:


1. Statewide Support to measure challenges, readiness, and fidelity (e.g., Fidelity Integrity Assessment-FIA through MTSS)

2. Maintain measurement at the local level to account for local context that is relevant for socio-emotional learning, student mental health, and wellbeing

    • Regardless of the measure, frequency, and cadence, increase response rates

    • Daily, weekly, monthly

    • Annual measures

3. Sustain participation rates necessary for valid and reliable results

4. Use of data-driven decisions to support evidence-based practices (e.g., Humboldt)

5. Strengthen statewide infrastructure to implement evidence-based practices through the system of support/county offices of education; CDE Transformative SEL Workgroup and Designation of a SEL Lead; Linkage between statewide partners that mirrors investments (e.g., California Department of Education, California Collaborative for Educational Excellence and California Health and Human Services and Mental Health Oversight and Accountability Commission)

6. Universal Supports- Dashboard and Screener- through the System of Support