Saint Louis County Project RESTORE

Goals and Objectives of the practice:

The goal of Project RESTORE was to plan, implement and evaluate a series of coordinated, multi-disciplinary interventions tailored for minority, at-risk youth that improve academic, disciplinary and health-related protective factors and reduce risk factors for violent behaviors among participating youth. To achieve this goal, the project had specific objectives: (1) increase cultural competence among those working with or serving at-risk youth; (2) protective / resiliency factors; (3) improve academic outcomes; (4) decrease disciplinary actions; (5) arrests, court referrals, crimes, gun violence; (6) reduce stress and increase behavioral health; and (7) increase family engagement.

What did you do to achieve the goals and objectives?

Project RESTORE staff at DPH worked with partner school administrators to identify staff members at the intervention schools as well as the high school from which peer mentors would be recruited. These staff members then worked with Project RESTORE team from DPH to coordinate the implementation of the programs at the intervention schools. In consultation with school administrators, a class period during which the peer mentorship program (TAHC) would be delivered was identified at each of the schools. The project RESTORE coordinator then worked with the three school districts to: (1) identify and sustain a TAHC sponsor (s) in each partner high school who are responsible for recruiting peer educators (10-30 students); and (2) identify and maintain contacts in intervention schools (e.g., Principals, Counselors, Health or Family and Consumer Studies teachers) to assist with scheduling TAHC presentations.  Once the peer educators were identified, DPH staff then delivered a one-half day training (annually) to peer educators for them to learn about the history of the program, understand their role as peer educators and role models, be trained on presentation formats and tips, and practice participating in, and leading lessons. The TAHC curriculum offers grade- and age-appropriate lessons, and is delivered in a highly structured and prescriptive sequence of sessions for the first time as part of the Project RESTORE program. Specifically, in Year One of the program, the peer educators were trained to deliver a curriculum consisting of eight 50-minute lessons. These lessons covered a variety of topics including: (1) every choice counts; (2) dealing with conflict; (3) alcohol and tobacco and other drugs; and (4) dating issues.  In Year Two the peer mentors were again trained for one-half day and the lessons the curriculum included eight 50-minute lessons focusing on: (1) positive outlook on life; (2) forming a positive self-concept; (3) decision making and problem solving; (4) managing stress; (5) coping with depression; (6) substance abuse; and (7) bullying. During scheduled TAHC lesson day and class period, trained peer educators were transported from high school in the same district to the intervention middle school to deliver lessons to the cohort students. During the designated class time, either TAHC sponsor (teacher) or the teacher for the specific class period was present in the classroom only to observe and ensure order. For providing peer mentorship and education, as an incentive, the peer educator received credit for volunteer hours that they applied toward their graduation requirements. 

The summer program was delivered by the Police Athletic League (PAL). PAL planed, implemented and reported data for the summer program that offers opportunities to participate in activities on the school campus, involves sports and physical activity, provides positive and productive individuals to serve as mentors (law enforcement officers) and involves parents/ caregivers in activities to promote their understanding of risk and protective factors and foster interaction and communication with their children. The summer program also includes field trips at the end of each week. Field trip destinations in Year One included trips to Busch Stadium for a baseball game, Brunswick Zone, City Museum, Redman Pool, Six Flags and a culminating day-long trip to the Civil Rights Museum in Memphis, Tennessee. In Year Two, a six-week summer program was delivered again from May 15, 2019 to June 30, 2019. During this time, PAL participants went on several field trips, including, Six Flags of Mid America, camping, movies, fishing, swimming and several other activities.  Specialized officers in the different police units including crime analysis lab and tactical unit teams, also, came to speak to the students about their work.

Then, beginning in the 2018 – 2019 school year, as part of the four-year implementation plan for Project RESTORE, PAL transitioned to a year round after-school program. Before the transition, all partners worked closely to ensure that PAL's after-school activities aligned with existing after-school programs within the schools. The after school program therefore leveraged existing after-school activities such as basketball and other extracurricular activities to implement the year-round PAL after-school program. As part of the after-school program, PAL officers and other volunteer officers enlisted by the program help with different activities including homework and academic tutoring. Additionally, to better facilitate PAL's after-school program, the school administrators provided office space at each of the participating high schools where the cohort students transitioned. Using funds from Project RESTORE, these offices were turned into Club Houses” fully equipped with computers, video games, and other equipment that PAL participants can utilize after school. In these Club Houses” officers interact with cohort participants to play games, or help the students with their homework.  

An important aspect of this project was parent engagement. In this regard, we worked with school administrators and teachers to identify suitable event and time to bring together parents and students to talk to them about the project. On the recommendation of parents on the advisory board, we organized fun and engaging activities. Specifically, at the beginning of the 2018 – 2019 school year, we worked with school principals to organize a barbecue events at the three intervention schools that brought parents, teachers and students together to talk about the project and how their children can benefit from it. The schools brought in Disk Jockeys (DJs) to play music and entertain the students and parents. We also attended parent teacher conferences where we talked about the project.  Additionally, we brought in and expert to talk to both parents and teachers about how to deal with mental health issues, specifically focusing on how to cope with depression and stress, and how to identify these problems in their children. Two of the school districts were also able to bring in local artists that have become famous nationally (Koran Bolden, nationally acclaimed motivational speaker and author of the book Rock, Paper, Scissors; and Gerald Fulton, Jr. better known as Hitman Holla a battle rapper and star of the show Wild N Out with Nick Cannon), to talk to them about how to avoid getting in trouble and making good choices. 

An important component of the project is engaging adults who work with the cohort students. These included teachers, school staff as well as the police officers and other volunteers. To help these adults improve their engagement with the students, Project RESTORE provided professional development trainings focusing on cultural competence. Specifically, in Year One, we worked with all the schools to engage a company called Educational Equity Consultants (EEC), a local entity that provides diversity training that identifies and eradicates systems of oppression that damage our workplaces, communities and schools” (EEC, 2017). In addition, through project RESTORE funding, schools sent staff and school leaders to national training centers to be trained as trainers on Restorative Justice practices so that they can then come back and train their colleagues to ensure sustainability of these practices and changing the culture in their schools.

Another important component of this project is evaluation. To this end, it was important to have a strong evaluation team whose responsibility was to evaluate the program to ensure that the program is delivered with fidelity. The evaluation team evaluated three different components of the project: (1) the implicit bias training taken by St. Louis County Police officers; (2) the delivery of TAHC and PAL programs; and (3) Restorative Justice Practices at the schools. As described in the evaluation section, the evaluation teams worked with the Project RESTORE implementation team as well as administrators at intervention schools to provide feedback and guidance on different aspects of the project. For example, one of the major issues we learned during the planning stages of the project was that the schools were at different stages of implementing Restorative Justice practices. As part of this project, the evaluation team conducted an initial assessment in Year One and provided each of the schools with an assessment report on where they were, what they were doing, and an individualized action plan for moving forward with implementation of Restorative Justice practices. 

Criteria and Timeframe

The selection criteria of schools invited to participate in the project was based on requirement by the Office of Minority Health to include participants from schools with a high percentage of racial minority, and environmentally and/or economically disadvantaged students, as measured by the percentage of students receiving free/reduced lunch. Additionally, during the planning phase school administrators voiced concern that most of the violent behavior often started as cyber bullying. Because research indicates that electronic aggression peaks around the end of Middle School and the beginning of High School (~ 8th grade),[1]  we decided to enroll a cohort starting in the 7th grade. These students are being followed for four years until they complete 10th grade and all partners are involved in all aspects of the project until the end.

LHD and Partners' Roles

The DPH, as the lead agency with already established relationships with many entities in the county, convened all the partners and led in the planning and implementation of the project. DPH continues to provide overall leadership and coordination of the project. In this role, DPH plans and hosts all regular and ad hock partner meetings and ensures that the project adheres to the four-year implementation plan developed by all stakeholders. Further, as the cornerstone program (TAHC) is run by DPH, it coordinates with the schools to provide this training to peer educators and students in the cohort. Moreover, DPH reports benchmark data and progress to OMH; ensure financial and programmatic compliance with all HHS and OMB regulations. The other stakeholders play crucial roles as well. The three school partners provide school administrative data for project cohort; coordinate academic tutoring at least four hours per week; facilitate student and parent recruitment for Advisory Board and in-school interventions; facilitate teacher participation in: cultural competence and/or conflict management training, Restorative Justice assessments, seeking parental consent for cohort students; promote engagement in program evaluation; ensure availability of school campus facilities year-round (summer and academic year). Partners from institutions of higher learning provide evaluation for all aspects of the project. Law-enforcement agency approves officer participation and encourages full engagement in PAL and Implicit Bias training; provides access to law enforcement officers for evaluation of training and PAL; recruits cohort students to attend PAL after-school activities and Summer Program; plans, manages, implements and reports on PAL activities per project requirements and evaluation team requests; promotes program participation (students and law enforcement officers); provides training, orientation, guidance and supervision of PAL officers.

Funding/Budget

Project RESTORE is implemented through a 4-year $1.7 million funding from the Office of Minority Health, in the U.S. Department of Health and Human Services. This amount is spread out over 4 four years ($425,000 per year). Additionally, partners including DPH and St. Louis County Police Department have made staff available to help with the implementation of the project. For example, at DPH, the time that the Program Director, a Health Educator, and three members of the chronic disease epidemiology team is provided as in-kind service to the grant. The St. Louis County Police Department has assigned three officers and a commanding Sargent to the Police Athletic League. While these officers work full-time on the implementation of Project RESTORE at the three partner schools and other related projects, their salaries are covered by the St. Louis County Police Department.

[1] Williams, KR et al. (2007). Prevalence & predictors of internet bullying. Journal of Adolescent Health. 41:S14-21.

Project RESTORE Evaluation

The project goal for Project RESTORE is to plan, implement and evaluate a series of coordinated, multi-disciplinary interventions tailored for minority, at-risk youth that improve academic, disciplinary and health-related protective factors and reduce risk factors for violent behaviors among participating youth. In order to achieve this goal, program objectives include: 1) increasing teacher understanding of implicit bias and cultural competence, 2) increasing police officer understanding of implicit bias and cultural competence, 3) increasing peer, parent and/or adult connections and student social skills, 4) increasing academic achievement, 5) decreasing disciplinary actions, 6) increasing fidelity and integration of Restorative Justice practices, 7) decreasing encounters with criminal justice system, 8) decreasing aggression and peer violence, and 9) increasing parent engagement and understanding of protective factors.  Performance and outcome measures are presented in Table 2 below.

In order to measure achievement of these objectives, the St. Louis County Department of Public Health (DPH) recognizes the need for a robust evaluation design. To achieve this the St. Louis County DPH collaborates with a multi-disciplinary evaluation team. The evaluation team consists of evaluators from the University of Missouri St. Louis (UMSL) Department of Criminology and Criminal Justice, Southern Illinois University (SIU) Carbondale Department of Criminology and Criminal Justice, and the University of Missouri's Institute of Public Policy.

The evaluation team created primary data collection systems to evaluate and analyze the work of Project RESTORE. The first component of the evaluation are pre-post program surveys for trainings conducted with project personnel. The first set of surveys are for implicit bias trainings conducted with law enforcement personnel. The next set of surveys are for program staff and teachers at participating schools on school climate, cultural competency and implicit bias professional development. These surveys are collected online through Qualtrics online survey software. They aim to measure the change in cultural competency skills of school personnel and law enforcement officers working with minority, at-risk youth while identifying opportunities to refine or improve upon implicit bias training for law enforcement and school personnel. Analyses of these data include pre-post means test comparison for law enforcement implicit bias training and annual difference in mean comparison for school personnel.

The second component of the evaluation are surveys for youth at the intervention and control schools. These surveys are conducted annually with in-person paper surveys. They aim to measure the change in disciplinary actions, suspensions and expulsions among youth participants, the change in youth violence protective factors (i.e. positive, healthy relationships), the change in stress and behavior health, change in family engagement and environment, change in youth arrests, court referrals, crimes, homicides, and gun violence, and the change in academic outcomes of youth participants. Specifically, Police Athletic League (PAL) after-school and summer programming, TeenAge Health Consultant (TAHC) curriculum and participation, and academic tutoring offered through Project RESTORE. Analyses of these data include annual difference in mean comparison between treatment and control groups utilizing Difference-in-Difference regression methodology.

The final component of the evaluation are qualitative interviews with school administrators and project personnel, TAHC peer mentors and youth participants, and a survey of stakeholders.  This component aims to measure Restorative Justice fidelity and integration in participating school districts. Phone interviews are conducted with intervention and control school administrators and Project RESTORE personnel. Online surveys are conducted with community stakeholders and key personnel. Additionally, in-person focus groups are conducted with TAHC peer mentors and youth participants. Findings are analyzed through qualitative themes assessment methodology where participant quotes are utilized in reporting.

Secondary data are collected from school districts, the Missouri Department of Elementary and Secondary Education (DESE), and the Census Bureau's American Community Survey (ACS) for district and regional-level data comparisons. These include regional demographic profiles, income levels, district level free and reduced lunch status, and graduation rates.

Following data collection and analyses, the evaluation of law enforcement implicit bias training found that trainees indicated they recognize their own bias, felt confident in their knowledge about implicit bias, and their ability to address it prior to taking course. Results were similar across gender, race, and rank. The evaluation observed the most change in participant understanding of subjective culture and how it interacts and intersects with law enforcement culture. Addition changes were seen in participants understanding of their own values and biases. Results indicate implicit bias awareness exists for the law enforcement personnel and that the next phase of trainings should focus on reactions and treatment of implicit biases in the field. Evaluation results have been shared with the training team for future law enforcement implicit bias trainings.

The evaluation of school personnel implicit bias and cultural competency professional development found statistically significant differences in the cultural competency of administrators with the control school administrators indicating more adeptness in this area. Moreover, the control school administrators seemed to be viewed as more willing or capable in speaking about racism/sexism/classism. Finally, there were statistically significant differences between the two groups in terms of disciplinary procedures. Control school personnel indicated that verbal reprimands, conferences with students, notification of a parent, and detention were utilized more often than in the treatment schools. Results highlight that the schools with the highest need for intervention are the treatment schools. Evaluation findings are shared with school administrators to inform district-level discipline policy and guide future professional development opportunities.

The youth participant evaluation is in phase three of four and final data collection and analysis are ongoing. Initial review of youth survey results compare changes between phases one and two. The evaluation reviewed results for healthy habits, school commitment and problems, resilience, and disciplinary actions. Evaluators found significant differences between control and treatment schools in these areas and findings have been shared with project personnel to inform programmatic changes like an increase in survey participation and school district reporting. These findings are communicated annually with youth issue briefs in order to inform quality improvement efforts.

The qualitative Restorative Justice evaluation analysis found programming themes that the program is building stronger relationships, addressing disruptive behavior with appropriate interventions, and creating a safe and supportive environment. This assessment is conducted on a biennial basis and results presented here are for the first year of Restorative Justice implementation. Schools were categorized into one of four integration stages: ownership, expansion, initiation, and development. The ownership stage exemplifies complete knowledge and leadership in restorative practices. The expansion stage exemplifies considerable knowledge and leadership in restorative practices. The initiation stage exemplifies basic knowledge and leadership involvement in restorative practices. The development stage exemplifies minimal to some knowledge of restorative practices and leadership acknowledges issues related to restorative practices. The majority of schools were in the development stage and one school was found to be in the initiation stage. All schools were provided with an inventory of the qualitative assessment and an action plan for their stage of restorative practices. Additional technical assistance was provided for integration of restorative justice in participating schools on moving between implementation stages.

The evaluation team has a strong partnership with the St. Louis County DPH and participating school districts. Results are presented to program stakeholders on a regular basis in order to influence programmatic changes and quality improvement. Implementation of the third phase of the evaluation is underway and results will continue to be shared with all stakeholders. The robustness of the evaluation will aid in the dissemination of results and the research impact of program findings on communities implementing interventions tailored for minority, at-risk youth. While no modifications have yet been made based on the results, we are continuing to evaluate and adjustments will be made accordingly.

Table 2. Project Logic Model with showing Outcome Measures and Performace Metrics.

Lessons Learned

The implementation of the program was not smooth, especially in the first year. The first import lesson learned was that bringing different institutions together to work on such a big project was difficult and time consuming. In this process, we learned that understanding the capacity level of the different institutions is important in order to set expectations for what can be achieved and what cannot be accomplished by the different organizations. We also learned that transparency and open communications among the partner organizations, not only at the leadership level, but at the staff level is crucial. In this regard, regular meetings with detailed minutes with clearly identified action items, and responsibilities become very important documents for implantation of the project. Additionally, it is important to understand that not everyone in the partnership may have the full grasp of the granular details of the project. This may be especially important where there is staff or leadership turnover at the different partner organizations. In this regard it becomes important to periodically visit the overarching objectives of the project, and perhaps have a cheat sheet that everyone can refer to from time to time to have the same understanding of the project objectives.      

Cost/benefit analysis

Although we did not conduct a cost benefit analysis for this project, we know the social and economic cost of violence, particularly youth violence are substantial. Recent reports have found that school-based violence prevention programs help reduce violent risky behaviors at all grade levels (CDC, School-Based Violence Prevention). From this perspective, although the investment of $1.7 million from the Office of Minority Health in the U.S. department of Health and Human Services for the implementation of Project RESTORE is substantial, this will eventually pay great dividend overtime. This is because the capacity that has been built through this funding at the partner institution will continue to implement these violence prevention strategies that will ensure sustainability of the project. Although there is no direct way to measure the cost-savings in terms of how many violent incidences will be averted, we anticipate that the conducive environment that is being created for students to learn violence prevention skills, stay in school longer and succeed, form partnership with law-enforcement will have substantial benefits for the target community and the society in general.  

Stakeholder commitment

The St. Louis County Department of Public Health considers violence as a major public health concern and is committed to addressing this issue using novel public health approaches. Indeed, this commitment is what lead the department to partner with the Department of Criminology and Criminal Justice Department at the University of Missouri- St. Louis (UMSL) even before the formal start of the Project RESTORE partnership. The two agencies are committed to working together and learning from each other about the different approaches to address violence. This partnership is what lead to the creation of the interventions being implemented as part of Project RESTORE. The schools are also committed for very obvious reasons. In fact, when the St. Louis County department of Public Health sent out invitations to the various schools in the area, other schools did not answer the call. The schools that ultimately remained are the ones that have shown serious commitment to address the issue of youth violence in schools. Further, at the beginning of the project, each partner's leadership signed a commitment letter committing their organization to this partnership clearly outlining each partners' responsibilities in the partnership. Additionally, during the funding period, each partners signs a contract with the department of Public Health that further outlines the responsibilities and expectations for the Project.  

Sustainability

As we developed this proposal, one of the questions we had as partners as well as a requirement from the funding agency was sustainability of the project once funding was over. We therefore designed the intervention programs with this objective in mind. As a result, we made a determination that the central component of the project would be the peer mentorship program called TeenAge Health Consultants (TAHC). This is a peer led health education program supported by the Saint Louis County Department of Public Health (DPH) in partnership with local schools that has been delivered in this area for over 30 years. There is a fulltime health education coordinator, whose responsibility is to train students at the various high schools so that they can deliver this program to their peers. Even if this particular health educator was to leave the health department, this is a position will always be there and replacement to do exactly the same job will be hired. Other staff in the chronic disease epidemiology program have also learned the different aspects of the implementation of the program and this capacity building will ensure the sustainability of this program. In addition, the school districts and the Police Athletic League (PAL) were working together even before the Project RESTORE partnership started and have a commitment to work together in the future. Moreover, the PAL program is a program that is delivered as part of the community policing imitative of the St. Louis County Police Department and there are four full-time officers and a sergeant assigned to this unity, with potentially more officers to be added to the program as the program's success has become well known. Because of this, the PAL component of this project will be sustainable as additional resources in terms of officer-salaries will not be required. Moreover, PAL also relies on volunteer uniformed officers as well as civilian members for the police force as well as members of the community who pass a background check. Additionally, an important aspect of this project was the evaluation component so that we demonstrate what and how it works. With this evidence-based information, we will be able to replicate this process and will freely share any data collection tools, lessons learned from this process with anyone interested in implementing this program.  More importantly, as the St. Louis County Department of Public Health considers violence as an important public health problem and is one of the driving organizations to address violence prevention in the region, discussions are currently underway to create a permanent violence prevention coordinator position. This will help in continuing to implement the programs currently being implemented as part of Project RESTORE ensuring the sustainability of the program.