Part 2: Systems Level: Evaluating School Psychological Services Delivery Programs
As noted above, evaluating individuals is only half of the equation in supporting high quality service delivery. The NASP Practice Model can also form the foundation for evaluating school psychological services delivered by a department or group of school psychologists (referred to here as the "school psychology services program"). It is just as important that you are involved in developing program level evaluation systems as personnel evaluations. You can do this most effectively working collaboratively with your colleagues and supervisor(s).
Smith-Harvey and Struzziero (2008) suggest that evaluating the school psychology services program should:
- provide feedback regarding the overall quality of services
- identify areas of need for enhancement of services
- lead to improvements in procedural or programmatic areas
- guide professional development
- provide opportunities for professional recognition
- allow for the promotion of the effectiveness of school psychological services
To begin this evaluation process, Smith-Harvey and Struzziero (2008) indicate that several questions should be asked related to the effectiveness and quality of school psychological services. Answering each of these questions requires collecting data in different ways and from varying sources.
- Do school psychology services have a positive effect on students and staff?
- Do school psychological service procedures comply with local, state, and federal guidelines?
- How do school psychological services compare with professional standards?
- Do administrators, parents, and teachers consider school psychological services adequate?
- What are the school psychology department's policies and resources?
- Are school psychology services cost effective?
Given these questions, the components of a comprehensive system of evaluating school psychological services would include:
- Evaluation of the overall quality and positive impact of school psychological services on students, families, and staff
- Evaluation of school psychological services in relation to meeting local, state, and federal guidelines
- Evaluation of school psychological services according to the NASP Practice Model-the current professional standards for practice.
Evaluating the overall quality and positive impact of school psychological services on students, families, and staff. School psychologists must increasingly use their data-based decision making skills to provide evidence of the impact of their services on the academic and social well-being of students. Section II of this guide to implementation of the NASP Practice Model articulates a process you can use for aligning your services to support the strategic goals of schools. Data collected through these processes are useful for showing the contributions that you and your colleagues collectively make to meeting school-wide, grade-wide, or classroom-defined goals. Strategies and indicators for evaluating the impact of school psychological services might include collecting:
- Student performance data (e.g., progress monitoring, goal attainment data)
- Pre and post data from individual or group behavioral plans or counseling cases
- Pre and post data from implementation of social-emotional programs
- Survey data following parent sessions or parent support groups
- Data reflecting response to crises (e.g., describing follow-up activities, direct student intervention, requests for external or community support)
- Survey data regarding the results of teacher consultations and impact on students
- Survey data following systems-level interventions (e.g., perceptions of school safety, attendance rates, reports of bullying)
- Data indicating that services are based on evidence-based practices
Additionally, you should be aware of national standards for data collection that might exist. The What Works Clearing House, housed as the Institute of Education Sciences, identifies evidence-based practices. Selection of programs should include those that have empirical data using rigorous standards for program evaluation. In every case, you should consider the breadth of services being offered in response to the breadth of needs of students and work to identify objective methods for monitoring and reporting student progress in response to those services.
Evaluating school psychological services in relation to state and federal guidelines. Maintaining compliance with state and federal statutory or regulatory requirements is necessary, is an important goal for school administration, and will most likely result in the best outcomes for our students and families. Evaluations should consider the data collected in compliance with federal, local, or state laws. For example, federal law requires states to report data related to graduation rates, achievement in reading and math, and various indicators of school discipline. State and local laws may require collecting and reporting data related to student attendance, incidences of bullying, frequency and type of classroom disruptions, drug or alcohol use on campus, and more. Other accessible data that could reflect on school psychological program effectiveness related to legal requirements are available through state special education compliance monitoring and due process hearings, such as:
- Consistently meeting timelines for meetings and completion of evaluations
- Reducing or addressing common themes in due process hearings/complaints (e.g., clustered in one building, or related to a specific topic)
- Providing prescribed services as identified within IEPs
- Addressing caregiver requests for access to records
- Tracking referrals for special education services
There are many sources of data currently being collected in schools that could support the evaluation of school psychological programs. It is critically important that you and your colleagues are aware of the breadth of data being collected and consider how those data are potentially linked to the quality of school psychological services in the district.
Evaluating district level school psychological services according to the NASP Practice Model. The NASP Practice Model offers several possibilities for assessing whether school psychological services are being delivered according to national standards. The NASP Practice Model can serve as a framework for evaluating (a) the depth and breadth of services offered by the school district in response to student needs, (b) the quality of supervision and mentoring provided to district school psychologists, and (c) the availability and quality of professional development and support to grow and nurture school psychologists.
The Depth and Breadth of Services. Comprehensive screening for academic, behavioral, and social-emotional risks, and factors that contribute to those risks, helps identify the prevention and intervention programs necessary. Important questions to be answered within this process include:
- Does the district provide a comprehensive range of student-level services?
- Are interventions for instructional support adequate?
- Are mental and behavioral health supports adequate?
- Are school psychologists working at a systems level, particularly the school building level, to address prevention of academic and social problems?
- Are school psychologists routinely collaborating with parents and other school and community professionals in the provision of services?
- Are school psychological services reflective of the practices that should permeate these services-data-based decision making, accountability, and use of consultation and collaboration?
- Do services properly reflect cultural competency, and are they delivered with respect for legal and ethical considerations?
- Are student-level and systems-level services based upon known research or evidence-based practices?
- Are student outcome indicators regularly collected and student progress towards school psychological service goals regularly monitored and reported to relevant stakeholders?
The Quality of Supervision and Mentoring. The NASP Practice Model Organizational Principles advocate for the following quality supervision and mentoring standards:
Qualified supervisors hold a valid school psychology state credential, have a minimum of 3 years of experience as a school psychologist, and have either education and/or experience providing supervision. Supervisors should:
- Provide supervision using methods that are appropriate for the level of expertise of the school psychologist. For example, novice school psychologists may require more intensive supervision, with more opportunities for observation, reflection, and feedback, than a seasoned professional.
- Ensure that practica and internship experiences have appropriate supervision in compliance with NASP's Standards for Graduate Preparation of School Psychologists.
- Provide leadership through participation in school psychology professional organizations and active involvement in public policy development at the local, state, and national levels.
- Allow time for school psychologists to participate in supervision and mentoring experiences.
- Consider reasonable professional alternatives when a qualified school psychologist supervisor is not employed by the district. Alternatives could include providing an opportunity for practitioners to receive supervision by a qualified school psychologist outside of the district or utilizing peer supervision models.
- Have a coordinated plan for conducting professional evaluations of all school psychologists and the school psychology service delivery programs.
Professional Development. The NASP Practice Model also provides guidance for school systems regarding the need to support the professional growth and advancement of school psychologists. It is critically important that a school district support practitioners in identifying areas where professional growth is needed and then also the time and resources needed to access the relevant training, supervision, mentoring, and support to improve practice. Districts can facilitate this process by encouraging practitioners and supervisors to take the NASP Practice Model online self-assessment and to develop personal professional development plans designed to enhance or remediate skills. Many schools can also recognize practitioners' commitment to excellence through employee recognition programs or other merit-based award programs.
Common Barriers to Effective School Psychology Services Program Evaluation
- School psychologist supervisors do not have knowledge or experience working as school psychologists, so they are not able to provide technical feedback about quality of specific services.
- School psychologist supervisors are supervising too many practitioners and are therefore not able to provide the mentoring support and feedback needed to support performance improvement.
- School psychologists do not collect data reflecting how their services contributed to positive student outcomes or the overall district-wide improvement plan.
- The overall effectiveness of the school psychology program is determined by looking only at the annual test scores of the students who have received school psychological services.
Table IV.5 summarizes ineffective practices that often result from these barriers and what corresponding effective practices might look like.
Advocacy Tips for Effective School Psychology Services Program Evaluation
- In districts where a teacher-specific evaluation tool is to be used for all professionals, including school psychologists, a representative of the school psychology team can meet with district administrators and provide resources on the best practices for personnel evaluation of school psychologists. The representative can offer to organize a team of school psychologists to work with district administrators to develop a unique evaluation tool and process for school psychologists.
- In settings where the system does not provide unique inservice opportunities for school psychologists, the school psychologist supervisor can organize professional development opportunities to be delivered during team meetings (e.g., guest speaker, viewing and discussing as a group an online learning webinar). Topics for discussion should be selected based on the professional skill needs of district practitioners as they relate to district improvement goals and initiatives.
- School psychologists in a district can work together to create an annual report of school psychological services for district administrators and the school board. The report should include data like the types of services offered, the settings, numbers of students, and measurable outcome data. The report and related recommendations can be presented to the school board, district building-level administrators, and the parent-teacher organization annually.
- A team of school psychologists can work with school district administrators to review school board policies, including specific documents like job descriptions and personnel appraisal tools, and determine how the district's policies align and reflect national standards for school psychology practice. Recommendations can be made to the school board and other influential decision makers for improving alignment.
Table IV.5. Practices in School Psychological Services Program Evaluation
|Ineffective Practices||Effective Practices|
|District policy discourages offering ongoing counseling services to students. School counseling staff may meet with students one time about an issue and then are encouraged to refer families for outside of school counseling support.||Counseling services are available in the district for all students in need. For special education students, these services are written into the IEP as needed. The IEP team determines the best qualified provider (school psychologist, school counselor, etc.) to deliver these services. Counseling records are maintained indicating the students who received counseling and the issues that were addressed. Records are reviewed annually to determine if systemic supports offered around particular issues are sufficient and effective.|
|In a school district with historically low graduation rates and high rates of office referrals, suspensions, and expulsions, data are collected at the school level and are never shared with the district office administrators. No system-wide analysis is undertaken. Interventions offered in schools are driven by interested practitioners and are not driven by system needs.||A multidisciplinary district level data team reviews school and feeder system discipline data on a regular basis. The team then works with school mental health team supervisors to determine a district-wide plan for reducing disciplinary problems, improving student-teacher interactions, and delivering targeted interventions to students at greatest risk for suspension and expulsion. Monitoring and review of the implementation of the plan occurs regularly.|
|The district ratio of school psychologist to student is three times greater than the nationally recommended ratios. The school psychologist supervisor is told by his boss that with the impending budget cuts in the district, it doesn't even make sense to ask for more positions. The primary responsibility of school psychologists is to complete assessments for multidisciplinary evaluations, leaving little time for any delivery of a comprehensive school psychological program.||Recognizing the valuable contributions of the school psychologist to improving student engagement and achievement, the school board commits to working towards adoption of the NASP Practice Model, including improving the ratios. A 10-year plan is developed to move the district to full adoption of the model. Data is collected annually about ratios, caseloads, the role of school psychologists, and their contributions to student achievement. The district accountability team review data, reports to appropriate administrators, and adjusts the plan as needed.|