Microsoft Word - APAS brochure_Jan_25

The Afterschool Program Assessment System (APAS)

APAS is offered exclusively by The National Institute on Out-of-School Time (NIOST) at the

Wellesley Centers for Women.

What is APAS?

The Afterschool Program Assessment System (APAS)

helps afterschool programs improve their program

quality and reach their desired outcomes for children

and youth. APAS

is one of the only assessment

systems available that helps afterschool programs link

quality and youth outcomes together in a

comprehensive, flexible and integrated fashion.

APAS includes two measurement tools—the Survey

of Afterschool Youth Outcomes (SAYO) and the

Assessing Afterschool Program Practices Tool (APT)

along with the nFocus Kidtrax database system.

• SAYO - measures changes in intermediary youth

outcomes linked with long-term healthy

development and educational success (

see page 2

• APT - helps programs examine important aspects

of program quality linked to the SAYO Outcomes

(

see page 3

• nFocus Kidtrax database - offers an easy way to

enter your SAYO results, record your APT

observations, and generate customized reports

that are easy to analyze.

What are the benefits of APAS?

Å Helps afterschool programs align program

improvement efforts with desired youth

outcomes, yet flexible in that the tools can

be used together or separately

Å Designed specifically for afterschool

programs

Å Research-based, scientifically field-tested

instruments

Å Based on a menu approach, wherein

programs measure those outcomes that fit

best with their program design and goals

Å Affordable and user-friendly format

Å Multiple internal and external uses for

tracking, monitoring and measuring

improvements in program quality and youth

outcomes

Background & Development

The SAYO and the APT tools have been extensively

piloted, and scientifically tested. SAYO and APT

have been used by nearly one hundred afterschool

programs in Massachusetts, Georgia, North

Carolina, and New Jersey. As part of the statewide

Massachusetts Afterschool Research Study (MARS)

in 2004, links were found between APT program

quality and SAYO youth outcomes.

The APAS Evaluation System. Developed by Beth M. Miller and Wendy B. Surr, National Institute on Out-of-School Time

at the Wellesley Centers for Women, Wellesley College, Wellesley, MA, for the Massachusetts Department of Education

The Survey of Afterschool Youth Outcomes (SAYO)

The SAYO uses brief pre-participation and post-participation outcome surveys to collect

data about youth from regular education teachers and afterschool staff. The SAYO

based on a menu approach in which programs collect data on outcomes that are most

aligned with their goals for the young people they serve. The

SAYO is designed for

programs serving children from kindergarten through middle school.

Sample SAYO Behavior Items

Please circle the number that best describes how often this student:

Never

Rarely

Some-

times

Usually

Always

N/A

( √ )

a. Is able to regain control of behavior when given warning.

1 2 3 4 5

b.Works well independently when expected to do so.

1 2 3 4 5

Programs that use SAYO

will gain information on how their participants have changed and what gains, if

any, they have made over the course of the year. However, as is true with all similar program evaluation

research, the results cannot necessarily be attributed to program participation. We encourage those using

the SAYO to collect additional data on youth which may enable their programs to better understand the

ways in which it may be contributing to outcomes. It should be noted that SAYO

was designed primarily as

a program assessment tool which can help programs track changes in youth linked to participation.

SAYO

should not be used as an individual youth diagnostic tool.

SAYO Outcome Areas

• Behavior in the Program/Classroom

• Initiative

• Engagement in Learning

• Relations with Adults

• Relations with Peers

• Problem Solving

• Communication Skills

• Homework

• Academic Performance (SAYO-Teacher

only)

Each outcome area is measured by asking teachers and staff to respond to four or five questions related to

observable youth behaviors. These items have been extensively tested and found to work as a single scale

that effectively captures the outcome being measured. Teacher and/or staff responses to these surveys can

be com

leted for all

artici

atin

outh or for a sam

le of

outh in a

ram.

The APAS Evaluation System. Developed by Beth M. Miller and Wendy B. Surr, National Institute on Out-of-School Time

at the Wellesley Centers for Women, Wellesley College, Wellesley, MA, for the Massachusetts Department of Education

Use APAS to identify goals and document results.

How Does It Work?

The Assessing Afterschool Program Practices Tool (APT)

The APT is a quality assessment tool designed to help programs strengthen practices that research

suggests are linked to SAYO outcome areas. The APT includes two tools: The APT Observation Tool

(APT-O), which focuses on observable program practices; and the APT-Program Questionnaire (APT-Q),

a self-assessment tool which examines those aspects of program quality that occur “behind the scenes,”

such as program planning and connections with schools and parents.

What does the APT Measure?

Positive Program Climate:

• Welcoming & Inclusive Environment

• Staff use Positive Behavior Guidance

• High Program and Activity Organization

• Space is Conducive to Learning

• Program Supports Staff

Practices that Support Individualized Needs & Interests

• 1:1 support

• Varied/Flexible Approaches & Scheduling

• Opportunities for Autonomy

• Connections with Families

Supportive Relationships:

• Staff-Youth Relationships

• Peer Relations

Practices that Stimulate Engagement

& Skill Building

• Staff Promote Engagement & Thinking

• Connections with Schools

• Targeted SAYO Skill Building/Activities

• Youth are Positively Engaged in Program &

Skill Building

Sample Items - APT Observation Tool

B. Staff Promote Youth Engagement & Stimulate Thinking How true? Notes

Staff are energetic, enthusiastic, and/or upbeat.

1=Staff appear distant, bored or

“flat” in their demeanor.

1 2 3 4

Staff are actively engaged in activities with youth

(e.g. interacting with groups, or

individual youth; providing ongoing facilitation, participating with youth).

1=Staff do

not engage with youth (e.g. chatting with other staff or engaged in own activity).

1 2 3 4

Staff encourage youth to share control/responsibility for activity (e.g. Youth help set

out materials for art project, youth are encouraged to take the lead in selecting and

initiating activities).

1=Staff maintain control throughout activity period, resist input

from youth, do not seek youth involvement.

1 2 3 4

The APAS Evaluation System. Developed by Beth M. Miller and Wendy B. Surr, National Institute on Out-of-School Time

at the Wellesley Centers for Women, Wellesley College, Wellesley, MA, for the Massachusetts Department of Education

Together, SAYO & APT, combined with the Kidtrax database provide a comprehensive,

easy to use system that can help you assess your afterschool program and track improvement.

How Does It Work?

How are programs using the

APAS system?

Å Comprehensive, self-assessment of program

practices by program administrators and multi-

level teams

Å Help promoting greater consistency in

practices across multiple sites

Å Pinpointing topics for staff development

sessions and as a guide for delivering feedback

to staff that is concrete and constructive

Å Using pre- and post- assessments of program

practices to measure change

over time

Å Using baseline youth behaviors in SAYO

outcome areas to aid in program planning

Å Communicating with schools about the needs

of individual youth

Å Measuring change over time in youth

outcomes areas to track program progress

toward goals

Is APAS right for my program?

APAS tools work best when used by:

Å Afterschool programs that serve youth in

Grades K-8

grade

Å Programs of all sizes and most types

Å Well-established programs (1+ year). It may

not be suitable for start-up programs, those

experiencing high staff turnover, facility

changes and/or major shifts in program focus

Å Programs with regular/consistent

programming and youth attendance across a

full school year. It may not be appropriate for

drop-in, specialized/limited programs (like

Karate class 2/hrs wk), or those focused

exclusivel

on academic tutorin

For more information on APAS visit

www.niost.org

Phone: 781-283-2546

E-Mail: [email protected]

How is APAS different from

other assessment tools?

Å Includes outcome areas that research links to

success in school and the workplace

Å Gathers data from teachers and after school

staff—whereas most other youth outcome

tools rely exclusively on youth perspectives

Å Allows programs to focus on assessing

specific, observable practices rather than

reflecting upon more general quality

characteristics.

Å Allows programs to examine how youth are

participating in the program

Å Results from the Massachusetts Afterschool

Research Study found links between APT

quality and SAYO-(S) youth outcomes.

Å Includes a comprehensive database system.

The APAS Evaluation System. Developed by Beth M. Miller and Wendy B. Surr, National Institute on Out-of-School Time

at the Wellesley Centers for Women, Wellesley College, Wellesley, MA, for the Massachusetts Department of Education

I want to use APAS, how do I proceed?

Step 1: Contact a NIOST representative to

determine the appropriate training needed for

your program(s):

Å SAYO

Å APT

Å APAS ( SAYO & APT)

Step 2: Learn about and set-up the nFocus Kidtrax

database system (approx. 4 hour commitment)

Step 3: Attend either an APAS training at our

annual Boston Summer Seminars, hire a NIOST

trainer(s) to conduct (any of the above) trainings

at your venue, or take the online SAYO tutorial (a

SAYO only option).

Important: NIOST encourages 2 people per

site/program to complete a training.

Step 4: You’re ready to get started!