Pilot Evaluation

In September-October of 2012 the Creative Spark conducted a pilot using systematic qualitative research data collection and analysis best practices.  This included entrance and exit surveys, focus groups as well as face to face interviews with teachers, mental health counselors and students.

Pilot Conducted at:  Lawrence Grassi Middle School, Canmore, AB – September/October 2012

Creative Spark Pilot – Evaluation Conclusions

Based on all evaluation criteria, feedback and comments the Creative Spark pilot has succeeded above and beyond expectation on every objective category and desired outcome.  Results found herein have been returned by best practices in systematic evaluation procedures, surveys and face to face interviews with teachers, mental health counselors and students.  (See Evaluation Process – Objectives/Outcomes page 7)

Descriptive Analysis

The Creative Spark inspired students to be more creative.

Following the Creative Spark a majority of students increased significantly their perception that they can be creative in many more aspects of their life.

Participating in the Creative Spark most students made a direct connection that being creative can help them be healthier, in particular mental wellness as a component of overall health (along with fitness and nutrition)

Teachers were amazed at the level of focus and attention by all students.

Students in grade 7 (into early teens and adolescence) appear to start to lose interest in traditional art class activities, especially boys. The Creative Spark however had an opposite response; for example approximately 50% of Grade 7 boys were showing considerable “attitude” not interested at the start of the Creative Spark program.   Most of this group (61%) moved up the scale more than all other participants becoming highly engaged by the end of the first class and was sustained through the second and third class.  In short the Creative Spark was reaching a tough group.

Every teacher (and councilors) in every class observed fundamental teaching themes such as building on persistence, embracing challenges and risk taking (good risks).  Also communicating with clarity and applying past knowledge to new situations*

Mental Health councilors (and Teachers) observed development of emotional confidence, increased social interactions, dealing with peer pressure and judgment, empathy, being comfortable with frustrating, shared hardship (fear), increased participation of shy students, all within a safe environment.

Teachers, mental health counselors and parents have all recommended this program go to other schools and do so on a large scale.  Endorsement of this project has been provided by everyone who has seen it, without exception and most emphatically.

*teachers made frequent reference to Habits of Mind and Growth Mind set teaching metrics when talking about the Creative Spark.  The Creative Spark programming was developed without the Mind-Set model however this will now be a focus.  See Decisions and Next Steps.

Teacher and Mental Health Councilor Feedback Survey Highlights

  • The first thing that comes to mind is ENGAGEMENT
  • All students were attentive, focused, engaged and creative – amazing to watch the Spark!
  • Students were not afraid of judgment in my perception at that moment and they were having a good experience of being judged, It’s OK to be different than others
  • Seeing Students to be extremely involved and taking risks, trying new things

I would like to share a story. Following one session, the teacher approached this writer astonished at a comment I made to a young student who was so focused and on the task.  Apparently a death in the community had occurred and this student could not focus all morning in class due to his grief.  The Creative Spark activity provided him with opportunity to focus, engage, create and reflect.  What a wonderful opportunity this class provided the student.
              Laura Wellmann – Metal Health Capacity Building for Children and Families

  • Amazing energy and engagement! You captivated student attention and challenged them to think outside the box and take safe risks
  • I was amazed by what I observed. WOW!
  • We need this program as prevention because we have less support in the class room
  • EXCITING!!! The kids were vibrating with energy.  The students were completely focused on task and engaged the entire 2 hours – amazing!!

A lot was covered today; What is being healthy, Judgment, Creativity, Art Technique, persistence, impulsivity, humour (laugh at your work, i.e. mistakes), OK to be precise, take risks and be innovative, questioning (how are you going to deal with that), use your knowledge from previous experience, listening with empathy, show how you feel at the time
             Sonja Howatt – Teacher, LGMS Canmore

Student Entrance/Exit Evaluation Highlights

  • Met or exceeded objectives was achieved on all categories i.e. 15% increase or more
  • Exceeded beyond expectations on 13 out of 14 categories i.e. 25% increase or more
  • In both Grades 5 & 7 students considered they did only one or two things that were creative in their lives.  Upon completing the Creative Spark students increased their perspective of how many creative things they did that is creative by 65% and 50% (grades 5/7 respectively)
  • Upon completion of the Creative Spark both age groups increased the amount in which they consider being creative will help with other things in their lives i.e. their health and mental health.  This was even more pronounced with older students, in particular boys 61% increase

Teacher and Mental Health Councilor Evaluations

General Feedback

  • It was really interactive, every student was participating; Lively, engaging and interesting
  • I can’t wait to see where this project goes in the future
  • Excellent program, you could see the benefit right away on some student behavior
  • The program gives a chance to help students express themselves through creativity in a safe and caring and fun environment, students were focused
  • This program reaches what we are trying to teach to students Habits of Mind and Mind Set
  • The vocabulary used in the program is known by the students
  • The kids became more engaged as the class progressed. This was due to the fact that Rob invited every student to participate in some way
  • Rob created a caring and safe environment; all students are fully engaged and are not scared to share thoughts, even the shy students were participating
  • Students were not afraid of judgment in my perception at that moment and they were having a good experience of being judged, It’s OK to be different than others
  • I really liked how you made art and concept of healthy activities energizing!  Allowing students to identify how being creative and taking risks can be healthy
  • Scaffolding: started easy (students always able to succeed) then built in more complexity.
  • Not expecting perfection, but practicing (no erasing) and taking risks
  • The examples of his personal study(s) really drove the message home
  • Great management of the class in a fun and caring way
  • Easy to identify student that are reluctant to take risks
  • Valuable message was given today
  • Student were able to laugh at their work and others without fear in a respectful climate
  • Students are constantly stimulated visually, verbally and physically
  • How at ease the student were this week at taking creative risks with their art work and recognizing the importance of perseverance
  • How engaged student were – especially boys
  • Seeing students to be extremely involved, and taking risk, trying new things

Would you recommend The Creative Spark for other schools?

  • Absolutely! Not only other schools but all grades as well – utilizing creative arts to help student engage, take risk and preserver is a wonderful strategy
  • This will serve as a great foundation to build on all year:

 

  1. Feedback to improve
  2. Practice
  3. Hard work
  4. The Journey is the Thing
  5. Ways to be Creative

 

  • I was thrilled to be part of this pilot, it is a worthwhile program for adults as well as youth
  • YES! This program has laid a foundation that I will be able to build upon through the year
    • Risk Taking
    • Ways to be Creative
    • Hard work and practice = improvement
  • Yes, every student whatever the social needs of the class deserves to receive this program

 

 

Any Concerns:

  • Nothing;  None; No Concerns; Nothing; No big concerns; Nope! Everything went smoothly
  • Nothing – except perhaps how exhausted you may have been after 2 hours! Can there be some more shared delegation of roles to help take the facilitation load off one person
  • Samples were small which can create less focus for the students.  I would use Smartboard
  • I noticed a lot of kids (mostly grade 7s) who had quite violent drawings (which makes sense since they are grade 7 boys and because of the concept of victory and leadership).  My concern isn’t that they were drawing violent scenes as it is the idea behind the piece maybe getting lost

What suggestions would you make to roll out this program on a regional level

  • Develop a program for elementary, Jr. H and High school
  • Publicity, newspaper, TV news
  • Go to Calgary Schools so you could reach more, talk to Alberta Teachers Association
  • Be present during teacher convention in February for district Palliser but also Edmonton

General Suggestions: 

  • I think we should reinforce the message sent during the session.  At the end of the session we could have student reflecting by writing their thoughts or feelings.  Example: ‘Today I learned…;
    I realized in some situations that I…….; (Students are used to reflection in class)
  • So that the messages are not forgotten Teachers should repeat the messages from the program when appropriate; Example:  judgment, persistence
  • It takes a great deal of energy and focus to maintain engagement for two hours straight – it can be exhausting!  Is there a way to have shared partipating/identified roles & talks for teachers  & collaborative partner to help remove the entire facilitation load from Rob?  Just a thought.
  • Get things going earlier to eliminate rushing at the end.  This will allow for more discussion.
  • Have all supplies ready before students arrive, have assistant hand out resources
  • More time for student to showcase work, More time to reflect!

Student Entrance/Exit Evaluations

Before/After Conclusions:
  • Met or exceeded objectives was achieved on all categories i.e. 15% increase or more
  • Exceeded beyond expectations on 13 out of 14 categories i.e. 25% increase or more
  • Four categories reviewed  due to decreased results – findings captured in analysis
  • Prior to Creative Spark most students in Grade 5 said they liked doing creative activities often to all the time.  Upon completing the program 30% where more interested in doing creative activities.  70% no change, none decreased
  • Grade 7 students’ interest in doing creative activities ranged much more than grade 5 from doing creative activities a very little to often.  Upon completing the program overall there was slightly higher interest in doing creative activities amoung those in the mid-range of interest, however, those students that indicated low to very low interest in Creative Activities jumped significantly in their interest in creative activities (mostly boys) following the Creative Spark
  • In both Grades 5 & 7 students considered they did only one or two things that were creative in their lives.  Upon completing the Creative Spark students increased their perspective of how many creative things they did in life that is creative by 65% and 50% (grades 5/7 respectively).
  • In both Grades 5 most students considered themselves to be very creative.  Even with such high entrance scores 50% of the student increased how creative they considered themselves to be.   Most moved from Very Creative to Extremely Creative
  • Both age groups increased their enjoyment of doing creative activities at school
  • Both age groups increased significantly their enjoyment of creative activities outside of school
  • Upon completion of the Creative Spark both age groups increased the amount in which they consider being creative will help with other things in their lives i.e. their health and mental health.  This was even more pronounced with older students, in particular boys 61% increase

25% or more increase to outcome criteria = exceeded beyond expectations

15% or more increase to outcome criteria = met or exceeded objectives

14% increase or decrease to outcome criteria is considered a flat response = no change

15% decrease to outcome criteria = triggers in-depth review of decreased categories

Please Note: it is possible to have in the same category exceeded beyond expectations and also a decrease to outcome criteria. Example 65% students increased their measurable criteria, 15% no change and 20% decreased

Evaluation Process – Objectives/Outcomes

Evaluation Process Used: Qualitative research data collection and analysis best practices were solicited from and reviewed with the advisory committee (Educators, Mental Health Professionals, Councilors and Students).  Some evaluation concepts applied include limited reference of Utilization-Focused Evaluation 4th Edition – Michael Quinn Patton.  Systematic entrance/exit surveys of students; post activity surveys and interviews with students; teachers, mental health co-facilitator (formal and informal) as well as face to face dialogue.

Evaluation Process Objectives: Get the best possible data to affect decisions; Validity of data and face value (i.e. what do we want to find out and why:  how, when and where the data is gathered); Privacy (students see and understand their names are not kept in the research)

Overall Evaluation Objectives

  • How important is creative expression to students
  • How do students related creative expression outside of art classes
  • Do students relate creativity to dealing with broader life issues
  • Does the Creative Spark inspire creativity
  • Does Creative Spark effectively link creative activities and wellness
  • Does the Creative Spark help youth with mental health issues
  • What needs to happen next

Outcome Objectives:            

Creative Spark Links to Mental Wellness (Sparks)

  • Achieve 15% or greater positive increase on overall evaluation objective criteria
  • Establish Creative Spark helps to build confidence and resilience in students
  • Establish Creative Spark helps inspire persistence in students
  • Establish Creative Spark helps inspire empathy in students
  • Establish Creative Spark inspires students to socialize more with peers
  • Establish Creative Spark helps reduce psychological stress

Note: some survey data and interview findings are subjective dialogue, sample size too small and/or non-transposable into viable numerical data.  This information is captured in survey comments and final analysis.

Decisions and Next Steps

Based on all evaluation criteria, feedback and comments the Creative Spark pilot has succeeded above and beyond expectation on every objective category and desired outcome.  Still there is lots of room for improvement and ways to make the Creative Spark a large scale and sustainable program empowering young Canadians to be more creative in the lives and to take responsibility for their mental wellbeing.

Decisions

Give students more time to reflect, in particular on links to health

Class materials must be prepared completely in advance

Larger samples and/or use Smartboard to display materials and examples

Evaluation questions should examine mental health subjects in more depth

Rob (any facilitator) needs an assistant to deliver the classes.  Mental Health co-facilitator can provide this assistance provided simple training is provided

Need to capture and model Rob’s style and energy on key deliverables

More time to debrief and learn from teachers and co-facilitators after each class

More teacher instruction and support material so that they can carry on themes with their class to reinforce the program

 

Leave a Reply