Learning from Faith Formation during the Pandemic:
Reflection and Evaluation
John Roberto
In the midst of the disruption, loss, pain, and stress of this pandemic, pastors and faith formation leaders have been learning a lot by listening attentively to the needs of their people and responding creatively with new approaches to faith formation. Leaders have done amazingly creative work in faith formation, reaching and engaging people of all ages with tools and methods and media they never imagined using way back in 2019.
There’s much to be learned from your pandemic creativity and innovations. Many of the new approaches you launched—at home faith formation, digital methods and media, online learning, to name a few—can become permanent features of faith formation in your church for years to come.
Now is a good time to reflect and evaluate while everyone’s experience is still fresh. This is the time to capture your personal learning and engage all those participating in faith formation in sharing their stories and experiences. This article presents three types of evaluation: Personal, Team, and Participant. I have created worksheets for each of the three types of evaluation presented in the article. They can be found in the Appendix. Adapt the questions and worksheets for your context and purposes.
Personal Learning
The first type of reflection and evaluation is personal. Take some personal time to process your experience of the pandemic and the ways you responded to the needs of people in your community. Select questions from the list below that can help you name your experience and identify your learning during the pandemic. You may want to start a journal to record your reflections.
Lawrence Peers in his article “Showing Up” offers these questions for reflection:
Team Learning
The second type of reflection and evaluation is with your leadership team. You can send your team questions to prepare, conduct a Zoom meeting and record the results, and then review the results and prepare a report. Zoom creates a video and audio version of your meeting. You can use the audio version to transcribe the meeting into your report. According to Zoom,
Audio transcription automatically transcribes the audio of a meeting or webinar that you record to the cloud. After this transcript is processed, it appears as a separate VTT file in the list of recorded meetings. The transcript is divided into sections, each with a timestamp that shows how far into the recording that portion of the text was recorded. You can edit the text to more accurately capture the words, or to add capitalization and punctuation, which are not captured by the transcript.
First, a simple way to evaluate the year is to conduct a Four-Quadrant Evaluation. Invite the team to reflect on the experience of leading faith formation during the pandemic.
Second, you can evaluate the year using “10 Questions to Ask Now” by Susan Beaumont. She organizes her questions into three categories: what have we lost, what did we assume, and what wants to emerge. You can select one or more questions in each category for your team evaluation session.
What Have We Lost?
All significant transitions begin with an ending. Something must come to an end before we can explore a new beginning. Most of us resist endings, accompanied as they are by loss. We gloss over the painful work of grief to move onto action, which feels more productive.
What Did We Assume?
Many of our old assumptions no longer hold true. If we do not acknowledge the truth of this, we will make decisions that are inappropriate for the next season. We need good questions to unfreeze some of our old assumptions and expand our consciousness.
What Wants to Emerge?
When we ask better questions, we invite innovation into the new order. We can be led by the future itself into something fresh and exciting. We begin noticing what wants to emerge through us.
These questions are not meant to be magic bullets that solve problems. The questions are merely an invitation to pay exquisite attention—to notice why we are making the choices we make, and what wants to happen next. The dialogue that these questions provoke can make the difference between merely surviving this season and thriving in the next.
Third, you can use the six questions below to prepare for planning the 2021-22 program year. They are adapted for faith formation from: “Church Re-entry Roadmap: Creating a Plan for Your Ministry” developed by Ministry Spark (from David C Cook). (You can download the entire Roadmap for free from: https://ministryspark.com/resources/covid-19-church-re-entry-roadmap.)
1. Make a list of what worked and didn’t work in faith formation pre-COVID with age groups, families, and all ages together.
2. Write down your vision and goals for faith formation and make sure your new plan for faith formation is focusing around that vision and goals.
3. Think through what worked and didn’t work during COVID.
4. Identify what you have learned from the challenges you faced and are facing?
5. Identify the new opportunities that are now present to you and your church?
6. Make a list of the ways that you are innovating. Then identify ways that you can continue innovating in 2021-2022. What areas (challenges, opportunities) are ripe for innovation?
Participant Learning
The third type of reflection and evaluation is participant feedback. There are four levels in evaluating a program’s effectiveness.
Donald Kirkpatrick in Evaluating Training Programs identified these four levels and symbolized them as a pyramid. While it is not easy in most faith formation programming to get feedback on the results of our programming, we can find ways to invite people to share their reactions, learning, and new behaviors
Transfer or behavior change is the most difficult to measure. We need to find out if the participants are applying what they are learning to their lives. Measuring behavior change is necessary, not only to see if behavior has changed, but also to determine the reasons why change has not occurred.
Here is a brief overview of the four levels of evaluation.
Level 1: Reaction
(Immediately at the conclusion of a program or activity.)
How do participants react to the program, or better, what is the measure of their satisfaction?
Level 2: Learning
(Within a short time after the conclusion of a program or activity)
To what extent has learning occurred: understanding, practice, attitudes?
Level 3: Transfer
(Ideally before the program begins and then at a later time so people can integrate their learning into their lives.)
To what extent has behavior changed as a result of the program?
Level 4: Results
(Ideally before the program begins and then at a regular intervals to see how people are living their learning.)
To what extent have results occurred because of the program?
Methods and Tools for Evaluating Participant Learning
An evaluation tool can be as simple as two questions—“What benefited you?” and “What could we do to improve the experience?”—given at the conclusion of a program or activity. At a gathered event people can receive a file card and write a response to each question on each side of the card. For a non-gathered program or activity (online, at home) you can use a form on your website or a Google Form or an email with the two questions.
The method that you use needs to match the tool and the questions you are asking. For example to evaluate a season or a year of programming you will want something more structured. Creating an evaluation form on SurveyMonkey (www.surveymonkey.com) gives you a way to ask multiple questions (structured and open-ended), invite people to complete the survey, and then have SurveyMonkey compile the results into a report for everyone to review and analyze. You can send participants an email or text with the survey link (URL) or post the link on your website. You can also use Google Forms (www.google.com/forms/about) to create a similar type of online evaluation. When creating a survey, try to limit the number of essay questions. People prefer the multiple choice/rating scales.
The evaluation tools presented in the Appendix can be used with a variety of methods. Briefly, some of the more popular methods for conducting an evaluation include:
The Research Guide that accompanies this article describes how to conduct interviews and focus groups, and then process the results.
Worksheets
(Download the MS Word format of the Worksheets file so that you can adapt the questions.)
Works Cited
There’s much to be learned from your pandemic creativity and innovations. Many of the new approaches you launched—at home faith formation, digital methods and media, online learning, to name a few—can become permanent features of faith formation in your church for years to come.
Now is a good time to reflect and evaluate while everyone’s experience is still fresh. This is the time to capture your personal learning and engage all those participating in faith formation in sharing their stories and experiences. This article presents three types of evaluation: Personal, Team, and Participant. I have created worksheets for each of the three types of evaluation presented in the article. They can be found in the Appendix. Adapt the questions and worksheets for your context and purposes.
Personal Learning
The first type of reflection and evaluation is personal. Take some personal time to process your experience of the pandemic and the ways you responded to the needs of people in your community. Select questions from the list below that can help you name your experience and identify your learning during the pandemic. You may want to start a journal to record your reflections.
Lawrence Peers in his article “Showing Up” offers these questions for reflection:
- How would you describe your own personal experiences of doing ministry during this time?
- Are some emotions more welcome than others—within you? among your colleagues? with members in your church community?
- What strengths are you relying on?
- What are your disappointments? What are your concerns?
- What is wanting to emerge in your life and ministry that you may only have a slight glimpse of right now?
Team Learning
The second type of reflection and evaluation is with your leadership team. You can send your team questions to prepare, conduct a Zoom meeting and record the results, and then review the results and prepare a report. Zoom creates a video and audio version of your meeting. You can use the audio version to transcribe the meeting into your report. According to Zoom,
Audio transcription automatically transcribes the audio of a meeting or webinar that you record to the cloud. After this transcript is processed, it appears as a separate VTT file in the list of recorded meetings. The transcript is divided into sections, each with a timestamp that shows how far into the recording that portion of the text was recorded. You can edit the text to more accurately capture the words, or to add capitalization and punctuation, which are not captured by the transcript.
First, a simple way to evaluate the year is to conduct a Four-Quadrant Evaluation. Invite the team to reflect on the experience of leading faith formation during the pandemic.
- Quadrant 1. The upper left quadrant is for identifying what the participants in faith formation (children, teens, adults, and/or families) found beneficial and what really engaged them.
- Quadrant 2. The upper right is for what your team wants to celebrate. What you did you accomplish since the pandemic began?
- Quadrant 3. The bottom left is for ideas to build on the bright spots from the past year.
- Quadrant 4. The bottom right is for what would you like to improve or give greater attention to as you plan faith formation in the 2021-22 year.
Second, you can evaluate the year using “10 Questions to Ask Now” by Susan Beaumont. She organizes her questions into three categories: what have we lost, what did we assume, and what wants to emerge. You can select one or more questions in each category for your team evaluation session.
What Have We Lost?
All significant transitions begin with an ending. Something must come to an end before we can explore a new beginning. Most of us resist endings, accompanied as they are by loss. We gloss over the painful work of grief to move onto action, which feels more productive.
- What were we on the verge of discovering or accomplishing before the onset of the pandemic? What needs to move forward in different ways now?
- What was possible before that may not be possible for some time—if ever?
- What seemed important before that feels superfluous now?
What Did We Assume?
Many of our old assumptions no longer hold true. If we do not acknowledge the truth of this, we will make decisions that are inappropriate for the next season. We need good questions to unfreeze some of our old assumptions and expand our consciousness.
- What was undervalued before that may hold greater value now?
- What mattered about geography before that no longer matters?
- What margin or lack of margin was built into our old model of doing church? What new abundance are we experiencing now? Where are we experiencing scarcity now that was not evident before?
What Wants to Emerge?
When we ask better questions, we invite innovation into the new order. We can be led by the future itself into something fresh and exciting. We begin noticing what wants to emerge through us.
- What is our greatest asset now?
- What relationships will we need to build on or strengthen in the months ahead?
- What unique role might our congregation play in local, national, and even global recovery?
- What long term changes in the bigger picture would we like to be part of bringing to fruition?
These questions are not meant to be magic bullets that solve problems. The questions are merely an invitation to pay exquisite attention—to notice why we are making the choices we make, and what wants to happen next. The dialogue that these questions provoke can make the difference between merely surviving this season and thriving in the next.
Third, you can use the six questions below to prepare for planning the 2021-22 program year. They are adapted for faith formation from: “Church Re-entry Roadmap: Creating a Plan for Your Ministry” developed by Ministry Spark (from David C Cook). (You can download the entire Roadmap for free from: https://ministryspark.com/resources/covid-19-church-re-entry-roadmap.)
1. Make a list of what worked and didn’t work in faith formation pre-COVID with age groups, families, and all ages together.
- What were you doing in physical settings?
- What were you doing in online settings?
- What were you doing in hybrid settings that blended physical and online?
2. Write down your vision and goals for faith formation and make sure your new plan for faith formation is focusing around that vision and goals.
3. Think through what worked and didn’t work during COVID.
- What worked or didn’t work in faith formation in online-only settings?
- What worked or didn’t work in faith formation in physical settings?
- What worked or didn’t work in faith formation in hybrid settings?
- What works for both a physical and digital setting?
- What did you start that you want to continue?
4. Identify what you have learned from the challenges you faced and are facing?
5. Identify the new opportunities that are now present to you and your church?
6. Make a list of the ways that you are innovating. Then identify ways that you can continue innovating in 2021-2022. What areas (challenges, opportunities) are ripe for innovation?
Participant Learning
The third type of reflection and evaluation is participant feedback. There are four levels in evaluating a program’s effectiveness.
- Reaction: How do participants react to the program, or better, what is the measure of their satisfaction?
- Learning: To what extent has learning occurred?
- Transfer: To what extent has behavior changed as a result of the program?
- Results: To what extent have the results occurred because of the program?
Donald Kirkpatrick in Evaluating Training Programs identified these four levels and symbolized them as a pyramid. While it is not easy in most faith formation programming to get feedback on the results of our programming, we can find ways to invite people to share their reactions, learning, and new behaviors
Transfer or behavior change is the most difficult to measure. We need to find out if the participants are applying what they are learning to their lives. Measuring behavior change is necessary, not only to see if behavior has changed, but also to determine the reasons why change has not occurred.
Here is a brief overview of the four levels of evaluation.
Level 1: Reaction
(Immediately at the conclusion of a program or activity.)
How do participants react to the program, or better, what is the measure of their satisfaction?
- Determine what you want to find out.
- Design a form or process that will quantify reactions.
- Encourage written comments and suggestions.
- Get honest responses.
Level 2: Learning
(Within a short time after the conclusion of a program or activity)
To what extent has learning occurred: understanding, practice, attitudes?
- Evaluate knowledge, practices, and/or attitudes before and after the program.
- Measure knowledge, practices, and attitudes.
- If practical, use a control group that did not participate in the program to compare with the group that participated.
Level 3: Transfer
(Ideally before the program begins and then at a later time so people can integrate their learning into their lives.)
To what extent has behavior changed as a result of the program?
- If possible, evaluate before and after the program. It is usually impossible to do this, so it becomes necessary to do it after the program and determine what the participant is doing differently than he or she was doing before the program.
- Allow time for the behavior to change.
Level 4: Results
(Ideally before the program begins and then at a regular intervals to see how people are living their learning.)
To what extent have results occurred because of the program?
- Measure both before and after the program.
- Allow time for possible results to be achieved.
- Repeat the measurement at appropriate times.
- Use a control group if practical.
Methods and Tools for Evaluating Participant Learning
An evaluation tool can be as simple as two questions—“What benefited you?” and “What could we do to improve the experience?”—given at the conclusion of a program or activity. At a gathered event people can receive a file card and write a response to each question on each side of the card. For a non-gathered program or activity (online, at home) you can use a form on your website or a Google Form or an email with the two questions.
The method that you use needs to match the tool and the questions you are asking. For example to evaluate a season or a year of programming you will want something more structured. Creating an evaluation form on SurveyMonkey (www.surveymonkey.com) gives you a way to ask multiple questions (structured and open-ended), invite people to complete the survey, and then have SurveyMonkey compile the results into a report for everyone to review and analyze. You can send participants an email or text with the survey link (URL) or post the link on your website. You can also use Google Forms (www.google.com/forms/about) to create a similar type of online evaluation. When creating a survey, try to limit the number of essay questions. People prefer the multiple choice/rating scales.
The evaluation tools presented in the Appendix can be used with a variety of methods. Briefly, some of the more popular methods for conducting an evaluation include:
- Conducting interviews with selected participants
- Conducting focus groups of 8-10 participants
- Designing and disseminating a survey with structured-response evaluation questions and open-ended questions using Survey Monkey ( https://www.surveymonkey.com) or Google Forms (www.google.com/forms/about)
- Conducting a poll or survey on Facebook or conducting a chat on Facebook at a set time to solicit feedback
- Inviting people to upload a photo or story to Facebook or Instagram describing the experience of participating in the program or describing what they learned
- Inviting people to create a video describing their experience, using a couple of prepared questions, and then posting the videos to your church’s YouTube channel and/or uploading them to your church website
The Research Guide that accompanies this article describes how to conduct interviews and focus groups, and then process the results.
Worksheets
(Download the MS Word format of the Worksheets file so that you can adapt the questions.)
- Personal Learning: 5 Questions
- Team Learning: Four-Quadrant Evaluation
- Team Learning: 10 Questions to Ask Now
- Team Learning: Creating a Roadmap for Planning
- Participant Learning: Questionnaire Level (1 Evaluation)
- Participant Learning: Unfinished Sentences (Level 1 or 2 Evaluation)
- Participant Learning: Questionnaire (Level 2 Evaluation)
- Participant Learning: Reflection & Application Questions (Level 2 Reaction Evaluation)
- Participant Learning: Application Tools (Level 3 Evaluation)
Works Cited
- Beaumont, Susan. “10 Questions to Ask Now.” Congregational Consulting, June 1, 2020. (Accessed at https://www.congregationalconsulting.org/10-questions-to-ask-now)
- “Church Re-entry Roadmap: Creating a Plan for Your Ministry.” Published by David C Cook, 2020. (Accessed at: https://ministryspark.com/resources/covid-19-church-re-entry-roadmap)
- Kirkpatrick, Donald L. Evaluating Training Programs. Berrett-Koehler, 1998.
- Peers, Lawrence. “Showing Up.” Congregational Consulting, September 25, 2020. (Accessed at: https://www.congregationalconsulting.org/showing-up)
|
Download the PDF of the Article.
|
Download the Customizable Worksheets
| ||||
| Research Guide.docx |
