Sheila and Kim here, writing today with Jennica Nichols and Maya Lefkowich, creative evaluators from AND implementation in Vancouver, BC (Canada). Jennica and Maya practice arts-based evaluation, so this was a fun opportunity for us to play a little in their methodological sandbox.
Over the last few months we’ve been chatting and swapping stories about our relationships with questions, the importance of thoughtful question asking (and answer receiving), and the possibilities that intentionally creative questions can create.
Asking questions is central to evaluative practice, but doing it “well” is something we can easily take for granted. We’ve had fun sharing and reflecting together about what this means to us and how this shows up in our work. We hope the stories in this post spark reflection about your own relationship with questions too.
Jennica & Maya’s story
My feet ached as I kicked my heels off under the table and reached for the tennis shoes at the bottom of my bag.
Jennica collapsed into the banquet chair next to me, “How was your first evaluation conference?”
Boring. A symphony of pleasantries composed in Times New Roman and Microsoft Blue. “Good,” I smiled. Just a few months after starting an evaluation business together, I wasn’t sure that I wanted to be an evaluator. The conference didn’t help. And, I was afraid of disappointing my partner and ruining our company before it got off the ground.
“Oh yeah,” Jennica smirked, “How was it really?”
“I hated it.” I admitted,“It’s like a cult. I don’t know the words and everyone worships the same person.” My stomach churned as I imagined Jennica’s disappointment.
She laughed, “I knew you would,” she was unbothered. It wasn’t the catastrophe I imagined.
A pitcher of cheap Happy Hour lager later, I confessed that I couldn’t couldn’t see myself as an evaluator. Everything was too sterile and impersonal. And, what on earth was a KPI?
Jennica listened carefully, and then asked, “What’s one thing you would do differently to make the evaluation community more your flavour?”
I thought about it for a year. At the next evaluation conference, I made a change. During the talks, I scribbled down what the presenters said verbatim. Then, I rearranged the words and phrases into a found poem. It was a flirtation between what they said and how I experienced their talk. After the sessions, I gave the poems back to the speakers and introduced myself. It was terrifying. And, it was the intimacy and creative flair that I needed to feel like I belonged.
Looking back, I realize that Jennica’s question changed everything. Now, instead of exchanging forgettable pleasantries and business cards at conferences, Jennica and I write and share poems. It forges more authentic connections – even if just for a moment. And as a result, I see myself as part of the community because I am building relationships with evaluators one poem at a time.
Sheila’s story
“Alright everyone,” I began, standing in front of the professional development policy board with a stack of pages.
“Today, we’re diving into some actual qualitative data analysis,” I said, standing in front of the policy board with a stack of pages.
“We had over 1,800 responses to the open-ended question, ‘What did you learn?‘ from our professional development feedback surveys. I want you to work in small groups and start coding these responses, using the skills you developed in our last session together.
The room filled with focused discussion. Occasionally, I’d hear some chuckling and faint exclamations of surprise that eventually settled into a murmur of conversation that spread through the room.
“Well, what did you find?” I asked as the groups came back together.
Nancy raised her hand. “So many of these answers are basically just restating the course titles! They’re not really saying what the person learned!”
Several others nodded feverishly in agreement.
I agreed. “These responses don’t provide the depth or richness of information we need to make informed decisions about our professional learning programs.”
Tom, another board member added, “So, what’s the solution?”
“Well, if we’re not getting the insights we need,” Margie offered. “Maybe the way we’re asking the question is the problem.”
A thoughtful silence settled over the room as we considered this.
“Instead of asking, ‘What did you learn?’” I offered, “What if we asked about their feelings and experiences during the sessions? And then follow up with a question that asks them to describe their feelings?”
Linda raised an eyebrow. “Emotions are closely linked to learning. That might give us more depth.”
The room started to buzz with enthusiasm for this new direction and I suggested,
“For example, we could ask participants to select one or more emotions they might feel during a session, like energized, renewed, bored, inspired, overwhelmed, angry, excited, frustrated, and so on. And the follow up question would ask why they chose the words they did.”
Tom nodded. “It sounds like we’re on to something. Let’s try it out.”
As the meeting wrapped up, I felt a sense of accomplishment. We turned a frustrating data collection challenge into an opportunity for innovation! We acknowledged the limitations of what we were doing and embraced an experimenting with new ways to gather the information we needed.
We’ve now used this approach for several years and found that respondents are generally more forthcoming in their descriptions of how they felt. Through their descriptions, we’re able to discern more about their learning experience and better understand how they plan to apply their learning. Best of all, we’re receiving far fewer uninformative responses.
Kim’s story
My coworkers and I nervously called the room of about 40 community program leaders together. Our mission for that morning was to orient the group to the new initiative’s evaluation. They were required to be there as programs funded through the initiative, and unsure about what their participation would entail. We knew we’d have to shake things up a bit to convey our approach.
“Hi all! Today, we want to start with a little reflection. Please spend a few minutes thinking about your past experiences with, or your perceptions of, evaluation. What does the word evaluation mean to you? Good, bad or ugly, we want to hear it. No hard feelings, we promise. Use the note cards on your table to jot down a few words, or a phrase, to capture your thoughts.”
After a few minutes we queued the group to share a bit with a neighbor at their tables.
We could sense trepidation, but we quickly noticed nodding, laughter, and a collective sense of relief start to wash over the group as many shared similar sentiments, frustrating experiences, and curiosity about what we had planned.
“Who is feeling brave enough to share with the whole group?” my colleague asked.
From toward the back of the room, someone shouted “busy work we have to do for funders” glancing sideways at the foundation program officers sitting nearby. But they and we nodded in acknowledgement, understanding that this is, unfortunately, a common experience.
“Judgment of our work by others who don’t understand it” another participant added. Heads nodded again.
“Outcomes that aren’t meaningful to us.” More nodding, and a “hear hear” rang out.
“Reports no one reads” said a funder staff member, which elicited laughter all around.
“Thank you for sharing” I said after a few others added their reflections. “We can’t promise that the initiative evaluation experience will be all roses and sunshine, but we really do want it to be truly useful to you, and a positive experience.
“We know that sometimes letting go of old experiences or expectations can help us move forward. We wish we were standing around a campfire right now, but we’ll have to make do with a shredder since we’re inside. If you’d like to, we invite you to come up and shred those negative experiences in hopes of leaving them behind.”
Participants queued up in a line, enthusiastically shredding their notecards. The “scree-brrttttttttt” of the shredder whirred and we could feel the dynamics in the room shifting as participants realized they might have some power over the process ahead.
We took careful notes about what came up so we could be sure to acknowledge and incorporate it going forward, and the mood lightened. Questions started to bubble up from the group.
“Are you sure you aren’t going to use our program quality results to determine whether our funding is renewed?”
“No, we’ve set up systems and processes to prevent that from even being possible.”
“How will we know if we’re successful?”
“We have some outcomes in mind, but we’re going to figure that out together. What do you think that might look like?”
By starting with a question about participants’ past experiences, and then responding to them as we did, we were able to start thinking together about what might be possible going forward. Participants who might have groaned about evaluation at the start were now eager to help shape it with us.
What’s the “moral” of these stories?
Each of these stories illustrates how thoughtful, creative and well-timed question asking and answering can help shake things loose when we could otherwise be stuck—how they can help us think or act differently, or be more willing to try doing so. In each scenario, questions prompted a reflective moment that led to deeper insights and a solution, or at least some creative steps toward solving a problem.
Our reflections on these reflections? We challenge ourselves and others to:
- Ask ourselves tough questions, and honor the answers by acting on them
- Be willing to try something new; to ask questions differently than we might normally do, or to resist defaulting to our usual practices related to question asking, and answer receiving
- Embrace our humanity! Remember that we and our evaluation participants are whole beings who might be able to more easily access how we are feeling than what we think. Doing something ‘unusual’ together can help us build the relationships needed to do things differently.
