ASU Ch 12 Evaluation of Adult & Career Education Programs Case Study Review Chapter 12 Case Study Instructions Critique the following interview. Christie

ASU Ch 12 Evaluation of Adult & Career Education Programs Case Study Review Chapter 12 Case Study Instructions Critique the following interview. Christie, C., & Preskill, H. (2006). Appreciative Inquiry as a method for evaluation:An interview with Hallie Preskill. American Journal of Evaluation, 27(4), 466-474. Typing Guidelines Use the template for this assignment. Department of Adult and Career Education
Evaluation of Adult and Career Education Programs
ARTICLE REVIEW
Case Study:
Student Name:
Title of Article:
Author(s):
Summary of Article:
My Views:
Dialogue With Hallie Preskill
Christina A. Christie
Christie: In my experience, when I’ve had people ask me, “So what is an appreciative inquiry?”
it’s always been a bit of a challenge for me to offer them a reasonable answer. I’ve only read about
appreciative inquiry (AI) as an organizational change method, so there were many things that
struck me about the AI evaluation report you’ve provided for my review in preparation for this
interview. I’ve never read about what it’s like to actually conduct an AI, particularly as part of an
evaluation. Could you describe for me what your experience has been with AI?
Preskill: I came across AI in the organizational learning and development literature about eight
or nine years ago and became excited about the possibilities it offered evaluators, particularly those
who use participatory, collaborative, democratic, and learning approaches to evaluation. And the
more I learned, the more I became convinced that the approach could not only enhance the use of
evaluation findings but that it could contribute to process use and building evaluation capacity. So
I started reading everything I could on the topic; took a five-day workshop from David Cooperrider
and Diana Whitney, the founders of AI; began conducting workshops on the applications of AI to
evaluation; and eventually began writing about its uses in evaluation.
Christie: Before we get into the details of the CEDT evaluation project and how you used AI,
can you give me a brief explanation of what AI is?
Preskill: Sure. AI has been used as an approach to organizational change and development
since the mid-1980s. In a nutshell, AI is a process that builds on past successes (and peak
experiences) in an effort to design and implement future actions. The philosophy underlying AI is
that deficit-based approaches to organizational change are not necessarily the most effective and
efficient. In other words, when we look for problems, we find more problems, and when we use
deficit-based language, we often end up feeling hopeless, powerless, and generally more
exhausted. To the contrary, proponents of AI have found that by reflecting on what has worked
well, by remembering the times we were excited and energized about the topic of study, and when
we use affirmative and strengths-based language, participants’ creativity, passion, and excitement
about the future is increased (see Cooperrider, Sorensen, Whitney, & Yager, 2000; Whitney &
Trosten-Bloom, 2003).
Christie: Okay, let’s talk a bit about how you applied AI to the development of the CEDT
evaluation system.
Preskill: CEDT is the internal training function of Sandia National Laboratories, which is one
of the United States National Laboratories for Defense. They do a lot of research and testing on
antiterrorism technologies, and most of their work is classified. They are located on the Kirtland
Air Force Base in Albuquerque, New Mexico.
The CEDT department offers an extensive array of learning opportunities delivered through
instructor-led classroom training, Web-based and computer courses, self-study, coaching,
mentoring, and consulting on a wide variety of topics. While some evaluation of the department’s
services and offerings had been occurring, there was no system or coordination of these
evaluations.
Christie: How did CEDT come to hire you to do an evaluation-focused AI?
Preskill: One of the instructional designers had been in my evaluation course a few years back.
One day, I received an e-mail from him saying, “Our evaluation team met today, and it was decided
we would like to meet with you to discuss the Sandia Corporate training and development
evaluation process.”
Christie: The CEDT “evaluation team” wanted to meet with you, suggesting that they had an
evaluation team?
Preskill: They had recently put together a team of three. As with many organizations, they were
increasingly being asked to prove their department’s effectiveness. They knew that CEDT
employees were collecting evaluation data, but few knew what others were doing, and there was
little sharing of data collection instruments and/or evaluation findings. The organization had no
system for gathering data, aggregating it, making sense of it, and then acting on it. They just didn’t
think their process was very effective. I think that’s why they put together the “evaluation team”
and then called us in.
Christie: How did you come to use AI with this group?
Preskill: For a while, I had been thinking about what an “evaluation system” would look like
in an organization. I have always been surprised at how ad hoc many evaluations in organizations
seem to be. Without a coordinated or strategic system for evaluation, it is difficult to compare
evaluations from year to year and to share what is learned from evaluation practice. Without an
evaluation system, you also struggle with identifying programs that are being under- or overevaluated and with deciding how to effectively allocate evaluation resources. Perhaps most
important, the lack of an evaluation system significantly limits the impact evaluation may have at
a strategic level and, thus, its ability to provide information on critical organizational issues. As
we talked with the evaluation team and learned what their needs were, it became clear that we
would need to not only understand what kinds of data they were collecting and how, but that we
would need to be very clear as to what would work in their particular context. Conceivably, that
meant talking with a large number of employees. Given that our time and resources for this project
were limited, my business partner and I decided that in order to design the “ideal system,” AI
would be an appropriate strategy for the six-hour Needs Analysis Meeting. We chose this approach
because it would offer participants a way to raise issues and concerns about current evaluation
practices while also helping them envision an evaluation system that could work based on previous
positive evaluation experiences.
Christie: How did you get this CEDT group to agree that AI was the appropriate method to use
in this case?
Preskill: The evaluation team knew of another AI project taking place elsewhere in the
organization, for another purpose, so AI wasn’t completely countercultural. They also purported
to be a learning organization. Thus, they wanted to model open-mindedness and were willing to
try new things. Given the presumed organizational acceptance of AI and the desire to be a learning
organization, I think they felt it was worth a try. When we talked about using AI, they seemed very
receptive. We gave them an introduction to AI and why it might be a good fit with this project.
Christie: Your former student, the instructional designer, had a position of authority such that
he could actually say, “Yes, this is something we want to do.”
Preskill: Yes, however he reported to one of the people on the evaluation team.
Christie: So you had buy-in from the top?
Preskill: Yes. The vice president of the department was consistently kept in the loop as to what
was happening. She was the one who initially authorized the funding to bring us in. And she
authorized the final proposal. On the day of the AI meeting, she and her colleagues—two VPs—
were full participants.
Involvement of the leadership was critical at two levels. First, there had to be support for any
changes the system would require in the organization’s infrastructure. Second, if the new
evaluation system was going to require new resources, the organization’s leadership was going to
have to develop a business case for securing these. Another advantage of having the leadership in
the room is that it symbolically communicates that what we are doing is important, that we’re
going to see this through, that you’re going to have what it needs and what it takes to make it
happen, and that the results, whatever those are, are going to be used.
Christie: How did you determine the topics of primary interest for the inquiry?
Preskill: Our scope of work was referred to as a needs analysis—to determine the evaluation
need, or system, and then to come up with a framework from which they could develop that system.
My colleague and I developed a scope of work—what we saw as the requirements of the project.
The scope of work was to review and analyze the materials of the existing evaluation system and
to obtain input from the department’s employees regarding what elements were critical in an
evaluation system for their organization. Our proposal included planning for and facilitating the
AI meeting, debriefing the meeting with our evaluation team and a few others, analyzing the data
generated from the meeting, and developing what they called “a user-friendly report” that
described current evaluation methods and the ideal evaluation system.
Christie: In short, CEDT presented you with an opportunity to use AI to conduct a needs
analysis for a comprehensive evaluation system.
Preskill: Exactly. And we were very excited. It was an opportunity to do two things: one, to
develop an evaluation system for an organization and, two, to use AI for this purpose. I knew it
would be a learning experience for all of us but that it would also test what I believed about the
potential for AI in evaluation. I had a lot of faith that it would work.
Christie: As a methodology?
Preskill: Right. AI has been used very successfully in organizational change contexts around
the world, in a wide variety of settings. People have used it for large-scale organization change,
for changing organizational and community culture, and for getting movement where there was
stagnation in organizations and communities. At the time we started this project, we were not
aware of anyone who had used AI for evaluation purposes. Since that time, however, we have
found several evaluators who are using AI processes, which is very encouraging. We have tried to
highlight some of these in a book that Tessie Catsambas and I wrote, Reframing Evaluation
through Appreciative Inquiry (Preskill & Catsambas, 2006). We have included 16 cases from
around the world in which AI principles and practices have been used in evaluation.
Christie: You mentioned that you wanted to involve as many people as possible in developing
the evaluation system. An all-day meeting can be quite a commitment for people. What was
recruitment like for the AI meeting?
Preskill: An e-mail invitation, co-signed by the department’s director and the evaluation team,
was sent to all of the department’s 52 employees. The e-mail said that evaluation has been an issue
and we’ve hired a local consultant to come in and help. By communicating beforehand why this
was important and why employees should show up, that it was going to be a different kind of day,
43 (83%) of the employees attended the meeting. Part of me wants to say that participation was a
reflection of the organization’s commitment to evaluation. I think these people are generally
committed to what they’re doing. But I didn’t get the sense that they were all as crazy about the
evaluation as I wanted them to be. I think they were curious, maybe. I think it was communicated
that it would be worth their time to participate. And I think they felt they needed to be responsive
to proving their value and worth in the organization. In fact, one of the department’s goals focused
on demonstrating effectiveness.
Christie: Was the meeting framed as a professional development experience, or was it framed
as an activity that the organization was engaging in?
Preskill: Good question. No, it wasn’t billed as a professional development day. It was referred
to as a workshop. However, we ended up calling it the “Big Day Meeting” because everyone was
invited to participate.
Christie: It seems obvious that it’s critical that active, engaged participation is needed for this
type of process to be successful. To me, the setup is incredibly important because that will impact
whether people want to participate and the level at which they choose to engage in the process.
Aside from the e-mail invitation that you described, what was done or what factors contributed to
the successful setup of the Big Day Meeting?
Preskill: It was a time in the organization when they liked the notion of being a learning
organization. They had just reorganized, so I think they felt energized to a certain extent but also
frustrated with some of the limitations of their technology (particularly with regard to evaluation).
I think they respected their leaders. I also got the sense that they felt they were in this together. I
believe they saw this as worthwhile in the larger picture of what they needed to do to contribute to
the organization. There was a lot of internal pressure to show results. I think they were frustrated
that they couldn’t either gather the necessary data or aggregate it or report it in a way that was
meaningful and useful.
The commitment to our meeting came from knowing they had to figure out a way to more
effectively collect information and use that information to advance as an organization. Overall,
people were saying, “We don’t have the technology” or “We don’t have access to our data” or
“The forms we’re using are poor, and I don’t know how to make them better.” Some might have
been motivated by personal need, irritation with the current system, and knowing that they had to
change, so the AI we were going to embark upon was as good as anything.
Christie: Did you describe the AI model to participants before the Big Day Meeting?
Preskill: Not in this case. I don’t think you always have to teach participants what AI is. If you
start by describing the underlying theories, it often serves as a barrier—some participants will want
to debate its principles instead of experiencing it. Rather, I just say, “This process may be a little
bit different than what you might have experienced in the past. If you want to know more about it,
at the end of the day, I’ll be happy discuss it with you, and I can refer you to some great resources.”
For clients who have no knowledge of AI, I like to give them a taste of the AI process before I
tell them about AI’s underlying theories and principles. I think it is one of those processes that
either requires a leap of faith and trust that it will work or it needs to be experienced. We are so
used to being negative and finding faults that when we talk about focusing on what has worked,
what has been successful, many people just assume it is one of those “Pollyanna and rose-colored
glasses approaches” and that it won’t address the pressing issues and problems. However, our
experience has shown that problems do get addressed, but in a very different way. After engaging
in AI, this becomes clear to most participants.
Christie: What do you do if you have resistant participants?
Preskill: I’ll have my client go through an exercise where they experience the first phase of
AI—they conduct appreciative interviews. For the CEDT evaluation team, we put together a
couple of handouts, described the process, and then answered their questions. They got it right
away; for some people, it intuitively makes sense. And it’s in line with the culture they are trying
to create—a collaborative culture, one that is focused on success versus problems. You’re usually
willing to believe that you can solve problems by describing and discussing success. But some do
have a harder time with it.
Christie: I found it interesting that, in fact, it’s only in the very beginning of the evaluation
report that you claim this to be an AI project. It struck me that it doesn’t matter what you call it.
The focus is the process, beginning with stories about what has worked—the positive pieces. But
in my experience, people do like to talk about what is “wrong.” How do you keep AI participants
focused on the positive?
Preskill: First, it’s critically important to remember that the first phase of an AI is Inquire (or
Discovery, depending on which labels you use), where participants conduct paired interviews
using a specifically designed interview guide. There are three sets of core questions that are tailored
to the inquiry. The first question is the Peak or Best Experiences question that asks interviewees
to tell a story about a time when they felt excited, energized, hopeful, and successful relative to
the topic of the inquiry (e.g., conducting an evaluation). They are asked to describe what was
happening, who was there, their role, and the core factor that contributed to this successful
experience. The next two questions are the Values questions. Here, interviewees are asked to
describe what they value most about the topic of the inquiry as well as what value they bring or
add to the topic of the inquiry. The third question is called The Three Wishes question. Reflecting
on the story they just told, interviewees are asked, “If you had three wishes for having more
exceptional experiences like the one you just described, what would you wish for?”
So while some people might think they are going to have the opportunity to explain what is
wrong, whose fault it is, and what the problems are, the questions asked in these interviews steer
them in a very different direction. I will admit, there are times when a person has difficulty talking
about successful experiences. We are so conditioned to dissect a problem and to come up with
quick solutions that it may take some time to shift our mental gears. However, when someone
starts talking about what didn’t work and can’t seem to focus on the appreciative questions, the
rule of thumb is to let the person say what he/she wants to say and then to try again. It then usually
works. I think people see really quickly that they’re not going to get where they want to go if they
start talking about what didn’t work. It’s not going to be creative. It’s not going to be innovative.
It’s not going to give them excitement, joy, or hope for something to be different. At the same
time, there are people who have to tell you the negative things first. No matter how you try to
redirect, they have to tell you. So you let them, and then you say, “Okay, now let’s talk about
something that did work.”
Christie: Okay, let’s get back to the CEDT Big Day Meeting. Can you explain exactly what
you did and what happened?
Preskill: After some preliminary introductions by the vice president and the evaluation team,
we began the meeting by asking each person to pair up with one other person and to interview each
other for 10 minutes each. They were asked to consider the following:
Think of a time when you knew that a CEDT evaluation process was working well. You were
confident and excited that important and useful data were being collected, and you felt energized
about the evaluation process. What was happening? Who did it involve? What made this evaluation
process (or outcome) so successful? Why was it successful? What was your role? What value did
you add to this evaluation process?
After 20 minutes, the pairs were asked to join two other pairs to form a group of six. In this
group, they were to tell the highlights of their partner’s story (2 minutes each) and then to look for
themes across the group’s stories. As the themes emerged, they wrote them on flipchart paper. (An
easel with paper was placed at each table.) Each group was then asked to share their themes with
the larger group.
After a brief discussion about the similarities and differences between the groups (there were
many commonalities among the lists), the Imagine phase of the AI process was initiated.
Participants…
Purchase answer to see full
attachment