Arizona State University Interview with Gary Henry Article Review Chapter 5 Case Study Instructions:Select one of the following topics and then prepare a t

Arizona State University Interview with Gary Henry Article Review Chapter 5 Case Study Instructions:Select one of the following topics and then prepare a typed report using the guidelines below: Topics: CHOSEN TOPIC —– 2.If you did not critique the interview with Gary Henry in Chapter 7, you may critique his interview. Guidelines: Critique their presentation of information from an expertise-oriented and from a consumer-oriented approach.What information is helpful?Would other information be helpful to you in making a decision?If so, what?Does that information relate to a different criterion or standard you have?How does the information fit into the approaches reviewed in Chapter 5? Use the Case Study Form to submit your assignment. CHAPTER 5 Alternative Approaches to Program Evaluation
TEXTBOOK
Worthen, B.R., Sanders, J.R., & Fitzpatrick, J.L. (2011). Program evaluation, 4th edition, New Jersey:
Pearson Education, Inc.
PAGE 126
Everyone evaluates. As we discussed in Chapter 1, we all form opinions or make judgments about the
quality of things we encounter. Such evaluations include everything from the meal we just finished
eating or the movie or con- cert we saw last week to more serious endeavors—the program to help
students at risk of dropping out at our high school or the parent contact program for par- ents new to
our school. Our focus here is not on our individual judgments of something, but on evaluations that are
more formal, structured, and public. We connect these personal evaluations with the more formal ones
here, though, be- cause the earliest evaluation approaches were concerned, almost exclusively, with
judging the quality of something. Those judgments were often derived by
PAGE 127
a group of individuals coming together to consider their criteria and the program or product to be
judged.
The first modern-day approaches to evaluation were expertise-oriented and consumer-oriented
evaluations. These approaches continue to be used today, though not so widely in the professional
evaluation field. However, they have influenced the ways we think of evaluation and its purposes and
methods.
We will review each briefly, with a focus on the most widely used current method— accreditation—to
illustrate the key principles of these approaches and how they affected, and continue to affect,
evaluation practices.
The Expertise-Oriented Approach The expertise-oriented approach to evaluation is probably the oldest
type of formal, public evaluation and, as its name implies, it relies primarily on professional expertise to
judge the quality of an institution, program, product, or activity. For example, the merits of a leadership
training program for school principals could be assessed by experts from various fields including
leadership, educational adminis- tration, and training who would observe the program in action,
examine its mate- rials and underlying theory, perhaps interview some trainers and participants, or, in
other ways, glean sufficient information to render a considered judgment about its value.
In another case, the quality of a hospital could be assessed by looking at its spe – cial programs, its
operating facilities, its emergency room operations, its in-patient operations, its pharmacy, and so on,
by experts in medicine, health services, and hospital administration. They could examine facilities and
equipment/supplies of the hospital, its operational procedures on paper and in action, data on the fre quency and outcomes of different procedures, the qualifications of its personnel, patient records, and
other aspects of the hospital to determine whether it is meeting appropriate professional s tandards.
Although professional judgments are involved to some degree in all evalua- tion approaches, this one is
decidedly different from others because of its direct, open reliance on professional expertise as the
primary evaluation strategy. Such expertise may be provided by an evaluator or by subject-matter
experts, depend- ing on who might offer most in the substance or procedures being evaluated. Usually
one person will not own all of the requisite knowledge needed to adequately evaluate the program,
institution, or agency. A team of experts who complement each other are much more likely to produce a
sound evaluation. Several specific evaluation processes are variants of this approach, including doctoral
examinations administered by a committee, proposal review panels, site visits and conclusions drawn by
professional accreditation associations, reviews of institutions or individuals by state licensing agencies,
reviews of staff performance for decisions concerning promotion or tenure, peer reviews of a rticles
submitted to professional journals, site visits of educational programs conducted at the behest of the
program’s sponsor, reviews and recommendations by prestigious blue -ribbon
PAGE 128
panels, and even the critique offered by the ubiquitous expert who serves in a watchdog role. To impose
some order on the variety of expertise-oriented evaluation activ- ities, we have organized and will
discuss these manifestations in four categories: (1) formal professional review systems, (2) informal
professional review systems, (3) ad hoc panel reviews, and (4) ad hoc individual reviews. Differences in
these categories are shown in Table 5.1, along the following dimensions:
1. Is there an existing structure for conducting the review?
2. Are published or explicit standards used as part of the review?
3. Are reviews scheduled at specified intervals?
4. Does the review include opinions of multiple experts?
5. Do results of the review have an impact on the status of whatever is being evaluated?
Developers of the Expertise-Oriented Evaluation Approach and Their Contributions It is difficult to
pinpoint the origins of this approach, since it has been with us for a very long time. It was formally used
in education in the 1800s, when schools be- gan to standardize college entrance requirements.
Informally, it has been in use since the first time an individual to whom expertise was publicly accorded
ren- dered a judgment about the quality of some endeavor—and history is mute on when that occurred.
Several movements and individuals have given impetus to the various types of expertise -oriented
evaluations. Elliot Eisner, an early evaluator discussed later in this chapter, stressed the role of
connoisseurship and criticism in evaluation, roles that required exper- tise in the subject matter to be
evaluated. James Madison and Alexander Hamilton took on the role of “expert evaluators” in discussing
and elaborating on the meaning and merits of the newly proposed Constitution in The Federalist Papers.
(They were experts because they were both present and active at the Constitutional Convention that
drafted the document. As such, they were also
PAGE 129
internal evaluators!) Their writings were influential at the time and are still used by jurists in the U.S.
courts to interpret the meanings of the Constitution, illustrating the important actions that can come
from reasoned judgments by experts about a product. Accreditation of institutions of higher education
is the primary present-day application of expertise-oriented evaluations. The New England Association
of Schools and Colleges, which granted the first accredita- tion and continues accreditations for colleges
and universities in New England today, began in 1885 when a group of headmasters of preparatory
secondary schools began meeting with presidents of colleges in New England to discuss what graduates
should know to be prepared for college. Thus, more than 100 years ago, school and college leaders were
talking about ways to align their curricula!
Formal Professional Review Systems: Accreditation Historical Foundations. To many, the most familiar
formal professional review system is that of accreditation, the process whereby an organization grants
approval of institutions such as schools, universities, and hospitals. Beginning in the late 1800s, regional
accreditation agencies in the United States gradually supplanted the borrowed western European
system of school inspections. These agencies became a potent force in accrediting institutions of higher
education during the 1930s. Education was not alone in institutionalizing accreditation processes to
determine and regulate the quality of its institutions. Parallel efforts were under way in other
professions, including medicine and law, as concern over quality led to wide -scale acceptance of
professionals judging the efforts of those educating fellow professionals. Perhaps the most memorable
example is Flexner’s (1910) examination of medical schools in the United States and Canada in the early
1900s, which led to the closing of numerous schools he cited as inferior. As Floden (1983) has noted,
Flexner’s study was not accreditation in the strict sense, because medical schools did not participate
voluntarily, but it certainly qualified as accreditation in the broader sense: a classic example of pri- vate
judgment evaluating educational institutions. Flexner’s approach differed from most contemporary
accreditation efforts in two other significant ways. First, Flexner was not a member of the profession
whose efforts he presumed to judge. An educator with no pretense of medical ex- pertise, Flexner
nonetheless ventured to judge the quality of medical training in two nations. He argued that common
sense was perhaps the most relevant form of expertise: Time and time again it has been shown that an
unfettered lay mind, is . . . best suited to undertake a general survey. . . . The expert has his place, to be
sure; but if I were asked to suggest the most promising way to study legal education, I should seek a
layman, not a professor of law; or for the sound way to investigate teacher training, the last person I
should think of employing would be a professor of education. (Flexner, 1960, p. 71)
PAGE 130
It should be noted that Flexner’s point was only partially supported by his own study. Although he was a
layman in terms of medicine, he was an educator, and his judgments were directed at medical education
rather than the practice of medicine, so even here appropriate expertise seemed to be applied. Second,
Flexner made no attempt to claim empirical support for the criteria or process he employed, because he
insisted that the standards he used were the “obvious” indicators of school quality and needed no such
support. His methods of collecting information and reaching judgments were simple and
straightforward: “A stroll through the laboratories disclosed the presence or absence of apparatus,
museum specimens, library, and students; and a whiff told the inside story regarding the manner in
which anatomy was cultivated” (p. 79). Third, Flexner dispensed with the professional niceties and
courteous criti- cisms that often occur in even the negative findings of today’s accreditation processes.
Excerpts of his report of one school included scathing indictments such as this: “Its so-called equipment
is dirty and disorderly beyond description. Its outfit in anatomy consists of a small box of bones and the
dried-up, filthy frag- ments of a single cadaver. A cold and rusty incubator, a single microscope, . . . and
no access to the County Hospital. The school is a disgrace to the state whose laws permit its existence”
(Flexner, 1910, p. 190). Although an excellent example of expertise-oriented evaluation (if expertise as
an educator, not a physician, is the touchstone), Flexner’s approach is much like that of con temporary
evaluators who see judgment as the sine qua non of evalu- ation and who see many of the criteria as
obvious extensions of logic and common sense (e.g., Scriven, 1973). Accreditation in Higher Education
Today.
Accreditation in Higher Education Today. Accreditation in the United States and in many other countries
today meets our criteria for an expertise-oriented, formal review system. The systems make use of an
existing structure (generally an inde- pendent regional or national accreditation organization in the
United States or governmental agencies in other countries), standards published by the organiza- tion
responsible for accreditation, a specified schedule (for example, reviews of institutions every 2, 5, or 10
years), and opinions of multiple experts, and the status of the institution, department, college, or school
is affected by the results. Accreditation is an excellent example of expertise -oriented evaluation because
it uses people with expertise in the subject matter of the program or institution to form a judgment
regarding the quality of the entity to be evaluated. The accredi- tation of an institution or program
provides consumers and other stakeholders with some indication of the quality of the institution, as
judged by experts in the field, and may facilitate summative decisions. For example, many students use
an institution’s or program’s accreditation status to aid their decisions about whether to apply to or
attend an institution or program. Further, the feedback the accred- itation process provides to the
institution can be used for program and institutional improvement and decision making. Thus, the
accreditation process serves a formative purpose as well.
PAGE 131
Accreditation in the United States is most common for institutions of higher education.1 We will spend a
little time describing this process because it has recently become quite political and controversial, and
even for those readers not involved in accreditation, the arguments illustrate the types of political issues
and choices that often arise in any evaluation. These include disagreements over the purpose of the
evaluation (formative or summative); the neutrality and inde- pendence of the experts or evaluators;
the criteria to be used to judge the product and, thus, the data to be collected or reviewed; and the
transparency of the process (what should be available to the public or other stakeholders outside the
organi- zation). These controversies have emerged as the U.S. Department of Education, which has a
stake in accreditation through provision of student loans to accred- ited institutions, has begun to take
issue with the accreditation practices of the independent regional accrediting bodies that have
traditionally reviewed colleges and universities for accreditation. As noted earlier, in many countries,
including Germany, the Netherlands, India, and the countries of the United Kingdom, institutions of
higher education are required by law to be accredited. Government agencies, generally through a
ministry or department of education, conduct the accreditation process. In some countries, such as
Canada, there is no accreditation process for higher education, partly because most institutions of
higher education are run by the provincial gov- ernments and that governance is considered to provide
sufficient oversight. In the United States, accreditation evolved in a way that very much mirrors U.S.
citizens’ distrust of government. With a desire to minimize government’s role nonprofit or voluntary
associations carry out the accreditation tasks often fulfilled by government agencies in other countries.
PAGE 131
As noted earlier, the New England Association of Schools and Colleges was the first accreditation
organization in the United States. Originally established as a mechanism for dialogue between
administrators of secondary schools and leaders of colleges in the region in 1885, it eventually evolved
into the accrediting association for colleges and institutions in the region (Brittingham, 2009). Other
regional associations followed, with each taking responsibility for accrediting institutions of higher
education in their region. Today, there are six regional accrediting organiza- tions in the United States,
each pursuing similar activities within their region.2 These associations focus primarily on accrediting
institutions of higher education, though often they are also involved in accrediting K–12 schools. Finally,
there are many ac- crediting associations that review programs in particular disciplines rather than entire institutions. For example, the American Bar Association accredits law schools, the Association of
American Medical Colleges accredits medical schools, and the National Council for Accreditation of
Teacher Education (NCATE) accredits teacher education programs, with the Teacher Education
Accreditation Council (TEAC) emerging as a recent competitor to NCATE. Accreditation of institutions of
higher education by the six regional associa- tions has followed a similar plan and approach, the missionbased approach, since the 1950s. With the mission-based approach, accreditors focus on the extent to
which the institution is pursuing and achieving its stated mission. Although each association also has
standards for higher education that it uses in the evaluation, the mission-based approach reflects the
philosophy of the associations in its eval- uations. Barbara Brittingham describes the mission-based
approach and the accreditation process in the United States as “unusually focused on the future” to help
the institution improve (2009, p. 18).
PAGE 132
The Process of Accreditation. In the first stage of accreditation, the institution prepares a self-study
report describing its mission and its progress toward that mis- sion, as well as how the institution meets
the standards of the accrediting body. The second major stage is the core of the expertise -oriented
approach: a team of peers, faculty, and administrators from other institutions in the region receives the
report and conducts a site visit during which they interview faculty, administra- tors, staff, and students;
review institutional records on admissions, course curric- ula, student satisfaction and outcomes;
observe facilities and classrooms, and so forth. Based on their review of the report and their experiences
during the site visit, the team, usually three or four experts, writes a report expressing their views
regarding the institution, their recommendations concerning its accreditation sta- tus, and their
suggestions for improvement. The site visit report is then reviewed by a standing commission at the
accrediting association, which may amend the conclusions. The commission then presents the final
conclusions to the institution.
The process is expertise-oriented in several ways: (a) the association has exper- tise concerning
standards for higher education, the state and status of other institu- tions, and the practice of
accreditation and review; (b) the faculty and administrators who form the site team have expertise in
participating in the governance of their own universities and others where they have been employed
and receive some training from the association to serve as site reviewers. Therefore, the expertise of the
site visit team and the association allows those involved to make use of the standards of the association,
their review of the report, and their site visit to form a final judgment of the quality of the institution.
This process is a common one followed not only by the regional accrediting organizations but also by the
organizations that accredit programs in individual disciplines in higher education and by organizations
that accredit other educational institutions, including school districts, private schools, charter schools,
secondary schools, vocational schools, and religious schools.
PAGE 133
Accreditation Controversies: Accreditation Politicized. So what can be contro- versial here? As one
author defending the system notes, “Who better, one might ask, to evaluate the quality of a college or
university than those who work in the field?” (O’Brien, 2009, p. 2). O’Brien argues that the evaluation
and the relation- ship between the accrediting organizations and the institution should not be
adversarial, noting, “The evaluators are not inspectors coming in with their white gloves” (O’Brien, 2009,
p. 2). But the history of the controversy traces back to the GI Bill passed by Congress after World War II
to provide financial assistance to returning soldiers to attend colleges and universities. The government
wanted to ensure that the financial assistance went for worthwhile post secondary educa- tional
activities, but did not want to get directly into the business of examining col- leges and universities for
quality. So, it decided to rely on the independent regional accrediting associations, which were already
reviewing colleges and universities, to determine the institutions students could receive financia…
Purchase answer to see full
attachment