INTRODUCTION

 

Stuart Nagel

MKM-PSO-DSI Center and University of Illinois

 

Policy theory can be divided into conceptual theory, theory of knowing, causal theory, and normative theory.  They all have in common a high level of generality about public policy, higher than methods, process, or substance.¹

 

NOTE:

 

1.   For more on general aspects of policy theory, see Frohock, Fred (1979). Public Policy:  Scope and Logic. Englewood Cliffs: Prentice‑Hall.; Dunn, William, ed., (1982).Values, Ethics, and the Practice of Policy Analysis. Lexington: Lexington‑Heath.; Nagel, Stuart (1988). Policy Studies:  Integration and Evaluation. Westport: Greenwood Press.; and White, Louise (1994) "Values, Ethics and Standards in Policy Analysis," in Nagel, Stuart,  ed. Encyclopedia of Policy Studies. New York: Marcel Dekker.

 

 

CONCEPTUAL THEORY AND POLICY EVALUATION¹

 

Stuart Nagel

MKM-PSO-DSI Center and University of Illinois

 

I.  DEFINING POLICY ANALYSIS

 

Public policy analysis can be defined as determining which of various alternative public or governmental policies will most achieve a given set of goals in light of the relations between the policies and the goals.  That definition brings out the five key elements of policy evaluation:

 

1.         Goals, including normative constraints and relative weights for the goals.

 

2.         Policies, programs, projects, decisions, options, means, or other alternatives that are available for achieving the goals.

 

3.         Relations between the policies and the goals, including relations that are established by intuition, authority, statistics, observation, deduction, guesses, or by other means.

 

4.         Drawing a tentative conclusion as to which policy or combination of policies is best to adopt in light of the goals, policies, and relations.

 

5.         Determining what it would take to bring a second place or other place alternative up to first place.

 

            Other concepts that are often used to mean the same thing as public policy analysis include policy evaluation, policy studies, program evaluation, public management science, and policy science.  One could make distinctions between those concepts as follows:

 

1.                  Policy evaluation emphasizes evaluating alternative public policies, as contrasted to describing them or explaining why they exist.

 

2.                  Policy studies includes describing policies, explaining their existence, and evaluating them.

 

3.                  Program evaluation emphasizes evaluating a specific program like a halfway house in Chicago in 1984, as contrasted to developing general principles of how to evaluate.

 

4.                  Public management science emphasizes decision-making that is involved in implementing broader decisions, generally made by legislatures and agencies that have quasi-legislative authority.

 

5.                  Policy analysis emphasizes systematic analytic methods that can be quantitative or qualitative.

 

6.                  Policy science emphasizes quantitative methods.

 

Methods of public policy analysis refer to:

 

1.                  How to draw a conclusion as to which policy to adopt from information on goals, policies, and relations.

 

2.         How to establish the relations between policies and goals.

 

3.         How to determine what policies are available for adoption and what goals are appropriate to consider.

 

II.  DEFINING GOOD POLICY ANALYSIS

 

The key characteristics of good policy analysis relate to validity, importance, usefulness, originality, and feasibility.

 

A.  VALIDITY

 

Validity in general refers to being accurate.  In the context of policy evaluation research and its key elements, validity refers to:

 

1.                  The listed goals that include all the major goals and no non-goals of the relevant policy makers in this context.

 

2.                  The policies that encompass the total set of feasible alternatives and no infeasible alternatives.  Feasibility in this context refers to being capable of being adopted and implemented by the relevant policy makers and policy appliers.

 

3.                  The external consistency with empirical reality in describing the relations between the alternative policies and the goals.

 

4.                  The internal consistency of logically drawing a conclusion that follows from the goals, policies, and relations.

 

Validity can occur in degrees, or at least non-validity can.  A valid study meets all four of the above requirements.  An invalid study fails to meet at least one of the four requirements.  A study is even more invalid if it fails to meet more than one of the requirements, or fails to meet a requirement by a large margin.  Validity in policy evaluation is thus like equity, where equity is defined as providing a minimum benefits level for all relevant persons, groups, or places.  Thus, if that minimum level is met, equity is present although equality may not be present.  There can, however, be degrees of inequity depending on the size of the individual deviations from that minimum level and the number of people involved, as is the case with non-validity.

 

B.  IMPORTANCE

 

Importance can be defined in two ways:

 

1.                  Does the research deal with issues in which there are big societal benefits and/or big societal costs being analyzed?  Thus research on avoiding nuclear war is more important than research on whether a city should have a strong mayoral form of government or a city manager.

 

2.                  Does the research deal with a subject matter or a set of causal hypotheses that potentially have broad explanatory power?  This is theoretical importance, as contrasted to policy importance.

 

There can be degrees of policy importance depending on the product of the total benefits times the total costs that are at stake.  That benefits‑times‑costs criterion makes sense in judging the relative importance of two research projects, even though  benefits‑minus‑costs is the criterion for judging which of two investments or alternative policies to adopt.  In other words, if one research project deals with benefits of 100 units and costs of 150 units, and the second project deals with benefits of 10 units and costs of 8 units, then the first project is more important given the size of the benefits and the costs, even though the second project would be a better or more profitable investment. We would not want to add benefits to costs to judge importance since those variables are likely to be measured in different units that cannot be meaningfully added.  In this context, however, different units can be meaningfully multiplied.

 

C.  USEFULNESS

 

Usefulness should be considered a continuum concept rather than a yes/no concept.  Usefulness or utilization at its lowest level involves doing a policy research project that is not in any way referred to by the people who make policy in the subject matter area.  Nor is there any evidence that the policy makers were aware of the project, even though they did not explicitly cite it.  That is clearly non-utilization.  At the other extreme or the highest level of utilization, the research project converts the decision makers from being negative to being positive or vice versa on an issue.  That is a rare occurrence and may never occur for controversial issues.

 

Research that reinforces preconceived decisions is the next to highest level.  This is reasonably common.  Some skeptics of the value of systematic policy analysis consider this occurrence not to be utilization.  Policy researchers should, however, be quite pleased if their research accelerates a worthwhile decision that otherwise might not be made for a while.  An example might be the research that showed more defendants could be released prior to trial without increasing the no‑show rate, provided there is some systematic screening to determine who is to be released.  Liberals found support in such studies since they wanted to see more defendants released in view of the presumption of innocence.  Likewise, conservatives also found support in such studies since they wanted to see a reduction in the expensive jail burden on the taxpayers.  That kind of reinforcement did accelerate the presumably desirable occurrence of increased pre-trial release with a constant or lower no‑show rate.

 

At the next to the lowest level is research which is referred to by the policy makers orally or by being officially cited.  The citing could be by either the majority group among the decision makers or the minority/dissenters.  Either kind of citing is an example of low‑level utilization, even if the research cited was not on the winning side, and was not influential enough to convert decisions or even reinforce preconceived decisions.  This is also a common occurrence.  People who are skeptical of the value of policy research tend to emphasize how common the lowest categories are of no citing or citing without influence.  One should, however, recognize that progress in dealing with policy problems may require many unutilized research projects before a research project is developed that does get well-used.  The policy research system is still a success if only one in ten projects gets used, if that one tends to produce benefits that outweigh the costs of the other nine.

D.  OTHER CRITERIA OF GOOD POLICY ANALYSIS

 

In defining good policy evaluation, one should add as a criterion the concept of originality to the concepts of validity, social/theoretical importance, and usefulness.  Originality can be measured in degrees since all research differs to some extent from previous research unless an exact plagiarism is involved.  Even highly original research builds on and synthesizes prior research.  Feasibility is an additional criterion for judging proposed policy research, as contrasted to completed policy research.  Feasibility is concerned with how easily the research can be implemented given the limited time, expertise, interests, funds, and other resources of the researcher.

 

In defining good policy evaluation, we should also indicate certain criteria that should probably be considered irrelevant.  One such criterion is whether the recommendations of a research study have been adopted by policy makers, even though the policy makers show no awareness of the existence of the research.  To be considered useful, the research must at least be referred to.  A second irrelevant criterion is the direction of the research recommendations in terms of being liberal or conservative.  That criterion is not relevant to good policy analysis in an objective scientific sense, since there is generally no objectivity as to whether a liberal or a conservative recommendation is better.  One could argue that liberal recommendations tend to directly benefit more people given their mass orientation.  On the other hand, one could argue that conservative recommendations benefit more people in light of the trickle‑down theory, which says the masses are better off if well‑to‑do potential investors are encouraged to develop new technology and businesses.  The other criteria of validity, importance, and usefulness do have some objective reality on which both liberals and conservatives can agree.²

NOTES:

 

1.   This article is modified from Nagel, Stuart, ed. (1990). Policy Theory and Policy Evaluation:  Concepts, Knowledge, Causes, and Norms. New York: Greenwood.

 

2.   For more on conceptual theory in policy evaluation, see Gregg, Phillip, ed. (1976).Problems of Theory in Policy Analysis. Lexington: Lexington‑Heath.; Dunn, William, ed. (1986). Policy Analysis:  Perspectives, Concepts, and Methods. Greenwich:  JAI Press.; and Dror, Yehezkel (1994). "Basic Concepts in Policy Studies" in Nagel, Stuart, ed., Encyclopedia of Policy Studies. New York: Marcel Dekker, Inc., pp. 1-30.