THEORY OF KNOWING AND POLICY EVALUATION

 

Stuart Nagel

MKM-PSO-DSI Center and University of Illinois

 

I.  SOURCES OF POLICY ANALYSIS ELEMENTS

 

Where do goals, policies, and relations come from?  The answer includes four main possibilities:

 

1.         Authority: One or more persons, books, articles, or other reliable sources of information regarding the relevant goals, policies, or relations.

 

2.         Statistical or observational analysis: The analyzing of specific instances in order to generalize what the goals, policies, or relations might be.

 

3.         Deduction: The drawing of a conclusion from premises that have been established from authority, observation, and/or intuition.

 

4.         Sensitivity analysis: The guessing of the goals, policies, or relations, and the determination of what effect, if any, the guessed values have on the final decision regarding which policy is best.

 

The four basic sources can be subclassified in various ways.  For example, authority can be meaningfully discussed in terms of expert authority and general public opinion.  Authority could also be contemporary or historical.  Observation can be impressionistic or systematic, including statistical.  Deductive approaches can be based on intuitively accepted or empirically validated premises.  Sensitivity analysis is threshold analysis in which we want to know the break‑even point, above which we should take one course of action, and below which we should take another.

 

What constitutes an authority on goals, policies, or relations?  The answer depends on the subject matter.  The Supreme Court is an authority, for example, on what goals are legitimate in satisfying the right‑to‑counsel clause of the Sixth Amendment to the Constitution.  The Court has said that saving money is not an appropriate goal, but that saving innocent persons from being convicted is.  If, however, the issue is not whether right-to-counsel should be provided but rather how it should be provided, then saving money is an appropriate goal.  For this issue, the goals of a county board would be relevant because it generally appropriates money to pay court‑appointed lawyers to represent the poor.  Such goals might include satisfying the local bar while minimizing expenditures.  The board might, therefore, decide on a salaried public defender system, rather than on a less expensive but less politically feasible assigned counsel system or a less legally feasible volunteer system.  For other policy problems, the key authorities might be legislative opinion, public opinion, the head of an administrative agency, or the like.

 

Accounting is a variation on statistical analysis.  Like statistical analysis, it involves aggregating data, but accounting data is generally more precise than statistical analysis that is based on averages or the fitting of curves to scattered data points.  A public opinion survey is not a variation on statistical analysis in the context of the typology of sources.  Rather, it is a form of consulting authority in which the authority is the general public or a special segment of it.  A statistical analysis (as a distinct source of information on goals, policies, or relations) involves a cross‑tabulation, an analysis of the variation between averages, or a regression‑equation analysis.  These forms of statistical analysis involve determining a relation that is relevant to weighting goals, deciding which policies are feasible to choose among, or relating a policy to a goal.

 

Deduction involves arriving at a conclusion from premises that have been established by way of authority, empirical validation, prior deduction, or intuition.  The more acceptable the premises are, the more acceptable the conclusions should be, assuming the conclusions have been validly deduced from the premises.  Deduction is especially helpful where there is no authority and no empirical data for determining the information desired.

 

In policy evaluation, sensitivity analysis is a useful source of information about goals, policies, and relations when authority, statistics, and deduction do not provide clear answers regarding them.  Sensitivity or threshold analysis enables one to determine how much room for error there is in weighting the goals, listing out the policies, or measuring the relations.  Often, the controversy over precision in these matters is wasted because, within the range in which the controversy occurs, the overall conclusion as to which policy or combination is best is still the same.  Sensitivity analysis also enables the policy evaluator to convert difficult questions about goals, policies, and relations into relatively easy questions, such as “Is a given weight, policy, or relation above or below some threshold?” rather than, “What is the exact weight, policy, or relation?”

 

There is no need to argue over which source between authority, statistics, and deduction is the most desirable.  Authority is clearly a big time‑saver if an accessible and respected authority is involved.  Deduction enables one to draw conclusions about goals, policies, and relations without having to gather original data but instead of synthesizing already known information.  Statistical analysis does constitute a more ultimate, but more difficult, form of proof.  In any concrete policy evaluation situation, the best source depends on the subject matter and what is to be done with it.  If the policy evaluation involves constitutional policy, an appeal to Supreme Court authority may be most relevant.  If it involves the effects of a strike in the coal industry on another segment of the economy, a deductive input‑output model may be the preferable type of analysis.  If it concerns the trade‑off problem of inflation and unemployment, a time‑series statistical analysis may be especially appropriate in relating inflation and unemployment to suicide rates, to the percentage of the two‑party vote that goes to the incumbent party, or to other social indicators.

 

II.  OVERCOMING OBSTACLES TO POLICY ANALYSIS KNOWLEDGE

 

There are five key methodological problems in decision-making:

 

1.                  Multiple dimensions on multiple goals.  This is the "apples and oranges" problem.

 

2.                  Multiple missing information.

 

3.                  Multiple alternatives that are too many for us to be able to determine the effects of each one.

 

4.                  Multiple and possibly conflicting constraints.

 

5.                  The need for simplicity in drawing and presenting conclusions in spite of all that multiplicity.

 

Decision‑making problems often involve multiple goals measured on a variety of different dimensions, such as miles, hours, dollars, 1‑5 attitude scales, yes‑no dichotomies, and so on.  Some of the ways in which multiple dimensions are handled are to (1) multiply the apples by two if you like each apple twice as much as each orange; then everything will be expressed in orange units; (2) ask whether the gain in apples from choosing one alternative is worth more or less than the gain in oranges from choosing a second alternative; or (3) convert the apple units into percentages by dividing the raw scores on the apples goal by the sum of the apples, and convert the orange units into percentages by dividing the raw scores on the oranges goal by the sum of the oranges.

 

We often do not know relation scores of each alternative or each goal, and we often do not know the relative weights of the goals.  The key way in which missinginformation is handled is to allow the user to quickly and accurately determine the effects of inserting various values for the missing information.  More specific techniques include:

 

1.         "What if” analysis, whereby the computer shows what would happen if we made changes in the goals, alternatives, and/or relations.

 

2.         Threshold analysis, whereby the computer shows for each relation score and goal weight the value that would cause a tie between the second‑place alternative and the first‑place alternative.

 

3.         Convergence analysis, whereby the computer shows for each goal weight at what magnitude the goal tends to dominate the other goals, such that nothing is to be gained by increasing the weight.

 

4.         Best‑worst analysis, whereby the computer shows first what the conclusion would be using values that most favor a given alternative, and then the values that least favor a given alternative.  The two conclusions are then averaged.

 

Decision‑aiding software can help in allocating resources, as contrasted to the generally easier problem of just finding a best alternative or combination.  A good way of allocating resources is to convert into percentages the raw merit scores of the objects to which the resources are to be allocated.  One can then apply the percentages to the grand total available to be allocated.  A good way to convert the raw scores into percentages is by dividing them by their total within the same goal in order to get part/whole percentages.  Those percentages can then be summed across the goals, using a weighted sum in which the goals have different weights.

 

Decision‑aiding software can help in dealing with constraints that require minimums or maximums on the alternatives or the goals or other conditions that must be met, regardless of how high the scores are on an alternative or on the goals.  The constraints can be met before one allocates scarce resources or determines the relation scores.  Doing so tends to result in giving an alternative more than it is entitled to when it deserves only the minimum.  That result cannot occur if adjustments are made after allocating, so as to bring alternatives up to their minimums.  The best ways of resolving conflicting constraints are either to expand the total benefits available or to reduce the total costs to be imposed so that all the constraints can be satisfied simultaneously.  If that is not possible, then one can resolve conflicting constraints by developing compromises that satisfy each constraint in proportion to its importance.  Other, less desirable, alternatives involve partially satisfying all constraints equally, or fully satisfying certain constraints in the order of their priority.

 

Decision‑aiding software that is based on multi‑criteria decision‑making (MCDM) can greatly simplify the analysis of a variety of decision‑aiding problems that have traditionally used more complicated and often less valid methods, such as arrow diagrams, payoff matrices, decision trees, optimum level curves, indifference curves, functional curves, and multi-objective programming.  The essence of MCDM software is that it works with a table, matrix, or spreadsheet with alternatives on the rows, evaluative criteria on the columns, relation scores in the cells, and a summation column at the right showing the overall score or allocation percentage of each alternative.¹

 

NOTE:

1.   For more on the theory of knowing in policy evaluation, see Dunn, William (1992). Public Policy Analysis. Englewood Cliffs: Prentice-Hall.; Nagel, Stuart (1989). Evaluation Analysis with Microcomputers. Greenwich: JAI Press.; and Rabin, Jack and Edward Jackowski, eds. (1988). Handbook of Information Resources Management. New York: Marcel Dekker.