Step 7

Evaluate Alternatives

Purpose / Why / How / Tools / Look Forward / Go On

We can now evaluate the full range of workable alternatives to see what will best solve the problem in this situation.

Purpose

• To evaluate and refine alternatives
• To find the optimal, "best," or most desirable alternative
• To present a clear analysis and comparison of alternatives
• To document this analysis for the decisionmakers and affected publics

Why?

In our search for the most desirable alternative, an analysis of trade-offs among competing needs and solutions is essential. Multi-purpose programs and projects meeting more than one need must consider that optimizing the solution for individual needs may cause problems in other areas (e.g. the optimum economic alternative may create problems in the environmental area...and vice versa.). By simultaneously viewing the impacts in both areas, balance can be achieved. A compromise alternative can allow both interests to consent to a desirable resolution of problems which is stronger than any alternative addressing merely a single interest.

The evaluation process is not to justify what you are doing but to refine the alternatives. The "best" alternative cannot be developed in the first run. Rather, the alternative that best meets the needs and situation will evolve as a result of the evaluation process.

How?

This is where most of the analytical work takes place. Disciplines work together to compare alternatives through various analytical techniques. Through analysis, the information needed for a decision is generated. At this stage, professional judgment is replaced with factual data as much as possible.

These analyses will vary, depending on what the decisionmakers need to make a decision. Sometimes, e arlier work requires repeating at a higher level of detail; for instance, a quick windshield survey may have been adequate for initial assessments, but detailed maps and samples may be needed to authoritatively evaluate and screen for higher levels of detail (accuracy).

Context

Analyze the systems (e.g., physical, biological, social, economic, and organizational) to determine how alternatives will interact with other processes. Ask:

• What is the setting in which the decision will be made?
• How will the solution be implemented?
• How will it be monitored and followed up?
• How will it interact with other processes and actions?

Develop indicators for each significant issue to track and compare all alternatives consistently.

Evaluation Criteria

Evaluation criteria reflect what drives the decision. The criteria applied here are based on a comparative analysis of alternatives to provide the information necessary to select the alternative. The basic question here is: What do I need to know about the alternatives to choose one?

Relative weights assigned to the importance of the evaluation criteria provide a basis for evaluating the relative merits of these . Weights are what tell you what is more important: 85-percent fish flows with some power generation or 100-percent fish flows precluding any other uses.

Carefully craft these criteria and weights so that you can develop a rationale behind your judgment-- why one alternative outranks another. No matter what the alternative, no one will be totally satisfied. Tradeoffs among benefits and impacts will have to be made. Look carefully at that last incremental benefit to determine if it is worthwhile. For example, you may have to settle for 90-percent solution if the 100-percent solution costs four times as much.

Criteria may conflict--high flows may be needed for an endangered species while stable flows are necessary for riparian habitat. Determine and agree upon which criteria are more important (Is meeting need x more important than meeting need y? Is speed more important than cost? Is easy maintenance more important than comprehensiveness?). These priorities should more or less mirror the prority stack of objectives.

Identify Impacts

Once you have alternatives that fit the evaluation criteria, look at what effects they will have on the overall systems. It won't do any good to solve one problem if you are going to create larger problems elsewhere. For example, if providing a water supply to one area robs the water supply from another area, the overall problem isn't solved. Look at indirect effects as well by thinking through the process (if x happens, then y might occur, which would influence z). Examine the alternatives within the context of other actions to determine cumulative effects. One house on a mountain may not prove to be a problem, but many subdivisions on that mountain could be.

Determine Data Needs

Data needs will vary, depending on participants, solution requirements, and problemshed interactions. To find out what you need, ask:

• "Are these data significant to the decision?"
• "Do we need these data to make an effective decision?"

If your answer is:

"Yes" then:

• Identify the data needed
• Identify and agree on indicators
• Determine how the data will be analyzed to answer the relevant questions
• Collect only essential data germane to the current level of detail

"No" then:

• Don't waste time or money with it
• Document and support your decision not to gather the data

Don't forget to get input from participants--and from interested public! If you miss something now, it will come haunt you later.

Determine what issues are significant and develop indicators to show impacts to these issues. Determine the level of detail needed for analyses by working backwards--figure out what level of detail is needed for an effective decision and for implementing the solution. Reaching agreements on evaluation techniques and weights of objectives and criteria is essential. Unresolved disagreements at this stage can drive a wedge into the process, creating "splinter groups" with their own methodology and even counter-analyses. These counter-analyses may be equally valid, but they will cloud the overall issues and focus attention and energy away from the real problem onto relatively unimportant side issues. For example, a counter-analysis may find that fish mortality is 14.9 percent, rather than 10.5 percent. This relatively small difference in analysis may not shed more light on the comparative merits of alternatives and may focus the conflict on numbers rather than solutions.

Determine Analysis Methods

Agree upon a procedure to analyze tradeoffs, evaluation criteria, and impacts for each alternative. Ensure that each alternative is treated in the same manner. Determine methodology, schedule, and priority. Laying out methodologies and analyses now will save money later. It will help ensure that disciplines work together so that results can be compared and the necessary data for each analysis is collected. (Don't waste time finding out flow temperatures if the biologists need flow rates instead!) Also, schedules will show the interaction of analyses (e.g., the hydrology work may need to be done before the biology.)

Refine Alternatives and Re-iterate Analysis

The first iteration will reduce the range of alternatives to those considered most reasonable (despite the ambiguity of the term) for further, more detailed study. Although there is no magic cutoff, costs and time usually limit alternatives carried through each further iteration to five or less. Now that you have been able to compare and evaluate alternatives and recognize tradeoffs, you can revisit and revise earlier steps in the process to reflect the diversity of input from all technical disciplines, publics, agencies, etc. Keep decisionmakers in the loop to refine alternatives and determine which ones to carry to the next iteration. A popular myth-conception, that the set of alternatives is locked in place, may rear up here. Showing the benefits of refining the alternatives may help avoid this. However, if the decisionmakers have reasons to narrow the alternatives under consideration, this should be clearly documented. The second iteration will lead to an even more detailed study and comparison of the best alternatives. (Again, figure out the level of detail you really need.)

Document the Analysis

Showing the results of comparative analyses in a draft document gives the overall picture. This helps form agreements-- small compromises in content, wording, and presentation may bridge the gap in larger conflicts. The draft document can then be refined into a document for decisionmakers (e.g., a NEPA compliance document, initiative, resource management plan, or proposed guidance).

Tools

A variety of analytical tools will be used at this point--both to gather data and display information on comparisons.

Get together with the core team and determine what tools will be used to measure what. Be sure that measurements, comparisons, and evaluations used are consistent assumptions and methods. While you are comparing apples to oranges, at least use the same properties and measurements to show how effective the alternatives will be to meet the objectives and evaluation criteria.

Evaluation tools include:

• Action review
• Bar charts
• Constraints table
• Devil's advocate
• Fishbones
• Force field analysis
• Frequency charts
• Issue map
• Matrix diagram
• Matrix table
• Multi-attribute trading system (MATS)
• Pare it down
• Pro/con table
• Ranking techniques
• Rating tables
• Trading cards
• Value engineering

Tradeoff Analysis

A tradeoff analysis examines many components, factors, and criteria within the problem's context. This moves away from the simplistic emphasis on only one factor which a single discipline might apply.

In these analyses, factors (criteria, impacts, costs, etc.) are weighted to reflect their relative importance. Impacts on the factors are compared for each alternative to analyze benefits, costs, and tradeoffs. Participants can change weights or data to determine the significance of changes in criteria or priority stacks. Sensitivity analysis provides room for comparisons and enables participants to:

• Set sideboards for minimum performance and point out fatal flaws (Even if you made the cost of the water as low as possible, you could not compensate for water quality below x standard.)
• Determine what is most important (What are the deciding factors in a preferred alternative: wetlands, benefit-cost ratio, amount of water delivered, etc.?)
• Build consent (Can participants agree on an 85-percent solution if the 100-percent solution takes twice as long?)
• Look beyond analytical differences (What would really happen if juvenile salmon mortality rates at the gates were 20 percent rather than 5 percent?)

Consider software programs, such as MATS (multi-attribute tradeoff system). MATS is designed to help decisionmakers (or publics) make choices among alternatives when many pieces of information must be considered. MATS provides a framework for decision analysis and documentation, with content provided by the user. The MATS process helps reduce the complexity of developing alternatives by tracking all assumptions, factors, weights, alternative comparisons, and tradeoffs

Sensitivity to changes in facts or values is easily displayed for evaluation during the decisionmaking process. MATS can be used to encourage tradeoff discussion among publics and technical disciplines.
 

Matrices

Evaluating alternatives comes down to which altenatives offer what. Use matrix , rating , and ranking tables to quickly show how alternatives stack up against weighted evaluation factors. (For example, lining a canal may cost less than piping, but will need more repairs and maintenance.)

Pilot Studies

Consider testing the waters first with a pilot study, phased implementation, or test market. For example, an administrative function might be tested within one group or division first. Participants in the test group can provide insights into what worked and why. Armed with this information, you can expand, modify, or drop the alternative. Draft documents can also serve to test ideas and analyses.

Interrelationships

Analyses that examine interrelationships, such as GIS mapping systems, can help determine:

• How the components of alternatives would work together
• How alternatives would affect existing or future related projects.

Look Forward

Based on what you know at this point, begin paving the way for selection and implementation. Most opposition or conflicts occur because people feel either that their concerns were ignored or the process is unreasonable. This stage presents many opportunities to head off these conflicts. Make sure that participants and especially potential implementors have a chance to review the analysis. If they are not involved now, it may be very difficult to persuade people to implement the selected alternative.

Recommending

The technical/partnership team can now recommend an alternative to the decisionmakers. If an alternative doesn't clearly stand out over the others, it may be necessary to select additional evaluation criteria and refine the evaluation process.

Present the results to the decisionmakers and the public in a final document. This usually concludes the activities of the technical team. Celebrate! Final documents:

• Provide a reality check (Did participants hear and understand each other? Were the relevant factors, objectives, and tradeoffs evaluated?)
• Provide something concrete for representatives to take back to their groups

Go On

Executive Summary and Process Tours:

Alternatives <-----> Select