Bureau of Reclamation Banner
This information is intended to convey the underlying concepts for Reclamation's decision processes. It is not mandatory.
See the Reclamation Manual for official Reclamation-wide requirements.

Reclamation's Decision Process Guide

Step 7

Evaluate Alternatives

Purpose / Why / How / Tools / Look Forward / Go On


pretty border

We can now evaluate the full range of workable alternatives to see what will best solve the problem in this situation.


pretty border

navigate in the page--Purpose

Pitfall:pitfall

Preselecting a solution before you evaluate all alternatives is a big mistake.

  • To evaluate and refine alternatives
  • To find the optimal, "best," or most desirable alternative
  • To present a clear analysis*definition and comparison of alternatives
  • To document this analysis for the decisionmakers and affected publics

pretty border

navigate in the page--Why?

Frequently, unimplementable or extreme alternatives are evaluated along with workable alternatives to show why they won't work and to find some component or compromise that will work.

In our search for the most desirable alternative, an analysis of trade-offs among competing needs and solutions is essential. Multi-purpose programs and projects meeting more than one need must consider that optimizing the solution for individual needs may cause problems in other areas (e.g. the optimum economic alternative may create problems in the environmental area...and vice versa.). By simultaneously viewing the impacts in both areas, balance can be achieved. A compromise alternative can allow both interests to consent to a desirable resolution of problems which is stronger than any alternative addressing merely a single interest.

The evaluation process is not to justify what you are doing but to refine the alternatives. The "best" alternative cannot be developed in the first run. Rather, the alternative that best meets the needs and situation will evolve as a result of the evaluation process.


pretty border

navigate in the page--How?

At this point, no alternative is the "best" one. Not all needs can be met to everyone's satusfaction with one alternative. Evaluating alternatives involves tradeoffs and compromises. You will probably end up with very different alternatives from the ones you started with.

This is where most of the analytical work takes place. Disciplines work together to compare*definition alternatives through various analytical techniques. Through analysis, the information needed for a decision is generated. At this stage, professional judgment is replaced with factual data as much as possible.

These analyses will vary, depending on what the decisionmakers need to make a decision. Sometimes, e arlier work requires repeating at a higher level of detail; for instance, a quick windshield survey*definition may have been adequate for initial assessments, but detailed maps and samples may be needed to authoritatively evaluate and screen for higher levels of detail (accuracy).

Context

Communicate with potential implementors to ensure you evaluate the factors that actually count.

click for comics

(Use the right details)

Analyze the systems (e.g., physical, biological, social, economic, and organizational) to determine how alternatives will interact with other processes. Ask:

  • What is the setting in which the decision will be made?
  • How will the solution be implemented?
  • How will it be monitored and followed up?
  • How will it interact with other processes and actions?

Develop indicators for each significant issue to track and compare all alternatives consistently.

Evaluation Criteria

Criteria used to screen options were to ensure that options would work. Now evaluation criteria weigh workable alternatives to chose the one that best meets the needs.

Evaluation criteria reflect what drives the decision. The criteria applied here are based on a comparative analysis of alternatives to provide the information necessary to select the alternative. The basic question here is: What do I need to know about the alternatives to choose one?

Relative weights assigned to the importance of the evaluation criteria provide a basis for evaluating the relative merits of these . Weights are what tell you what is more important: 85-percent fish flows with some power generation or 100-percent fish flows precluding any other uses.

Carefully craft these criteria and weights*definition so that you can develop a rationale behind your judgment-- why one alternative outranks another. No matter what the alternative, no one will be totally satisfied. Tradeoffs among benefits and impacts will have to be made. Look carefully at that last incremental benefit to determine if it is worthwhile. For example, you may have to settle for 90-percent solution if the 100-percent solution costs four times as much.

Criteria may conflict--high flows may be needed for an endangered species while stable flows are necessary for riparian habitat. Determine and agree upon which criteria are more important (Is meeting need x more important than meeting need y? Is speed more important than cost? Is easy maintenance more important than comprehensiveness?). These priorities should more or less mirror the prority stack of objectives.

Identify Impacts

If you push down in one place, the problem may pop up somewhere else.

Once you have alternatives that fit the evaluation criteria, look at what effects they will have on the overall systems. It won't do any good to solve one problem if you are going to create larger problems elsewhere. For example, if providing a water supply to one area robs the water supply from another area, the overall problem isn't solved. Look at indirect effects as well by thinking through the process (if x happens, then y might occur, which would influence z). Examine the alternatives within the context of other actions to determine cumulative effects. One house on a mountain may not prove to be a problem, but many subdivisions on that mountain could be.

Determine Data Needs

If the decision can't be made without that particular data, then the data are significant,*definition

click for comics

(Looking for a great black box?)

Data needs will vary, depending on participants, solution requirements, and problemshed*definition interactions. To find out what you need, ask:

  • "Are these data significant to the decision?"
  • "Do we need these data to make an effective decision?"

If your answer is:

"Yes" then:

  • Identify the data needed
  • Identify and agree on indicators
  • Determine how the data will be analyzed to answer the relevant questions
  • Collect only essential data germane to the current level of detail

"No" then:

  • Don't waste time or money with it
  • Document and support your decision not to gather the data

Don't forget to get input from participants--and from interested public! If you miss something now, it will come haunt you later.

Determine what issues are significant and develop indicators to show impacts to these issues. Determine the level of detail needed for analyses by working backwards--figure out what level of detail is needed for an effective decision and for implementing the solution. Reaching agreements on evaluation techniques and weights of objectives and criteria is essential. Unresolved disagreements at this stage can drive a wedge into the process, creating "splinter groups" with their own methodology and even counter-analyses. These counter-analyses may be equally valid, but they will cloud the overall issues and focus attention and energy away from the real problem onto relatively unimportant side issues. For example, a counter-analysis may find that fish mortality is 14.9 percent, rather than 10.5 percent. This relatively small difference in analysis may not shed more light on the comparative merits of alternatives and may focus the conflict on numbers rather than solutions.

Determine Analysis Methods

Watch out for confusion around the words " baseline" or "existing" conditions*definition . These say "Here is what the situation is now ." You are analyzing what will happen : compare projected conditions under each alternative to the projected conditions under the no action alternative. This is sometimes referred to as "alternative future scenario comparison"*definition .

Agree upon a procedure to analyze tradeoffs, evaluation criteria, and impacts for each alternative. Ensure that each alternative is treated in the same manner. Determine methodology, schedule, and priority. Laying out methodologies and analyses now will save money later. It will help ensure that disciplines work together so that results can be compared and the necessary data for each analysis is collected. (Don't waste time finding out flow temperatures if the biologists need flow rates instead!) Also, schedules will show the interaction of analyses (e.g., the hydrology work may need to be done before the biology.)

Refine Alternatives and Re-iterate*definition Analysis

The first iteration will reduce the range of alternatives to those considered most reasonable (despite the ambiguity of the term) for further, more detailed study. Although there is no magic cutoff, costs and time usually limit alternatives carried through each further iteration to five or less. Now that you have been able to compare and evaluate alternatives and recognize tradeoffs, you can revisit and revise earlier steps in the process to reflect the diversity of input from all technical disciplines, publics, agencies, etc. Keep decisionmakers in the loop to refine alternatives and determine which ones to carry to the next iteration. A popular myth-conception*definition, that the set of alternatives is locked in place, may rear up here. Showing the benefits of refining the alternatives may help avoid this. However, if the decisionmakers have reasons to narrow the alternatives under consideration, this should be clearly documented. The second iteration will lead to an even more detailed study and comparison of the best alternatives. (Again, figure out the level of detail you really need.)

Document the Analysis

Showing the results of comparative analyses in a draft document gives the overall picture. This helps form agreements-- small compromises in content, wording, and presentation may bridge the gap in larger conflicts. The draft document can then be refined into a document for decisionmakers (e.g., a NEPA*definition compliance document, initiative, resource management plan, or proposed guidance).


pretty border

navigate in the page--Tools

Toolbox iconSee the toolbox for more tools.

A variety of analytical tools will be used at this point--both to gather data and display information on comparisons.

Get together with the core team and determine what tools will be used to measure what. Be sure that measurements, comparisons, and evaluations used are consistent assumptions and methods. While you are comparing apples to oranges, at least use the same properties and measurements to show how effective the alternatives will be to meet the objectives and evaluation criteria.

Evaluation tools include:

Tradeoff Analysis

A tradeoff analysis examines many components, factors, and criteria within the problem's context. This moves away from the simplistic emphasis on only one factor which a single discipline might apply.

In these analyses, factors (criteria, impacts, costs, etc.) are weighted to reflect their relative importance. Impacts on the factors are compared for each alternative to analyze benefits, costs, and tradeoffs. Participants can change weights or data to determine the significance of changes in criteria or priority stacks. Sensitivity analysis provides room for comparisons and enables participants to:

  • Set sideboards for minimum performance and point out fatal flaws (Even if you made the cost of the water as low as possible, you could not compensate for water quality below x standard.)
  • Determine what is most important (What are the deciding factors in a preferred alternative: wetlands, benefit-cost ratio, amount of water delivered, etc.?)
  • Build consent (Can participants agree on an 85-percent solution if the 100-percent solution takes twice as long?)
  • Look beyond analytical differences (What would really happen if juvenile salmon mortality rates at the gates were 20 percent rather than 5 percent?)

Consider software programs, such as MATS (multi-attribute tradeoff system). MATS is designed to help decisionmakers (or publics) make choices among alternatives when many pieces of information must be considered. MATS provides a framework for decision analysis and documentation, with content provided by the user. The MATS process helps reduce the complexity of developing alternatives by tracking all assumptions, factors, weights, alternative comparisons, and tradeoffs

Sensitivity to changes in facts or values is easily displayed for evaluation during the decisionmaking process. MATS can be used to encourage tradeoff discussion among publics and technical disciplines.
 

Matrices

Evaluating alternatives comes down to which altenatives offer what. Use matrix , rating , and ranking tables to quickly show how alternatives stack up against weighted evaluation factors. (For example, lining a canal may cost less than piping, but will need more repairs and maintenance.)

Pilot Studies

Consider testing the waters first with a pilot study, phased implementation*definition, or test market. For example, an administrative function might be tested within one group or division first. Participants in the test group can provide insights into what worked and why. Armed with this information, you can expand, modify, or drop the alternative. Draft documents can also serve to test ideas and analyses.

Interrelationships

Analyses that examine interrelationships, such as GIS mapping systems, can help determine:

  • How the components of alternatives would work together
  • How alternatives would affect existing or future related projects.

pretty border

navigate in the page--Look Forward

Based on what you know at this point, begin paving the way for selection and implementation. Most opposition or conflicts occur because people feel either that their concerns were ignored or the process is unreasonable. This stage presents many opportunities to head off these conflicts. Make sure that participants and especially potential implementors have a chance to review the analysis. If they are not involved now, it may be very difficult to persuade people to implement the selected alternative.

Recommending

The technical/partnership team can now recommend an alternative to the decisionmakers. If an alternative doesn't clearly stand out over the others, it may be necessary to select additional evaluation criteria and refine the evaluation process.

Present the results to the decisionmakers and the public in a final document. This usually concludes the activities of the technical team. Celebrate! Final documents:

  • Provide a reality check (Did participants hear and understand each other? Were the relevant factors, objectives, and tradeoffs evaluated?)
  • Provide something concrete for representatives to take back to their groups

pretty border

navigate in the page--Go On

Executive Summary Take this car on a fast tour and Process spiralling forward Tours:

Alternatives <-----> Select

pretty border

PreviousStep 6, Develop Alternatives

NextStep 8, Select

Please contact Deena Larsen 303-445-2584 with questions or comments on this material.