You are here

Multi-criteria Analysis

Rating items on multiple criteria

To assess a list of items on multiple criteria, Facilitators have the choice of letting participants apply the different criteria

  1. sequentially, i.e. with one rating sheet per criterion, or
  2. in one 'shot', i.e. by setting up a Multi-criteria rating sheet. 

Sequential rating combines the benefits of greater discriminatory power with lower cognitive load: While working through the first Rating sheet, participants only need to think of that criterion, e.g. 'Importance', then turn to the next rating sheet and apply that criterion, e.g. 'Urgency' to all items, and so on for further criteria. Sequential rating is recommended in most circumstances.

One-shot multi-criteria rating means that the Rating sheet asks participants to rate each item on all criteria, e.g. 'Importance', 'Urgency' and 'Feasibility' before progressing to the next item. This can be convenient but requires higher levels of concentration. You should also be aware that applying multiple criteria simultaneously may produce unconscious and conscious (political) cross-over between criteria.

When you add a second criterion to an existing rating and thereby turn it into a 'Multi-criteria analysis', this automatically adds a Multi-criteria results table and a Multi-criteria results chart.

Rating on multiple criteria works just like multiple independent ratings, EXCEPT that in Multi-criteria analysis

  • The list of items is identical on all rating sheets (changes apply to all)
  • Anonymity settings run across criteria (changes apply to all)

Moreover, you should stick to one Rating method if possible: It is much easier to analyze results across criteria if they are in the same 'space'.

Also, limit the number of items on which you run a full-fledged Multi-criteria analysis. If you have a long list of items, run a 'cut-off' vote on a criterion items must meet to qualify for in-depth analysis. For example, when looking for measures, rate them on 'effectiveness'. Then analyze the 10 (or 20) 'most effective' measures on their specific 'solving power' regarding 'Problem A', 'Problem B' and 'Problem C'. Add other relevant criteria such as 'feasibility' and '(political) palatability' or whatever may interest you.

    Analysis of results in the Multi-criteria Results table

    The Multi-criteria table displays the results for all criteria side-by-side.

    • For a summary score across all criteria, display column 'All criteria'
    • For a summary value for each criterion, display row 'Mean of column'

    Columns can be hidden or displayed with a switch in the column header when you open 'Table settings' with the cogwheel  toolbar button.

    To see quickly which items score highest/lowest for any criterion, click on the relevant column header.

    If you have voted 'by team', use the team comparison widget of the bottom status bar to see how teams are aligned or differ in their assessments.

    Highlight 'High Standard Deviation' to spot controversy. If specific items are controversial regarding multiple criteria, this may indicate a lack of (shared) understanding of that item.

    If your table is inherently complex, reduce complexity initially by hiding columns, then reveal more data as you dig deeper.

    Once you are familiar with the basics, explore advanced features like

    • Color coding
      which lets you analyze patterns based on and sensitivity to certain thresholds
    • Calculated columns
      which let you calculate indicator values such as 'risk' from, for example, assessments of 'impact' and 'probability'.

    Accessibility of the table

    Since you and your group may want to analyze some results of your multi-criteria analysis while some ratings are still in progress, Multi-criteria tables are not blocked by a progress meter. Columns

    • whose single results table or chart would be barred by a 'progress meter'
    • are calculated on the basis of such a column

    will simply show a label, 'in progress'.

    To access the Results for that criterion, simply 'close' the Rating sheet.

    Note that if you have good cause to enable access to the results while rating is in progress, you can do so by selecting the relevant option in the 'View' settings tab of the Rating sheet.

    Decision making

    If you are trying to reach a decision in the group, Multi-criteria analysis will often provide a robust basis for that decision, especially, if you have run your analysis after a Discussion on the relevant criteria.

    However, the table will not make the decision for you. Use it as a basis for reaching a conclusion. That conclusion will depend on whether you are looking for a 'broad' solution i.e. something that scores well across all criteria, or wether one criterion is clearly more important than the others which merely need to be 'satisfied' (good enough), not optimized.

    Multi-criteria analysis is, of course, an excellent way of presenting the opinion of the group to decision makers outside the 'room'.