How to make evaluations using a decision framework

Lately I have been doing work evaluating different pieces of software. The process is quite simple at its core – you start with requirements, map the requirements to some software vendors, dive deeper into the metrics that will steer your decision, and finally evaluate their product against the metrics to determine which one fits the best. This is a strategic solution for Pariveda Solutions.

After studying  some existing selection frameworks, I figured that the underlying principles could be applied to almost anything. Of course I can’t give away the secret sauce, but I’ll do my best to generalize some ideas so that you can understand the value in the process. I like calling it a decision framework, rather than a selection framework.

[As a note, there is a link below to download the sample that I will begin walking through – you can download it now if you’d prefer]

The Scenario

We’ve all been in a spot where a tough decision was at hand. There are usually many factors involved and it is hard to keep track of all your thoughts. There are some simple ideas that can be applied to this decision to help you make a smart evaluation. The scenario I’ll work through in this example is ‘Evaluating where to live’. Hopefully it is something that most everyone can relate to.

The Method

Requirements – you should start the evaluation with some high level requirements. This will help you initially narrow your choices. Here are the requirements I’ll use:

  • It has to be a major city
  • It has to be in the U.S.
  • It needs to be close to a beach (an hour away at most)
  • It can’t be on the east coast

These are obviously some strange requirements, but that is part of it. It is a tough decision because it is unique. The requirements don’t have to make sense to anyone else, they just have to fit your situation.

Choices – based on the high-level requirements, try to narrow your field to a manageable number of choices. Usually selecting 3 – 6 choices should work. It is very typical for choices to fall out of the running as the evaluation takes place. For my city example, I’ll use the following cities:

  • Houston :)
  • Seattle
  • San Diego

Metrics – now that you’ve got an idea of what you are looking for, flesh out all of the metrics by which you want to make your decision. Just start listing things out (lots of things, the ones below are a very small subset):

  • What is the pollution level like?
  • What is crime like?
  • How is traffic around downtown?

As you finalize your framework, you will continue to update the metrics. You might find that some are too detailed and that others need to be more detailed. Sometimes two metrics will be combined into one and other times splitting one metric into several makes sense. It is all up to you.

I find it very useful to put metrics into logical groupings. For example, we might have metrics around pollution, temperature, and the length of different seasons. It is helpful to group these metrics so that evaluation can be done at a granular level and automatically roll-up to a final decision – you’ll see how this works shortly.

Evaluation – now that you have all of your metrics defined, evaluate each choice on that metric. Use whatever scale makes sense to you (although I would suggest using some type of scale that easily translates to numerical values). I find that using a 1 – 5 scale works pretty well because it gives you some room to make a substantial difference between choices without muddying the waters (i.e. it becomes increasingly difficult to justify differences in a scale of 1 – 50).

So for each choice (Houston, Seattle, and San Diego) you will evaluate each metric (pollution, crime, traffic, etc.) on the chosen scale (1 – 5). Don’t think about the overall decision to be made when doing these evaluations. Keep your focus on the small task of evaluating the one metric, and the choice will become apparent by weighing the different logical groupings.

The Payoff

I won’t blab anymore about the methods. I think that a sample will help to sell the ideas. Download my sample evaluation here. Walk through the different work sheets (tabs at the bottom) to understand how the model was built. The meat is in the scorecard. You can easily see how each of your choices matches up to the others in each area. You get to see a raw and weighted score, as well as a standard deviation and margin of victory (for the weighted scores).

The scorecard portion might seem a little busy at first, but it is a great piece of documentation. It lets you easily see which areas are the most important and which made the biggest difference in scoring. All the calculations are done dynamically. As an example, go into the ‘Weighting’ sheet and change up some numbers (maybe you think that Safety is more important than Affordability). Now go back to the scorecard and see the changes get updated automatically. The scorecard is great because it shows how your weights affect the overall decision. In the sample, Seattle has a higher raw score than Houston but Houston wins the decision due to my weight distribution.

There are a lot of ways that this framework could be modified, but the underlying principles are what makes it special. How would you modify the framework? As always, feedback and questions are welcome.

Posted in Personal, Professional Tagged with: , ,
6 comments on “How to make evaluations using a decision framework
  1. Chris Aikens says:

    Awesome post as usual. Even though I have worked with frameworks in the past I never thought to extend their use into my personal life. Now I can figure out which car to buy

    • brhlavinka says:

      Thanks for the read, Chris! Yeah – I also never thought to extend it to my personal decisions until recently. Seems like it could be more work than necessary unless it is something very important.

  2. I wish you would have written this article when I was deciding between UH, UT, and A&M. I probably still would’ve reached the same conclusion, but it would’ve been much more organized than three separate pros/cons lists.

  3. Josh says:

    San Diego is the clear choice, unless you are a football fan…

Leave a Reply

Your email address will not be published. Required fields are marked *


Subscribe to my Insights via Email

Join 61 other subscribers

%d bloggers like this: