Testing with Humans

Testing with Humans, written by Giff Constable and Frank Rimalovski, is the sequel to Talking to Humans. The book focuses on how to use experiments to drive faster, more informed decisionmaking. This contains my personal book notes.

This book is about three things. How to:

1. Prioritize risks so that you only run experiments on things that matter;
2. Design and run effective, compact experiments that deliver insights; and
3. Structure a decision-making process that helps you move fast with inevitably imperfect information.

1. Prioritize risks so that you only run experiments on things that matter

• The business assumptions exercise from Talking to Humans is a good way to catalog your assumptions and identify the riskiest assumptions.

• The business assumptions exercise is an alternative to the Lean Canvas or Business Model Canvas, which are fine to use as well.

• All frameworks are trying to unearth answers to very simple questions around your idea:

  • Who is this for?
  • What problem or need are we solving for them?
  • How will we solve it?
  • How will we acquire and retain our customers?
  • How will we create value for our company? (This could be monetary or non-monetary)
  • How could it go wrong?
  • That last question can be the most useful for teasing out risks. Phrased in a slightly more formal way, it reads: “What assumptions do we have that, if proven wrong, could cause our business to fail?”

• Those questions are largely for startups, but can be adapted for non-startup ideas as well. You would arrive at slightly different questions:

  • Who is this for?
  • What are we predicting they will do?
  • What value will they get?
  • What value will we get?
  • What could cause this [feature/initiative/change] to fail to deliver value to either our customers or our business?
  • From those basic answers, you can figure out what you need to learn before you commit to spending a lot of time and money

Financial Model

A lightweight financial model is another way to expose assumptions and risks. This model should be 24 months after you first start accepting users or customers. Model basic numbers for:

• How many customers you get each month, by what means and with what cost
• How many customers pay you and how much do they spend
• How many customers stick around and for how long
• How much it costs to fulfill each customer
• How much it costs to run the business with your desired investments and estimated customer growth

This model is fantasy, yes. But it will expose key assumptions and pressure points. Doing it thoughtfully will expose key inputs where you really don't know the answer.

Once you're written out the assumptions and risks, prioritize them in a 2x2 chart using sticky notes. The x-axis goes low uncertainty to high uncertainty. The y-axis goes low impact to high impact. The top-right quadrant are your risky assumptions.

2. Design and run effective, compact experiments that deliver insights

All good experiments share five core traits:

1. They are structured and planned. Don't wing it.
2. They are focused. Test a core hypothesis and don't do too many things at once.
3. They are believable. You designed them in a way you can trust what you're learning.
4. They are flexible. The team running the experiment is open to making small improvements but without introducing too many confusing new variables.
5. They are compact. You can run them quickly.

Always ask yourself: How can we learn just as much with half the time and effort?. This could be limiting edge cases or narrowing the customer segment.

Experiment Template

The simplest template is a sentence:

For [customer segment], we believe that [outcome] will happen when we run [experiment description].

But here's a more complete template:

1. What hypotheses do we want to prove / disprove?
2. For each hypothesis, what quantifiable result indicates success? i.e. your pass/fail metric(s)
   • Don't freeze up here. Sometimes this isn't clear, but you need to pick a number. Business experiments are not like science, and you can't expect statistical significance. Lean into the uncertainty. The financial model may help here.
3. Who are the target participants of this experiment?
4. How many participants do we need?
5. How are we going to get them?
   • Sending out personalized emails to people in our networks
   • Researching prime targets and calling them directly on the phone
   • Playing six degrees of separation and networking through relationships
   • Attending relevant conferences or meetups, engaging with people, and asking for follow-up conversations
   • Politely intercepting people before they walk into a store
   • Approaching doctors in their lunch cafeteria
   • Running online ads on Google, Facebook, Craigslist, etc. (typically feeding those ads into a sign-up page or a short, qualifying survey so we could filter out the right kind of participants)
   • Reaching out via existing email lists and newsletters
   • Embedding our experiment into an existing product experience.
   • See also Talking to Humans for more ideas.
6. How do we run the experiment?
7. How long does the experiment run for?
8. Are there other qualitative things to learn during this experiment?
   • Are there any "ride-along" questions we can ask our recruited subjects?

Ideas for Testing Demand

1. Landing Pages. Express your value prop on a page and give the visitor the ability to express interest with a call to action.
   • Consider how soft you make your CTA. I.e., it's easier to get an email address than have someone enter their credit card.
   • Consider where you tie it to your brand or not. You can take more risks if you're off-brand, but it might hurt conversion to not have it tied to a brand.
   • Don't present a bunch of options and ask a prospect to pick. If you want to test different options, A/B test.
2. Advertising. Boil your value prop down into something that can be presented to a target audience to see if they convert.
   • A/B test variations on your value prop.
   • Test different channels.
   • Ultimately this breaks down into two conversion points: (1) do they click through the ad, and (2) do they convert when they land on the resulting page or survey
   • Start with a small amount of money and optimize your ad settings based on the initial results before expanding your spend. In particular, be cautious with the “match type” of your keywords. If you select “broad match”, you might find yourself paying for a lot of irrelevant clicks.
3. Promotional. A promo video or other digital asset with a CTA to join a waitlist or something.
   • Promo tests are definitely cheaper than building a product, but getting the story-telling and form factor right usually takes some testing unto itself.
4. Pre-Selling. E.g., crowd-funding like Kickstarter. Can you literally get someone to pay you before the product is done?
   • The more you are trying to test your pricing model and customer willingness to buy, the more you should steer towards pre-purchases and paid pilots.

Ideas for Testing Product / Features

1. Paper testing / mockups with something like Balsamiq.
2. Button to nowhere. Once clicked, the use gets a pop up that says the feature isn't ready yet (possibly offering to interview them about their interest.)
3. Task completion. Can you get someone to fill out a spreadsheet or take a specific action? This one isn't so clear.
4. Product prototypes. Don’t confuse building your prototype with building your real product. Take on a hacker’s mindset, and don’t worry about the re- usability of software code or materials. Instead, focus on the speed and quality of what you can learn.
5. Wizard of Oz tests. The customer things they're interacting wit ha product, but behind the scenes its a service. Always do an abbreviated trial run before starting a Wizard of Oz test. It will help you understand how many participants you can handle, and inevitably expose ways you can improve the experiment.
6. Concierge test. With this, you manually and overtly act as the product you eventually want to build. Since you’re acting in a consultative capacity, is that you get to talk directly with your customers, maximizing quantitative learning. It's also often easier to charge money for the work.
7. Pilots, ideally paid pilots. This is mostly used in B2B. These are less about experiments, but you still learn a lot.
8. Usability testing. You don't need too many people (like 5 people) to do this.

3. Structure a decision-making process that helps you move fast with inevitably imperfect information

• Try to Avoid Your Own Biases:

  • Anchoring. Don't fixate on the first piece of information that comes in.
  • Confirmation bias. Don't prioritize and interpret information in a way that supports our position. This is why we set quantitative targets in advance.
  • Bandwagon effect. Don't believe more in an idea just because its popular.
  • Sunk cost fallacy. Don't be afraid to kill an idea even after putting time, money or personal capital into it.

• At the end of every experiment, choose one of the following four options:

  1. You aren’t satisfied and still need more data to make a key business decision.
  2. You are ready to move forward with the hypothesis with confidence.
  3. You decide to change your hypothesis based on the data (which might mean a new experiment).
  4. You decide the kill the initiative entirely.

12 Tips for Running Effective Experiments - A Summary of Key Points

1. Save your experiment effort for risks that will truly impact the success or failure of your project or business.
2. Don’t just think about experiments for your product. Remember to examine your customer segments, value propositions, customer acquisition methods, pricing plans and revenue models, unit economics, etc.
3. Stretch your thinking at the start because there are always more ways to test something than you think.
4. Be disciplined about the details because sloppy experiments lead to sloppy results.
5. Set target pass/fail goals ahead of time or you’ll be tempted to rationalize what happened after the fact.
6. Ask how you can just learn just as much, if not more, with half the time and effort.
7. Optimize for learning, not for building product, or you’ll move too slowly.
8. For big experiments, do a trial run first because you’ll often discover things to improve.
9. Run your experiments with intensity and speed, because time will disappear faster than you think.
10. Include opportunities for qualitative research (talking to humans!) as you go.
11. Fight your own confirmation biases. In other words, don’t twist results to hear what you want to hear, or dismiss undesirable results too quickly.
12. Combine evidence and judgment to make smart decisions and execute!