My test suite takes 9 months to run

Test output

Finished tests in 9 months 12 days 6 hours, 1.0046 tests/month, 3.5037 assertions/months.

Testing as developers, testing as founders

As developers, we are proponents and practitioners of TDD. This means that we try to keep a good suite of tests for our Sybil application. This leads us notably to start decoupling it as much as possible. Once the test are in place, one of the important thing is to keep them fast.

As founders of 8th color, we tend to follow the Customer Development and Lean Startup movement (is this the right word?), as the two go quite well together. One of the key aspects of both approaches is to not assume but rather test any assumptions you may have about your business, whether it is your market fit, problem fit or way to market.

What the testing cycle means as founders

 

Cover of "The Four Steps to the Epiphany&...

Cover of The Four Steps to the Epiphany

Test (in the lean startup approach) means basically finding a potential customer and putting your application in his hands, as soon as possible (the first “application” may well be just a presentation or a video or some wireframes). This allows to “test” what works, collect feedback, come back with a more advanced product, then repeat the cycle.

The capacity to run this cycle is critical to your business in many different ways: first and foremost, however good and innovative your idea, you will never make a business of something that has value for you only. The real value of your product is not the one it has for you (i.e., the one you think it has), but the value it has for your customers. One of the axiom of the “get out of the building” principle is that when we make suppositions, we are (most often) wrong (if we were right all the time, testing would be a pure loss of time). The whole model is then “just” a way to cope with it and still be successful, by recognizing it. Another aspect is that, as a startup, quick cycles are one of the unfair advantages you can have against bigger, better funded, more experimented companies.

For this to work, you need to do your part, meaning, minimize the time between the moment you collect feedback from a test and the one you’ve integrated it to be ready for the next test. We try hard on this one, from reacting to emails as soon as possible (same day for 80%, especially if they come from customers) to having a quick development cycle (our typical feature development schedule uses hours as time unit – not weeks or months). But last week, we stumble on a big roadblock: we discovered that our test suite was taking too long to run. Like, 9 months.

Testing in the B2B world

When we started our product development for Sybil, we made an assumption (supported by some interviews) about the problem of skills management: that is was probably more visible and painful in big structures than in

small one. This was an assumption to be tested, and we knew it. We started very soon with pilots, in order to build Sybil with them, and not in our basement. While this strategy has been successful on many aspects, there is one point that came back to bite us.

As well said in a tweet from the recent Belgian #failconf, “First tip at #failcon12 : dear #startup , your #b2b software sales funnel is an agonizing 9 months long”. This is especially true if you target big accounts (with complex sales processes involving different actors) and if you are a price business (i.e., that your business model is based on a small number of sales with high margins).

The fact that the sales funnel is that long is of course a problem, but a well known one (basically, you need to find a way to be sustainable with a permanent 9 months liquidity problem). The real problem for startups is that you need to get through the whole funnel with one (if possible, some) companies to validate your model.

In other words :

“You need 9 months to validate an assumption.”

Finished tests in 9 months 12 days 6 hours, 1.0046 tests/month, 3.5037 assertions/months.

5 tests, 15 assertions, 1 failures, 0 errors, 0 skips

Now I suppose I just have to fix that one, and run it again.

Some hope, still

Does this means that no startup can work in B2B? Or that it is not possible to do customer development there? Clearly not. I’ve outlined most of the problems before, but solutions do exist, and no situation is that grim:

  • The funnel may be 9 months long, but you can (and should!) validate a maximum “along the way” – i.e., it is not just about the last mile
  • Identifying a maximum of problems early can mitigate this problem, and the learning will probably be worth it anyway (even learning about what does not work is worthy, and you have a lot of nuggets of knowledge you can pick up along the way if you are attentive).

My principal lesson on this is that the longer the funnel, the earlier you need to start validating. B2B (and especially big companies) are not adverse to pilots (where you bring a prototype/unfinished product to them, to get feedback) or even funded development. Use those opportunities as soon as possible in the development.

For the record, we had our first customer contact with a version of Sybil that has less than a full day of development behind.

Enhanced by Zemanta




Leave a Reply

Your email address will not be published. Required fields are marked *

You may use these HTML tags and attributes: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <strike> <strong>