Think twice before measuring

There’s a lot being measured these days. Every enterprise with some modicum of self-respect floods its customers with customer satisfaction reviews. Even my pharmacy sent me, not once but thrice, a questionnaire about my customer experience. It seems like they’re zipped off towards your inbox after every visit. However, what is the value of all these data, aside from my doubt that there’s someone actually using them? I know that a heavy sigh would emanate from my children if they were to ever see this column, ‘Oh, no, here’s dad again with his never-ending obsession’. Nevertheless, I will persist, the amount of useless or inadequate information gathered is far too much, making it a wasted effort towards the original goal.

A common mistake is that the organization in question approaches the wrong target audience

Take for example a baker who, for the duration of a month, puts a questionnaire in the bag of every order. Does the baker get anything useful from this? Wasted effort and wasted time. I can say, without any questionnaire, that most of his customers are either very or extremely satisfied with the service. After all, unsatisfied customers won’t be around anymore to fill in any questionnaires! What the baker actually wants is to grow his revenue. This is fine, but then he needs to ask different questions to other people than his current customers. To grow his clientbase, he needs to approach those who aren’t a customer yet with questions about their reasoning. Only then can he take appropriate action to add them to his clientbase. If he still should want to ask his current customers anything; then don’t ask them whether they are satisfied, but ask what they’re missing instead. Then he might learn what he could sell them additionally to increase his revenue.

My favourite unwilling victim in these criticizing rants is the major Dutch Railway Company NS. They pride themselves on regularly published punctuality data “last month 90.8% of trains were on time.” However, I happen to know that there’s far more to that simple number than appears at first sight. The simple fact that a cancelled train is not counted at all can make me livid. However, even then that number doesn’t say much: are a stopping train with at most two dozen on board and a packed intercity weighted equally? And what is ‘on time’: did they leave on time or arrive on time; and are delays at intermediary stops also included in these numbers? It looks like the NS thinks their sole purpose is to make the trains ride, which is false. They might be better off staring at a model train in the attic if they just want to make trains move.

Their real and sole purpose is to move travellers in a punctual manner.

Their real and sole purpose is to move travellers in a punctual manner. The travellers in question are to be measured whether they arrive on time, not the trains themselves. In particular, one should look at the complete route, as 3 to 4 minutes of delay at one stop can mount up to a delay of 30 minutes because people will miss their transfer. Furthermore, there should also be a focus on how the traveller in question experiences a delay. Someone who travels seldom will experience a delay in another way than a daily commuter. Just because most delays occur during rush hours, and daily commuters are condemned to these hours, it is clear they will encounter them more often and with less patience. In the past, I roughly estimated that if 90% of trains are on time, then a daily commuter will most likely suffer two to three delays per week.

Measuring is fantastic, I have done it a lot, and playing around with the results you get from measuring can be immensely satisfying. 

However, if you don’t gather the right information, you might as well use randomly generated data.

Read here the reaction to this column by Dennis Huisman