Just recently, I was quite enthusiastic to read a software science paper, because its title sounded promising. I do not want to refer to this specific paper, because it is neither the goal to discredit one specific work nor one specific author. It is a set of similar papers that needs to be criticised.
Over the last years more and more questionnaires can be found at scientific venues in software science. The commonality of these papers is that authors ask developers about their opinions on some topic. Then, responses from a huge number of people are collected. Then, the results are analyzed.
So far, there is no problem.
There is nothing wrong with opinions. It is interesting to know what peoples' opinions are. It is especially interesting from a marketing perspective, because it says something about the perception of people.
The problem lies in the conclusions.
What a number of works in software science do and which is fundamentally wrong is to infer from subjective perceptions something about the perceived phenomena.
Let's assume there is a technology X that tries to make developers' life easier. Then, someone asks developers whether it makes their life easier and let's assume with high evidence the answer is yes. What can we conclude from it?
We can conclude that developers think that it makes their life easier. It is also possible that developers just pretend that it makes their life easier. But we do not know whether it makes their life easier. Making any claim about how or whether the technology influenced a developer's life is not possible from the evidence gathered so far.
In order to find out whether the technology X helps, no result from subjective perceptions would bring us closer to an answer. Whatever the result of the questionnaire is, the question whether or not the technology helps is still unanswered.
One could argue that a developer's life becomes better because he thinks that there is a technology that helps him - independent of whether or not it actually helps him. This kind of argument is comparable to a placebo argument that we frequently find in homeopathy. But it should be clear that this argument should not be used in software science, because it is more a meta argument: if something makes people think that it makes their lives better, then it is good. The argument is comparable to the question whether a free beer makes a developer's life better.
Of couse, this leads us (as always) to the need for studies. But the argument is not that questionnaires are no studies. They are. But the problem with them is that they purely depend on subjective perceptions.
There are good reasons why you find whole textbooks about perception in psychology. Perception is not only subjective from the perspective that people can perceive the same phenomenon in different ways (because of differences in physionomy, differences in experience, etc.). Perception is influenceable. You can easily find a bunch of studies that show that perception can be influenced and the Pepsi versus Coke experiment [1] is just one example (again, whole textbooks are on that topic, there is no point in giving a longer list on that here).
So, what is actually the problem? When we study technology, we need to measure interactions with the technology in a non-subjective way. You can still ask developers questions. "In this scenario, what is the outcome?" could be an appropriate question. But it differs from a question "Do you think that technology X helps?".
We need to stop asking for subjective perceptions.
The implications of the statement are much more serious than we think. Community processes that can be found today for example in programming language design typically ask people about opinions. But it leads to far to discuss this issue here.
Please, stop collecting developer opinions.
Opinions are important. But they do not permit to draw any conclusion beyond peoples' opinions. And should a technical discipline have the focus on opinions? I think the answer is no.
Feel free to leave a comment.
