2012-10-16

What makes good research?

When joining my group, one of my PhD students got blamed by one of my colleagues: With her being best in class across all  theory courses, why would she "waste her talent" with a practitioner like me? Ah, the good old debates between theory and practice. As a researcher in software engineering, none of my work will ever be as elegant as what my colleagues in theory do, attacking the most complex problems with beautiful abstraction. But then, I know colleagues in theory who feel inferior to their more applied colleagues, because what these do actually has impact and interest in the real world.  

Obviously, different people have different ideas on what makes good research. But what are these categories?

Six years ago, I had a conversation with the founder of our faculty, Günter Hotz. Our faculty is well-known for its constant excellent hiring, based on ideas Hotz installed 40 years ago. So I asked him: How do you recognize good researchers? Or better: good research? To which he presented which I hereby name the Hotz model of good research:

It is actually fairly simple. There are three criteria, which are orthogonal. (He raised his hand, spreading off thumb, index, and middle finger at right angles to each other.) You ask the following questions:

First, was it hard? When you look at the research, is this something which required years of training and perspiration, or is this something anyone on the street could have come up with?

Second, is it elegant? Is this something which solves a specific problem, or is this something that you can apply over and over again to new problems?

Third, is it useful? Is this of value only to the academic community, or is this something people outside can use to create value or make money?

If you're good on one of the axes, you are doing good research.  If you're good on two axes, we'll hire you.  And if you're good on all three axes – well, I have yet to meet somebody who excelled in all three.

On this, he looked me in the eye and said:

Actually, now that I think of it, the "usefulness" axis is a fairly recent addition to the system.

There's a number of lessons to be learned from this. First, if your work was very hard, is very elegant, or very useful, you have every reason to be proud – and you deserve the respect from your fellow researchers, even if you don't share the same axes. In software engineering, usefulness is first and foremost, but truly hard or elegant work can still leave me highly impressed. If I get an idea of how it may be put to use, I feel struck by shock and awe.

Second, there is little chance you will be able to excel on all three axes. Usefulness calls for applicability in the real world, whose details will quickly spoil the elegance of your approach. Elegance implies simplicity, and simple is the antonym of hard – unless you claim it was real hard to get your approach simple. Making something really useful is hard, but was it a true intellectual challenge, or just a long engineering process? Difficulty, elegance, usefulness – pick any two.

Third, usefulness is still a recent criterion by academic standards, and as a new kid on the block, it may struggle to get accepted by traditional academics. But computer science has long left the academic ivory towers, and is progressing at a furious pace in the real world. As a young researcher, why limit your impact to dusty publications, or wait until the real world finally adopts your approach?

You have every right to choose the axis (or axes!) on which you plan to excel. Different people make different choices, but remember: The axes are orthogonal, and no axis dominates the others. The only important thing is to push the boundaries as far as you can. Face the hardest problems. Find the most elegant abstraction. Make it really useful. Push, push, push, with all love and respect. That's what makes good research.

8 comments:

  1. Hi Andreas, I like this but there's a problem with the "hard" criterion which is that a lot of good systems research, when it is written very well, sounds easy. People read this stuff and think it's trivial. On the other hand, a lot of poor systems research sounds really hard because it is so badly presented. It can be extremely difficult, when reading some paper, to distinguish which of these categories it falls into since, presumably, the people writing the paper are the actual experts.

    ReplyDelete
  2. Interesting that "novel" wasn't one of the axes ... I suppose that's a good thing, though, since it's rare for some idea to truly be "novel" (whatever that means).

    ReplyDelete
  3. I guess that hard would imply novel (if a solution to a problem is already available, it can't be that hard to find it :-) )

    ReplyDelete
  4. Andreas: thanks for sharing Hotz's 3D model of good research (hard, elegant, useful). IMHO, it is a very elegant and useful model... and considering all the arguments and misconceptions, coming up with this model could not have been that easy. Thanks!

    ReplyDelete
  5. I'd change "Was it hard" to "Was it non-obvious." Sometime a new result is easy once you have the right idea, but the idea was not obvious at all. Focusing on hardness encourages authors to obfuscate their work, to make it look hard.

    Moshe Vardi

    ReplyDelete
  6. Hi, Thanks! for sharing such nice blog. I think hardness is attached with problem (we should select hard problem to work on) not with solution. Simple solutions are better than hard one :))

    ReplyDelete
  7. Where does "Did you gain insights (either for yourself or something undiscovered previously) into the problem fit it in?" Perhaps in the hard part ? For instance, quite often one underestimates some part of a problem as being easy to solve, but it turns out to be the hardest in hindsight.

    ReplyDelete