tl;dr
The shoulders of giants are meant to be stood on. Honor them by adding something meaningful to the dialogue.
Summary
Creating a breakthrough product is a bit of an art and a science. There’s are certainly heaps of intuition and luck involved, but you can easily use scientific methods and approaches to help you know whether the path you are on is going to bear the kind of fruit you are hoping for.
Assumptions
For me, I follow Andreessen Horowitz’s definition of breakthrough products: that your product must be 10x better than the current way of doing that thing in order to overcome competition and user switching costs. It should often work “out of the box” and the customer must understand what your product does and why they love in the first 8-seconds of seeing it.
Pick a big fight
These elements are only arrived at by solving large problems, because those are the types of problems that allow 10x improvements in the first place. Typically large problems are complex and are usually equal parts technical and behavioral. The behavioral aspect requires patience and empathy on the part of the entrepreneur. Too many times a technical person gets excited by the intrinsic value of the product’s mechanics that he forgets to true up with the reality of how non-technical, busy, overwhelmed users think about it.
Knowing the ingredients
Creating something requires that you have a healthy understanding of the ingredients involved in making the final product. We know that a chemist, having mastered the properties and behaviors of life’s building blocks can construct many wonders that far supersede those found in nature. The same is true for building great products, great businesses and great civilizations.
Know the level of abstraction you need to play at
The chemistry analogy is useful because it introduces another important concept in creating effective products: knowing the unit of measurement. The chemists often work at the atomic level. And for truly breakthrough products, innovators must also be comfortable playing at the atomically-equivalent level of their domain.
Find the substrate
I don’t have any scientific evidence for this, but I believe that complexity in life boils down to a few key ingredients - usually at or near the “atomic” level. If you make tweaks at the bedrock level, you can get many other innovations for “free” at higher levels of abstraction. This is the whiplash effect that supply chain management talks about applied to innovative outcomes.
Trace the lineage
Finding the substrate is usually an outcome in tracing the lineage of a product category back to its roots. Much in the same way that most of music follows the same basic structure for the last 50 years, all the way back to "I wanna hold your hand" so too do most products. While there may be some variation depending on the product’s domain (just like the structure varies slightly, by music genre), you can easily find the bedrock of a product’s intellectual origin by taking some time to trace its lineage.
True-Up the Trees with the Forest
While doing this kind of research helps you find a starting point, it does not guarantee a successful endpoint: a product that is durable enough to justify a supporting business organization. It does, however, increase your odds of finding something that can truly stand out from the market. As an innovator, you cannot solely live at the atomic level because your users do not live there. They live at the presentation level, far above the product’s proverbial “forest.”
As such, once you have come up with your initial concepts, you must be continuously truing them up with users. The more ignorant the user, the better feedback you’ll get. Users don’t care about the feature set, or the road map, or the pixel ratio or the technological mountain-moving needed to make it happen. They want to know whether or not it is going to work immediately and how much brain damage they will incur switching to your product.
Why prototyping and feedback matters
User feedback is both humbling and relieving. Because users only understand products based on their perceptions presentation level, this frees you up from worrying about what competitors are doing (at least temporarily). User testing often illuminates how much product managers and entrepreneurs key off competitors’ feature sets as a proxy for what users want. This is a fallacy because it assumes that your competition knows more about your users than you do - if so, you should probably resign. While there are certainly competitive antes that need to be made when defining product roadmaps, relying too much on the competition is fatal. The answer is to go to the source; to seek the answer for one’s self, not live off a derivative of someone else’s answer.