Walter R. Paczkowski - Deep Data Analytics for New Product Development

When I worked at AT&T Bell Labs (the Labs) in the 1990s, I was fortunate to be in a division called the Consumer Lab. This was an eclectic group of people: market researchers, computer scientists, mathematicians, statisticians, psychologists, human factors experts, and economists even an anthropologist at one time. Our charge was simple: to identify and test new products and service concepts (products for short) with consumers and business decision makers to determine market potential. In some instances, the concern was how people interacted with the products so bugs could be worked out or eliminated at the beginning of the design stage. We performed both functions by inviting customers into a lab setting where they could test the products, or mock-ups of them, under controlled conditions, but conditions that mimicked the intended environment where they would be used. We then used advanced statistical and econometric methods to estimate demand, sales, and revenue. The results had direct input into the new product business case process and the whole new product development process from ideation to launch and tracking. Members of the Consumer Lab were often involved in new product development right from the very beginning.

My particular group in the Consumer Lab was charged with terminating a new product concept to prevent it from going to market if it was going to fail. We did not earn any money for the company, but, instead, we saved it money. We actually stopped many products from ever seeing the light of day in the market simply because they did not really solve any problems faced by customers or because the product was too awkward or confusing for them to use. We stopped the products in the early stages of concept design. This was important because it was very expensive to fully develop a product concept and eventually bring it to market. If it failed, then the financial losses could be devastating. So it benefited AT&T to maintain a lab such as ours for the sole purpose of testing new product concepts. We were AT&Ts Test Kitchen.

I learned a lot from this experience. I learned about developing new product concepts; about the role, importance, and composition of a business case; and about testing demand at different stages of the new product development process. In short, I learned about the new product development process itself. I expanded on this learning with my own consulting after I left the Labs. Most importantly, in my time at the Labs and afterward, I learned about the vital role that deep data analytics plays in new product development. A business manager cannot just say that customers, whether consumers or business customers, will buy whatever is produced. Data, better yet, actionable, insightful, and useful information extracted from data must be used to support decision making and this extraction can only be done by deeply analyzing data using advanced tools and methodologies. It is this set of learnings about deep data analytics for new product development that is in this book.

If the learnings are about using deep data analytics in the new product development process, then two logical questions are begged to be answered. The first is What is the process? and the second is What is deep data analytics? Let me address these two questions in reverse order.

Analytics is the process of revealing or extracting what is hidden inside data. What is hidden? Information. Insight. Knowledge. The mistake people often make is believing that data are the information needed for decision making. That having a lot of data gives them, almost