Pramod Singh - Machine Learning with PySpark: With Natural Language Processing and Recommender Systems

This Apress imprint is published by the registered company APress Media, LLC part of Springer Nature.

The registered company address is: 1 New York Plaza, New York, NY 10004, U.S.A.

Businesses are swimming in data, yet many organizations are struggling to remain afloat in a rapidly expanding sea of data. Ever-increasing connectivity of data-transmitting devices is driving growth in how much and how fast data is being generated. This explosion in digitalization has been accompanied by a proliferation of applications and vendors. Companies often use multiple vendors for the same use cases and store data across multiple systems. Digital data lives in more and more formats and across fragmented data architecture layers.

In a world where data lives everywhere, many organizations are on a race to be more data-driven in their decision-making to get and stay ahead of the competition. The winners in this race proactively manage their data needs and leverage their data and analytics capabilities to drive tangible business outcomes.

The starting point for using data and analytics as a strategic tool is having good data. Take a B2B company that was looking to improve how it forecasted next months sales, for example. Each month this company would aggregate individual sales reps hot leads into a sales forecast that might be wide of the mark. Improving the underlying data quality required four changes to how the company captured customer insights from the front line. First, the company clearly defined at which stage in the sales funnel a lead should be tagged to the sales forecast. Second, it reduced the granularity of information that the sales force had to enter along with each sales lead. Third, it gave more time to the sales reps to complete the data entry. Fourth, it secured commitment from the sales force to improve the data quality. Along with ensuring a common understanding of the changes, the company was able to improve how it translated frontline customer insights into better data.

With better data in their arsenals, businesses need to orchestrate the data in a complex web of systems, processes, and tools. Despite having a state-of-the art CRM (customer relationship management) system, numerous spreadsheets were metronomically circulated across the organization to syndicate the companys sales forecast. The company improved how it orchestrated the sales forecast data in four ways. First, it consolidated disparate planning and tracking tools into a consistent view in the CRM system. Second, it secured commitment from the organization to use the CRM forecast as one source of truth. Third, it configured the access for all relevant stakeholders to view the sales forecast. Fourth, it compressed the organizational layers that could make adjustments as fewer layers brought fewer agendas in managing the sales forecast.

The effective use of data and analytics can help businesses make better decisions to drive successful outcomes. For instance, the company mentioned previously developed a system of sanity checks to assure the sales forecast. It used machine learning to classify individual sales reps into groups of optimistic and conservative forecasters. Machine learning helped predict expected conversion rates at an individual level and at an aggregate basis that the company could use to sense-check deviations from the forecast envelope, either to confirm adjustments to the sales forecast or trigger a review of specific forecast data. Better visibility on the sales forecast enabled the supply chain function to proactively move the companys products to the branches, where sales was forecasted to increase, and place replenishment orders with its suppliers. As a result, the company incurred fewer lost sales episodes due to stock-outs and achieved a more optimal balance between product availability and working capital requirements.

Building data and analytics capabilities is a journey that comes in different waves. Going back to the example, the company first concentrated on a few use cases such as improved sales forecasting that could result in better visibility, more valuable insights, and automation of work. It defined and aligned on changes to its operating model and data governance. It set up a center of excellence to help accelerate use cases and capability building. It built a road map for further use case deployment and capability development upskilling current employees, acquiring new talent, and exploring data and analytics partnerships to get the full value of its proprietary data assets.

Great data and analytics capabilities can eventually lead to data monetization. Companies should start with use cases that address a raw customer need. The most successful use cases typically center on improving customer experience (as well as supplier and frontline worker experience). Making the plunge to build data and analytics capabilities is hard; yet successful companies are able to muster the courage to ride the waves. They start with small wins and then scale up and amplify use cases to capture value and drive tangible business outcomes through data and analytics.