John Lee - Using Python 3.9 and BeautifulSoup to Web Scrape Market, Economic, Financial and Historical Company Earnings Data: Version: January 01, 2022

It used to be very expensive in both time, money and energy to obtain market, economic and financial data. People had to rely on CRSP tapes to get stock prices, read the paper version of the Wall Street Journal or buy the paper version of Value Line financial research. This all changed with the start of the Internet in the 1990s. The Internet essentially removed the need to have intermediaries as the channel for information delivery to consumers.

One big overarching theme of the Internet is FREE. With internet browsers like Firefox and Chrome, we can access information at relatively no cost to quickly gain an excellent general understanding of the economic and market conditions. If we want a deeper understanding of economic and market conditions, we can pay a reasonable fee.

We will see that the Internet and Python will significantly reduce the time to gather market and economic data and give us more time to analyze the market, economic and financial data. Another benefit we will see is that the Internet and Python will reduce a lot of data errors that can result from the manual procurement of data.

We will use Python to retrieve many different types of market, financial and economic data. The types of market, financial and economic data we will retrieve are stock, market index, currency rates, bonds, futures, mutual funds, electronically traded funds (ETF), commodities, balance sheet, income statement, statement of cash flow, GDP, consumer price index, unemployment rate, household income and money supply.

Chapter 28 discusses how to retrieve historical company earnings.

There are two ways to retrieve market data; one of them is web scraping the data; the other is to use the REST web service or REST API that is provided by the website. When web scraping, we will use the beautifulsoup Python library. The install instruction is located at https://pypi.org/project/beautifulsoup4/ .

The target audience for this book is those interested in retrieving market, financial and economic data. This book will show many sources for obtaining this type of information using the case study approach. People who read this book will need a working knowledge of Python. This book will not explain Python; instead, readers will study the code to learn different Python techniques and where to get the financial, market and economic data. Like experience programmers, the reader will google any Python programming concept that needs more clarifications.

The official Python URL is https://www.python.org/

This book uses Python 3.9. Go to https://www.python.org/downloads/ to download Python.

The URL https://www.codecademy.com/articles/what-is-an-ide said the following about IDEs,

An IDE, or Integrated Development Environment, enables programmers to consolidate the different aspects of writing a computer program.

IDEs increase programmer productivity by combining common activities of writing software into a single application: editing source code, building executables, and debugging.

This book will use the Visual Studio Code Integrated development environment (IDE) when doing serious development in Python. Visual Studio Code is included with the Anaconda Platform and is part of the Anaconda Navigator.

This book will use the Visual Studio Code Integrated development environment (IDE) when doing serious development in Python. Go to the URL https://code.visualstudio.com/download to download Visual Studio Code.

JetBrains did a survey and found out that Visual Studio Code was the second most popular IDE use to do Python development. The URL for this survey is https://www.jetbrains.com/lp/python-developers-survey-2019/ . Jupyter Notebook has gotten a lot of press for Python development, but according to JetBrains survey VS Code is much more popular than Jupyter Notebook, 24% vs. 5%.

The URL https://insights.stackoverflow.com/survey/2019#technology-_-most-popular-development-environments indicated Visual Studio Code was the most popular IDE across the board.

This book will use Visual Studio Code.

Visual Studio Code can also be installed at https://code.visualstudio.com/

Sublime Text is a popular text editor. It is also a popular application to do Python programming and to run Python code.

Kite is a popular free plugin for both Visual Studio Code and Sublime Text. The URL to download Kite is https://kite.com .

This book will use the Python IDLE to run Python code.

Go to the URL https://realpython.com/python-idle/ to learn more about Python IDLE.

Yahoo! Finance has historically been the go-to place to get financial data. We can see this by the fact that there are Python libraries that scrape Yahoo! Finance for market data. In this book, we will use three of the most popular Yahoo! Finance Python libraries: