Luiz Felipe Martins - Python: End-to-end Data Analysis

, Getting Started with Python Data Analysis , shows how to work with time oriented data in Pandas. How do you clean, inspect, reshape, merge, or group data these are the concerns in this chapter. The library of choice in the course will be Pandas again.

, Python Data Analysis Cookbook , demonstrates how to visualize data and mentions frequently encountered pitfalls. Also, discusses statistical probability distributions and correlation between two variables.

, Mastering Python Data Analysis , introduces linear, multiple, and logistic regression with in-depth examples of using SciPy and stats models packages to test various hypotheses of relationships between variables.

Module 1:

There are not too many requirements to get started. You will need a Python programming environment installed on your system. Under Linux and Mac OS X, Python is usually installed by default. Installation on Windows is supported by an excellent installer provided and maintained by the community.This book uses a recent Python 2, but many examples will work with Python 3as well.

The versions of the libraries used in this book are the following: NumPy 1.9.2,Pandas 0.16.2, matplotlib 1.4.3, tables 3.2.2, pymongo 3.0.3, redis 2.10.3, and scikit-learn 0.16.1. As these packages are all hosted on PyPI, the Python package index, they can be easily installed with pip. To install NumPy, you would write:

$ pip install numpy

If you are not using them already, we suggest you take a look at virtual environments for managing isolating Python environment on your computer.For Python 2, there are two packages of interest there: virtualenv and virtualenvwrapper. Since Python 3.3, there is a tool in the standard library called pyvenv (https://docs.python.org/3/library/venv.html), which serves the same purpose.

Most libraries will have an attribute for the version, so if you already have a library installed, you can quickly check its version:

>>>importredis

>>>redis.__version__'2.10.3'

This works well for most libraries. A few, such as pymongo, use a different attribute(pymongo uses just version, without the underscores).While all the examples can be run interactively in a Python shell, we recommend using IPython. IPython started as a more versatile Python shell, but has since evolved into a powerful tool for exploration and sharing. We used IPython 4.0.0 withPython 2.7.10. IPython is a great way to work interactively with Python, be it in the terminal or in the browser.

Module 2:

First, you need a Python 3 distribution. I recommend the full Anaconda distribution as it comes with the majority of the software we need. I tested the code with Python 3.4 and the following packages:

joblib 0.8.4

IPython 3.2.1

NetworkX 1.9.1

NLTK 3.0.2

Numexpr 2.3.1

pandas 0.16.2

SciPy 0.16.0

seaborn 0.6.0

sqlalchemy 0.9.9

statsmodels 0.6.1

matplotlib 1.5.0

NumPy 1.10.1

scikit-learn 0.17

dautil0.0.1a29

For some recipes, you need to install extra software, but this is explained whenever the software is required.

Module 3:

All you need to follow through the examples in this book is a computer running any recent version of Python. While the examples use Python 3, they can easily be adapted to work with Python 2, with only minor changes. The packages used in the examples are NumPy, SciPy, matplotlib, Pandas, stats models, PyMC, Scikit-learn. Optionally, the packages basemap and cartopy are used to plot coordinate points on maps. The easiest way to obtain and maintain a Python environment that meets all the requirements of this book is to download a prepackaged Python distribution. In this book, we have checked all the code against Continuum Analytics' Anaconda Python distribution and Ubuntu XenialXerus (16.04) running Python 3.

To download the example data and code, an Internet connection is needed.