LitArk » Books » Computer

it-ebooks - Data Science iPython Notebook 20160913

Here you can read online it-ebooks - Data Science iPython Notebook 20160913 full text of the book (entire story) in english for free. Download pdf and epub, get meaning, cover and reviews about this ebook. year: 2016, publisher: iBooker it-ebooks, genre: Computer. Description of the work, (preface) as well as reviews are available. Best literature library LitArk.com created for fans of good reading and offers a wide selection of genres:

Romance novel Science fiction Adventure Detective Science History Home and family Prose Art Politics Computer Non-fiction Religion Business Children Humor

Choose a favorite category and find really read worthwhile books. Enjoy immersion in the world of imagination, feel the emotions of the characters or learn something new for yourself, make an fascinating discovery.

Book:
Data Science iPython Notebook 20160913
Author:
itebooks
Publisher:
iBooker it-ebooks
Genre:
Books / Computer
Year:
2016
Rating:
5 / 5
Favourites:
Add to favourites
Your mark:
- 100
- 1
- 2
- 3
- 4
- 5

Description
Author's other books
Similar books

Data Science iPython Notebook 20160913: summary, description and annotation

We offer to read an annotation, description, summary or preface (depends on what the author of the book "Data Science iPython Notebook 20160913" wrote himself). If you haven't found the necessary information about the book — write in the comments, we will try to find it.

it-ebooks: author's other books

Who wrote Data Science iPython Notebook 20160913? Find out the surname, the name of the author of the book and a list of all author's works by series.

Data Science iPython Notebook 20160913 — read online for free the complete book (whole text) full work

Below is the text of the book, divided by pages. System saving the place of the last page read, allows you to conveniently read the book "Data Science iPython Notebook 20160913" online for free, without having to search again every time where you left off. Put a bookmark, and you can go to the page where you finished reading at any time.

Light

Font size:

↓

↑

Reset

Interval:

↓

↑

Bookmark:

Make

This notebook was prepared by Donne Martin. Source and license info is on GitHub.

Amazon Web Services (AWS)

SSH to EC2
Boto
S3cmd
s3-parallel-put
S3DistCp
Redshift
Kinesis
Lambda

SSH to EC2

Connect to an Ubuntu EC2 instance through SSH with the given key:

In [ ]:

! ssh -i key.pem ubuntu@ipaddress

Connect to an Amazon Linux EC2 instance through SSH with the given key:

In [ ]:

! ssh -i key.pem ec2-user@ipaddress

Boto

Boto is the official AWS SDK for Python.

Install Boto:

In [ ]:

! pip install Boto

Configure boto by creating a ~/.boto file with the following:

In [ ]:

aws_access_key_id = YOURACCESSKEY aws_secret_access_key = YOURSECRETKEY

Work with S3:

In [ ]:

import boto s3 = boto . connect_s3 ()

Work with EC2:

In [ ]:

import boto.ec2 ec2 = boto . ec2 . connect_to_region ( 'us-east-1' )

Create a bucket and put an object in that bucket:

In [ ]:

import boto import time s3 = boto . connect_s3 () # Create a new bucket. Buckets must have a globally unique name (not just # unique to your account). bucket = s3 . create_bucket ( 'boto-demo- %s ' % int ( time . time ())) # Create a new key/value pair. key = bucket . new_key ( 'mykey' ) key . set_contents_from_string ( "Hello World!" ) # Sleep to ensure the data is eventually there. # This is often referred to as "S3 eventual consistency". time . sleep ( ) # Retrieve the contents of ``mykey``. print key . get_contents_as_string () # Delete the key. key . delete () # Delete the bucket. bucket . delete ()

Each service supports a different set of commands. Refer to the following for more details:

AWS Docs
Boto Docs

S3cmd

Before I discovered S3cmd, I had been using the S3 console to do basic operations and boto to do more of the heavy lifting. However, sometimes I just want to hack away at a command line to do my work.

I've found S3cmd to be a great command line tool for interacting with S3 on AWS. S3cmd is written in Python, is open source, and is free even for commercial use. It offers more advanced features than those found in the AWS CLI.

Install s3cmd:

In [ ]:

! sudo apt-get install s3cmd

Running the following command will prompt you to enter your AWS access and AWS secret keys. To follow security best practices, make sure you are using an IAM account as opposed to using the root account.

I also suggest enabling GPG encryption which will encrypt your data at rest, and enabling HTTPS to encrypt your data in transit. Note this might impact performance.

In [ ]:

! s3cmd --configure

Frequently used S3cmds:

In [ ]:

# List all buckets ! s3cmd ls # List the contents of the bucket ! s3cmd ls s3://my-bucket-name # Upload a file into the bucket (private) ! s3cmd put myfile.txt s3://my-bucket-name/myfile.txt # Upload a file into the bucket (public) ! s3cmd put --acl-public --guess-mime-type myfile.txt s3://my-bucket-name/myfile.txt # Recursively upload a directory to s3 ! s3cmd put --recursive my-local-folder-path/ s3://my-bucket-name/mydir/ # Download a file ! s3cmd get s3://my-bucket-name/myfile.txt myfile.txt # Recursively download files that start with myfile ! s3cmd --recursive get s3://my-bucket-name/myfile # Delete a file ! s3cmd del s3://my-bucket-name/myfile.txt # Delete a bucket ! s3cmd del --recursive s3://my-bucket-name/ # Create a bucket ! s3cmd mb s3://my-bucket-name # List bucket disk usage (human readable) ! s3cmd du -H s3://my-bucket-name/ # Sync local (source) to s3 bucket (destination) ! s3cmd sync my-local-folder-path/ s3://my-bucket-name/ # Sync s3 bucket (source) to local (destination) ! s3cmd sync s3://my-bucket-name/ my-local-folder-path/ # Do a dry-run (do not perform actual sync, but get information about what would happen) ! s3cmd --dry-run sync s3://my-bucket-name/ my-local-folder-path/ # Apply a standard shell wildcard include to sync s3 bucket (source) to local (destination) ! s3cmd --include '2014-05-01*' sync s3://my-bucket-name/ my-local-folder-path/

s3-parallel-put

s3-parallel-put is a great tool for uploading multiple files to S3 in parallel.

Install package dependencies:

In [ ]:

! sudo apt-get install boto ! sudo apt-get install git

Clone the s3-parallel-put repo:

In [ ]:

! git clone https://github.com/twpayne/s3-parallel-put.git

Setup AWS keys for s3-parallel-put:

In [ ]:

! export AWS_ACCESS_KEY_ID = XXX ! export AWS_SECRET_ACCESS_KEY = XXX

Sample usage:

In [ ]:

! s3-parallel-put --bucket = bucket --prefix = PREFIX SOURCE

Dry run of putting files in the current directory on S3 with the given S3 prefix, do not check first if they exist:

In [ ]:

! s3-parallel-put --bucket = bucket --host = s3.amazonaws.com --put = stupid --dry-run --prefix = prefix/ ./

S3DistCp

S3DistCp is an extension of DistCp that is optimized to work with Amazon S3. S3DistCp is useful for combining smaller files and aggregate them together, taking in a pattern and target file to combine smaller input files to larger ones. S3DistCp can also be used to transfer large volumes of data from S3 to your Hadoop cluster.

To run S3DistCp with the EMR command line, ensure you are using the proper version of Ruby:

In [ ]:

! rvm --default ruby-1.8.7-p374

The EMR command line below executes the following:

Create a master node and slave nodes of type m1.small
Runs S3DistCp on the source bucket location and concatenates files that match the date regular expression, resulting in files that are roughly 1024 MB or 1 GB
Places the results in the destination bucket

In [ ]:

! ./elastic-mapreduce --create --instance-group master --instance-count 1 \ -- instance - type m1 . small -- instance - group core -- instance - count \ -- instance - type m1 . small -- jar / home / hadoop / lib / emr - s3distcp - 1.0 . jar \ -- args "--src,s3://my-bucket-source/,--groupBy,.*([0-9]{4}-01).*, \ --dest,s3://my-bucket-dest/,--targetSize,1024"

For further optimization, compression can be helpful to save on AWS storage and bandwidth costs, to speed up the S3 to/from EMR transfer, and to reduce disk I/O. Note that compressed files are not easy to split for Hadoop. For example, Hadoop uses a single mapper per GZIP file, as it does not know about file boundaries.

What type of compression should you use?

Time sensitive job: Snappy or LZO
Large amounts of data: GZIP
General purpose: GZIP, as its supported by most platforms

You can specify the compression codec (gzip, lzo, snappy, or none) to use for copied files with S3DistCp with outputCodec. If no value is specified, files are copied with no compression change. The code below sets the compression to lzo:

In [ ]:

-- outputCodec , lzo

Redshift

Copy values from the given S3 location containing CSV files to a Redshift cluster:

In [ ]:

copy table_name from 's3://source/part' credentials 'aws_access_key_id=XXX;aws_secret_access_key=XXX' csv ;

Copy values from the given location containing TSV files to a Redshift cluster:

In [ ]:

copy table_name from 's3://source/part' credentials 'aws_access_key_id=XXX;aws_secret_access_key=XXX' csv delimiter ' \t ' ;

Light

Font size:

↓

↑

Reset

Interval:

↓

↑

Bookmark:

Make

Similar books «Data Science iPython Notebook 20160913»

Look at similar books to Data Science iPython Notebook 20160913. We have selected literature similar in name and meaning in the hope of providing readers with more options to find new, interesting, not yet read works.

it-ebooks

The Art of Data Science

it-ebooks

Technical Notes On Using Data Science & Artificial Intelligence

it-ebooks

R for Data Science

it-ebooks

Tiny Python & ES6 Notebook

it-ebooks

An Introduction to Data Science

McKinney

Python for Data Analysis: Data Wrangling with Pandas, NumPy, and IPython 2nd Edition

Martins

IPython notebook essentials: compute scientific data and execute code interactively with NumPy and SciPy

Dan Toomey

Jupyter for Data Science

McKinney

Python for data analysis : Data wrangling with Pandas, Numpy, and IPython

Wes McKinney

Python for Data Analysis: Data Wrangling with Pandas, NumPy, and IPython

Cyrille Rossant

IPython Interactive Computing and Visualization Cookbook

Cyrille Rossant

Learning IPython for Interactive Computing and Data Visualization

Reviews about «Data Science iPython Notebook 20160913»

Discussion, reviews of the book Data Science iPython Notebook 20160913 and just readers' own opinions. Leave your comments, write what you think about the work, its meaning or the main characters. Specify what exactly you liked and what you didn't like, and why you think so.