Mikhail Klassen - Mining the Social Web

In this chapter, well ease into the process of getting situated with a minimal (but effective) development environment with Python, survey Twitters API, and distill some analytical insights from tweets using frequency analysis. Topics that youll learn about in this chapter include:

• Twitters developer platform and how to make API requests

• Tweet metadata and how to use it

• Extracting entities such as user mentions, hashtags, and URLs from tweets

• Techniques for performing frequency analysis with Python

• Plotting histograms of Twitter data with the Jupyter Notebook

Most chapters wont open with a reflective discussion, but since this is the first chapter of the book and introduces a social website that is often misunderstood, it seems appropriate to take a moment to examine Twitter at a fundamental level.

How would you define Twitter?

There are many ways to answer this question, but lets consider it from an overarching angle that addresses some fundamental aspects of our shared humanity that any technology needs to account for in order to be useful and successful. After all, the purpose of technology is to enhance our human experience.

As humans, what are some things that we want that technology might help us to get?

• We want to be heard.

• We want to satisfy our curiosity.

• We want it easy.

• We want it now.

In the context of the current discussion, these are just a few observations that are generally true of humanity. We have a deeply rooted need to share our ideas and experiences, which gives us the ability to connect with other people, to be heard, and to feel a sense of worth and importance. We are curious about the world around us and how to organize and manipulate it, and we use communication to share our observations, ask questions, and engage with other people in meaningful dialogues about our quandaries.

The last two bullet points highlight our inherent intolerance to friction. Ideally, we dont want to have to work any harder than is absolutely necessary to satisfy our curiosity or get any particular job done; wed rather be doing something else or moving on to the next thing because our time on this planet is so precious and short. Along similar lines, we want things now and tend to be impatient when actual progress doesnt happen at the speed of our own thought.

One way to describe Twitter is as a microblogging service that allows people to communicate with short messages that roughly correspond to thoughts or ideas. Historically, these tweets were limited to 140 characters in length, although this limit has been expanded and is liable to change again in the future. In that regard, you could think of Twitter as being akin to a free, high-speed, global text-messaging service. In other words, its a piece of valuable infrastructure that enables rapid and easy communication. However, thats not all of the story. Humans are hungry for connection and want to be heard, and Twitter has 335 million monthly active users worldwide expressing ideas, communicating directly with each other, and satisfying their curiosity.

Besides the macro-level possibilities for marketing and advertisingwhich are always lucrative with a user base of that sizeits the underlying network dynamics that created the gravity for such a user base to emerge that are truly interesting, and thats why Twitter is all the rage. While the communication bus that enables users to share short quips at the speed of thought may be a necessary condition for viral adoption and sustained engagement on the Twitter platform, its not a sufficient condition. The extra ingredient that makes it sufficient is that Twitters asymmetric following model satisfies our curiosity. It is the asymmetric following model that casts Twitter as more of an interest graph than a social network, and the APIs that provide just enough of a framework for structure and self-organizing behavior to emerge from the chaos.

In other words, whereas some social websites like Facebook and LinkedIn require the mutual acceptance of a connection between users (which usually implies a real-world connection of some kind), Twitters relationship model allows you to keep up with the latest happenings of any other user, even though that other user may not choose to follow you back or even know that you exist. Twitters following model is simple but exploits a fundamental aspect of what makes us human: our curiosity. Whether it be an infatuation with celebrity gossip, an urge to keep up with a favorite sports team, a keen interest in a particular political topic, or a desire to connect with someone new, Twitter provides you with boundless opportunities to satisfy your curiosity.