Week 2.1 OSM APIs and tools for data collection

Summary

The instructor introduces Week 2’s topic: tools and frameworks for collecting and working with online social media data. The lecture explains how APIs connect programs to social media services, how returned data is formatted and stored, and which tools can be used to inspect it.

Course reminders and recap

• Students are encouraged to read existing forum questions and answers before posting, and to help answer other students’ questions.
• Assignment 1 is due during the second week’s weekend. The instructor describes it as a straightforward set of questions based on the slides and tutorials.
• The course has covered:
  • Types of social networks and social media content, including ephemeral and anonymous networks.
  • The large volume of content generated on social platforms.
  • The “5 Vs” of social media: volume, velocity, veracity, variety, and value.
  • Real-world effects and risks, such as compromised accounts, misleading content, and consequences for users.
  • Introductory Linux and Python. Windows can also be used, although course support will focus mainly on Linux.

APIs and data collection

An API (Application Programming Interface) lets a program communicate with a social media service. A program sends a request for data, and the service returns the data that the API permits it to provide. The lecture focuses mainly on Facebook and Twitter APIs, while noting that many other platforms also offer APIs.

• Rate limits restrict how much data can be collected. The amount available depends on the service and the type of data requested.
• The course will use API examples to illustrate how requests work and what data may be accessible.

Programming and data formats

• Python is introduced as a popular language for making API requests and collecting data. Its libraries can help with URLs, parsing, APIs, and JSON.
• APIs commonly return data in JSON (JavaScript Object Notation); some services also use XML (Extensible Markup Language).
• A JSON viewer can make a returned block of JSON easier to inspect by displaying its fields and structure. The instructor mentions jsonviewer.stack.hu as an example.

Storing and inspecting collected data

The lecture outlines a basic workflow:

1. Request data through an API.
2. Use a programming language such as Python to collect and process the response.
3. Store the data in a database:
   • MySQL is a relational database that organizes data in rows and columns and supports queries.
   • MongoDB is another database option mentioned for storing collected data.
4. Inspect the stored data using a suitable interface:
   • phpMyAdmin for MySQL.
   • RoboMongo for MongoDB.

The course will introduce only basic database use and queries; it is not intended to be a full course on MySQL.

Facebook’s graph model

Facebook represents content and activity as a graph:

• Users, pages, posts, photos, videos, and status updates are represented as nodes.
• Connections and interactions—such as friendships, likes, and comments—are represented as edges.
• Nodes have unique numeric IDs.

The instructor connects this model to Facebook’s Graph API. Upcoming tutorials will cover using the API, creating a secret key, authentication, and the kinds of data that can be collected.

Speakers and sources featured

• Speaker: The course instructor, speaking for the Privacy and Security in Online Social Media course.
• Platforms and tools mentioned: Facebook and its Graph API; Twitter; Python; JSON; XML; MySQL; MongoDB; phpMyAdmin; RoboMongo; and JSON Viewer (jsonviewer.stack.hu).