In the class today, we will learn how to collect digital data through APIs. We will focus on:
# setup
import requests
import os
import pandas as pd
The famous acronym API stands for “Application Programming Interface”. An API is an online server allows different applications to interact. Most often for our purposes, an API will facilitate information exchange between data users and the holders of certain data. Many companies build these repositories for various functions, including sharing data, receiving data, joint database management, and providing artificial intelligence functions or machines for public use.
Let's think of an example capable of motivating the creation of an API. Imagine you own Twitter. You would have zillions of hackers every day trying to scrape your data, this would make your website more unstable and insecure. What is a possible solution? You create an API, and you control who accesses the information, when they access it, and what type of information you make available. Another option is to close you API and restrict data access to researchers. But, if you do this, you are likely to pay a reputational cost for not being transparent, and users might leave your platform.
Have you ever watched Matrix? APIs are just like that! In the movies, Neil and others would physically connect their mindes to a super developed server and ask to learn a certain skill - kung-fu, programming, language, etc. This is exactly what an API does. You connect to the website and request data, and receive it in return. It's like sending an email, but doing everything via programming language.
There are two main ways in which we academics commonly use APIs.
Access data shared by Companies and NGOs.
Process our data in Algorithms developed by third parties.
Our focus will be on the first. Later, we will see how to use the ChatGPT API for text classification tasks.
An API is just an URL. See the example below:
http://mywebsite.com/endpoint?key¶m_1¶m_2
Main Components:
In order to work with APIs, we need tools to access the web. In Python, the most common library for making requests and working with APIs is the requests library. There are two main types of requests:
get(): to receive information from the API -- which we will use the most for web data collection
post(): to send information to the API -- think about the use of ChatGPT for classification of text.
Let's start querying our first API. We will start with the simple Open Trivia API. This is a very simple API, and serves the purpose of learning all the basic steps of querying APIs. The Open Trivia API gives you ideas for your trivia games!
The Trivia API:
When querying an API, our work will often involve the following steps:
Before we start querying an API, we always need to read through the documentation/reference. The documentation often revel to us:
https://opentdb.com/api.phpamountcategoryNotice one thing here. The Trivia API requires you to gave the amount filter in your call. Not all APIs are like this. Some have a random api endpoint for you to play around
# build query
query = "https://opentdb.com/api.php?amount=1"
requests.get(querystring) to call the API¶To interact with the API, we will use the requests package. The requests package allow us to send a HTTP request to the API. Because we are intereste in retrieving data, we will mostly be working with the .get() method, which requires one argument — the URL we want to make the request to.
When we make a request, the response from the API comes with a response code which tells us whether our request was successful. Response codes are important because they immediately tell us if something went wrong.
To make a ‘GET’ request, we’ll use the requests.get() function, which requires one argument — the URL we want to make the request to. We’ll start by making a request to an API endpoint that doesn’t exist, so we can see what that response code looks like
# Make a get request to get the latest position of the ISS from the OpenNotify API.
response = requests.get(query)
type(response)
When we make a request, the response from the API comes with a response code which tells us whether our request was successful. Response codes are important because they immediately tell us if something went wrong. Here is a list of response codes you can get
200 — Everything went okay, and the server returned a result (if any).
301 — The server is redirecting you to a different endpoint. This can happen when a company switches domain names, or when an endpoint's name has changed.
401 — The server thinks you're not authenticated. This happens when you don't send the right credentials to access an API.
400 — The server thinks you made a bad request. This can happen when you don't send the information that the API requires to process your request (among other things).
403 — The resource you're trying to access is forbidden, and you don't have the right permissions to see it.
404 — The server didn't find the resource you tried to access.
# check status code
status_code = response.status_code
# print status code
status_code
With an 200 code, we can access the content of the get request. The return from the API is stored as a content attribute in the response object.
print(response.content)
The deafault data type we receive from APIS are in the JSON format. This format encodes data structures like lists and dictionaries as strings to ensure that machines can read them easily.
For that kind of content, the requests library includes a specific .json() method that you can use to immediately convert the API bytes response into a Python data structure, in general a nested dictionary.
# convert the get output to a dictionary
response_dict = response.json()
print(response_dict)
# index just like a dict
response_dict["results"][0]["question"]
# convert to a dataframe
import pandas as pd
# need to convert to a list for weird python reasons
pd.DataFrame([response_dict["results"][0]])
Let's see the full code:
# full code
import requests
import pandas as pd
# build query
query = "https://opentdb.com/api.php?amount=1"
#
response = requests.get(query)
# check status code
status_code = response.status_code
# move forward with code
if status_code==200:
# convert the get output to a dictionary
response_dict = response.json()
# convert to a dataframe
res = pd.DataFrame([response_dict["results"][0]])
else:
print(status_code)
# print the activity
res
If we look at the documentation, you see the APIs provides filters (query parameters) that allow you to refine your search.
For example, when you send a get request to the Youtube API, you are not interested in the entire Youtube data. You want data associated with certain videos, profiles, for a certain period of time, for example. These filters are often embedded as query parameters in the API call.
To add a query parameter to a given URL, you have to add a question mark (?) before the first query parameter. If you want to have multiple query parameters in your request, then you can split them with an ampersand (&)
We can add filters by:
constructing the full API call
Using dictionaries
## get only recreational activities
# build query
query = "https://opentdb.com/api.php"
# add filter
activity = "?amount=10"
# full request
url = query + activity
# Make a get request
response = requests.get(url)
# see json
response.json()
## get only recreational activities
# build query
query = "https://opentdb.com/api.php"
# add filter
parameters = {"amount": "10",
"category":"9"}
# Make a get request to get
response = requests.get(query, params=parameters)
# see json
print(response.status_code)
response.json()
See... it is the same url..
response.url
pd.DataFrame(response.json()["results"])
Let's transition now to a more complex, and with interesting data, API. We will work with the Yelp API.
This API:
See the documentation for the API here. The API has some interesting endpoints, for example:
/businesses/search - Search for businesses by keyword, category, location, price level, etc./businesses/{id} - Get rich business data, such as name, address, phone number, photos, Yelp rating, price levels and hours of operation./businesses/{business_id_or_alias}/reviews - Get up to three review excerpts for a business.Most often, the provider of an API will require you to authenticate before you can get some data. Authentication usually occures through an access token you can generate directly from the API. Depending on the type of authentication each API have in place, it can be a simple token (string) or multiple different ids (Client ID, Access Token, Client Token..)
Keep in mind that using a token is better than using a username and password for a few reasons:
Typically, you'll be accessing an API from a script. If you put your username and password in the script and someone finds it, they can take over your account.
Access tokens can have scopes and specific permissions.
To authorize your access, you need to add the token to your API call. Often, you do this by passing the token through an authorization header. We can use Python's requests library to make a dictionary of headers, and then pass it into our request.
Information about acquiring your credentials to make API call are often displayed in the API documentation.
Every API has a bit of a distinct process. In general, APIs require you to create an app to access the API. This is a bit of a weird terminology. The assumption here is that you are creating an app (think about the Botometer at Twitter) that will query the API many times.
For the YELP API, after you create the app, you will get an Client ID and an API KEY
API keys are personal information. Keep yours safe, and do not paste into your code.
Don't do this:
api_key = "my_key"
Do this:
I will show you in class what a .env file looks like.
We repeat the same steps as before, but adding an authentication step.
# load library to get environmental files
import os
from dotenv import load_dotenv
# load keys from environment variables
load_dotenv() # .env file in cwd
# Print all environment variables
#for key, value in os.environ.items():
# print(key, "=", value)
yelp_client = os.environ.get("yelp_client_id")
yelp_key = os.environ.get("yelp_api_key")
# OR JUST HARD CODE YOUR API KEY HERE. NOT A GREAT PRACTICE!!!
#yelp_key = ""
# save your token in the header of the call
header = {'Authorization': f'Bearer {yelp_key}'}
# see here
header["Authorization"]
We will query the /businesses/search endpoint. Let's check together the documentation here: https://docs.developer.yelp.com/reference/v3_business_search
We will use two parameters:
# endpoint
endpoint = "https://api.yelp.com/v3/businesses/search"
# Add as parameters
params ={"location":" Washington, DC 20057",
"term":"best noodles restaurant"}
# Make a get request with header + parameters
response = requests.get(endpoint,
headers=header,
params=params)
Let's check the response code
# looking for a 200
response.status_code
# What does the response look like?
yelp_json = response.json()
# print
print(yelp_json)
yelp_json.keys()
yelp_json["businesses"]
It returns a long dictionary with the key "businesses" and a list with multiple sub-entries.
How to deal with this data?
# convert to pd
df_yelp = pd.DataFrame(yelp_json["businesses"])
# see
df_yelp
# not looking realy bad.
Assume you are interested in the id, name, url, lat and long, and rating
# function to clean and extract information from yelp
def clean_yelp(yelp_json):
'''
function to extract columns of interest from yelp json
'''
# create a temporary dictionary to store the information
temp_yelp = {}
# collect information
temp_yelp["id"]= yelp_json["id"]
temp_yelp["name"]= yelp_json["name"]
temp_yelp["url"]= yelp_json["url"]
temp_yelp["latitude"] = yelp_json["coordinates"]["latitude"]
temp_yelp["longitude"] = yelp_json["coordinates"]["longitude"]
temp_yelp["rating"]= yelp_json["rating"]
# return
return(temp_yelp)
# apply to the dictionary
results_yelp = [clean_yelp(entry) for entry in yelp_json["businesses"]]
# Convert results to dataframe
yelp_df = pd.DataFrame(results_yelp)
yelp_df.shape
Remember to always save your response from the API call. You don't want be querying the API all the time to grab the same data.
import json
with open("yelp_results.json", 'w') as f:
# write the dictionary to a string
json.dump(response.json(), f, indent=4)
Make a successful query using your favorite type of food to the Yelp API. Pretty much I only want you to repeat what we did before, but changing the search term a bit
# code here
Now let's move to our last example.
We will be working with the YouTube API. This is a complex API, but lucky for us some other programmers already created a Python wrapper to access the API. We will use the youtube-data-api library which contains a set of functions to facilitate the access to the API.
Youtube has a very extensive api. There are a lot of data you can get access to. See a compreensive list here
What is included in the package:
The software is on PyPI, so you can download it via pip
#!pip install youtube-data-api
You need a Google Account to access the Google API Console, request an API key, and register your application. You can use your GMail account for this if you have one.
Create a project in the Google Developers Console and obtain authorization credentials so your application can submit API requests.
After creating your project, make sure the YouTube Data API is one of the services that your application is registered to use.
a. Go to the API Console and select the project that you just registered.
b. Visit the Enabled APIs page. In the list of APIs, make sure the status is ON for the YouTube Data API v3. You do not need to enable OAuth 2.0 since there are no methods in the package that require it.
# call some libraries
import os
import datetime
import pandas as pd
#Import YouTubeDataAPI
from youtube_api import YouTubeDataAPI
from youtube_api.youtube_api_utils import *
from dotenv import load_dotenv
# load keys from environmental var
load_dotenv() # .env file in cwd
api_key = os.environ.get("YT_KEY")
# create a client
# this is what we call: instantiate the class
yt = YouTubeDataAPI(api_key)
print(yt)
Let's start with the LastWeekTonight channel
https://www.youtube.com/user/LastWeekTonight
First we need to get the channel id
channel_id = yt.get_channel_id_from_user('LastWeekTonight')
print(channel_id)
# collect metadata
yt.get_channel_metadata(channel_id)
pd.DataFrame(yt.get_subscriptions(channel_id))
You first need to convert the channel_id into a playlist id to get all the videos ever posted by a channel using a function from the youtube_api_utils in the package. Then you can get the video ids, and collect metadata, comments, among many others.
from youtube_api.youtube_api_utils import *
playlist_id = get_upload_playlist_id(channel_id)
print(playlist_id)
## Get video ids
videos = yt.get_videos_from_playlist_id(playlist_id)
df = pd.DataFrame(videos)
print(df)
# id for videos as a list
df.video_id.tolist()
#grab metadata
video_meta = yt.get_video_metadata(df.video_id.tolist()[:5])
#visualize
pd.DataFrame(video_meta)
## Collect Comments
ids = df.video_id.tolist()[:5]
ids
# loop
list_comments = []
for video_id in ids:
comments = yt.get_video_comments(video_id, max_results=10)
list_comments.append(pd.DataFrame(comments))
# concat
df = pd.concat(list_comments)
df.head()
The youtube API also allows you to search for most popular videos using queries. This is very cool!
df = pd.DataFrame(yt.search(q='they are eating the dogs', max_results=10))
df.keys()
df
Some cool research using the Youtube API:
!jupyter nbconvert _week-08_apis.ipynb --to html --template classic