Week 08

Scraping + APIs

Lectures Notes:

• Parsing Unstructured Data: Scraping Static Websites: html – Jupyter Notebook

Slides

• Parsing Unstructured Data: Scraping Static Websites

Readings

• On HTMLs

  • tags

  • Anchors and Elements

  • Inspecting HTML elements

  • CSS Selector Reference - w3schools.com

• Selector Gadget

• Web Scraping

  • Tutorial: A Practical Introduction to Web Scraping in Python
  • Tutorial: Python Web Scraping Using BeautifulSoup - dataquest.io
  • Ultimate Guide to Web Scraping with Python Part 1: Requests and BeautifulSoup - LearnDataSci - Martin, Brendan
  • On the legality of webscraping
    • Web Scraping and Crawling are Perfectly Legal, Right?
  • robots.txt

• Changes in the APIs

  • https://inews.co.uk/news/twitter-researchers-delete-data-unless-pay-2364535

  • https://www.reddit.com/r/modnews/comments/134tjpe/reddit_data_api_update_changes_to_pushshift_access/

  • https://www.tiktok.com/legal/page/global/terms-of-service-research-api/en

  • https://www.bloomberg.com/news/articles/2022-06-23/meta-pulls-support-for-tool-used-to-keep-misinformation-in-check?leadSource=uverify%20wall