Web Scraping - Jerry Ng

Sign in Subscribe

Web Scraping

It's... data liberation?

How I Saved Scraped Data in an SQLite Database on GitHub

Some time back, I stumbled upon this amazing idea of Git scraping made popular by Simon Willison. It involves using GitHub Actions to scrape data and save it directly into a Git repository. Essentially, for each new set of data scraped, a new commit is created in your repository. The

Michelin Chef Plating Dishes

How I Scraped Michelin Guide Using Golang

At the beginning of the automobile era, Michelin, a tire company, created a travel guide, including a restaurant guide. Through the years, Michelin stars have become very prestigious due to their high standards and very strict anonymous testers. Michelin Stars are incredibly coveted. Gaining just one can change a chef&

How To Deploy Python Scrapy Spiders for Free on Cloud

Running Scrapy spiders in your local machine can be very convenient during the early stage of development. But, we would eventually need to deploy and run our spiders in the cloud periodically. In search of free options for running periodic crawling jobs, I have come across a few options available

5 Useful Tips While Working With Python Scrapy

Today, there are a handful of Python Scrapy tutorials out there. However, I find that most of the tutorials do not cover these low-hanging fruits that can greatly improve the developer experience of web scraping. In this post, I will be sharing with you some quick and easy tips that

4 Web Scraping Challenges To Look Out For

Today, with the help of numerous open-sourced tools, libraries, frameworks, and many more web scraping solutions, web scraping can seem easier than ever. While that is true, it can be foolish for someone to not expect any challenges while trying to scrape data from the web. For those who are

Written by human. Hosted on Digital Ocean.