Less Hassle, More Data

Subtitle: Tips for scraping an infinite scrolling website

Quick Tips

  1. Go to your target website (infinite scrolling website)
  2. Select either LISTLY WHOLE or LISTLY PART
  3. Click Edit Setting on your Databoard
  4. Increase Repeat N Times as much as you want
  5. Click Refresh and get more data

Imagine yourself trying to do market research to start a phone case business. You might first want to collect some information about bestselling items from your competitors to analyze market trends. Here are some tips to help you grab data out of your competitor websites in just a few clicks!

We’ll take Casetify as an example.

Castify.png
Castify-infinite-scrolling.png

As you can see, this is a long, infinite scrolling website where product information appears as you scroll down the page.

If you scrape this website in the first place, you’ll only see a limited number of products collected.

Less-Hassle-More-Data-1.png

In order to load more data and collect them, we can simply tweak the settings by clicking add setting button.

Less-Hassle-More-Data-2.png

Increase Repeat N times to 5 and hit the refresh data button on your Databoard.

Less-Hassle-More-Data-3.png
Less-Hassle-More-Data-4.png

You’ll probably be able to find more items were collected after changing the setting.

Less-Hassle-More-Data-5.png

Then, why not increase it up to 20?

Less-Hassle-More-Data-6.png

Voila, now you can handle infinite scrolling and collect all the items out of the infinite scrolling website by automating this step.

Less-Hassle-More-Data-7.png

Web scraping doesn’t have to be hard.

Interested in learning more about automated web scraping? Check out here!