The Process of Web Scraping

If you are taking your first step to web scraping, you might wonder what is involved in collecting data using a web scraping tool. This post will walk you through how data is fetched from a website and what are the most common browser errors you might encounter in the process of web scraping.

Step 1: Navigate to a website you want to scrape

First things first, you’ll probably need to go to a target website to collect data from. In this step, you may find yourself in trouble with some browser errors. These errors generally occur due to IP address blocking, failed authentication attempts including CAPTCHAs, login failures, or in case something has gone wrong on the target website’s server. In such a case, it is recommended that you check if your IP address is being blocked or your target website requires a login.

Related Errors

NoConnectionError

Step 2. Select specific element you’d like to collect

Next up is depending on what you want to grab. If there is no element to extract from your target website, your browser will give you NoElementError, which is one of the most common errors. Sometimes, a website loads slowly due to too much traffic and this can also lead to time out related errors: TimeoutException and SoftTimeLimitExceeded.

Unfortunately, there is nothing much you can do in this step. That said, you can check out if your target element is missing and reselect the element to fix the error.

Related Errors

NoElementError

NoListlyContent

JavascriptException

SoftTimeLimitExceeded

TimeoutException

Extra Tip

You can check out a snapshot of the error message on Listly’s Databoard.

Let’s say you’re trying to collect the 12-month payment information for each item, highlighted in blue, as below.

As you might have noticed, the information is not seen on some of other product detail pages. In this case, the specific content you’d like to scrape is literally not there so you end up running into the NoElementError.

Step 3. Uncover patterns and and scrape data from a website

Now, you’re only a couple clicks away! Just wait for seconds while the scraping tool is extracting data from a website. Here, Listly will visit a designated website and find repeated patterns to fetch data — Listly does many things behind the scenes.

Step 4. Save data in an Excel spreadsheet or some other structured formats

The last step is to download your data into Excel or any other structured formats you want. You can even connect your data to Google Sheets using an API.

Before you go, just one more thing!

As you might have noticed in this post, there is very little chance that you need to handle the errors that are not listed above. That being said, let us know any further questions or errors you have by leaving a comment below, we are always trying to improve!