How to Scrape Google Search Results for Free [2024 Edition]

Ever needed Google SERP data for hundreds of keywords at once? Traditional SEO tools can’t do it, and it's frustrating.

In this article, we’ll learn how to extract all Google search results from hundreds of keywords for free.

But is it useful for SEOs only? Let’s explore some other vital use cases of scraping Google SERPs.

Why scrape Google search results?

This makes the most visited website in the world a goldmine of data for various use cases, including:

1. Market Research: You can identify competitor presence, market gaps, and consumer sentiment.
2. SEO (Search Engine Optimization): It can help you create competitor strategies, identify keyword opportunities, and track content performance.
3. Academic Research: Google SERP scraper can be used for collecting all available content on a topic from Google search.
4. Journalism: Journalists can use it for research, finding articles, social media posts, and all content found in Google related to a topic.
5. Fact-Checking: You can save time skipping manual search and find the facts faster.

But does Google allow web scraping? Let’s find out.

Is it legal to scrape Google SERP?

Hence scraping Google search results is legal. But you can get in trouble for copyright infringement if you reuse copyrighted content without consent.

But Google has tons of APIs, why scrape it if we can use the official API?

Is there any official Google search API?

1. It’s extremely expensive
2. You don’t get actual search results always
3. Limits and pricing keeps changing

Let’s elaborate on each limitation.

Firstly,the official Google API is extremely expensive.

You can only perform 100 requests per day for free. With a paid plan, the API allows only 10k requests per day and you have to pay $5 per 1000 requests.

Other than Google official API, we can also use 3rd party SERP APIs. There are a lot of them available. But again, they’re expensive and require coding.

So why not just code a scraper without any API?

If you’re a good programmer, you can scrape Google search using python or any other programming language. But it’s hectic, complex, and non sustainable.

So for peeps like me who don’t have a fortune to spend and don’t like coding, both these solutions are useless. The only solution we have is using a no-code tool.

How to scrape Google search results without coding

There are a lot of no–code web scraping tools. But for this article, I’ll be using the one that I believe is best in the market – Google Search Scraper by Lobstr.io.

Features

1. 13 data attributes including PPA, related searches, and rich snippets
2. All countries, regions, and languages
3. Ads and omitted results included
4. Super fast, scrape 130+ results per minute
5. Cloud-based and has cool schedule feature
6. Custom exports, including Google Sheets, Amazon S3, SFTP, and webhook
7. Developer-ready API

Pricing

Lobstr offers a transparent and affordable pricing range that suits every pocket.

1. Free plan: Free forever, 58k results per month
2. Premium plan: €0.07 per 1000 results
3. Business plan: €0.04 per 1000 results
4. Enterprise plan: €0.03 per 1000 results

Let’s learn how to scrape Google search results using Lobstr.io

Step by step tutorial to scrape Google SERP using Lobstr.io

We’ll scrape Google search results within a few minutes in 6 simple steps.

1. Get Google SERP URL
2. Create new squid
3. Add tasks
4. Adjust behavior
5. Launch
6. Enjoy

Let’s go! 🏃

1. Get Google SERP URL

To scrape Google SERP data, all you need is a SERP URL. How do we get that? Let me hit 2 birds with 1 arrow. I’ll track how our latest article is performing in different regions in the US.

We’ll track its SERP position in New York, California, and Texas. Let’s enter our search query in Google and copy the URL.

2. Create new squid

To launch Google Search scraper, click the create new squid button on the dashboard. Now search “Google search” and select Google Search Scraper.

Tada! ✨Your scraper is ready.

3. Add tasks

You can add as many tasks as you want. Once done, click the Save button to proceed to settings.

4. Adjust behavior

Let’s adjust our scraper’s behavior using settings. We have 3 options to adjust our Squid’s behavior.

1. Basic settings
2. Advanced settings
3. Notifications

Basic Settings

Firstly, we have basic settings. You can specify how many pages you want to scrape using the Max Pages option.

Leave it blank to scrape all available search result pages. You can toggle Mobile Results to scrape mobile search results.

Next is – When to end run. Select end run once no credit left to stop scraping when you’ve consumed your daily credits. This is best for getting fresh data every time you initiate a run.

If you’ve got hundreds of URLs to scrape, select end run once all tasks consumed. This will pause the run once you’ve used all the credits and resume it the next day.

Since we only have 1 URL and we need fresh data on every launch, I’ll choose the first option.

Advanced settings

With that, we move to advanced settings. First option is Results per page, which allows you to set number of results displayed per page. It’s not so useful in our case.

The second option is where it gets interesting. This is where we choose the country, language, and specific location of our SERP data.

I need to know what position my article is ranking in New York. Let’s adjust our boi.

Next is concurrency i.e. number of bots deployed per run. You can deploy multiple bots for faster data extraction.

Check the pricing page to learn about the number of bots your plan includes. For example: with premium plan, you can either create 10 Squids or deploy 10 bots on a single Squid.

You can remove duplicate results using the Unique Results filter. If you prefer Excel for viewing data, toggle No Line Breaks to remove line breaks from text fields.

Once done with settings, click save and you’ll see the notifications menu.

Notifications

You can opt to receive email notifications upon success or failure of a run.

You don’t have to open your dashboard to know how your last run ended. Select On success to receive an email notification when the run ends successfully. Choose On error for vice versa.

With that, we’re ready to launch our scraper.

5. Launch

This is our final step. Here we configure our launch sequence 🚀 and boom!!! Google Search Scraper starts collecting data. There are 2 ways to launch the Squid:

1. Manually
2. Repeatedly

If you want to launch the scraper instantly, select Manually and click Save & Extract button. Next thing you see is a live console for monitoring progress and real-time results.

But what if I want to track the SERP position daily or weekly? I obviously want to do that. That’s where the schedule feature will help us.

Schedule

You can schedule your Squid to run automatically on time and frequency of your choice. To do that, select Repeatedly and set a schedule.

I’m scheduling my Squid to run every Monday at 12PM. This way I can track weekly changes in rankings.

6. Enjoy

Since I’ve to track 3 states, I created 2 more Squids to track Florida and California. They’ll all run on scheduled time every Monday.

But I can’t wait till Monday. After launching all 3 squids manually. Let’s download the .csv files and see the results.

Starting with California, our scraper collected 352 pages, and we rank #8 in SERP woohoo!!

In Texas, Google gave us 268 search results, and our boi ranks at #8, again ✨

And finally New York, the region we targeted specifically; 228 total results, and we rank… #6. That’s some great news.

So the tool is awesome, but I prefer Google Sheets. Downloading a csv file, then importing it to Sheets is hectic. Is it possible to sync a Google Sheet to our Squid?

Delivery

Lobstr’s all Squids can be synced to Google Sheets, Amazon S3, SFTP, and Webhook. To do this, just click the delivery button in the console and configure your preferred service.

Now you don’t have to open the dashboard and download results manually. They’ll be updated to your preferred service automatically after every run.

But is it actually better than SEO tools? Let’s find out!

How is Lobstr’s Google Search Scraper better than SEO tools?

But why are they the best? Well instead of searching for “best seo tool”, I prefer reading consumers’ responses. Here is 1 TwitterX polls for example:

Let’s draw a side-by-side comparison first:

Ahrefs rank tracker doesn’t support local rank tracking. You can’t see your ranking in a particular city or state.

This can often mislead you while doing competitor research or local optimization. Let’s understand it with an example.

Here I added my targeted keyword in the rank tracker. It says we’re ranking on #10 in the US. But this seems incorrect 🤔. We just confirmed we’re ranking #6 in New York.

Lobstr allows you to choose the city, state, and language you’re targeting. Giving you accurate rankings for local SEO optimization.

Second issue with Ahref is outdated results.

Lobstr provides real-time SERP data from Google. Every time you initiate a run, the results you get are exactly what your audience would see when they search that keyword.

Another issue with Ahrefs is – it’s expensive.

The basic plan costs $99 per month. Plus you’ll have to pay extra $100 per month for daily rank updates. Adding additional keywords costs you extra $50 per month.

Lobstr costs you only €50 per month. You can track unlimited keywords. Plus get real-time SERP updates. No extra charges.

Let’s see how Semrush’s rank tracker performs.

Semrush allows you to target a specific city or even a specific zip code. Its data is also updated daily.

This was further validated by other users in comments:

But the main issue is – It’s the most expensive tool.

Pro plan costs you $130 per month. You can only create 5 projects and each project supports only 1 location. This means you can track 5 locations.

It also limits total keywords. You can only monitor a maximum of 500 keywords in Pro plan.

It’s way more expensive if you compare it to Lobstr.io which allows unlimited keywords in all plans. Plus with Lobstr’s pro plan, you can track up to 10 locations.

But Lobstr also has a limitation. Let’s discuss that too.

Limitation

While Lobstr.io is the best SERP in the market, it does have a limitation. Google shows a max 500 results per search, so the scraper can only extract up to 500 results per URL.

How to bypass the max Google results limitation?

You can split the search into multiple URLs using related keywords.

1. We can use keywords like scraping tiktok ads free, scrape tiktok ads, etc.
2. You can also use Google advanced search and custom time range.

FAQs

What is SERP?

SERP stands for Search Engine Results Page. It's the page you see after you enter a query into a search engine like Google. In our context, it’s the Google search results page.

Can I scrape Google ads too?

Yes, this scraper extracts both paid and organic results. This means you can scrape Google ads too.

Can I get SEO data like Keyword Difficulty, Volume, etc?

No, Google doesn’t display volume and KD in SERP. In fact keyword difficulty is an estimate by SEO tools. So, no you won’t get these details. This tool is to scrape data from SERP.

Does this scraper also extract People Also Ask and related searches?

Yes, you get People Also Ask and Related Searches data too. Which makes it a perfect choice for keyword research.

Can I scrape Google News with this tool?

Can I scrape Google Maps with this tool?

Can I use AI to scrape Google?

Unfortunately No. If you want to scrape Google SERP data at scale, there’s no reliable AI solution available yet. But you can use GPT to help you code a scraper.

Do I need proxies for scraping Google SERPs?

No, with Lobstr, you don’t need proxies. We’ve got you covered. But yes, if you’re creating your own scraper, you may need proxies.

Conclusion

That’s a wrap on how to scrape Google search results without coding. You can use Lobstr.io Google Search Scraper not only for data collection but also a perfect, affordable, accurate SEO tool.

Go on, create a free account and give it a spin.