How to download ebooks from .onion with Python3 and requests?

The darknet is a mysterious area of the net, which arouses both interest and suspicion. There are sites not indexed by Google, with more or less legal products: drugs, weapons, counterfeit goods.

In addition to the illegal products, which we of course advise against, we also found this exceptional library of anarchist books, which offers more than 10,000 references to anarchist-inspired books in .pdf format, accessible entirely free of charge.

Accessible from Tor, with its famous .onion link: http://libraryqxxiqakubqv3dc2bend2koqsndbwox2johfywcatxie26bsad.onion/special/index

However, if you try to access it quickly with Python3, and requests:

import requests 
r = requests.get('http://libraryqxxiqakubqv3dc2bend2koqsndbwox2johfywcatxie26bsad.onion/special/index')
print(r.text)

Unable to access - with the inability to download items from the page:

What to do?

In this tutorial, we will see how to download, with Python3 and requests, more than 100 free .pdf files available on the darknet.

And the code is available here in full: https://gist.github.com/lobstrio/8d1c87203755d569da7ce2433179f099.

Off to the dark places of the darknet!

🥷

Prerequisites

In order to complete this tutorial from start to finish, be sure to have the following installed on your computer.

1. python3
2. SublimeText

You can click on the links below, which will take you either to an installation tutorial or to the site in question.

To clarify the purpose of each of the above: python3 is the computer language with which we will be scraping sites and downloading pdf's, and SublimeText is a text editor. Sublime.

Let's get to work.

Installation

We will proceed as follows:

1. Install TorBrowser
2. Install the tor package

For the first step, just go here: https://www.torproject.org/download/

Then download the browser that corresponds to your operating system. Here for me, Mac OS:

And then simply follow the installation instructions:

Finally, we will install requests, which allows us to move around the Internet with Python, pysocks, which will allow us to connect to the network of Tor proxies with Python, and lxml, which allows us to retrieve the elements present on a page:

$ pip3 install requests
$ pip3 install pysocks
$ pip3 install lxml

Finally, we'll install Tor from the command line:

Mac OS

First we install brew, the Mac OS package installation tool:

$ /usr/bin/ruby -e "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/master/install)"

Then we install tor:

$ brew install tor

We check that tor is well installed:

$ brew info tor
==> tor: stable 0.4.7.10 (bottled)

Finally, the service is launched:

$ brew services start tor

Linux

The package is installed:

$ sudo apt install tor

And we start the machine:

$ sudo /etc/init.d/tor start

And now we're ready to scrape, with our requests browser directly connected to the Tor proxies.

🔥

Code

Here is the code in full:

import requests
from lxml import html
import time

print('~~ start')

anarchist_library_onion_link = "http://libraryqxxiqakubqv3dc2bend2koqsndbwox2johfywcatxie26bsad.onion/special/index"
latest_books_library_onion_link = "http://libraryqxxiqakubqv3dc2bend2koqsndbwox2johfywcatxie26bsad.onion/latest?bare=1"

session = requests.session()
session.proxies = {'http': 'socks5h://localhost:9050', 'https': 'socks5h://localhost:9050'}


tor_ip = session.get("https://api.ipify.org/").text
local_ip = requests.get("https://api.ipify.org/").text
assert all([tor_ip, local_ip])
assert tor_ip != local_ip

response = session.get("http://libraryqxxiqakubqv3dc2bend2koqsndbwox2johfywcatxie26bsad.onion/latest?bare=1", timeout=100)

doc = html.fromstring(response.text)
items = doc.xpath('//div[@class="list-group"]//div[@class="amw-listing-item"]')
for i, item in enumerate(items[:10]): 
    link = "".join(item.xpath('./a/@href'))
    print(link)
    
    response = session.get(link, timeout=100)
    assert response.ok
    item_page_doc = html.fromstring(response.text)
    
    pdf_link = "".join(item_page_doc.xpath('//span[@id="download-format-pdf"]/a/@href'))
    assert pdf_link
    pdf_name = pdf_link.split('/')[-1]

    pdf_response = session.get(pdf_link, stream=True)
    with open(pdf_name, 'wb') as fd:
        for chunk in pdf_response.iter_content(2000):
            fd.write(chunk)
    print('✅ %s (%s)' % (pdf_name, i+1))
    time.sleep(1)

print('~~ success')
print(""" _       _         _            
| |     | |       | |           
| | ___ | |__  ___| |_ __ __  
| |/ _ \| '_ \/ __| __/| '__|
| | (_) | |_) \__ \ |_ | |   
|_|\___/|_.__/|___/\__||_|   
                                
""")

To execute the code:

1. Download the .py code
2. Run the script via the command line

And this is what will appear directly on your terminal:

$ python3 scraping-anarchists-library-darknet-requests-tor-tutorial.py
~~ start
✅ kevin-carson-may-day.pdf (1)
✅ theodoros-karyotis-ioanna-maria-maravelidi-yavor-tarinski-asking-questions-with-the-zapatistas.pdf (2)
✅ rasmus-hastbacka-six-myths-about-union-action.pdf (3)
✅ asbo-bang-up-and-smash-2nd-edition.pdf (4)
✅ bob-black-fija.pdf (5)
~~ success
 _       _         _            
| |     | |       | |           
| | ___ | |__  ___| |_ __ __  
| |/ _ \| '_ \/ __| __/| '__|
| | (_) | |_) \__ \ |_ | |   
|_|\___/|_.__/|___/\__||_|

And the precious anarchist ebooks, downloaded for free from the darknet, directly saved on your computer:

🤓

Step-by-Step Guide

The guide will be broken down into 3 parts.

1. Connecting to Tor proxies with requests and Python
2. Browsing the site
3. Downloading pdfs

Tor Proxies

First, let's connect to the Tor proxies with Python3 and requests, as follows:

session = requests.session()
session.proxies = {'http': 'socks5h://localhost:9050', 'https': 'socks5h://localhost:9050'}

Port 9050 is where Tor connects to its proxy pool. For more information, see our article on this topic.

Now we'll access https://api.ipify.org, once using tor, once using our local IP:

tor_ip = session.get("https://api.ipify.org/").text
local_ip = requests.get("https://api.ipify.org/").text
print(tor_ip)
print(local_ip)

And the result is clear:

$ python3 scraping-anarchists-library-darknet-requests-tor-tutorial.py
5.45.106.207
80.125.29.188

2 different IPs - we are well connected to the Tor proxies!

Now let's try to connect to the site:

response = session.get(latest_books_library_onion_link, timeout=100)
print(response.status_code)

And here again, when running the script, the message is clear:

$ python3 scraping-anarchists-library-darknet-requests-tor-tutorial.py
200

No more inaccessible pages. We positively access this site present on the darknet programmatically, with requests and Python.

🐍

All Results Page

Now we will navigate the site.

First, open Tor Browser, and go to the .onion site: http://libraryqxxiqakubqv3dc2bend2koqsndbwox2johfywcatxie26bsad.onion/special/index

Once you are on site, open your inspection console, right click + Inspect, then select the Network section.

Then press the search icon (1) below, which will open the query search tool:

Reload the page, and in the search tool, select one of the words on the page, here "A short history of May Day":

A query appears! It is the URL of this request that we will retrieve, and insert directly into our Python code:

response = session.get("http://libraryqxxiqakubqv3dc2bend2koqsndbwox2johfywcatxie26bsad.onion/latest?bare=1", timeout=100)

Now we will retrieve the URL of each of the pages of each book. When we open the 'Inspector' part of the inspection tool, we see that each book page is located in a div which has the class 'amw-listing-item':

We get this xPath, and finally, for each div, we'll go and get the link present in the 'href' attribute of the 'a' tag.

Here is the code:

response = session.get("http://libraryqxxiqakubqv3dc2bend2koqsndbwox2johfywcatxie26bsad.onion/latest?bare=1", timeout=100)

doc = html.fromstring(response.text)
items = doc.xpath('//div[@class="list-group"]//div[@class="amw-listing-item"]')
for i, item in enumerate(items[:10]): 
    link = "".join(item.xpath('./a/@href'))
    print(link)

And when you run it from the command line, the links appear clearly:

$ python3 scraping-anarchists-library-darknet-requests-tor-tutorial.py
http://libraryqxxiqakubqv3dc2bend2koqsndbwox2johfywcatxie26bsad.onion/library/kevin-carson-may-day
http://libraryqxxiqakubqv3dc2bend2koqsndbwox2johfywcatxie26bsad.onion/library/theodoros-karyotis-ioanna-maria-maravelidi-yavor-tarinski-asking-questions-with-the-zapatistas
http://libraryqxxiqakubqv3dc2bend2koqsndbwox2johfywcatxie26bsad.onion/library/rasmus-hastbacka-six-myths-about-union-action
http://libraryqxxiqakubqv3dc2bend2koqsndbwox2johfywcatxie26bsad.onion/library/asbo-bang-up-and-smash-2nd-edition
http://libraryqxxiqakubqv3dc2bend2koqsndbwox2johfywcatxie26bsad.onion/library/bob-black-fija

Beautiful!

All we have to do now is:

1. Go to the page of each book
2. Download the book in .pdf format

One Result Page

First, we go to the book page with requests:

response = session.get(link)

Then from the TorBrowser, once on the book page, inspect the area with the link to the pdf:

It appears that the link is found by following this xPath:

'//span[@id="download-format-pdf"]/a/@href'

Finally, the document is downloaded with requests:

pdf_link = "".join(item_page_doc.xpath('//span[@id="download-format-pdf"]/a/@href'))
assert pdf_link
pdf_name = pdf_link.split('/')[-1]

pdf_response = session.get(pdf_link, stream=True)
with open(pdf_name, 'wb') as fd:
    for chunk in pdf_response.iter_content(2000):
        fd.write(chunk)

And there you have it!

Benefits

This code will allow you to download .pdf documents with requests and Python3, directly from an .onion site on the dark net.

An obvious cultural benefit.

Limitations

While this Python script allows you to connect to an .onion site with Python3 and requests, you will only be able to download one page of results. Pagination is not supported.

Furthermore, this tutorial is only about the anarchist library on the darknet, accessible via this URL .onion: http://libraryqxxiqakubqv3dc2bend2koqsndbwox2johfywcatxie26bsad.onion/special/index.

To access other data sources, you will have to modify the script together.

Beware, please note that this tutorial is for educational purposes only.

We therefore disclaim any responsibility for any immoderate use that may be made of it, particularly on sites linked to other types of services or other types of data. Furthermore, we would like to remind you that, according to the legislation of the country in which you operate, it is strictly forbidden to possess texts protected by copyright without offering the fair remuneration due to the author. We therefore strongly recommend that you consult the legislation in force in your country of practice before embarking on any IT development project as illustrated above.

Conclusion

And that's the end of the tutorial!

In this tutorial, we have seen how to download .pdf files from an anarchist .onion library on the darknet, using requests and Python3.

If you have any questions, or if you need a custom, solid and scalable scraping service, we are of course at your disposal here.

Happy scraping!

🦀