The Onion-Location world

This document describes how to estimate the size of the Onion-Location world. The intuition for obtaining an answer is as follows:

Onion-Location requires HTTPS. Therefore, a pretty complete list of domains that may offer Onion-Location can be determined by downloading all CT-logged certificates and checking which SANs are in them.
Visit the encountered SANs over HTTPS without Tor, looking for if the web server set either the Onion-Location HTTP header or HTML meta-tag.

Please note that this is a lower-bound estimate, e.g., because not all web browsers enforce CT logging and SANs like *.example.com may have any number of subdomains with their own Onion-Location configured sites that we won't find.

We start by describing the experimental setup and tools used, followed by results as well as availability of the collected and derived datasets.

Experimental setup

XXX: system(s) used to run the below.

ct-sans dataset

We put together a tool named ct-sans that facilitates simple creation of a data set composed of unique SANs in CT logs recognized by Google Chrome.

Install:

$ go install ...ct-sans@latest
...

Make a snapshot of which CT logs and entries to download:

$ ct-sans snapshot -d $HOME/.config/ct-sans
...

Collect, or continue collecting if you decided to shutdown prematurely:

$ ct-sans collect -d $HOME/.config/ct-sans
...

The collected data is per-log, with each line containing a single SAN:

$ tail -n3 $HOME/.config/ct-sans/logs/xxx/sans.lst
example.org
www.example.org
*.example.net

[Motivation: easy to maintain the data set, e.g., without needing any special indexing and can yearly delete log shards that only contain expired certificates with rm -rf.]

The final data set of combined and non-duplicate SANs can be created with the UNIX tool sort. For the exact commands and an associated dataset manifest:

$ ct-sans package -d $HOME/.config/ct-sans
sort -u ...
...

Note that you may need to tweak the number of CPUs, available memory, and temporary disk space to be suitable for your own system.

zgrab2 website visits

ZGrab2 is an application-level scanner that (among other things) can visit HTTPS sites to record encountered HTTP headers and web pages. This is exactly what we need to visit each site in our ct-sans dataset, however only saving the output if it indicates that the site is configured with Onion-Location.

Install:

$ go install github.com/zmap/zgrab2/cmd/zgrab2@latest
...

Run:

$ zgrab ...

XXX: describe the grep pattern for filtering, and/or wrap in a bash script.

Results and data sets

XXX

Remarks

The ct-sans dataset can be updated by running the snapshot, collect, and assemble commands again. (The snapshot command downloads the latest list of CT logs and their signed tree heads to use as reference while collecting.)
The entire zgrab2 scan needs to be conduced from scratch for each dataset because sites may add or remove Onion-Location configurations at any time.
The above does not make any attempts to visit the announced onion sites.

Santity check

We will download ~3.4 * 10^9 certificates in total.

We only store SANs, not complete certificates. Assume that each certificate has on average 256 bytes of SANs (1/6 of avg certificate size). Then:

256 * 3.4 * 10^9 = 0.8 TiB of SANs.

We will also need temp disk space for sorting and removing duplicates; so if we could get a machine with 2TiB disk that should probably be more than enough.

The data needed to be stored after website visits will be ~negligible.

The more RAM and CPU workers we can get, the better. Same with bandwidth. For running this more continuously in the future, a less powerful machine should do.

XXX: Tobias will request a machine from our department tomorrow, minimum 8 CPUs and 32GiB RAM and ~2TiB disk. Pitch easier if we do website visits with mullvad enabled, so we will do that.