# The Onion-Location world

This document describes how to estimate the size of the [Onion-Location][]
world.  The intuition for obtaining an answer is as follows:

  1.  Onion-Location requires HTTPS.  Therefore, a pretty complete list of
      domains that _may_ offer Onion-Location can be determined by downloading
      all [CT-logged certificates][] and checking which [SANs][] are in them.
  2.  Visit the encountered SANs over HTTPS without Tor, looking for if the web
      server set either the Onion-Location HTTP header or HTML meta-tag.

Please note that this is a _lower-bound estimate_, e.g., because not all web
browsers enforce CT logging and SANs like `*.example.com` may have any number of
subdomains with their own Onion-Location configured sites that we won't find.

We start by describing the experimental setup and tools used, followed by
results as well as availability of the collected and derived datasets.

[Onion-Location]: https://community.torproject.org/onion-services/advanced/onion-location/
[CT-logged certificates]: https://certificate.transparency.dev/
[SANs]: https://www.rfc-editor.org/rfc/rfc5280#section-4.2.1.6

## Experimental setup

XXX: system(s) used to run the below.

### ct-sans dataset

We put together a tool named [ct-sans][] that facilitates simple creation of a
data set composed of unique SANs in [CT logs recognized by Google Chrome][].

[ct-sans]: XXX
[CT logs recognized by Google Chrome]: https://groups.google.com/a/chromium.org/g/ct-policy/c/IdbrdAcDQto

Install:

    $ go install ...ct-sans@latest
    ...

Make a snapshot of which CT logs and entries to download:

    $ ct-sans snapshot -d $HOME/.config/ct-sans
    ...

Collect, or continue collecting if you decided to shutdown prematurely:

    $ ct-sans collect -d $HOME/.config/ct-sans
    ...

The collected data is per-log, with each line containing a single SAN:

    $ tail -n3 $HOME/.config/ct-sans/logs/xxx/sans.lst
    example.org
    www.example.org
    *.example.net

[Motivation: easy to maintain the data set, e.g., without needing any special
indexing and can yearly delete log shards that only contain expired
certificates with rm -rf.]

The final data set of combined and non-duplicate SANs can be created with the
UNIX tool `sort`.  For the exact commands and an associated dataset manifest:

    $ ct-sans package -d $HOME/.config/ct-sans
    sort -u ...
    ...

    sort -Vuo sans.lst --buffer-size=1024K --temporary-directory=/tmp/t --parallel=2 a.lst b.lst

Note that you may need to tweak the number of CPUs, available memory, and
temporary disk space to be suitable for your own system.

### zgrab2 website visits

[ZGrab2][] is an application-level scanner that (among other things) can visit
HTTPS sites to record encountered HTTP headers and web pages.  This is exactly
what we need to visit each site in our ct-sans dataset, however only saving the
output if it indicates that the site is configured with Onion-Location.

Install:

    $ go install github.com/zmap/zgrab2/cmd/zgrab2@latest
    ...

Run:

    $ zgrab ...

XXX: describe the grep pattern for filtering, and/or wrap in a bash script.

[ZGrab2]: https://github.com/zmap/zgrab2

### Results and data sets

XXX

### Remarks

  - The ct-sans dataset can be updated by running the snapshot, collect, and
    assemble commands again.  (The snapshot command downloads the latest list of
    CT logs and their signed tree heads to use as reference while collecting.)
  - The entire zgrab2 scan needs to be conduced from scratch for each dataset
    because sites may add or remove Onion-Location configurations at any time.
  - The above does not make any attempts to visit the announced onion sites.

### Santity check

We will download ~3.4 * 10^9 certificates in total.

We only store SANs, not complete certificates.  Assume that each certificate has
on average 256 bytes of SANs (1/6 of avg certificate size).  Then:

  256 * 3.4 * 10^9 = 0.8 TiB of SANs.

We will also need temp disk space for sorting and removing duplicates; so if we
could get a machine with 2TiB disk that should probably be more than enough.

The data needed to be stored after website visits will be ~negligible.

The more RAM and CPU workers we can get, the better.  Same with bandwidth.  For
running this more continuously in the future, a less powerful machine should do.

XXX: Tobias will request a machine from our department tomorrow, minimum 8 CPUs
and 32GiB RAM and ~2TiB disk.  Pitch easier if we do website visits with
mullvad enabled, so we will do that.

### Notes from starting ct-sans on our machine

    $ go install git.cs.kau.se/rasmoste/ct-sans@v0.0.1
    $ ct-sans snapshot >snapshot.stdout
    $ cat snapshot.stdout
    2023/03/18 20:05:30 cmd_snapshot.go:30: INFO: updating metadata file
    2023/03/18 20:05:30 cmd_snapshot.go:47: INFO: updating signed tree heads
    2023/03/18 20:05:30 cmd_snapshot.go:82: INFO: bootstrapped Google 'Argon2023' log at tree size 841710936
    2023/03/18 20:05:30 cmd_snapshot.go:82: INFO: bootstrapped Google 'Argon2024' log at tree size 52689060
    2023/03/18 20:05:30 cmd_snapshot.go:82: INFO: bootstrapped Google 'Xenon2023' log at tree size 966608751
    2023/03/18 20:05:30 cmd_snapshot.go:82: INFO: bootstrapped Google 'Xenon2024' log at tree size 63025768
    2023/03/18 20:05:31 cmd_snapshot.go:82: INFO: bootstrapped Cloudflare 'Nimbus2023' Log at tree size 513025681
    2023/03/18 20:05:31 cmd_snapshot.go:82: INFO: bootstrapped Cloudflare 'Nimbus2024' Log at tree size 31749516
    2023/03/18 20:05:31 cmd_snapshot.go:82: INFO: bootstrapped DigiCert Yeti2024 Log at tree size 36293063
    2023/03/18 20:05:32 cmd_snapshot.go:82: INFO: bootstrapped DigiCert Yeti2025 Log at tree size 687
    2023/03/18 20:05:33 cmd_snapshot.go:82: INFO: bootstrapped DigiCert Nessie2023 Log at tree size 198950777
    2023/03/18 20:05:33 cmd_snapshot.go:82: INFO: bootstrapped DigiCert Nessie2024 Log at tree size 37773373
    2023/03/18 20:05:34 cmd_snapshot.go:82: INFO: bootstrapped DigiCert Nessie2025 Log at tree size 694
    2023/03/18 20:05:35 cmd_snapshot.go:82: INFO: bootstrapped Sectigo 'Sabre' CT log at tree size 228782818
    2023/03/18 20:05:35 cmd_snapshot.go:82: INFO: bootstrapped Let's Encrypt 'Oak2023' log at tree size 457590228
    2023/03/18 20:05:35 cmd_snapshot.go:82: INFO: bootstrapped Let's Encrypt 'Oak2024H1' log at tree size 32863293
    2023/03/18 20:05:35 cmd_snapshot.go:82: INFO: bootstrapped Let's Encrypt 'Oak2024H2' log at tree size 13281
    2023/03/18 20:05:36 cmd_snapshot.go:82: INFO: bootstrapped Trust Asia Log2023 at tree size 379756
    2023/03/18 20:05:38 cmd_snapshot.go:82: INFO: bootstrapped Trust Asia Log2024-2 at tree size 110270
    $ ct-sans collect --workers 40 --batch-disk 131072 --batch-req 2048 --metrics 60m >collect.stdout 2>collect.stderr

Tail in a tmux pane:

    $ tail -f collect.stdout
    ...
    $ head -n1 collect.stdout
    2023/03/18 20:11:52 cmd_collect.go:150: INFO: collect is up-and-running, ctrl+C to exit

Note that it is safe to ctrl+C, things are written to disk before exit.  Just
run `ct-sans collect` again to start collecting from where we left off.

Tail in a tmux pane:

    $ tail -f collect.stderr
    ...

With the above settings it takes 60m before the first status report is printed.