diff options
| -rw-r--r-- | README.md | 277 |
1 files changed, 123 insertions, 154 deletions
@@ -1,168 +1,137 @@ -# The Onion-Location world +# ct-sans -This document describes how to estimate the size of the [Onion-Location][] -world. The intuition for obtaining an answer is as follows: +A tool that downloads certificates from [CT logs][] [recognized by Google +Chrome][], storing the encountered [Subject Alternative Names (SANs)][] to disk. +The final data set `sans.lst` is de-duplicated and contains one SAN per line. - 1. Onion-Location requires HTTPS. Therefore, a pretty complete list of - domains that _may_ offer Onion-Location can be determined by downloading - all [CT-logged certificates][] and checking which [SANs][] are in them. - 2. Visit the encountered SANs over HTTPS without Tor, looking for if the web - server set either the Onion-Location HTTP header or HTML meta-tag. +[CT logs]: https://certificate.transparency.dev/ +[recognized by Google Chrome]: https://groups.google.com/a/chromium.org/g/ct-policy/c/IdbrdAcDQto/ +[Subject Alternative Names (SANs)]: https://www.rfc-editor.org/rfc/rfc5280#section-4.2.1.6/ -Please note that this is a _lower-bound estimate_, e.g., because not all web -browsers enforce CT logging and SANs like `*.example.com` may have any number of -subdomains with their own Onion-Location configured sites that we won't find. +**Warning:** research prototype. The source code may also be moved. -We start by describing the experimental setup and tools used, followed by -results as well as availability of the collected and derived datasets. +## Quick start -[Onion-Location]: https://community.torproject.org/onion-services/advanced/onion-location/ -[CT-logged certificates]: https://certificate.transparency.dev/ -[SANs]: https://www.rfc-editor.org/rfc/rfc5280#section-4.2.1.6 +You will need a Go compiler and GNU sort on the local system: -## Experimental setup + $ which go || echo "Go compiler is not in $PATH" + $ which sort || echo "GNU sort is not in $PATH" -XXX: system(s) used to run the below. +Install `ct-sans`: -### ct-sans dataset + $ go install git.cs.kau.se/rasmoste/ct-sans@latest + $ which ct-sans || echo "ct-sans is not in $PATH" -We put together a tool named [ct-sans][] that facilitates simple creation of a -data set composed of unique SANs in [CT logs recognized by Google Chrome][]. +Download and verify the signature of Google's list of known logs, +then download and verify the signatures of the logs' tree heads: -[ct-sans]: XXX -[CT logs recognized by Google Chrome]: https://groups.google.com/a/chromium.org/g/ct-policy/c/IdbrdAcDQto + $ ct-sans snapshot -d $HOME/ct-sans-demo + 2023/03/23 12:43:49 cmd_snapshot.go:30: INFO: updating metadata file + 2023/03/23 12:43:49 cmd_snapshot.go:47: INFO: updating signed tree heads + 2023/03/23 12:43:49 cmd_snapshot.go:82: INFO: bootstrapped Google 'Argon2023' log at tree size 862104911 + 2023/03/23 12:43:50 cmd_snapshot.go:82: INFO: bootstrapped Google 'Argon2024' log at tree size 55767940 + 2023/03/23 12:43:50 cmd_snapshot.go:82: INFO: bootstrapped Google 'Xenon2023' log at tree size 990277299 + 2023/03/23 12:43:50 cmd_snapshot.go:82: INFO: bootstrapped Google 'Xenon2024' log at tree size 66655425 + 2023/03/23 12:43:50 cmd_snapshot.go:82: INFO: bootstrapped Cloudflare 'Nimbus2023' Log at tree size 527018586 + 2023/03/23 12:43:50 cmd_snapshot.go:82: INFO: bootstrapped Cloudflare 'Nimbus2024' Log at tree size 34050592 + 2023/03/23 12:43:51 cmd_snapshot.go:82: INFO: bootstrapped DigiCert Yeti2024 Log at tree size 38426463 + 2023/03/23 12:43:53 cmd_snapshot.go:82: INFO: bootstrapped DigiCert Yeti2025 Log at tree size 697 + 2023/03/23 12:43:54 cmd_snapshot.go:82: INFO: bootstrapped DigiCert Nessie2023 Log at tree size 200387219 + 2023/03/23 12:43:55 cmd_snapshot.go:82: INFO: bootstrapped DigiCert Nessie2024 Log at tree size 40017666 + 2023/03/23 12:43:55 cmd_snapshot.go:82: INFO: bootstrapped DigiCert Nessie2025 Log at tree size 704 + 2023/03/23 12:43:56 cmd_snapshot.go:82: INFO: bootstrapped Sectigo 'Sabre' CT log at tree size 229064032 + 2023/03/23 12:43:57 cmd_snapshot.go:82: INFO: bootstrapped Let's Encrypt 'Oak2023' log at tree size 467618545 + 2023/03/23 12:43:57 cmd_snapshot.go:82: INFO: bootstrapped Let's Encrypt 'Oak2024H1' log at tree size 34451205 + 2023/03/23 12:43:57 cmd_snapshot.go:82: INFO: bootstrapped Let's Encrypt 'Oak2024H2' log at tree size 14680 + 2023/03/23 12:43:59 cmd_snapshot.go:82: INFO: bootstrapped Trust Asia Log2023 at tree size 388349 + 2023/03/23 12:44:01 cmd_snapshot.go:82: INFO: bootstrapped Trust Asia Log2024-2 at tree size 112771 -Install: +Subsequent uses of the snapshot command will update the signed list of known +logs, then update the logs' signed tree heads after verifying consistency. - $ go install ...ct-sans@latest - ... - -Make a snapshot of which CT logs and entries to download: - - $ ct-sans snapshot -d $HOME/.config/ct-sans - ... - -Collect, or continue collecting if you decided to shutdown prematurely: - - $ ct-sans collect -d $HOME/.config/ct-sans - ... - -The collected data is per-log, with each line containing a single SAN: - - $ tail -n3 $HOME/.config/ct-sans/logs/xxx/sans.lst - example.org - www.example.org - *.example.net - -[Motivation: easy to maintain the data set, e.g., without needing any special -indexing and can yearly delete log shards that only contain expired -certificates with rm -rf.] - -The final data set of combined and non-duplicate SANs can be created with the -UNIX tool `sort`. For the exact commands and an associated dataset manifest: - - $ ct-sans package -d $HOME/.config/ct-sans - sort -u ... - ... - - sort -Vuo sans.lst --buffer-size=1024K --temporary-directory=/tmp/t --parallel=2 a.lst b.lst - -Note that you may need to tweak the number of CPUs, available memory, and -temporary disk space to be suitable for your own system. - -### zgrab2 website visits - -[ZGrab2][] is an application-level scanner that (among other things) can visit -HTTPS sites to record encountered HTTP headers and web pages. This is exactly -what we need to visit each site in our ct-sans dataset, however only saving the -output if it indicates that the site is configured with Onion-Location. - -Install: - - $ go install github.com/zmap/zgrab2/cmd/zgrab2@latest - ... - -Run: - - $ zgrab ... +Download and verify the logs' Merkle trees up until the current snapshot: -XXX: describe the grep pattern for filtering, and/or wrap in a bash script. - -[ZGrab2]: https://github.com/zmap/zgrab2 - -### Results and data sets - -XXX - -### Remarks - - - The ct-sans dataset can be updated by running the snapshot, collect, and - assemble commands again. (The snapshot command downloads the latest list of - CT logs and their signed tree heads to use as reference while collecting.) - - The entire zgrab2 scan needs to be conduced from scratch for each dataset - because sites may add or remove Onion-Location configurations at any time. - - The above does not make any attempts to visit the announced onion sites. - -### Santity check - -We will download ~3.4 * 10^9 certificates in total. - -We only store SANs, not complete certificates. Assume that each certificate has -on average 256 bytes of SANs (1/6 of avg certificate size). Then: - - 256 * 3.4 * 10^9 = 0.8 TiB of SANs. - -We will also need temp disk space for sorting and removing duplicates; so if we -could get a machine with 2TiB disk that should probably be more than enough. - -The data needed to be stored after website visits will be ~negligible. - -The more RAM and CPU workers we can get, the better. Same with bandwidth. For -running this more continuously in the future, a less powerful machine should do. - -XXX: Tobias will request a machine from our department tomorrow, minimum 8 CPUs -and 32GiB RAM and ~2TiB disk. Pitch easier if we do website visits with -mullvad enabled, so we will do that. - -### Notes from starting ct-sans on our machine - - $ go install git.cs.kau.se/rasmoste/ct-sans@v0.0.1 - $ ct-sans snapshot >snapshot.stdout - $ cat snapshot.stdout - 2023/03/18 20:05:30 cmd_snapshot.go:30: INFO: updating metadata file - 2023/03/18 20:05:30 cmd_snapshot.go:47: INFO: updating signed tree heads - 2023/03/18 20:05:30 cmd_snapshot.go:82: INFO: bootstrapped Google 'Argon2023' log at tree size 841710936 - 2023/03/18 20:05:30 cmd_snapshot.go:82: INFO: bootstrapped Google 'Argon2024' log at tree size 52689060 - 2023/03/18 20:05:30 cmd_snapshot.go:82: INFO: bootstrapped Google 'Xenon2023' log at tree size 966608751 - 2023/03/18 20:05:30 cmd_snapshot.go:82: INFO: bootstrapped Google 'Xenon2024' log at tree size 63025768 - 2023/03/18 20:05:31 cmd_snapshot.go:82: INFO: bootstrapped Cloudflare 'Nimbus2023' Log at tree size 513025681 - 2023/03/18 20:05:31 cmd_snapshot.go:82: INFO: bootstrapped Cloudflare 'Nimbus2024' Log at tree size 31749516 - 2023/03/18 20:05:31 cmd_snapshot.go:82: INFO: bootstrapped DigiCert Yeti2024 Log at tree size 36293063 - 2023/03/18 20:05:32 cmd_snapshot.go:82: INFO: bootstrapped DigiCert Yeti2025 Log at tree size 687 - 2023/03/18 20:05:33 cmd_snapshot.go:82: INFO: bootstrapped DigiCert Nessie2023 Log at tree size 198950777 - 2023/03/18 20:05:33 cmd_snapshot.go:82: INFO: bootstrapped DigiCert Nessie2024 Log at tree size 37773373 - 2023/03/18 20:05:34 cmd_snapshot.go:82: INFO: bootstrapped DigiCert Nessie2025 Log at tree size 694 - 2023/03/18 20:05:35 cmd_snapshot.go:82: INFO: bootstrapped Sectigo 'Sabre' CT log at tree size 228782818 - 2023/03/18 20:05:35 cmd_snapshot.go:82: INFO: bootstrapped Let's Encrypt 'Oak2023' log at tree size 457590228 - 2023/03/18 20:05:35 cmd_snapshot.go:82: INFO: bootstrapped Let's Encrypt 'Oak2024H1' log at tree size 32863293 - 2023/03/18 20:05:35 cmd_snapshot.go:82: INFO: bootstrapped Let's Encrypt 'Oak2024H2' log at tree size 13281 - 2023/03/18 20:05:36 cmd_snapshot.go:82: INFO: bootstrapped Trust Asia Log2023 at tree size 379756 - 2023/03/18 20:05:38 cmd_snapshot.go:82: INFO: bootstrapped Trust Asia Log2024-2 at tree size 110270 - $ ct-sans collect --workers 40 --batch-disk 131072 --batch-req 2048 --metrics 60m >collect.stdout 2>collect.stderr - -Tail in a tmux pane: - - $ tail -f collect.stdout + $ ct-sans collect -d $HOME/ct-sans-demo ... - $ head -n1 collect.stdout - 2023/03/18 20:11:52 cmd_collect.go:150: INFO: collect is up-and-running, ctrl+C to exit - -Note that it is safe to ctrl+C, things are written to disk before exit. Just -run `ct-sans collect` again to start collecting from where we left off. - -Tail in a tmux pane: - - $ tail -f collect.stderr - ... - -With the above settings it takes 60m before the first status report is printed. + INFO: status update before shutdown + + Google 'Argon2023' log | 162.5 entries/s | Estimated done in 1474.01 hours | Working on [11776, 862104911) + Google 'Argon2024' log | 157.5 entries/s | Estimated done in 98.31 hours | Working on [11584, 55767940) + Google 'Xenon2023' log | 472.6 entries/s | Estimated done in 582.01 hours | Working on [33888, 990277299) + Google 'Xenon2024' log | 458.5 entries/s | Estimated done in 40.37 hours | Working on [32896, 66655425) + Cloudflare 'Nimbus2023' Log | 276.1 entries/s | Estimated done in 530.24 hours | Working on [19328, 527018586) + Cloudflare 'Nimbus2024' Log | 301.2 entries/s | Estimated done in 31.39 hours | Working on [20736, 34050592) + DigiCert Yeti2024 Log | 379.1 entries/s | Estimated done in 28.14 hours | Working on [27520, 38426463) + DigiCert Yeti2025 Log | 0.0 entries/s | Estimated done in 0.00 hours | Working on [697, 697) + DigiCert Nessie2023 Log | 331.3 entries/s | Estimated done in 168.00 hours | Working on [23040, 200387219) + DigiCert Nessie2024 Log | 329.8 entries/s | Estimated done in 33.68 hours | Working on [21120, 40017666) + DigiCert Nessie2025 Log | 0.0 entries/s | Estimated done in 0.00 hours | Working on [704, 704) + Sectigo 'Sabre' CT log | 275.7 entries/s | Estimated done in 230.78 hours | Working on [19456, 229064032) + Let's Encrypt 'Oak2023' log | 462.8 entries/s | Estimated done in 280.67 hours | Working on [33664, 467618545) + Let's Encrypt 'Oak2024H1' log | 121.4 entries/s | Estimated done in 78.79 hours | Working on [5248, 34451205) + Let's Encrypt 'Oak2024H2' log | 0.0 entries/s | Estimated done in 0.00 hours | Working on [14680, 14680) + Trust Asia Log2023 | 215.8 entries/s | Estimated done in 0.48 hours | Working on [15872, 388349) + Trust Asia Log2024-2 | 246.2 entries/s | Estimated done in 0.11 hours | Working on [17664, 112771) + +This will take a while depending on the local system, configuration of the +optional `ct-sans collect` flags, as well as how heavily the logs apply +rate-limits. For good performance while respecting rate-limits, you may want +to try `--workers 40 --batch-disk 131072 --batch-req 2048 --metrics 60m`. This +allowed us to download the logs (March 2023) in approximately 10 days. Our +single-IP EU machine had a 2 TiB SSD, 64 GiB memory, 16 CPU cores, and 1 Gbps. + +Of note is that it is safe to ctrl+C while collecting. Just wait for the +collect command to exit on its own so that things are persisted to disk. + +Once the collect phase is done, assemble the data set: + + $ echo "for demo-purposes, only Nessie2025 and Oak2024H2 are shown below"^C + $ ct-sans assemble -d $HOME/ct-sans-demo + 2023/03/23 13:05:12 cmd_assemble.go:54: INFO: merging and de-duplicating 2 input files with GNU sort + 2023/03/23 13:05:12 cmd_assemble.go:67: INFO: created /home/rgdd/ct-sans-demo/archive/2023-03-23-ct-sans/sans.lst (0.3 MiB) + 2023/03/23 13:05:12 cmd_assemble.go:69: INFO: adding notice file + 2023/03/23 13:05:12 cmd_assemble.go:87: INFO: adding README + 2023/03/23 13:05:12 cmd_assemble.go:96: INFO: adding signed metadata file + 2023/03/23 13:05:12 cmd_assemble.go:108: INFO: adding signed tree heads + 2023/03/23 13:05:12 cmd_assemble.go:117: INFO: uncompressed dataset available in /home/rgdd/ct-sans-demo/archive/2023-03-23-ct-sans + $ cat $HOME/ct-sans-demo/archive/2023-03-23-ct-sans/README.md + # ct-sans dataset + + Dataset assembled at Thu Mar 23 13:05:12 CET 2023. Contents: + + - README.md + - metadata.json + - metadata.sig + - sths.json + - notice.txt + - sans.lst + + The signed [metadata file][] and tree heads were downloaded at + Thu Mar 23 12:43:49 CET 2023. + + [metadata file]: https://groups.google.com/a/chromium.org/g/ct-policy/c/IdbrdAcDQto + + In total, 15377 certificates were downloaded from 2 CT logs; + 0 certificates contained SANs that could not be parsed. + For more information about these errors, see notice.txt. + + The SANs data set is sorted and de-duplicated, one SAN per line. + +**Note:** the different ct-sans commands must not run at the same time. + +## Updating the data set + +Simply run the same snapshot, collect, and assemble commands again. + +## Contact + + - IRC: room #certificate-transparency at [OFTC.net][] + - Matrix: room [#certificate-transparency][] (bridged with IRC) + - Email: rasmus (at) rgdd (dot) se + +[OFTC.net]: https://www.oftc.net/ +[#certificate-transparency]: https://app.element.io/#/room/#sauteed-onions:matrix.org/ + +## Licence + +BSD 2-Clause License |