diff options
Diffstat (limited to 'docs/operations.md')
| -rw-r--r-- | docs/operations.md | 826 |
1 files changed, 824 insertions, 2 deletions
diff --git a/docs/operations.md b/docs/operations.md index 1528c32..3f67a92 100644 --- a/docs/operations.md +++ b/docs/operations.md @@ -1,3 +1,825 @@ -# Operations +# onion-grab dataset -Placeholder. +This document describes our `onion-grab` data collection, including information +about the local systems and a timeline for our operations leading up to the +results for [Tranco top-1m][] and [SANs in CT logs][] during April, 2023. + +[Tranco top-1m]: https://tranco-list.eu/ +[SANs in CT logs]: https://git.cs.kau.se/rasmoste/ct-sans/-/blob/main/docs/operations.md + +## Summary + +The time to conduct initial tests against [Tranco top-1m][] was ~1 day. 207 +unique two-label `.onion` domains were found from 285 Onion-Location sites. + +The time to conduct the full measurement for [SANs in CT logs][] was ~10 days. +3330 unique two-label `.onion` domains were configured from 26937 unique sites. +13956 of those unique sites have the same Onion-Location configuration as +Twitter, which likely means that they copied some of their HTML attributes. + +The collected data sets are available here: + + - https://dart.cse.kau.se/onion-grab/2023-04-03-tranco.zip + - https://dart.cse.kau.se/onion-grab/2023-04-03-ct-sans.zip + +For further information about system configurations and operations, read on. + +## Local systems + +We have three mostly identical Ubuntu VMs: + + $ lsb_release -a + No LSB modules are available. + Distributor ID: Ubuntu + Description: Ubuntu 22.04.2 LTS + Release: 22.04 + Codename: jammy + +VM-1 is configured with 62.9GiB RAM, one CPU core with 32 CPU threads, and a +~2TiB SSD: + + $ grep MemTotal /proc/meminfo + processor /proc/cpuinfoemTotal: 65948412 keand + $ grep -c processor /proc/cpuinfo + 32 + $ grep 'cpu cores' /proc/cpuinfo | uniq + cpu cores : 1 + $ df -BG /home + Filesystem 1G-blocks Used Available Use% Mounted on + /dev/mapper/ubuntu--vg-ubuntu--lv 2077G 220G 1772G 12% / + +VM-2 and VM-3 are configured with 62.9GiB RAM, one CPU core with 16 CPU threads, +and a ~60TiB SSD (each): + + $ grep MemTotal /proc/meminfo + MemTotal: 65822508 kB + $ grep -c processor /proc/cpuinfo + 16 + $ grep 'cpu cores' /proc/cpuinfo | uniq + cpu cores : 1 + $ df -BG /home + Filesystem 1G-blocks Used Available Use% Mounted on + /dev/mapper/ubuntu--vg-ubuntu--lv 61G 11G 48G 18% / + +These VMs share a 1x10Gbps link with other network VMs that we have no control +over. We installed `vnstat` to track our bandwidth-usage over time: + + # apt install vnstat + # systemctl enable vnstat.service + # systemctl start vnstat.service + +We also installed Go version 1.20, see [install instructions][]: + + $ go version + go version go1.20.2 linux/amd64 + +[install instructions]: https://go.dev/doc/install + +Stopped and disabled `systemd-resolved`, populating `/etc/resolv.conf` with + + $ cat /etc/resolv.conf + nameserver 8.8.8.8 + nameserver 8.8.4.4 + +which gives us a setup that [supports 1500 DNS look-ups][] per VM. + +[supports 1500 DNS look-ups]: https://developers.google.com/speed/public-dns/docs/isp + +We set + + $ ulimit -Sn 100000 + # sysctl -w net.ipv4.ip_local_port_range="1024 65535" + +before running `onion-grab`. The complete outputs of these commands with `-a` +are available in our dataset. The versions of `onion-grab` are listed below. + +Finally, we [installed Mullvad VPN][] so that our `onion-grab` measurements can +run from Melbourne (VM-1), New York (VM-2) and Stockholm (VM-1). Remember to +set the same DNS resolvers as above (`mullvad dns set custom 8.8.8.8 8.8.4.4`). + +In the full measurement, we had to replace Stockholm with Frankfurt (see notes). + +[installed Mullvad VPN]: https://mullvad.net/en/help/install-mullvad-app-linux/ + +## Timeline + +| date | time (UTC) | event | notes | +| ---------- | ---------- | --------------------------- | ------------------------------------------- | +| 2023/04/02 | 23:26:27 | test run with tranco top-1m | to estimate reasonable repetition count [1] | +| 2023/04/03 | 12:47:43 | test run with tranco top-1m | to estimate reasonable repetition count [1] | +| 2023/04/03 | 17:20:00 | shuffle ct-sans dataset | deterministic per-VM seed, 15m/shuffle [2] | +| 2023/04/03 | 18:18:47 | test run with tranco top-1m | to estimate reasonable repetition count [1] | +| 2023/04/03 | 20:03 | transfer shuffled dataset | from VM-1 to VM-2 (1-3MB/s, painfully slow) | +| 2023/04/03 | 20:03 | transfer shuffled dataset | from VM-1 to VM-3 (1-3MB/s, painfully slow) | +| 2023/04/03 | 22:36:06 | start onion-grab (au mel) | checkout v0.0.2, set measure.sh params [3] | +| 2023/04/03 | 22:35:36 | start onion-grab (us ny) | checkout v0.0.2, set measure.sh params [4] | +| 2023/04/03 | 22:35:38 | start onion-grab (se sto) | checkout v0.0.2, set measure.sh params [5] | +| 2023/04/04 | 15:30 | se sto relay bw drop | store vnstat -h stats w/ daily cron job [6] | +| 2023/04/05 | 06:30 | kill onion-grab (se sto) | all Stockholm relays are very slow [7] | +| 2023/04/05 | 07:02:13 | start onion-grab (de fra) | all Swedish relays are very slow [8] | +| 2023/04/11 | 04:26:26 | us nyc completed | minor exit bug [9] | +| 2023/04/11 | 04:30:28 | au mel completed | minor exit bug [9] | +| 2023/04/11 | 20:25:50 | de fra stopped | ran out of memory for unknown reason [10] | +| 2023/04/11 | 22:36:25 | de fra started again | use start line we know is processed [10,11] | +| 2023/04/11 | 20:25:50 | de fra stopped | ran out of memory for unknown reason [12] | +| 2023/04/12 | 08:42:30 | de fra started again | use start line we know is processed [12,13] | +| 2023/04/12 | 11:50 | prepare dataset (au mel) | only moving files on VM-1 [14] | +| 2023/04/12 | 14:00 | prepare dataset (us nyc) | moving files on VM-2, transfer to VM-1 [15] | +| 2023/04/12 | 16:50 | prepare dataset (se sto) | moving files on VM-3, transfer to VM-1 [16] | +| 2023/04/12 | 17:00 | save bandwidths at VM-{1,2} | forgot to move them earlier [17] | +| 2023/04/13 | 00:35:38 | de fra completed | minor exit bug [18] | +| 2023/04/13 | 05:40 | prepare dataset (de fra) | moving files on VM-3, transfer to VM-1 [19] | +| 2023/04/13 | 05:50 | experiment is completed | datasets are ready, zipped, and documented | + +## Notes + +### 1 + +We downloaded [Tranco top-1m][], permalink [Z2XKG][] (2023-04-03): + + $ sha256sum tranco_Z2XKG-1m.csv.zip + 3e078a84e9aae7dbaf1207aac000038f1e51e20e8ccc35563da8b175d38a39dd tranco_Z2XKG-1m.csv.zip + $ unzip tranco_Z2XKG-1m.csv.zip + $ cut -d',' -f2 top-1m.csv > top-1m.lst + +[Z2XKG]: https://tranco-list.eu/list/Z2XKG/1000000 + +This gives us a list of 1M domains to perform test-runs on. The idea: + + 1. Make visits at a wanted rate (1450/s, below the 1500 DNS lookup limit) + 2. Make visits at several slower rates (100/s, ..., 1400/s) + 3. Repeat this from three locations (Stockholm, New York, Melbourne) + 4. Hypothesis: observe that the same number of Onion-Location setups are + discovered when running at the most rapid rate from three locations when + compared to a lower rate at the same three locations; and that the error + rates are roughly the same regardless of if we use a lower or higher rate. + +We used `onion-grab`'s `scripts/test.sh` to perform the above experiment from +VM-1. The link for downloading the data is listed above in the summary. You +should see 3 subdirectories with results from 28 different measurements. + +Let's look at the results in more detail: the error rates that are printed in +the `stderr.txt` files, as well as the parsed output using `scripts/digest.py`. + +#### Scan: Stockholm with limit 1450/s + + $ digest.py -i 20230402-232627/se17-wireguard-l1450.txt 2>&1 | + tail -n6 | head -n4 + digest.py:25 INFO: found 245 HTTP headers with Onion-Location + digest.py:26 INFO: found 42 HTML meta attributes with Onion-Location + digest.py:27 INFO: found 283 unqiue domain names that set Onion-Location + digest.py:28 INFO: found 205 unique two-label onion addresses in the process + +#### Scan: Stockholm, New York, Melbourne with limit 1450s (combined) + + $ digest.py -i 20230402-232627/*l1450.txt 2>&1 | tail -n4 | head -n2 + digest.py:27 INFO: found 285 unqiue domain names that set Onion-Location + digest.py:28 INFO: found 207 unique two-label onion addresses in the process + +Note that we found more Onion-Location setups here with the combined scan. + +#### Scan: Stockholm, New York, Melbourne with limits 100, 500, 1450 (combined) + + $ cat 20230402-232627/stderr.txt | tail -n5 | head -n2 + digest.py:27 INFO: found 285 unqiue domain names that set Onion-Location + digest.py:28 INFO: found 207 unique two-label onion addresses in the process + +Note that we did not find more Onion-Location setups now with 9x measurements. +This observation holds true if `scripts/digest.py` is run with all 28 outputs: + + $ ./scripts/digest.py -i\ + 20230402-232627/*-*-*\ + 20230403-124743/*-*-*\ + 20230403-181847/*-*-* 2>&1 | tail -n4 | head -n2 + digest.py:27 INFO: found 285 unqiue domain names that set Onion-Location + digest.py:28 INFO: found 207 unique two-label onion addresses in the process + +#### Error rates + +Below are some pretty-printed output from the error rates shown in the +respective `stderr.txt` files, ordered by the relay and limit that we set. The +maximum number of connects is 1M; all columns after that provide info about +failed connection attempts. E.g., the first row has 82814 DNS lookup errors. + +| relay hostname | limit/s | rate/s | onions | connected | dns (NotFound/Timeout/Other) | tcp (Timeout/Syscall) | tls (Cert/Other) | 3xx | eof | ctx | ??? | +| -------------- | ------- | ------ | ------ | --------- | ---------------------------- | --------------------- | ------------------- | ---- | ---- | ----- | ---- | +| us18-wireguard | 100 | 100.0 | 287 | 711816 | 82814 (72767 843 9204) | 51543 (21279 30264) | 87147 (77235 9912) | 2042 | 5449 | 58932 | 257 | +| us18-wireguard | 500 | 500.3 | 285 | 711373 | 83333 (72811 1304 9218) | 54058 (24064 29994) | 86728 (76803 9925) | 2160 | 5414 | 56689 | 245 | +| us18-wireguard | 1000 | 1001.0 | 286 | 711081 | 82882 (72804 852 9226) | 54763 (24599 30164) | 86840 (77011 9829) | 1760 | 5086 | 57333 | 255 | +| us18-wireguard | 1200 | 1201.5 | 286 | 711741 | 82841 (72800 855 9186) | 53041 (22654 30387) | 86885 (77111 9774) | 1803 | 4955 | 58485 | 249 | +| us18-wireguard | 1400 | 1402.1 | 287 | 710481 | 82894 (72805 1468 8621) | 59711 (29489 30222) | 86597 (76897 9700) | 1638 | 4975 | 53450 | 254 | +| us18-wireguard | 1450 | 1452.2 | 287 | 708649 | 82866 (72820 1272 8774) | 60294 (30460 29834) | 86506 (76602 9904) | 1887 | 5233 | 54298 | 267 | + +| relay hostname | limit/s | rate/s | onions | connected | dns (NotFound/Timeout/Other) | tcp (Timeout/Syscall) | tls (Cert/Other) | 3xx | eof | ctx | ??? | +| -------------- | ------- | ------ | ------ | --------- | ---------------------------- | --------------------- | ------------------- | ---- | ---- | ----- | ---- | +| au-syd-wg-002 | 100 | 100.0 | 285 | 723854 | 83319 (72800 1317 9202) | 48693 (14767 33926) | 91658 (81324 10334) | 1810 | 5235 | 45149 | 282 | +| au-syd-wg-002 | 500 | 500.3 | 285 | 723410 | 83119 (72791 1119 9209) | 51229 (16767 34462) | 91585 (81208 10377) | 1830 | 4680 | 43876 | 271 | +| au-syd-wg-002 | 1000 | 1001.0 | 285 | 724144 | 83052 (72771 1075 9206) | 50697 (16591 34106) | 91678 (81442 10236) | 1491 | 4922 | 43733 | 283 | +| au-syd-wg-002 | 1200 | 1192.3 | 286 | 723169 | 83090 (72820 1122 9148) | 51408 (16685 34723) | 91571 (81354 10217) | 1413 | 5024 | 44052 | 273 | +| au-syd-wg-002 | 1400 | 1391.8 | 286 | 721119 | 83305 (72796 1906 8603) | 55236 (21640 33596) | 91339 (81197 10142) | 842 | 5752 | 42124 | 283 | +| au-syd-wg-002 | 1450 | 1431.3 | 285 | 720439 | 83182 (72793 1498 8891) | 56817 (23193 33624) | 91376 (81049 10327) | 1100 | 5486 | 41334 | 266 | + +| relay hostname | limit/s | rate/s | onions | connected | dns (NotFound/Timeout/Other) | tcp (Timeout/Syscall) | tls (Cert/Other) | 3xx | eof | ctx | ??? | +| -------------- | ------- | ------ | ------ | --------- | ---------------------------- | --------------------- | ------------------- | ---- | ---- | ----- | ---- | +| se17-wireguard | 100 | 100.0 | 286 | 724643 | 83146 (72400 954 9792) | 48497 (14711 33786) | 92230 (81881 10349) | 2081 | 5815 | 43325 | 263 | +| se17-wireguard | 500 | 500.3 | 288 | 723176 | 84208 (72453 1367 10388) | 48685 (15239 33446) | 91664 (81341 10323) | 2073 | 5513 | 44416 | 265 | +| se17-wireguard | 1000 | 1001.0 | 289 | 723834 | 83156 (72427 962 9767) | 49559 (16347 33212) | 91847 (81572 10275) | 1852 | 5638 | 43856 | 258 | +| se17-wireguard | 1200 | 1201.5 | 289 | 724093 | 83078 (72450 905 9723) | 48780 (15597 33183) | 91868 (81656 10212) | 1823 | 5708 | 44389 | 261 | +| se17-wireguard | 1200 | 1201.5 | 289 | 723788 | 83081 (72397 950 9734) | 49070 (15848 33222) | 91745 (81595 10150) | 1790 | 5670 | 44589 | 267 | +| se17-wireguard | 1201 | 1202.5 | 288 | 723642 | 83063 (72413 909 9741) | 48923 (15769 33154) | 92120 (81575 10545) | 1823 | 5322 | 44839 | 268 | +| se17-wireguard | 1202 | 1202.1 | 290 | 723846 | 83055 (72452 912 9691) | 48999 (15916 33083) | 91860 (81519 10341) | 1813 | 5497 | 44669 | 261 | +| se17-wireguard | 1203 | 1204.5 | 289 | 723772 | 83051 (72479 882 9690) | 48926 (15775 33151) | 91945 (81630 10315) | 1825 | 5502 | 44716 | 263 | +| se17-wireguard | 1204 | 1205.5 | 290 | 723816 | 83109 (72462 902 9745) | 49256 (16161 33095) | 92015 (81551 10464) | 1762 | 5364 | 44420 | 258 | +| se17-wireguard | 1400 | 1402.1 | 288 | 721902 | 83808 (72426 1341 10041) | 51820 (18732 33088) | 91409 (81308 10101) | 1727 | 5725 | 43345 | 264 | +| se17-wireguard | 1446 | 1448.2 | 290 | 720637 | 83037 (72463 924 9650) | 49421 (16422 32999) | 91416 (81132 10284) | 1801 | 5517 | 47903 | 268 | +| se17-wireguard | 1447 | 1449.2 | 286 | 720927 | 83038 (72480 930 9628) | 49361 (16463 32898) | 91630 (81243 10387) | 1807 | 5399 | 47580 | 258 | +| se17-wireguard | 1448 | 1450.2 | 288 | 720841 | 83016 (72492 933 9591) | 49251 (16209 33042) | 91636 (81236 10400) | 1803 | 5410 | 47783 | 260 | +| se17-wireguard | 1449 | 1449.4 | 288 | 720456 | 83065 (72459 922 9684) | 49513 (16554 32959) | 91479 (81171 10308) | 1786 | 5459 | 47981 | 261 | +| se17-wireguard | 1450 | 1450.3 | 288 | 720684 | 83036 (72476 915 9645) | 49348 (16266 33082) | 91608 (81238 10370) | 1734 | 5404 | 47932 | 254 | +| se17-wireguard | 1450 | 1450.0 | 287 | 719193 | 83193 (72428 1319 9446) | 53567 (20562 33005) | 91390 (81135 10255) | 1956 | 5775 | 44641 | 285 | + +From the looks of it, the number of successful connections decrease somewhat as +we are approaching the 1450/s limit. Comparing the most successful and least +successful runs with regards to the number of connects we get per location: + + - Melbourne: 3705 + - New York: 3167 + - Stockholm: 5450 + +These differences are mostly due to more TCP timeouts and context deadlines. + +#### What does this mean + +Running from three different locations at limit 1450/s finds the same number of +Onion-Location setups as all 28 measurements combined. That's what we wanted. + +Connect errors (mainly TCP timeouts and context deadline errors) increase +slightly as we use the higher limits. This is not what we wanted. However, the +increase in connect errors per 1M sites is only 0.3-0.5%. These errors are +transient, and should mostly be accounted for by having 3x tries per domain. + +(Each scan is running with a shuffled list, similar to our full measurement.) + +**Conclusion:** scanning from three different locations at limit 1450/s strikes +a good balance between found Onion-Locations, errors, and timeliness of results. + +### 2 + +The [ct-sans dataset][] that we will `onion-grab` in the full measurement was +collected and assembled at 2023-04-03. It contains 0.91B unique SANs. + +[ct-sans dataset]: https://git.cs.kau.se/rasmoste/ct-sans/-/blob/main/docs/operations.md + +To avoid biases like encountering the same errors at all VMs due to the order in +which the sites were visited, the dataset is shuffled separately before use. + +We did all shuffling on VM-1 because it has the most disk available. + +Prepare shuffled dataset for VM-1: + + $ seed="2023-04-03-vm-1" + $ time shuf\ + --random-source <(openssl enc -aes-256-ctr -pass pass:"$seed" -nosalt </dev/zero 2>/dev/null)\ + -o vm-1.lst 2023-04-03-ct-sans/sans.lst + + real 13m40.637s + user 10m30.368s + sys 2m28.062s + $ time sha256sum vm-1.lst + 4bf4b2701e01dd7886757987a2a9f2750aff677c2bd9f3e28d6ca8a1b7c25a3b vm-1.lst + + real 2m51.630s + user 2m33.246s + sys 0m11.460s + +Prepare shuffled dataset for VM-2: + + $ seed="2023-04-03-vm-2" + $ time shuf\ + --random-source <(openssl enc -aes-256-ctr -pass pass:"$seed" -nosalt </dev/zero 2>/dev/null)\ + -o vm-2.lst 2023-04-03-ct-sans/sans.lst + + real 14m35.500s + user 11m31.577s + sys 2m31.447s + $ time sha256sum vm-2.lst + 46f6c4af1e215f2d2cfb3ee302c8e3d02f43d4d918eb42f300a818e68f73f7ff vm-2.lst + + real 3m7.084s + user 2m36.416s + sys 0m19.012s + +Prepare shuffled dataset for VM-3: + + $ seed="2023-04-03-vm-3" + $ time shuf\ + --random-source <(openssl enc -aes-256-ctr -pass pass:"$seed" -nosalt </dev/zero 2>/dev/null)\ + -o vm-3.lst 2023-04-03-ct-sans/sans.lst + + real 14m37.878s + user 11m37.963s + sys 2m20.373s + $ time sha256sum vm-3.lst + c2df53320c1e7ab21355c9ebc1e53b1a8f564c9e7a2bd3e24f2cc8fca8b9eaf6 vm-3.lst + + real 3m6.324s + user 2m36.804s + sys 0m17.056s + +Double-check that we have the expected number of lines: + + time wc -l vm-?.lst 2023-04-03-ct-sans/sans.lst + 907332515 vm-1.lst + 907332515 vm-2.lst + 907332515 vm-3.lst + 907332515 2023-04-03-ct-sans/sans.lst + 3629330060 total + + real 7m54.915s + user 0m59.213s + sys 1m25.353s + +**Note:** `shuf` is memory-hungry and needs ~2x the size of the input file. So, +anything less than ~60GiB memory will be insufficient for a 25GiB dataset. + +### 3 + + $ ulimit -Sn 100000 + $ ulimit -a >ulimit.txt + # sysctl -w net.ipv4.ip_local_port_range="1024 65535" + # sysctl -a >sysctl.txt + $ go install git.cs.kau.se/rasmoste/onion-grab@v0.0.2 + $ git log | head -n1 + commit abce43c4ad9000e0c5c83d31c2185986ab8a54c9 + $ cd scripts + $ sha256sum vm-1.lst + 4bf4b2701e01dd7886757987a2a9f2750aff677c2bd9f3e28d6ca8a1b7c25a3b vm-1.lst + $ git diff + diff --git a/scripts/measure.sh b/scripts/measure.sh + index a520c6d..269b5ad 100755 + --- a/scripts/measure.sh + +++ b/scripts/measure.sh + @@ -8,11 +8,11 @@ + # lists 8.8.8.8 and 8.8.4.4, output of uname -a and sysctl -a is ..., etc. + # + + -relay_country=se + -relay_city=sto + +relay_country=au + +relay_city=mel + limit=1450 + num_workers=10000 + -input_file=example.lst + +input_file=vm-1.lst + timeout_s=30 + response_max_mib=64 + metrics_interval=1h + +So, we selected Melbourne relays. + + $ ./measure.sh 2>measure.stderr + +### 4 + + $ ulimit -Sn 100000 + $ ulimit -a >ulimit.txt + # sysctl -w net.ipv4.ip_local_port_range="1024 65535" + # sysctl -a >sysctl.txt + $ go install git.cs.kau.se/rasmoste/onion-grab@v0.0.2 + $ git log | head -n1 + commit abce43c4ad9000e0c5c83d31c2185986ab8a54c9 + $ cd scripts + $ sha256sum vm-2.lst + 46f6c4af1e215f2d2cfb3ee302c8e3d02f43d4d918eb42f300a818e68f73f7ff vm-2.lst + $ git diff + diff --git a/scripts/measure.sh b/scripts/measure.sh + index a520c6d..31b2f9e 100755 + --- a/scripts/measure.sh + +++ b/scripts/measure.sh + @@ -8,11 +8,11 @@ + # lists 8.8.8.8 and 8.8.4.4, output of uname -a and sysctl -a is ..., etc. + # + + -relay_country=se + -relay_city=sto + +relay_country=us + +relay_city=nyc + limit=1450 + num_workers=10000 + -input_file=example.lst + +input_file=vm-2.lst + timeout_s=30 + response_max_mib=64 + metrics_interval=1h + +So, we selected New York relays. + + $ ./measure.sh 2>measure.stderr + +### 5 + + $ ulimit -Sn 100000 + $ ulimit -a >ulimit.txt + # sysctl -w net.ipv4.ip_local_port_range="1024 65535" + # sysctl -a >sysctl.txt + $ go install git.cs.kau.se/rasmoste/onion-grab@v0.0.2 + $ git log | head -n1 + commit abce43c4ad9000e0c5c83d31c2185986ab8a54c9 + $ cd scripts + $ sha256sum vm-3.lst + c2df53320c1e7ab21355c9ebc1e53b1a8f564c9e7a2bd3e24f2cc8fca8b9eaf6 vm-3.lst + $ git diff + diff --git a/scripts/measure.sh b/scripts/measure.sh + index a520c6d..4cc0913 100755 + --- a/scripts/measure.sh + +++ b/scripts/measure.sh + @@ -12,7 +12,7 @@ relay_country=se + relay_city=sto + limit=1450 + num_workers=10000 + -input_file=example.lst + +input_file=vm-3.lst + timeout_s=30 + response_max_mib=64 + metrics_interval=1h + +So, we selected Stockholm relays (default). + + $ ./measure.sh 2>measure.stderr + +### 6 + +Notice that Stockholm relays are "slow". Bandwidth appear to have dropped to +1/10 of the initial part of the measurement. Unclear if there are more errors +yet or not, and if this will sort itself out. Adding a cron job that prints +hourly bandwidth stats every day at 23:59 to store more fine-grained data: + + $ mkdir /home/rasmoste/vnstat + $ crontab -e + +And add at the end of the file: + + 59 23 * * * vnstat -h >"/home/rasmoste/vnstat/$(date)" + +(Added this on all three VMs.) + +### 7 + +(In VM-3) + +Bandwidth stats: + + $ cat Tue\ Apr\ \ 4\ 11\:59\:01\ PM\ UTC\ 2023 + + ens160 / hourly + + hour rx | tx | total | avg. rate + ------------------------+-------------+-------------+--------------- + 2023-04-04 + 00:00 82.61 GiB | 12.78 GiB | 95.39 GiB | 227.61 Mbit/s + 01:00 80.93 GiB | 12.70 GiB | 93.63 GiB | 223.41 Mbit/s + 02:00 80.90 GiB | 12.68 GiB | 93.58 GiB | 223.30 Mbit/s + 03:00 81.13 GiB | 12.63 GiB | 93.77 GiB | 223.74 Mbit/s + 04:00 88.59 GiB | 12.97 GiB | 101.57 GiB | 242.35 Mbit/s + 05:00 85.10 GiB | 12.93 GiB | 98.04 GiB | 233.92 Mbit/s + 06:00 82.97 GiB | 12.84 GiB | 95.81 GiB | 228.61 Mbit/s + 07:00 79.05 GiB | 12.62 GiB | 91.67 GiB | 218.72 Mbit/s + 08:00 87.83 GiB | 12.81 GiB | 100.64 GiB | 240.13 Mbit/s + 09:00 81.22 GiB | 12.62 GiB | 93.84 GiB | 223.91 Mbit/s + 10:00 79.26 GiB | 12.57 GiB | 91.83 GiB | 219.12 Mbit/s + 11:00 81.70 GiB | 12.67 GiB | 94.37 GiB | 225.17 Mbit/s + 12:00 97.83 GiB | 13.21 GiB | 111.04 GiB | 264.94 Mbit/s + 13:00 82.47 GiB | 12.59 GiB | 95.06 GiB | 226.83 Mbit/s + 14:00 78.42 GiB | 11.46 GiB | 89.88 GiB | 214.45 Mbit/s + 15:00 27.42 GiB | 5.95 GiB | 33.37 GiB | 79.62 Mbit/s + 16:00 23.30 GiB | 5.37 GiB | 28.67 GiB | 68.42 Mbit/s + 17:00 28.12 GiB | 6.03 GiB | 34.15 GiB | 81.48 Mbit/s + 18:00 48.01 GiB | 8.76 GiB | 56.77 GiB | 135.46 Mbit/s + 19:00 40.23 GiB | 7.73 GiB | 47.97 GiB | 114.46 Mbit/s + 20:00 55.55 GiB | 9.63 GiB | 65.18 GiB | 155.52 Mbit/s + 21:00 35.10 GiB | 7.06 GiB | 42.16 GiB | 100.60 Mbit/s + 22:00 20.94 GiB | 5.00 GiB | 25.94 GiB | 61.91 Mbit/s + 23:00 21.19 GiB | 4.95 GiB | 26.14 GiB | 68.03 Mbit/s + ------------------------+-------------+-------------+--------------- + +We were hoping that this was a transient error, but all relays in Stockholm +appear to underperform. The rate has dropped as a result, and the number of +successes as well. See separate data and log files in our dataset (`se-sto/`). + +It will be faster, and give more accurate results, to start from a new location. + +Kill: `pidof onion-grab`, `kill <PID>`. + +Move `measure.stderr` to the data dir to not overwrite it when we restart. + +### 8 + +(In VM-3.) + +We experienced the same "slowness" with both Gothenburg and Malmo relays. When +moving our measurement to Frankfurt, good bandwidth is observed again. + + diff --git a/scripts/measure.sh b/scripts/measure.sh + index a520c6d..d46f9c1 100755 + --- a/scripts/measure.sh + +++ b/scripts/measure.sh + @@ -8,11 +8,11 @@ + # lists 8.8.8.8 and 8.8.4.4, output of uname -a and sysctl -a is ..., etc. + # + + -relay_country=se + -relay_city=sto + +relay_country=de + +relay_city=fra + limit=1450 + num_workers=10000 + -input_file=example.lst + +input_file=vm-3.lst + timeout_s=30 + response_max_mib=64 + metrics_interval=1h + +So, we selected Frankfurt relays. + +Without any other restarts in the same tmux pane as before: + + $ ./measure.sh 2>measure.stderr + +### 9 + +The summary prints (which means that the Go receiver routine waited for an +answer for at least one timeout and shutdown) are shown in `onion-grab`'s stderr +output, however `onion-grab` hangs after that so the measure.sh script doesn't +exit. + + - VM-1 (au mel) processed up until: 907330676 + - VM-2 (us nyc) processed up until: 907330662 + +To be compared with the number of entries in the ct-sans dataset: 907332515. + + $ python3 -c "print(f'{907332515 - 907330676}')" + 1839 + $ python3 -c "print(f'{907332515 - 907330662}')" + 1853 + +So, it appears that we have ~1800 workers that were unable to provide their +final answers (most likely timeouts) before the receiver routine shutdown. This +explains why `onion-grab` hangs, i.e., there are still workers that are waiting +to send their answers to the receiver who is not reading answers anymore. + +In addition to the outstanding answers most likely being timeouts, it is not the +same ~1800 answers on all machines since the dataset was shuffled for all VMs. + +**Action:** ctrl+C the measurement script that is waiting for `onion-grab` to +complete, we already have the `onion-grab` output that we want stored to disk. + +### 10 + +Latest `onion-grab` stderr print was 2023/04/11 20:25:50, then died due to too +little memory. Latest progress print was: + + 2023/04/11 20:02:33 INFO: metrics@receiver: + + Processed: 819368251 + +So, we can safely continue without missing any sites with Onion-Location +configured by starting a new measurement from line ~819368251. + + $ python3 -c "print(f'{907332515 - 819368251}')" + 87964264 + $ tail -n87964264 vm-3.lst > vm-3-remaining.lst + $ wc -l vm-3-remaining.lst + 87964264 vm-3-remaining.lst + +### 11 + +Restart `onion-grab` from VM-3 with the final domain names to visit. + + $ git diff + diff --git a/scripts/measure.sh b/scripts/measure.sh + index a520c6d..6d77c66 100755 + --- a/scripts/measure.sh + +++ b/scripts/measure.sh + @@ -8,11 +8,11 @@ + # lists 8.8.8.8 and 8.8.4.4, output of uname -a and sysctl -a is ..., etc. + # + + -relay_country=se + -relay_city=sto + +relay_country=de + +relay_city=fra + limit=1450 + num_workers=10000 + -input_file=example.lst + +input_file=vm-3-remaining.lst + timeout_s=30 + response_max_mib=64 + metrics_interval=1h + $ ./measure.sh 2>measure-remaining.stderr + +(`onion-grab` results are written to a separate directory that is timestamped, +so there is no risk that the above command will overwrite any collected data.) + +### 12 + +Latest `onion-grab` stderr print was 2023/04/11 23:43:19, then died due to too +little memory. Latest progress print was: + + 2023/04/11 23:36:31 INFO: metrics@receiver: + + Processed: 5217381 + +So, we can safely continue without missing any sites with Onion-Location +configured by starting a new measurement from line ~5217381. + + $ python3 -c "print(f'{87964264 - 5217381}')" + 82746883 + $ tail -n82746883 vm-3-remaining.lst > vm-3-remaining-2.lst + $ wc -l vm-3-remaining-2.lst + 82746883 vm-3-remaining-2.lst + +### 13 + +Restart `onion-grab` from VM-3 with the final domain names to visit. However +reducing the number of workers to see if that can keep us from blowing up. If +this doesn't work we will have to bump the amount of memory in our VM. + +(The large amount of workers is anyway not necessary with low latency.) + + $ git diff + diff --git a/scripts/measure.sh b/scripts/measure.sh + index a520c6d..3b2e54b 100755 + --- a/scripts/measure.sh + +++ b/scripts/measure.sh + @@ -8,11 +8,11 @@ + # lists 8.8.8.8 and 8.8.4.4, output of uname -a and sysctl -a is ..., etc. + # + + -relay_country=se + -relay_city=sto + +relay_country=de + +relay_city=fra + limit=1450 + -num_workers=10000 + -input_file=example.lst + +num_workers=4000 + +input_file=vm-3-remaining-2.lst + timeout_s=30 + response_max_mib=64 + metrics_interval=1h + $ ./measure.sh 2>measure-remaining-2.stderr + +### 14 + +Renaming and moving output in VM-1: + + $ mv data/20230403-223517 au-mel + $ rmdir data + $ mv au-mel/au-mel-l1450.stderr au-mel/onion-grab.stderr + $ mv au-mel/au-mel-l1450.stdout au-mel/onion-grab.stdout + $ mv sysctl.txt au-mel/ + $ mv ulimit.txt au-mel/ + $ mv measure.stderr au-mel/ + $ ls -l au-mel/ + total 6992 + -rw-rw-r-- 1 rasmoste rasmoste 800 Apr 3 22:36 measure.stderr + -rw-rw-r-- 1 rasmoste rasmoste 3749490 Apr 11 08:21 onion-grab.stderr + -rw-rw-r-- 1 rasmoste rasmoste 3346026 Apr 11 04:29 onion-grab.stdout + -rw-rw-r-- 1 rasmoste rasmoste 42500 Apr 3 22:11 sysctl.txt + -rw-rw-r-- 1 rasmoste rasmoste 823 Apr 3 22:11 ulimit.txt + $ mv au-mel ~/exp/onion-grab/data/2023-04-03-ct-sans/ + +### 15 + +Renaming and moving output in VM-2: + + $ mv data/20230403-223519 us-nyc + $ rmdir data + $ mv us-nyc/us-nyc-l1450.stdout us-nyc/onion-grab.stdout + $ mv us-nyc/us-nyc-l1450.stderr us-nyc/onion-grab.stderr + $ mv sysctl.txt us-nyc/ + $ mv ulimit.txt us-nyc/ + $ mv measure.stderr us-nyc/ + $ ls -l us-nyc + total 6784 + -rw-rw-r-- 1 rasmoste rasmoste 800 Apr 3 22:35 measure.stderr + -rw-rw-r-- 1 rasmoste rasmoste 3553624 Apr 11 08:21 onion-grab.stderr + -rw-rw-r-- 1 rasmoste rasmoste 3326545 Apr 11 04:25 onion-grab.stdout + -rw-rw-r-- 1 rasmoste rasmoste 42531 Apr 3 22:12 sysctl.txt + -rw-rw-r-- 1 rasmoste rasmoste 823 Apr 3 22:11 ulimit.txt + +Zip and checksum before moving to VM-1: + + $ zip -r us-nyc.zip us-nyc/ + $ sha256sum us-nyc.zip + 8759b8e7192390cc8f125a795c55b55ad9ecadb27344ce88004998ca89b7c4be us-nyc.zip + +Transfer to VM-1, check that checksum is OK then unzip. + +### 16 + +Renaming an moving in VM-3: + + $ mv data/20230403-223521 se-sto + $ mv se-sto/se-sto-l1450.stderr se-sto/onion-grab.stderr + $ mv se-sto/se-sto-l1450.stdout se-sto/onion-grab.stdout + $ cp ulimit.txt se-sto/ + $ cp sysctl.txt se-sto/ + $ mkdir se-sto/bw + $ cp ~/vnstat/"Tue Apr 4 11:59:01 PM UTC 2023" se-sto/bw + $ cp ~/vnstat/"Wed Apr 5 11:59:01 PM UTC 2023" se-sto/bw + $ ls -l se-sto + total 912 + drwxrwxr-x 2 rasmoste rasmoste 4096 Apr 12 16:55 bw + -rw-rw-r-- 1 rasmoste rasmoste 801 Apr 3 22:35 measure.stderr + -rw-rw-r-- 1 rasmoste rasmoste 441711 Apr 5 06:36 onion-grab.stderr + -rw-rw-r-- 1 rasmoste rasmoste 424925 Apr 5 06:27 onion-grab.stdout + -rw-rw-r-- 1 rasmoste rasmoste 42529 Apr 12 16:54 sysctl.txt + -rw-rw-r-- 1 rasmoste rasmoste 823 Apr 12 16:54 ulimit.txt + $ zip -r se-sto.zip se-sto/ + $ sha256sum se-sto.zip + 6fcd5640b1022828d19f3585b2a9c9488ce5c681a81a61c22b1bd4cbbe326b49 se-sto.zip + +Move to VM-1, check checksum and unzip. + +### 17 + +VM-1: + + $ mv ~/vnstat au-mel/bw + +Then stop the cronjob that creates bw output (`crontab -e`). + +VM-2: + + $ mv ~/vnstat bw + $ zip -r bw.zip bw/ + $ sha256sum bw.zip + c4753326fcdb4dd136af81c1359cfe37fe6756726c497f39d3c33f799fc975f3 bw.zip + +Transfer to VM-1, check checksum, unzip and put in us-nyc directory. Then stop +the cronjob that creates bw output in VM-2 as well. + +### 18 + +`onion-grab` hangs on shutdown similar to VM-1 and VM-2 [9]. The final summary +print shows processed until 82746708, which should be compared to the size of +82746883 (vm-3-remaining-2.lst). I.e., 175 missing workers/answers. + +Same action as in [9], ctrl+C measurement script. + +### 19 + +Renaming and moving in VM-3, first run: + + $ mv data/20230405-070154 de-fra + $ mv de-fra/de-fra-l1450.stderr de-fra/onion-grab.stderr + $ mv de-fra/de-fra-l1450.stdout de-fra/onion-grab.stdout + $ mv measure.stderr de-fra/measure.stderr + $ mv ulimit.txt de-fra/ + $ mv sysctl.txt de-fra/ + +Second run: + + $ mv data/20230411-223623/de-fra-l1450.stderr de-fra/onion-grab-2.stderr + $ mv data/20230411-223623/de-fra-l1450.stdout de-fra/onion-grab-2.stdout + $ rmdir data/20230411-223623 + $ mv measure-remaining.stderr de-fra/measure-2.stderr + +Third run: + + $ mv data/20230412-084228/de-fra-l1450.stderr de-fra/onion-grab-3.stderr + $ mv data/20230412-084228/de-fra-l1450.stdout de-fra/onion-grab-3.stdout + $ rmdir data/20230412-084228 + $ mv measure-remaining-2.stderr de-fra/measure-3.stderr + +Grab bandwidths, exclude output from 4th since this measurement started 5th: + + $ rm ~/vnstat/"Tue Apr 4 11:59:01 PM UTC 2023" + $ vnstat -h >"/home/rasmoste/vnstat/$(date)" + $ mv ~/vnstat de-fra/bw + +Overview: + + $ ls -l de-fra + total 6768 + drwxrwxr-x 2 rasmoste rasmoste 4096 Apr 13 05:39 bw + -rw-rw-r-- 1 rasmoste rasmoste 1019 Apr 11 23:43 measure-2.stderr + -rw-rw-r-- 1 rasmoste rasmoste 810 Apr 12 08:42 measure-3.stderr + -rw-rw-r-- 1 rasmoste rasmoste 1009 Apr 11 20:25 measure.stderr + -rw-rw-r-- 1 rasmoste rasmoste 24004 Apr 11 23:43 onion-grab-2.stderr + -rw-rw-r-- 1 rasmoste rasmoste 23002 Apr 11 23:42 onion-grab-2.stdout + -rw-rw-r-- 1 rasmoste rasmoste 318627 Apr 13 05:38 onion-grab-3.stderr + -rw-rw-r-- 1 rasmoste rasmoste 312774 Apr 13 00:34 onion-grab-3.stdout + -rw-rw-r-- 1 rasmoste rasmoste 3117995 Apr 11 20:25 onion-grab.stderr + -rw-rw-r-- 1 rasmoste rasmoste 3034130 Apr 11 20:25 onion-grab.stdout + -rw-rw-r-- 1 rasmoste rasmoste 42529 Apr 3 22:12 sysctl.txt + -rw-rw-r-- 1 rasmoste rasmoste 823 Apr 3 22:11 ulimit.txt + +Then stop the cronjob that creates bw outputs (`crontab -e`). + +Zip, checksum, and transfer to VM-1: + + $ zip -r de-fra.zip de-fra/ + $ sha256sum de-fra.zip + 2ea1f053decea3915b29bc60c2f954da55ea48f6d8ab9f47112caddf3a2e2f7f de-fra.zip |