From 8e0fa61c06fd12c502ea171bee65f5fd63ccb158 Mon Sep 17 00:00:00 2001 From: Rasmus Dahlberg Date: Fri, 7 Apr 2023 21:34:13 +0200 Subject: Add docs --- docs/notes.md | 24 ++++++++++++++++++++++++ docs/operations.md | 3 +++ docs/setup.md | 19 +++++++++++++++++++ 3 files changed, 46 insertions(+) create mode 100644 docs/notes.md create mode 100644 docs/operations.md create mode 100644 docs/setup.md (limited to 'docs') diff --git a/docs/notes.md b/docs/notes.md new file mode 100644 index 0000000..d95c6f0 --- /dev/null +++ b/docs/notes.md @@ -0,0 +1,24 @@ +# Notes + +`onion-grab` uses use the default `net.Dial` function, which in turn uses +[goLookupIPCNAMEOrder][] for DNS lookups with the recursive name servers in +`/etc/resolve.conf`. For example, with + + $ cat /etc/resolve.conf + nameserver 8.8.8.8 + nameserver 8.8.4.4 + +[goLookupIPCNAMEOrder]: https://github.com/golang/go/blob/8edcdddb23c6d3f786b465c43b49e8d9a0015082/src/net/dnsclient_unix.go#L595-L804 + +the query will first be directed to `8.8.8.8`; then `8.8.4.4` if no valid answer +is available yet ([lines 663-778][]). If you are running `onion-grab` with +[Mullvad VPN][], specify custom DNS: `mullvad dns set custom 8.8.8.8 8.8.4.4`. + +[lines 663-778]: https://github.com/golang/go/blob/8edcdddb23c6d3f786b465c43b49e8d9a0015082/src/net/dnsclient_unix.go#L663-L778 +[Mullvad VPN]: https://www.mullvad.net/ + +Further, default settings are used to follow at most 10 HTTP 3XX redirects. A +[conservative][] value for the `MaxResponseHeaderBytes` option is 10MiB; the +`onion-grab` default is 16MiB and our measurements bumped this up to 64MiB. + +[conservative]: https://go-review.googlesource.com/c/go/+/21329/2/src/net/http/transport.go diff --git a/docs/operations.md b/docs/operations.md new file mode 100644 index 0000000..1528c32 --- /dev/null +++ b/docs/operations.md @@ -0,0 +1,3 @@ +# Operations + +Placeholder. diff --git a/docs/setup.md b/docs/setup.md new file mode 100644 index 0000000..7e4bdb8 --- /dev/null +++ b/docs/setup.md @@ -0,0 +1,19 @@ +# Setup + +`onion-grab` has been tested on Ubuntu/Debian based systems. If you are running +a large measurement, you may run into issues that are **OS related**. + +## Hints + + - We disabled and stopped `systemd-resolved`, which eventually causes some or + all DNS requests to be blocked when running with many concurrent workers. + - We used Google's `8.8.8.8` and `8.8.4.4`, which [supports 1500qps][] per IP. + - You may need to tinker with `ulimit` and `sysctl`, e.g., if observing that + there are too many open file descriptors or similar. See for example the + value of `ulimit -n` and `sysctl net.ipv4.ip_local_port_range` + +[supports 1500qps]: https://developers.google.com/speed/public-dns/docs/isp + +## Credit + +Björn Töpel helped us debug some of these OS-related issues. -- cgit v1.2.3