Add docs

3 files changed, 46 insertions, 0 deletions

diff --git a/docs/notes.md b/docs/notes.md
new file mode 100644
index 0000000..d95c6f0
--- /dev/null
+++ b/docs/notes.md
@@ -0,0 +1,24 @@
+# Notes
+
+`onion-grab` uses use the default `net.Dial` function, which in turn uses
+[goLookupIPCNAMEOrder][] for DNS lookups with the recursive name servers in
+`/etc/resolve.conf`.  For example, with
+
+    $ cat /etc/resolve.conf
+    nameserver 8.8.8.8
+    nameserver 8.8.4.4
+
+[goLookupIPCNAMEOrder]: https://github.com/golang/go/blob/8edcdddb23c6d3f786b465c43b49e8d9a0015082/src/net/dnsclient_unix.go#L595-L804
+
+the query will first be directed to `8.8.8.8`; then `8.8.4.4` if no valid answer
+is available yet ([lines 663-778][]).  If you are running `onion-grab` with
+[Mullvad VPN][], specify custom DNS: `mullvad dns set custom 8.8.8.8 8.8.4.4`.
+
+[lines 663-778]: https://github.com/golang/go/blob/8edcdddb23c6d3f786b465c43b49e8d9a0015082/src/net/dnsclient_unix.go#L663-L778
+[Mullvad VPN]: https://www.mullvad.net/
+
+Further, default settings are used to follow at most 10 HTTP 3XX redirects.  A
+[conservative][] value for the `MaxResponseHeaderBytes` option is 10MiB; the
+`onion-grab` default is 16MiB and our measurements bumped this up to 64MiB.
+
+[conservative]: https://go-review.googlesource.com/c/go/+/21329/2/src/net/http/transport.go
diff --git a/docs/operations.md b/docs/operations.md
new file mode 100644
index 0000000..1528c32
--- /dev/null
+++ b/docs/operations.md
@@ -0,0 +1,3 @@
+# Operations
+
+Placeholder.
diff --git a/docs/setup.md b/docs/setup.md
new file mode 100644
index 0000000..7e4bdb8
--- /dev/null
+++ b/docs/setup.md
@@ -0,0 +1,19 @@
+# Setup
+
+`onion-grab` has been tested on Ubuntu/Debian based systems.  If you are running
+a large measurement, you may run into issues that are **OS related**.
+
+## Hints
+
+  - We disabled and stopped `systemd-resolved`, which eventually causes some or
+    all DNS requests to be blocked when running with many concurrent workers.
+  - We used Google's `8.8.8.8` and `8.8.4.4`, which [supports 1500qps][] per IP.
+  - You may need to tinker with `ulimit` and `sysctl`, e.g., if observing that
+    there are too many open file descriptors or similar.  See for example the
+    value of `ulimit -n` and `sysctl net.ipv4.ip_local_port_range`
+
+[supports 1500qps]: https://developers.google.com/speed/public-dns/docs/isp
+
+## Credit
+
+Björn Töpel helped us debug some of these OS-related issues.