From 15f5577a2cb2f05977de38eb8572afbff7c875da Mon Sep 17 00:00:00 2001 From: Rasmus Dahlberg Date: Sun, 26 Mar 2023 13:25:19 +0200 Subject: Add some debug notes --- README.md | 13 +++++++++++++ 1 file changed, 13 insertions(+) diff --git a/README.md b/README.md index 25299c8..b28dac4 100644 --- a/README.md +++ b/README.md @@ -132,6 +132,19 @@ domains/s, it would be wise to roll-back _at least_ 6000 lines. Get in touch if you know a fix, e.g., based on `ulimit` and `sysctl` tinkering. +More debug notes: + + - My system is not fully utilized wrt. CPU/MEM/BW; an odd thing is that it + seems to work fine to run multiple onion-grab instances as separate + commands, e.g., 3x `-w 280` to get up to ~225 Mbps utilization (max). + Adding a fourth instance gets me into the same problem as documented above. + - Tinkering with with options in http.Transport doesn't seem help. + - Using multiple http.Client doesn't help (e.g., one per worker) + - An odd thing is that after errors, it appears that only DNS is dead. E.g., + `curl https://www.rgdd.se` fails while `curl --resolve + www.rgdd.se:443:213.164.207.87` succeeds. Replacing the system's DNS with a + local unbound process doesn't seem to help though. + ## Contact - rasmus (at) rgdd (dot) se -- cgit v1.2.3