diff options
| -rw-r--r-- | README.md | 138 |
1 files changed, 136 insertions, 2 deletions
@@ -1,3 +1,137 @@ -# find-onion +# onion-grab -docdoc +A tool that visits a list of domains over HTTPS to see if they have +[Onion-Location][] configured. + +[Onion-Location]: https://community.torproject.org/onion-services/advanced/onion-location/ + +**Warning:** research prototype. The source code may also be moved. + +## Quickstart + +You will need a Go compiler on the local system: + + $ which go >/dev/null || echo "Go compiler is not in PATH" + +Install `onion-grab`: + + $ go install git.cs.kau.se/rasmoste/onion-grab@latest + +List all options: + + $ onion-grab -h + +### Basic usage + +Store domains in a file; one domain per line: + + $ cat domains.lst + www.eff.org + www.qubes-os.org + www.torproject.org + $ onion-grab -i domains.lst + 2023/03/25 17:43:30 INFO: starting await handler, ctrl+C to exit + 2023/03/25 17:43:30 INFO: starting 2 workers + 2023/03/25 17:43:30 INFO: starting work aggregator + 2023/03/25 17:43:30 INFO: generating work + www.qubes-os.org header= attribute=http://qubesosfasa4zl44o4tws22di6kepyzfeqv3tg4e3ztknltfxqrymdad.onion/ + www.torproject.org header=http://2gzyxa5ihm7nsggfxnu52rck2vv4rvmdlkiu3zzui5du4xyclen53wid.onion/index.html attribute= + 2023/03/25 17:43:40 INFO: about to exit, reading remaining answers + 2023/03/25 17:43:50 SUMMARY: 3/3 connected, 2 sites configured Onion-Location + +Sites with Onion-Location are printed to stdout, here showing that +`www.torproject.org` configures it with an HTTP header while `www.qubes-os.org` +does it with an HTML attribute. All three sites connected successfully. + +### Working with a larger list + +Below the [Tranco top-1M][] list is used as an example; 100 workers are +specified, metrics are printed every 15s, and sanity-checks against a site with +Onion-Location which should be reachable are carried out every 60s. + + $ cut -d ',' -f2 top-1m.csv > top-1m.lst + $ onion-grab -i top-1m.lst -w 100 -m 15s -C 60s -c rgdd.se + 2023/03/25 17:44:20 INFO: starting await handler, ctrl+C to exit + 2023/03/25 17:44:20 INFO: starting checker + 2023/03/25 17:44:20 INFO: starting 100 workers + 2023/03/25 17:44:20 INFO: starting work aggregator + 2023/03/25 17:44:20 INFO: generating work + nytimes.com header=https://www.nytimesn7cgmftshazwhfgzm37qxb44r64ytbb2dj3x62d2lljsciiyd.onion/ attribute= + twitter.com header= attribute=https://twitter3e4tixl4xyajtrzo62zg5vztmjuricljdp2c5kshju4avyoid.onion/ + theguardian.com header=https://www.guardian2zotagl6tmjucg3lrhxdk4dw3lhbqnkvvkywawy3oqfoprid.onion/international attribute= + 2023/03/25 17:44:22 Transport: unhandled response frame type *http.http2UnknownFrame + 2023/03/25 17:44:31 Transport: unhandled response frame type *http.http2UnknownFrame + dw.com header=https://www.dwnewsgngmhlplxy6o2twtfgjnrnjxbegbwqx6wnotdhkzt562tszfid.onion/ attribute= + brave.com header=https://brave4u7jddbv7cyviptqjc7jusxh72uik7zt6adtckl5f4nwy2v72qd.onion/index.html attribute= + 2023/03/25 17:44:35 INFO: currently 72.3 sites/s, 72.3 sites/s since start + guardian.co.uk header=https://www.guardian2zotagl6tmjucg3lrhxdk4dw3lhbqnkvvkywawy3oqfoprid.onion/international attribute= + proton.me header=https://protonmailrmez3lotccipshtkleegetolb73fuirgj7r4o4vfu7ozyd.onion/ attribute= + voanews.com header=https://www.voanews5aitmne6gs2btokcacixclgfl43cv27sirgbauyyjylwpdtqd.onion/ attribute= + 2023/03/25 17:44:50 INFO: currently 64.3 sites/s, 68.3 sites/s since start + ^C2023/03/25 17:44:51 INFO: about to exit, reading remaining answers + 2023/03/25 17:44:51 NOTICE: only read up until line 2089 + 2023/03/25 17:45:01 SUMMARY: 1488/2089 connected, 8 sites configured Onion-Location + +[Tranco top-1M]: https://tranco-list.eu/latest_list + +Note that `ctrl+C` can be used to exit early as shown above. To continue from +where you left off (line `2089`), specify the `-n` option on the next run: + + $ onion-grab -i top-1m.lst -w 100 -m 15s -C 60s -c rgdd.se -n 2089 + 2023/03/25 17:45:57 INFO: starting await handler, ctrl+C to exit + 2023/03/25 17:45:57 INFO: starting checker + 2023/03/25 17:45:57 INFO: starting 100 workers + 2023/03/25 17:45:57 INFO: starting work aggregator + 2023/03/25 17:45:57 INFO: generating work + cia.gov header= attribute=http://ciadotgov4sjwlzihbbgxnqg3xiyrg7so2r2o3lt5wz5ypk4sxyjstad.onion + 2023/03/25 17:46:12 INFO: currently 79.9 sites/s, 79.9 sites/s since start + propublica.org header=http://p53lf57qovyuvwsc6xnrppyply3vtqm7l6pcobkmyqsiofyeznfu5uqd.onion/ attribute= + theintercept.com header=https://54dus3ggt7uxz7wjvhkia2ntxmz5lkhbvgohrwur43trt3d6vrcvfmqd.onion/ attribute= + torproject.org header=http://2gzyxa5ihm7nsggfxnu52rck2vv4rvmdlkiu3zzui5du4xyclen53wid.onion/index.html attribute= + 2023/03/25 17:46:27 INFO: currently 75.1 sites/s, 77.5 sites/s since start + ^C2023/03/25 17:46:28 INFO: about to exit, reading remaining answers + 2023/03/25 17:46:28 NOTICE: only read up until line 4487 (line 2398 relative to start) + 2023/03/25 17:46:38 SUMMARY: 1609/2399 connected, 4 sites configured Onion-Location + +## Known issues + +### Too many parallel workers + +Here's what would happen if the local system cannot handle the number of workers: + + $ onion-grab -i top-1m.lst -w 1000 -m 15s -C 60s -c rgdd.se + 2023/03/25 17:47:36 INFO: starting await handler, ctrl+C to exit + 2023/03/25 17:47:36 INFO: starting checker + 2023/03/25 17:47:36 INFO: starting 1000 workers + 2023/03/25 17:47:36 INFO: starting work aggregator + 2023/03/25 17:47:36 INFO: generating work + 2023/03/25 17:47:36 Transport: unhandled response frame type *http.http2UnknownFrame + twitter.com header= attribute=https://twitter3e4tixl4xyajtrzo62zg5vztmjuricljdp2c5kshju4avyoid.onion/ + 2023/03/25 17:47:37 Transport: unhandled response frame type *http.http2UnknownFrame + brave.com header=https://brave4u7jddbv7cyviptqjc7jusxh72uik7zt6adtckl5f4nwy2v72qd.onion/index.html attribute= + 2023/03/25 17:47:51 INFO: currently 151.3 sites/s, 151.3 sites/s since start + 2023/03/25 17:48:06 INFO: currently 78.1 sites/s, 114.7 sites/s since start + 2023/03/25 17:48:21 INFO: currently 121.9 sites/s, 117.1 sites/s since start + 2023/03/25 17:48:36 INFO: currently 78.1 sites/s, 107.4 sites/s since start + 2023/03/25 17:48:46 ERROR: checker expected onion for {Domain:rgdd.se OK:false HTTP: HTML:} + 2023/03/25 17:48:46 NOTICE: only read up until line 7442 + 2023/03/25 17:48:46 INFO: about to exit, reading remaining answers + 2023/03/25 17:48:56 SUMMARY: 232/7442 connected, 2 sites configured Onion-Location + +On a Debian system, it appears that all future HTTP GET requests made by +`onion-grab` will fail if a worker overload happens. The exact cause is +unclear. Other programs may be affected too, e.g., `curl` and `Firefox`. + +To get back into a normal state, try: + + # systemctl restart systemd-resolved + +**Note:** domains with Onion-Location are likely to be missed if `-n 7442` is +used here in a subsequent run. For example, with `-C 60s` and an average of 100 +domains/s, it would be wise to roll-back _at least_ 6000 lines. + +Get in touch if you know a fix, e.g., based on `ulimit` and `sysctl` tinkering. + +## Contact + + - rasmus (at) rgdd (dot) se |