Conversation
-p/--proxy was parsed and documented, but it never reached a request. Nothing read args.proxy, and no call passed proxies= to requests. scan_site() even took an args parameter that it never looked at. So a scan with -p went out over the user's own connection while it looked like it was proxied, and nothing said otherwise. I set the proxy once in modules.py and every outbound call reads it from there: scan_site, its control request, check_update, motd and get_header_file. That was simpler than adding a parameter to each of them and to logo(), which is where motd() gets called from. The webscraper's Chrome gets the same proxy as a launch flag. The control request is the second fetch the accuracy check makes per candidate hit. It landed in a separate change, so it needs proxying here too, or --tor would send the scan through Tor while that one request went out over the real connection. -tr/--tor points at socks5h://127.0.0.1:9050. The h matters. It keeps DNS on the Tor side, so the hostnames being checked never reach the local resolver. Chrome gets plain socks5:// instead, since it resolves through the SOCKS proxy on its own and won't accept the socks5h spelling. If nothing is listening on 9050 it now exits once with a message, rather than failing on every site in the list. Tested on one username throughout: 85 accounts with no proxy, 0 through a dead proxy, 81 over SOCKS5 to a remote host via ssh -D, and 77 over Tor. Before this change the dead proxy run still found all 85. That is how you can tell the flag was never read. The SOCKS5 and Tor runs find fewer, because some sites refuse connections from datacenter and Tor exit addresses.
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
-p/--proxyis parsed and documented but it never reaches a request. Nothing in the codebase readsargs.proxy, and no call passesproxies=to requests.scan_site()even takes anargsparameter and never touches it. So a scan run with-pgoes out over your own connection while looking like it's proxied, and nothing tells you. The example in--help(-p http://127.0.0.1:8080) doesn't work either.That also blocks "Tor Searching" on the roadmap, because there's nothing to hang it on.
Rather than add a
proxiesparameter to every function that makes a request, and tologo()because it callsmotd(), I set it once inmodules.pyand each outbound call reads it:scan_site,check_update,motd,get_header_file. The webscraper's Chrome gets the same proxy as a launch flag.This sits on top of the accuracy change that just merged, so it also routes that change's per-hit control request through the proxy. Without that,
--torwould send the scan over Tor while the control request went out on the real connection.-tr/--torpoints atsocks5h://127.0.0.1:9050. Thehis deliberate: DNS resolves on the Tor side, so the hostnames being checked never hit the local resolver. Chrome gets plainsocks5://instead, because it resolves through the SOCKS proxy itself and rejects thesocks5hspelling. If--toris passed and nothing is listening on 9050, it exits once with a message instead of failing on every site in the list.Tested on the same username each time:
127.0.0.1:9999: 0ssh -D: 81--tor: 77Before this change the dead proxy run still found all 85, which is how you can tell the flag was never read. The SOCKS5 and Tor runs find fewer because some sites refuse connections from datacenter and Tor exit addresses.
One thing I left alone:
send_webhook()infiles.pystill posts directly. Sending someone's own webhook through Tor seemed more likely to break it than help, but I'll change it if you'd rather it followed the proxy too.