I wrote the url_scrape.py command-line filter to scrape HTML, XML, and plaintext documents for the URLs that they contain. It reads from standard input and gives the results on standard output, one URL per line.

The heart of the script is the following regular expression, in Python:

url_pattern = re.compile('''["']http://[^+]*?['"]''')

which basically looks for URLs between quotes ("" or '') that start with http://.

The script is most useful if you mix and match it with sort, uniq, and grep. Unix geeks know the drill. For example:

[beaker] ~> wget -O - http://madphilosopher.ca/ | ./url_scrape.py | sort | uniq
http://coralcdn.com/
http://del.icio.us/madphilosopher
http://del.icio.us/madphilosopher/comments
http://denyhosts.sourceforge.net/
http://en.wikipedia.org/wiki/Slashdot_effect
http://geourl.org/near/?p=http://madphilosopher.ca/
http://gmpg.org/xfn/
http://gmpg.org/xfn/11
http://home.online.no/~kafox/blogfiles/
http://iloveradio.org
http://madphilosopher.ca
http://madphilosopher.ca/2003/01/
http://madphilosopher.ca/2003/02/
http://madphilosopher.ca/2003/03/
http://madphilosopher.ca/2003/04/
...
http://madphilosopher.ca/images/delicious_links.gif
http://madphilosopher.ca/images/geourl.gif
http://madphilosopher.ca/images/get_firefox.gif
http://madphilosopher.ca/images/rss2.gif
...

The script

Download the url_scrape.py script.

Since the server in question is running FreeBSD, which uses a combined allow/deny syntax in hosts.allow and doesn’t use hosts.deny, I had to modify the DenyHosts script script slightly to get it to work in the FreeBSD context. Basically, I configured DenyHosts to write to a dummy hosts.deny file and then wrapped it in a cron(8) script to concatenate this dummy file with a hosts.allow.template file. Thus hosts.allow is dynamically generated with the dynamic deny rules first and the static allow rules last.

It seems to be working so far.

Update from the comments: FreeBSD is now supported in the latest version of DenyHosts.