I've updated the rate limiting downloader (that uses token bucket algorithm) to be more user friendly out of the box. It attempts to use urllib2 by default so it can rate limit pretty much any url.
Previously the script required an http proxy and the only means of adjusting operating parameters was global variables in the script. A proxy server is no longer required and all operating parameters can be adjusted via script options.
If an http proxy is selected the python http library (httplib) is used, this is a more rudimentary library so not as many situations are handled. It's possible to install a proxy handler in urllib2 but I didn't do this.
This is more-or-less a "complete" rate limiting download manager. Option output:
Usage: rlfetch.py [options] url
Options:
-h, --help show this help message and exit
-f FILE, --file=FILE output filename
-d DIR, --dest=DIR destination directory
-p SERVER:PORT, --proxy=SERVER:PORT
http proxy server
-z BYTES, --buffer=BYTES
buffer size
-l KBS, --limit=KBS kbs limit
-b KBS, --burst=KBS burst limit
-t SECONDS, --tick=SECONDS
tick interval
Current state of the code.
No comments:
Post a Comment