Using TwitterFeed to send certain del.icio.us bookmarks to Twitter

TwitterFeed is a really cool service. Using Twitter feed you can push any RSS feed to Twitter, which enables a super simple way to push pretty much any kind of syndicated data to Twitter (and now Facebook, Laconica, and HelloTxt).

I also use del.icio.us to save interesting bookmarks in a easily accessable persistent location.

The problem was several fold:

• Twitter is a great way to talk about things (including links)

• Twitter isn't so great at categorizing links and making them easy to find later

• I wanted to save links on del.icio.us but share them on Twitter

• Not everything I shared on del.icio.us was something I wanted to spam my Twitter followers with.

Turns out the solution is pretty simple. On del.icio.us you can create tags. These tags help oranize bookmarks.

For each tag you can get an RSS feed (see where this is going?). RSS feeds for tags look like this:

http://feeds.delicious.com/v2/rss/<username>/<tag-name>

You can also visit the tag's webpage and select the RSS feed button next to the URL bar (then record new link in the URL bar):

I selected the tag name tweet-this for this set-up.

The next step is to visit TwitterFeed and get set-up with an account. Once you've got your account you'll be at a Feed Dashboard, select Create a New Feed and enter the previously recorded URL:

Select test rss feed to make sure everything is working.

The rest is point and shoot with the TwitterFeed set-up process. Now, only bookmarks recorded on del.icio.us with the tweet-this tag will show up on Twitter.

[Source for this post in Markdown mark-up]

Exponential Visitor Graph (Ebay)

I thought this was fairly entertaining...

I wrote a simple page hit counter for a couple Ebay / Craigslist listing we were doing (using Google's AppEngine, and Yahoo's YUI).

Recently I added a feature to allow it to graph visitor counts over time (basically a cheesy analytics engine).

I guess one of the items was particularly desirable (a very complete, good codition SNES system with a lot of games) because it had what I image is probably a pretty typical graph of visits to an Ebay auction:

It's pretty obvious that the spike (a huge spike) in visitors is when the auction was about to close when everyone was manically hitting reload. Just eyeballing the graph, it looks like an exponential increase.

[Source for this post in Markdown mark-up]

Updated: Token Bucket Downloader

I've updated the rate limiting downloader (that uses token bucket algorithm) to be more user friendly out of the box. It attempts to use urllib2 by default so it can rate limit pretty much any url.

Previously the script required an http proxy and the only means of adjusting operating parameters was global variables in the script. A proxy server is no longer required and all operating parameters can be adjusted via script options.

If an http proxy is selected the python http library (httplib) is used, this is a more rudimentary library so not as many situations are handled. It's possible to install a proxy handler in urllib2 but I didn't do this.

This is more-or-less a "complete" rate limiting download manager. Option output:

   Usage: rlfetch.py [options] url

   Options:
     -h, --help            show this help message and exit
     -f FILE, --file=FILE  output filename
     -d DIR, --dest=DIR    destination directory
     -p SERVER:PORT, --proxy=SERVER:PORT
                           http proxy server
     -z BYTES, --buffer=BYTES
                           buffer size
     -l KBS, --limit=KBS   kbs limit
     -b KBS, --burst=KBS   burst limit
     -t SECONDS, --tick=SECONDS
                           tick interval

Current state of the code.

Basic Token Bucket Rate Limiter

In locations were you have limited internet resources it's sometimes necessary to implement rate limiting. I was curious exactly how this worked so I worked out a simple Token Bucket based rate limiting HTTP downloader.

This Python script does a couple things:

• Limits rate of data consumption in kilobytes per second
• Prints out the instantaneous KB/s and the overal/actual KB/s [this is done by monitoring the file size on disk]

Token bucket is a pretty simple algorithm. The basic algorithm is to create an artificial stream of tokens, which are generated as fast you want to allow the real stream to go. If tokens are not removed from the "bucket" then tokens are only generated up to a "burst limit", which is the maximum amount over the average limit that's desirable (this could change to help trend a stream toward the average limit).

In the Python implementation, 3 threads are used. Thread one monitors the rate of download. Thread two consumes tokens and downloads real data from an HTTP source. Thread three generates tokens and places them in a bucket, stopping when the burst limit has been reached.

The code is available below or at codepad.org.

# -*- python -*-

import os
import random
import time
import httplib
import urlparse
import sys

import threading

from pprint import pprint

###########################
# Tuning knobs
BUFSIZE = 8192
BPS_LIMIT = 50 * 1024
BURST_BPS_LIMIT = 70 * 1024
TICK = 0.01
###########################

g_usage = "%s <download directory> <target url>"

g_bucketTokens = 0
g_exit = False

def parseArgv(argv):
    if len(argv) < 3:
        return None
    directory = argv[1]
    if not os.path.isdir(directory):
        return None
    url = argv[2]
    s = urlparse.urlsplit(url)
    _, filename = os.path.split(s.path)
    return url, os.path.join(directory, filename)
     
def takeTokens(tokencount):
    global g_bucketTokens
    if g_bucketTokens >= tokencount:
        g_bucketTokens -= tokencount
        return True
    return False

def printKbs(filename):
    def _():
        start, end = None, None
        tot, num = 0, 0
        while True:
            if g_exit:
                print "KB/s monitor exiting..."
                break
            end = time.time()
            size = g_byteCount
            if start is not None:
                inst = ((size - old) / 1024.0) / (end - start)
                tot += inst
                num += 1
                print "I: %.02f kb/s, A: %.02f kb/s" % (inst, tot / num)
            start = time.time()
            old = size
            for x in xrange(int(2 / TICK)):
                if g_exit: break 
                time.sleep(TICK)
    return _

def feedBucketTokens():
    global g_bucketTokens
    tokens_per = int(BPS_LIMIT * TICK)
    print "Tokens per tick: %d" % (tokens_per,)
    while True:
        if g_exit:
            print "Token feeder exiting..."
            break
        if g_bucketTokens >= BURST_BPS_LIMIT:
            time.sleep(TICK)
            continue
        g_bucketTokens += tokens_per
        time.sleep(TICK)

class BucketReader (object):
    def __init__(self, fp):
        self.fp = fp
    def read(self, bufsize):
        while True:
            if takeTokens(bufsize):
                break
            time.sleep(TICK)
        return self.fp.read(bufsize)
        
def prepareFile(filename):
    fp = open(filename, 'ab+')
    fp.seek(0, 2)
    fsize = fp.tell()
    return fp, fsize

def startHttpReq(url, fsize):
    headers = { "Range" : ("bytes=%d-" % fsize), }
    pprint(headers)
    h = httplib.HTTPConnection("aproxyserver", 8080)
    h.request("GET", url, headers=headers)
    r = h.getresponse()
    pprint(r.getheaders())
    return r

g_byteCount = 0

def readLoop(fpIn, fpOut):
    global g_byteCount
    while True:
        d = fpIn.read(BUFSIZE)
        fpOut.write(d)
        fpOut.flush()
        g_byteCount += len(d)
        if len(d) < BUFSIZE:
            break

def main():
    global g_exit
    try:
        params = parseArgv(sys.argv)
        if params is None:
            print g_usage % (sys.argv[0],)
            raise SystemExit(1)
        url, filename = params
        threading.Thread(target=feedBucketTokens).start()
        threading.Thread(target=printKbs(filename)).start()
        fpOut, fsize = prepareFile(filename) 
        fpInput = startHttpReq(url, fsize)
        readLoop(BucketReader(fpInput), fpOut)
    except KeyboardInterrupt:
        pass
    finally:
        g_exit = True

if __name__ == '__main__':
    main()

# vim: et:sts=4:ts=4:sw=4:

IPython on Wx \ Windows

This is almost strictly for Google to index in hopes that it helps someone else trying to do what I was doing. Also, this was probably in the documentation somewhere, but I hate reading documentation.

IPython is an enhanced interactive shell for Python. Out of the box if you run the WxIPython program you may get the following error when attempting to do tab completion:

'InteractiveShell' object has no attribute 'Completer'

This just means that you didn't install PyReadline. IPython should really be smart enough to figure this out on its own — tab completion is part of the core functionality of IPython. Therefore, they could also bundle PyReadline with the default Windows installation package. Especially since it looks like PyReadline was developed by the same group.

Can you spot the social commentary?

I guess freeze framed like this it's pretty easy to spot. This is from American Dad season 4, episode 9, time index around 6 minutes (as the picture shows) -- captured from Hulu.

Parallax Scrolling in Android

A while ago I heard about something called parallax scrolling [1] being used on Google's Android platform (specifically on the G1). I had just gotten the G1 at that point and hadn't noticed that it was doing something special when scrolling between the three screens in the "home" application.

The idea is pretty simple, you have two surfaces that scroll at two different speeds, one is bigger or father away, depending on how you look at it.

On the G1 it allows the background of the home screen to be smaller than the total "viewable area" that the three home screens cover. This is useful since the size of the home screen areas can change drastically when the phone is flipped open. It also creates the illusion of depth.

Below is my implementation using Proce55ing (click and drag the light purple square).

Source code: parallax (plain text)

1 - Turns out I was probably reading a Qt developer blog. These posts support that conclusion.

Don't Irritate Small Company Hiring Managers

Where I work is a relatively small company. We don't have much the way of an Human Resource (HR) department. We've got one person that's in charge of general personnel related issues but the manager of the development group is the one that initially contacts prospective employees (candidates).

The manager of the development group is really just a glorified developer. Our development group is small enough that there's not enough management tasks to occupy all of his time. His time distribution is about 75% development and 25% management.

This time distribution is important because it means he's got disparate responsibilities, and in regards to contacting employment candidates that's not all he's doing — he's not sitting around all day talking to candidates. So if a candidates isn't courteous in his communication it leaves an impression — because he's dealing with a small number of candidates, and because he actually does other things.

This is why it surprises me when every once in a while I get a colleague that asks for a recommendation to start our (remember small company) hiring process, but they don't make every effort to remain in good standing!

One scenario went like this:

• A past schoolmate contacted me, said he might be unemployed soon
• I got him in contact with our company and recommended him
• He started talking with our development manager
• Promised to send us a resume, but never sent it
• Development manager and I talked about him, and his unresponsiveness a few times
• The development manager contacted him again (it should be the other way around)
• He got back to us and said he'd decided on a another position

The failure was that he didn't tell us that he'd found another position even when he'd previously agreed to send us a resume. This wasted my time and more importantly the development managers time, annoyed both of us, and thusly made us both think he's an idiot.

Our opinion of him was underscored by the fact that he got a mere consulting / contracting position. So it was a stupid decision on his part given the stability of our company and the fact that we're a development “oasis” in Albuquerque were there are few software companies that offer such a low stress environment.

So, I think there are a few guidelines when talking with small companies:

• Keep in mind that whoever you're talking to about getting hired probably has other responsibilities
• Don't assume that they are dealing with so many candidates that they don't care if you don't follow up with them
• Let them know when things change
• Try not to irritate them because they'll remember you

If you follow the above guidelines it's more likely that even if you find another position you'll remain in good standing with the small company. Small companies can be great but they have different dynamics than bigger companies. Throwing away opportunities is bad in general, so it's important to keep the above in mind when dealing with a small company.

The Annoying XM Dj and "That Song You Love"

I have a theory about those annoying XM Djs and XM devices that allow you to record that favorite song you like.
Those really popular, good songs, always seems to have a voice over in front of them, so if you record it you've got the annoying voiceover in the front (or the end) of the song... I think this is done on purpose, but that's a different theory.
The theory is that the voice over distracts you long enough for you to recognize the song and initiate a recording. Otherwise, without the voice over, you just go in to enjoying the song, enjoying the escapism, relishing in the feeling it evokes — totally not thinking about mudane trivialities like pressing the record button on the XM device.
At that point, why do I even care that it's coming from an XM device — I'm just listening to good music.
Consider the following scenario:

Hey, it's that song I really like (singing)
♪ ... Oh if I were a boy ... ♪ (song ends)
♪ ... Womanizer, womanizer ... ♪
Damn you Britney you don't deservce to follow Beyoncé!
Crap! Why didn't I record that!?

Remember, that's just a dramatization, and the scenario is necessarily reality... but it happens all too often.
If you manage to record the song with the Dj in the front you have to suffer through the Dj's blather everytime you want to enjoy the recorded song. The annoyance is particularly bad on songs with a long non-vocal intro. Dj's classically use this as a license to talk over the song.

Pioneer GEX-XMP3 Portable XM Satellite Radio Receiver w/ Home Kit

Speed Bumps / Humps

Interesting argument on speed bumps:

They don't really work and they're annoying, I just speed to the next one and slow down

In this case I was arguing for the addition of more speed bumps. We just bought a house not to long ago and didn't properly evaluate the irritation factor of our proximity to a main entrance of a large housing division.

I figure there's several points related to speeding in our neighborhood:

• The neighborhood was designed before they started designing housing divisions like mazes to add a kind of "natural traffic calming" — without the funky hardware like traffic circles and speed humps.

  From what I've seen these kind of divisions have no houses facing arterial streets (arterial on the scale of a housing development). All houses are facing a "capillary" street, the undesirable locations are the ones whose backyards face the arterial streets or whose backyards face the large city streets that bound the division.

• Because there are relatively few places to get out of the division people are frustrated and impatient when they are on the verge of exiting — thus the speeding to get out

• People are just jerks and don't care that it's a residential street!

There are other factors like the fact that there's a several speed bumps on the road in question, just none between my house (and about 14 other house) and the exit of the division -- which doesn't quite make sense. People seem to see the exit, notice that there's no speed bumps, and haul tail to the exit.

Regarding the above argument... if speed bumps don't work, I guess I don't really care, at least they're being annoyed by the speed bumps while they're speeding between them. This way there's mutual irritation between me the driver.