Steven's Weblog

Y’ever get worked up about how much you ought to get worked up about something? And then at some point you have to evaluate the inequality about whether the effort of evaluating how much to get worked up, plus the effort of getting worked up itself, is greater or less than the effort of just getting maximally worked up about the original thing from the start. If you know what I mean.

Huh, I thought shopping for computer-telephony interfaces or scalable web-hosting infrastructure was hard. A little research lately has convinced me that shopping for any sort of insurance must be at least as difficult and probably more.

There is one inherent fact about insurance that seeds the complexity: you’re trying to predict complicated and rare events. The more predictable an event is, actually, the less likely it’s going to be insurable, or insurable at a rate you’re willing to pay. I mean, if you can predict an event with some certainty, you should be using a bank account instead of an insurance policy.

There’s also the psychological twist that these are events that you just plain don’t want to happen. That probably tends to make you undervalue the little information you do have that helps you make your predictions, and causes you to want to just hurry up and get done making the decisions.

On top of all that is, I’m pretty sure, a general tendency for insurance companies to purposely make it harder to buy their products. A great tension exists between the generally communal idea of insurance (we all pool our money together to help whoever needs it) and the capitalist imperative to profit. Basically, insurance companies can only profit to the degree that they hide their actuarial knowledge from their customers.

I was looking at health insurance tonight and was presented with a painful interface to choose a policy. I specified a few parameters in very general terms, and they presented me with dozens of possible policies to choose from, with a ‘comparison’ feature that did little to enlighten me about the differences. It kind of reminds me of that game Black Box where you’re trying to discern what’s in the box by shooting a tiny number of rays into the darkness, though this particular game of black box is probably 80-dimensional instead of 2D. My choices would still not be easy even if I had a full view of the model(s) they use to design these policies, but…

Ya know, for all the conceptual elegance of the idea of branching/merging in software revision control, it sure can be a pain in practice. Just in the last while, I’ve run across the following situations:

• someone mismerged my code because they merged based on files rather than revisions
• had to decide whether to merge someone else’s code because mine was dependent on it, but they hadn’t merged theirs
• had to decide when to merge my code, which was part of a larger feature and was not a complete feature in itself: do it now and possibly introduce instability, do it later and possibly have it get delayed in the shuffle
• stuff I’d merged disappeared because the branch was remade after I merged
• conflict while merging my code because the branch was based on a way older revision than I was informed (so other changes I’d made previously were not there to merge against)
• not knowing whether my changes were to be merged because it wasn’t decided when the feature was to be released
• a special case of the previous situation, when it was first decided the feature was for later, then it was moved earlier
• work for a future feature ended up causing a conflict when trying to merge a current feature because the two changes were made in the other order on the same code

I wonder what the current status of Java < -> .Net Web Services interop is. A year-n-somethin’ ago, it was a bit of a mess, and for the project I was working on we had to create a proxy running on a Windows box to make the calls. Ugh.

Actually, I don’t really wonder. I don’t care, because I’m not having to deal with such right now. Java/Python/Flex interop, yeah. Passin’ them AMF packets around like a wild man…

Here’s a simple thing I did today that will save me some annoyance when dealing with new mail.

I manage my mail in Thunderbird by the simple rule of: if it’s something I need to respond to or reread soon, it stays in the inbox. If it’s junk, junk it, if it’s totally ephemeral, trash it, otherwise it goes to the ‘old’ folder. There are already keyboard shortcuts for junk and trash, but I was still dragging and dropping to put stuff in the old folder. Until I installed keyconfig (and yeah, that link is as close as there is to a project page for the thing, unless you count the author’s homepage). I then installed a script on O that says
MsgMoveMessage(“mailbox://nobody@Local%20Folders/old”)

No more drag-n-drop to manage my email. Nice.

(Update: now Thunderbird has that nice ‘A’-for-Archive button, so I use that instead. Had to archive everything from my ‘old’ folder to make that happen, but that wasn’t too bad. I did have one problem, though: I had stuff in my ‘old’ folder addressed to, for example, acct at ourada.org, which was an alias I used to use. Thunderbird got confused and tried to archive that to my gmail account. I just set the archive folder for all my accounts to a Local Folders one, and that problem went away.)

It’s fascinating to me sometimes to describe a problem that comes up in software development, outside its full context, just to remind myself of how weird and deep it sounds in isolation. Makes me feel a little better about how long it takes me to solve said problem, maybe.

In this case, in the full context we just call it the ‘slicer’, but that one word hides a lot of complexity. What the slicer does is: given a document in XML format A and a set of start points and offsets of selected regions of formatted text (both specified by character counts as the user sees the document), extract the given regions from the document in XML format C, rebuilding any necessary start/end tags and preserving internal formatting. Another process does the translation from format A to B to C. (These translation layers can add extra characters that are not part of the document as the user sees it.)

One thing I’ve been learning over the last few months is that if your app is pretty beefy, you might well need more than one cloud provider to deploy it. With the major differences in approach between Amazon EC2 and Google AppEngine, they each have a variety of strengths and weaknesses with respect to a given application, and the weaknesses are such that you can’t really just live with them. So you partition the app into pieces and spread ’em around…

I’d like to be able to claim that that gives you a more robust solution, but of course, if you require two clouds to be operational in order for your app to work, you’re less robust. Both EC2 and GAE have been OK, but not great, in terms of uptime. I expect both to get better over time, though.

If you work with Google AppEngine and use task queues much, you know it can be annoying to have to press the ‘Run’ button on each task to make it actually run. Here’s a Greasemonkey script to push the button for you. It mindlessly pushes the first Run button it finds immediately on page load. Since the page reloads, that should eventually press them all away.

Have fun, but there are no guarantees that this won’t run tasks you didn’t really want to run or somesuch.

Dumb script

Hmmm, pi is a little bumpier than I thought. (It could just be that my statistical intuition is off, though.)

Each bar plot below represents the number of occurrences of a digit in the decimal expansion of pi. The y-axis is an index, and the frequency x is counted over the range of digits y*20 to y*20+400. I thought 400 would be a long enough length to make these graphs pretty flat. Higher lengths make it flatter, of course, but still not to the degree that seems ‘right’ to me.

I guess I could calibrate my perception by using a uniformly distributed sequence of digits…

Pi digit frequencies

Update: huh. I guess it is just me. Here’s the same sort of graph but with uniformly random digits (at least, assuming that RAND’s book, which I lazily selected as my source, is indeed uniform). Looks equally bumpy to me. Ah well, I’ll leave the post up as a reminder of my folly…

Random digit frequencies

Scott Stafford was nice enough to post code for asc-gzip/.xfd decompression to go with my asc-gzip/.xfd compression code. See this comment. I’m also reposting it here because the comment formatting is a little more bad than the main-post formatting.

Thanks for your post. Of course, I needed the opposite, I had one I needed to decompress. So I backwarded your algorithm and here is the result:

def decompress(fc):
    fc2 = fc.splitlines(True)
    fc3 = "".join(fc2[1:]) # could verify that it's asc-gzip here if we wanted to...
    unb64 = base64.standard_b64decode(fc3)
    
    ctr = 0
    ret = []
    while 1:
        if ctr == len(unb64): break
        
        ccltop = ord(unb64[ctr])
        ctr += 1
        cclbottom = ord(unb64[ctr])
        ctr += 1
        compressedchunklen = ccltop * 256 + cclbottom
        
        cltop = ord(unb64[ctr])
        ctr += 1
        clbottom = ord(unb64[ctr])
        ctr += 1
        chunklen = cltop * 256 + clbottom
        #~ print compressedchunklen, chunklen
        
        compressedchunk = unb64[ctr:ctr+compressedchunklen]
        ctr += compressedchunklen
        
        chunk = zlib.decompress(compressedchunk)
        assert(len(chunk) ==  chunklen)
        ret.append(chunk)
    
    return "".join(ret)