Steven's Weblog

If you recently bought an AVR Raven, and you were horrified to discover that the AVR Wireless Studio includes USB drivers that only work in Windows XP SP2 and previous, it may help you to know that you can get better drivers installed by installing AVR Studio, after which you can return to AVR Wireless Studio to do something with your expensive little dev kit.

Here’s a nice example of how a product spurred an inquiry by a customer which results in interesting answers that excite the product designers and improve the product and design understanding of the whole community:
iCufflinks improved battery life code hacking….

Not that you were necessarily looking for that. I happened to be thinking about just this sort of thing a minute before I saw this post, so to me, it’s an answer to a question…

For a while I’ve thought that I needed to reduce the amount of work I do in order to be more satisfied. And I’ve got a pretty good setup in that regard, as I’m working well less than full-time and on my own schedule.

I think I need to point out that I don’t spend my time outside work just soaking in radiation on the couch. I wouldn’t have thought to mention it, except I heard that someone asked that when they heard about my workstyle. I guess I just assume that anyone who has found a way to free up their time would spend a lot of it reading, designing, walking, philosophizing …

So I guess my recent discovery that what I want is _more_ work, just less stress, is not a big revelation. It’s probably more a terminological shift than anything. Other ways to state my intention would be that I want less job and more work; stress without the distress; to get paid a little for the playing around that I do that some people would call work; to find ways to apply my lust for learning to applications that others can benefit from… something like that.

Which is why I have to start a company. Not yet gonna give up the current consulting gigs that give me a semi-steady income, because the last thing I want is to design a system for pushing my stress levels into the stratosphere, all the while knowing that I did it to myself, just because I have to generate significant income quickly. I’m going to develop my company in parallel with my consulting. I’m sure things will dovetail nicely as I progress.

I’m spending a little time seeking out people who seem to have similar ideas in mind for creating their best work. I’ll try to write up some of the conversations I have; just spoke to the owner of a small book store here in Ames and learned a few things…

This one took a while to figure out.

I bought a USB TV tuner, even though I have a perfectly good PCI one, because I want to replace my big loud server machine with an old Mac Mini refitted with some more storage and a new OS. Oh, I should tell the story of the Mac Mini first.

So, as mentioned above, I have a nice working server machine with (analog NTSC cable) DVR capability thanks to my WinTV HVR-1600 PCI card and the excellent work of the LinuxTV, MythTV, and IVTV people (and, surely, the credits could roll on forever, but I’ll be brief).

But it’s too noisy. I realized that the Mac Mini was quiet and could probably take on the various duties of my server machine, simple as they are, if only I had DVR. USB TV tuners must be as good as PCI ones these days, right? That’s true to a point: the HVR-950Q can record all the same stuff (NTSC, ATSC, ClearQAM, OTA/cable, etc.), but I didn’t realize until I bought it that it doesn’t do MPEG2 encoding on-board like the HVR-1600 does. Ah well, the little processor in the Mini (mid-2007 model with 1.83GHz Core 2 Duo) should be able to handle that.

So I pushed ahead: utilizing guides from around the internet, I popped the Mini open and gave it 4GB of RAM (collected from an old laptop) and a 500GB HD (which I had as a spare for a little RAID box that is currently unused because of _its_ fan noise, which is a story for another day). Along the way I had to replace the stupid connector for the HD temperature sensor that stupid broke because I inadvertently yanked the stupid wire out (thanks for your sacrifice, girlfriend’s old trash laptop!). Then I installed Ubuntu Natty server 64-bit on it, which was a convoluted process. But worthwhile, because now I have a nice, somewhat capable Linux server that’s quite small and quiet. I don’t care that much about being able to run OSX, though I can still do that if necessary by plugging the old drive into the box through USB, and, yeah, I’ll probably end up buying another OSX box some day for some reason, because I just have the sort of life that requires me to run a lot of OSs.

Anyway, now I have everything in line to do the DVR thang with my quiety hardware. A test run recording something with mencoder worked, and only used 40% of one CPU core, so the little processor can handle the video encoding well enough. But MythTV refused to work with the 950Q, and I’m _not_ going to give up all the goodness of MythTV and MythWeb.

Forum/blog posts by users and even a driver writer/all-around 950Q-debugging-all-star declared victory with this hardware; one even specifically mentioned a Mac Mini. The MythTV logs, even at maximum verbosity, failed to uncover the root cause (spoiler alert: the failure point in this case is not logged, even though similar failures around it are). Poring over more posts and more logs, trying out dozens of settings changes, playing with alternate encoders, reading through code, etc. added up to more hours than I really should have spent with this, especially since I was getting nowhere.

Finally I broke down and pulled out gdb to see if I could debug into the heart of this thing. Luckily, the Ubuntu packagers included a package with debug symbols, so I was able to do pretty decent source-level debugging without having to recompile (spoiler: yet). Turns out that somewhere around kernel version 2.6.38, the old V4L1 header was removed (deprecated, no doubt), which caused problems for the Ubuntu people in packaging MythTV. They worked out a patch to separate out the V4L1 functionality from the V4L2 stuff, but they didn’t get it quite right for my case. A little, crucial, function called SetFormatV4L2 was defined as a dummy returning false (failure), but the driver for the 950Q is a V4L2 thing, so the recorder quietly died every time, quietly (remember how I said this particular failure mode is not logged?).

This being the open-source world, I was able to mimic the ifdef patterns that the MythBuntu people had applied elsewhere, so as to get the SetFormatV4L2 and DoV4L2 functions defined properly while still being able to compile without the V4L1 header. And, woot, it all works now. I posted a bug report to the MythBuntu team, so hopefully that’ll get fixed upstream before the next update to libmythtv hits my box.

In any case, thanks once again to open source and the bazillion people who inadvertently cooperated to ensure that I can time-shift Soul Quest Overdrive. Because, in the end, that’s really what it’s all about, isn’t it?

Oh, the cross-cutting concerns. Working on optimizing the amount of downloading clients have to live with when the big ol’ Flex app is updated. Need to compute the build dependencies of the SWFs. But one of the SWFs depends on a set of Python components on the server, the list of which can only be computed by running a complicated algorithm that imports a bunch of modules, performs some introspection on them, compiles lists of… Like I said, it’s complicated.

Do I live with this and split out the algorithm to compute the dependencies for me, or do I try to restructure the generation process so that the dependencies become simple again? Or do I skip the whole dependency-calculation thing and run at it from the other end?

Such are the problems of the Steven.

In case I’m not the only one trying to do this, I thought I’d write this up.

Given: a build that produces many SWFs, a source tree with lots of dependencies, not all of which are obvious, a revision control system from which to get a history of your source. You wish to write a script that correctly predicts which SWFs will change between two revisions of the source. You want to test that script by actually comparing the binaries to see if the predictions were correct.

The first thing you’ll note is that doing a binary compare of the SWFs, or a text compare of the swfdumps, does not answer the question. There’s a timestamp, and if debugging is on, there’s a debugger password. But much worse, internal ID numbers and ordering of classes and chunks of code changes semi-randomly. Do two builds of precisely the same source, and unless you write a lot of fancy code, you can’t tell whether the SWFs are really the same.

So, go grab the Flex SDK source and modify it with the following two bits of sed:
[bash]
sed -i ‘s/bHashSet/LinkedHashSet/g’ `find -name *.java`
sed -i ‘s/bHashMap/LinkedHashMap/g’ `find -name *.java`
[/bash]
In other words, replace the Java collections which have non-deterministic iterators with equivalents that have deterministic iterators.

Your new compiler produces pretty deterministic output. You still have to deal with the timestamp and debug password. One way to do that is to [bash]swfdump -abc[/bash] the binaries and compare the dumps textually, ignoring those few lines that always change. Another approach is to modify SimpleMovie.java to remove the code conditional on configuration.generateDebugTags(), and set the last argument in the ProductInfo constructor (the timestamp) to 0. Then the binaries should be bitwise identical.

I wonder what programming will be like in a couple thousand years, when, presumably, programming languages and the systems built up from them will have attained a richly layered history of idioms, allusions, allegories, metaphors, genres, fads and past fads… I suppose it won’t take thousands of years, necessarily, since programming languages inherit a lot from human languages, and because software is mixed up in everything humans do and absorbs some of those flavors. But I also don’t think there’s really a literature of programs right now; at least, I haven’t been introduced to it yet, and I’d think I would have been. I’d love to have someone prove me wrong, though.

Though the technical aspects are a source of joy for me in software development, the human aspects are no less fascinating. The practice of making software provides some unique insights into human psychology, because software is pretty much made of pure thought, and because software is intertwined in so many human activities. Software developers often get a very detailed look at what people do or where they expect to go in their lives through the ongoing development of a project or their requirements for a new project. That’s pretty cool, when you think about it.

Maybe some day I’ll be able to add to my resume that I have great expertise in understanding and eliciting people’s aspirations.

I’ve worked on more than what I consider my fair share of projects where I had to deal with crazy input. What I mean by crazy input is: it fails to meet the basic criteria of the standard to which it’s supposed to conform, or there is not a standard to which it’s supposed to conform, or standard elements are mashed together in unanticipated ways to try to achieve non-standard effects, or some combination of those.

I suppose that if you dig deep, you’ll find that most of the data in the world is crazy in this sense. So despite my distaste for it, I’ll probably never get away from it. Still, I need to whine about it occasionally. But in addition to whining about it, I’ll present a little practical advice on dealing with it.

The first and most obvious reality in dealing with crazy input is that some heuristic methods will have to be used. For example, in some HTML input, I had to deal with this little number, which, in some documents, occurred between every two paragraphs (well, divs, because these documents don’t use p tags much at all):
[html]
<div style="margin-top: 6pt; font-size: 1pt">&nbsp;</div>
[/html]

These aren’t really paragraphs. Even if you call a div a paragraph, they’re still not paragraphs; they’re just there for spacing. So I have a rule that says something like “If a paragraph is preceded by a paragraph that is essentially empty and has a font size less than 5, remove the empty paragraph and change the spacing on the current one to get the same spacing effect”.

That’s not the best/worst example, but it gets the point across. Heuristics will be necessary. That implies two things: the overall code structure must be adaptable to the addition of heuristics, and I’ll need a bestiary to test the code.

It’s part of the software developer’s mindset to try to neatly partition the entire universe into non-overlapping subsets, then write chunks of code to deal with each partition separately. The introduction of heuristics into such a beautiful scheme will cause some pain. In the beautiful world, I’d have code that says “it’s a paragraph, let’s do the paragraph thing with it”. In a world laden with heuristics, I have code that says “it’s a paragraph, but let’s see if it’s _really_ a paragraph, then we’ll either do the paragraph thing or do some wildly different thing”. I guess it’s less that the code structure has to be adaptable than it’s that my mindset has to be adaptable.

Regardless of the flexibility of my mindset or my code, though, heuristics, by their nature, do not neatly partition the universe. They leave some things out, they overlap, and/or they tangle together in increasingly strange ways. I’ll never remember, when it comes time to add some new code, all the situations that got me to this point or all the ways that things can go wrong.

I’m not yet a full convert to test-driven development, but when dealing with beastly input, I consider a bestiary to be quite necessary. A big set of unit tests, with a perfect specimen of each of the beasts I’ve encountered, each named after the ticket in the ticket-tracking system that brought it to me. I had one project in the past where I should have created a bestiary but didn’t, and that project was one of the worst disasters in my professional life. Another one like that and I would have traded my keyboard in for a shovel and started a new career…

A friend once told me about one of his introductory computer science classes, wherein a fellow student would occasionally stop the professor to ask “Yeah, but, how do we know what computers can do?!?”. The professor didn’t really have a great answer for him, and after a while the student gave up asking and dropped the class.

It’s easy to think that this person just did not have it, whatever it takes to be in computer science. Naturally, he should leave because he had no business in the class in the first place. And in a sense, yeah, that’s precisely right, he didn’t have It, and It is quite necessary. But I wonder whether what It was is something that could be gifted by the professor or another student, or if he could have continued with the lectures and exercises for a while before It began to dawn on him, and things would start to crystallize.

I also wonder whether It was, to quote a phrase that popped up unbidden in my head just a while ago, “design balls”. Because I remember a time when I didn’t have my computer program design balls, when I was basically bluffing it, and now I really have It, and I’m not bluffing. I’ll have to do some serious introspection to determine when the magic transition happened, though I can say that luckily it happened well before I was taking formal courses in computer science, so I didn’t have the great difficulty outlined above.

You can build a textbook definition of the design process by stringing together various phrases:

• define need
• derive system outline
• calculate component parameters
• analyze expected system response
• simulate or prototype system
• verify conformance to the specification
• ship design artifacts to manufacturer for implementation

blah blah blah, you know what I mean. Those phrases even make sense to someone who is already past a certain point in their design education. But there’s an unspoken, ummm, let’s say, glue, that holds the pieces of the process together. I don’t know if it’s unspoken because it’s impossible to speak of, as my lame analogy suggests, or whether it’s just that sort of silence that develops around certain conceptual vortexes in a field, for whatever reasons.

And for now, I’m going to give up on speaking about this. Not forever, because now that I’m pondering it, I feel like it’s a really important thing, and perhaps there is hope and help for those who want to have It but don’t yet have It, and maybe It is teachable. Hmmmm.