I think I have a fix (which I developed in the process of trying to explain the code), but just handing it over wouldn’t achieve the goal of giving him the deeper understanding of the code that he needs going forward (I still offered him the option; we’ll see what he thinks). The fix only affects about three lines of code, and it probably would have taken me about 45 minutes to find and fix myself. But explaining the whole context of the code, and precisely how it was that I screwed up in the first place, that’s a lot more work. I spent 3 hours trying to document the meaning of one relevant variable, and I still feel like I didn’t do a great job of it. It will be at least another couple hours of work to describe exactly how I fixed the bug, if he wants to hear it.

This might be one of those cases where he’d be better off refactoring the code. Sometimes people use refactoring as a way to gain deeper understanding of the code, as well as making it easier to understand in the future. In this case, for example, I wrote a function called adjust-rowspan-vec, which probably should have been named update-rowspan-vec-for-next-row. Though the name rowspan-vec in itself is not all that clear, but ugh, it’s hard to find a good name for that one; maybe data-for-cells-which-span-rows-and-are-currently-in-flight-so-that-we-can-track-where-to-insert-dummy-cells-as-we-scan-down-the-input-rows, or something. So, yeah, probably needs a refactoring, just so that we can better wrap our heads around whatever concept I was trying to express with rowspan-vec.

It’s hard to describe the concepts in use in the code even if it was working properly, but to describe my partially-broken mental model when I wrote the incorrect code would add another level of complexity. I think I could do it, though, and it’s awesome that human languages have the ability to express such deep weirdness.

[And just to vent a little more, none of this is made any easier by the fact that the code in question is in a functional language, which means we're using immutable variables and two levels of recursion rather than mutable variables and loops. Nor is it made easier by my lack of confidence in communicating in English with someone who's first language is not English...]

Sparkfun did have a couple interesting forum threads about these, here and here, and at Mill-Max, I found something close (though that doesn’t lead to any small-quantity distributors).

Anyway, when I was looking at the Basic Stamp, it popped into my mind that maybe one could save some hassle in doing home prototyping of 2-layer PCBs if one could route vias out to the edge of the board and use clips at the edges, rather than drilling and using rivets, wires soldered on both sides, etc. But then I realized that such a scheme is like trying to lay out your graph on a sphere rather than a plane, which leads to no advantage, as is well-known in topology. Topology is good for something, eh?

Thinking along those lines, though, makes me wonder about other methods that might yield some advantage. For example, route out a hole in the middle of the board and clip traces on those edges and you’ve got yourself a torus, which does (I think) get you more flexibility. It might be fun to do an investigation of graph embeddings in a variety of physically-realizable connection schemes, with an eye toward finding some method that is easy for a home prototypist given some small catalog of techniques and pre-manufactured connectors. There’s probably some paper written by Euler when he was 7 that lays this all out, so I suppose I should hit the library and brush up on my topology…

it’s one thing to say “oh, that soup i just made from some random ingredients turned out really well.”. it’s another thing to throw up the recipe on your blog and let people put it together themselves. it’s yet another thing, though, for the entrepreneur to turn that into a can of soup that millions of people can buy for $0.99 as just another option in the soup aisle.

After I wrote that, I had a feeling that there was something odd about my use of the word ‘entrepreneur’ there. I think it’s that the average entrepreneur wants to think that their product is so radical that it wouldn’t fit in the soup aisle. It has to be in an end-cap display with a space cleared out around it so that people fainting from joy won’t hit their heads on shelves.

Maybe your thing is so radical it can’t rightly be called soup any more. But there’s quite a bit of marketing power in being on the shelf next to the soup, such that a customer in the soup aisle can use existing decision processes to decide on your product. If it really is a significant departure from soup, customers will tell you (and others) that, and they’ll work with you to define a new category. That’s probably a lot easier than trying to define that category ahead of time and then drag people there with you.

For any given product, we can find a number of ways to manufacture it. (A rubber ball can be fashioned by hand-kneading and rolling a bit of rubber, or by injection molding.) For each method of manufacture, we can separate costs into two buckets: setup and per-unit costs. (Hand-rolling balls has basically zero setup costs but pretty high unit costs, injection molding has high setup costs but quite a bit lower unit costs.) These costs mean that for any given product, there are a number of regimes, under each of which a particular manufacturing method is most economical.

So, at its simplest, the concept of rescaling the economies of scale means using the same ingenuity we use in devising those different methods of manufacture to improve upon the tools which are part of those methods, as well. We figured out how to make rubber balls cheaper by using injection molding, now let’s figure out how to make setting up molds for new products cheaper, too.

And of course, we do that. At least, whenever a production process is not completely vertically integrated, and the product is under competitive stress, then there are separate firms making tools for the process, and they are subject to competitive pressure to reduce their prices. The successful ball maker will seek and find cheaper alternative vendors for the mold making process.

So, not necessarily a radical idea, looked at that way. It might be a little more radical to say that my reason for wanting to rescale things is so that smaller producers are empowered to use more sophisticated production methods for smaller batches. To shrink the gap between the concepts of ‘prototype’ and ‘small production run’. To help bring the promise of mass customization to fruition. To allow more people to scratch their own itches. To give more little ideas a chance to come to life without having to appease the mighty powers of capital. OK, that last one is probably skirting radical, so I’ll stop there for now.

But I think I know why I did. Paper magazines, after I’m ‘done reading’ them, become totems. Of what I was, or want to be, or want to do. But like any such totems, they can sorta disappear into the background of my perception after they’ve laid around the floor for a few days. Magazines, though, come again fresh every month, on someone else’s schedule. So I think I got the subscription mainly to help establish habits that I want to establish.

Or, whatever. I probably don’t actually know why I do things.

(* The fact that I have a couple thousand pounds of books does not contradict the assertion that I don’t much like paper. Deal with that as you must.)

Not that you were necessarily looking for that. I happened to be thinking about just this sort of thing a minute before I saw this post, so to me, it’s an answer to a question…

I think I need to point out that I don’t spend my time outside work just soaking in radiation on the couch. I wouldn’t have thought to mention it, except I heard that someone asked that when they heard about my workstyle. I guess I just assume that anyone who has found a way to free up their time would spend a lot of it reading, designing, walking, philosophizing …

So I guess my recent discovery that what I want is _more_ work, just less stress, is not a big revelation. It’s probably more a terminological shift than anything. Other ways to state my intention would be that I want less job and more work; stress without the distress; to get paid a little for the playing around that I do that some people would call work; to find ways to apply my lust for learning to applications that others can benefit from… something like that.

Which is why I have to start a company. Not yet gonna give up the current consulting gigs that give me a semi-steady income, because the last thing I want is to design a system for pushing my stress levels into the stratosphere, all the while knowing that I did it to myself, just because I have to generate significant income quickly. I’m going to develop my company in parallel with my consulting. I’m sure things will dovetail nicely as I progress.

I’m spending a little time seeking out people who seem to have similar ideas in mind for creating their best work. I’ll try to write up some of the conversations I have; just spoke to the owner of a small book store here in Ames and learned a few things…

I bought a USB TV tuner, even though I have a perfectly good PCI one, because I want to replace my big loud server machine with an old Mac Mini refitted with some more storage and a new OS. Oh, I should tell the story of the Mac Mini first.

So, as mentioned above, I have a nice working server machine with (analog NTSC cable) DVR capability thanks to my WinTV HVR-1600 PCI card and the excellent work of the LinuxTV, MythTV, and IVTV people (and, surely, the credits could roll on forever, but I’ll be brief).

But it’s too noisy. I realized that the Mac Mini was quiet and could probably take on the various duties of my server machine, simple as they are, if only I had DVR. USB TV tuners must be as good as PCI ones these days, right? That’s true to a point: the HVR-950Q can record all the same stuff (NTSC, ATSC, ClearQAM, OTA/cable, etc.), but I didn’t realize until I bought it that it doesn’t do MPEG2 encoding on-board like the HVR-1600 does. Ah well, the little processor in the Mini (mid-2007 model with 1.83GHz Core 2 Duo) should be able to handle that.

So I pushed ahead: utilizing guides from around the internet, I popped the Mini open and gave it 4GB of RAM (collected from an old laptop) and a 500GB HD (which I had as a spare for a little RAID box that is currently unused because of _its_ fan noise, which is a story for another day). Along the way I had to replace the stupid connector for the HD temperature sensor that stupid broke because I inadvertently yanked the stupid wire out (thanks for your sacrifice, girlfriend’s old trash laptop!). Then I installed Ubuntu Natty server 64-bit on it, which was a convoluted process. But worthwhile, because now I have a nice, somewhat capable Linux server that’s quite small and quiet. I don’t care that much about being able to run OSX, though I can still do that if necessary by plugging the old drive into the box through USB, and, yeah, I’ll probably end up buying another OSX box some day for some reason, because I just have the sort of life that requires me to run a lot of OSs.

Anyway, now I have everything in line to do the DVR thang with my quiety hardware. A test run recording something with mencoder worked, and only used 40% of one CPU core, so the little processor can handle the video encoding well enough. But MythTV refused to work with the 950Q, and I’m _not_ going to give up all the goodness of MythTV and MythWeb.

Forum/blog posts by users and even a driver writer/all-around 950Q-debugging-all-star declared victory with this hardware; one even specifically mentioned a Mac Mini. The MythTV logs, even at maximum verbosity, failed to uncover the root cause (spoiler alert: the failure point in this case is not logged, even though similar failures around it are). Poring over more posts and more logs, trying out dozens of settings changes, playing with alternate encoders, reading through code, etc. added up to more hours than I really should have spent with this, especially since I was getting nowhere.

Finally I broke down and pulled out gdb to see if I could debug into the heart of this thing. Luckily, the Ubuntu packagers included a package with debug symbols, so I was able to do pretty decent source-level debugging without having to recompile (spoiler: yet). Turns out that somewhere around kernel version 2.6.38, the old V4L1 header was removed (deprecated, no doubt), which caused problems for the Ubuntu people in packaging MythTV. They worked out a patch to separate out the V4L1 functionality from the V4L2 stuff, but they didn’t get it quite right for my case. A little, crucial, function called SetFormatV4L2 was defined as a dummy returning false (failure), but the driver for the 950Q is a V4L2 thing, so the recorder quietly died every time, quietly (remember how I said this particular failure mode is not logged?).

This being the open-source world, I was able to mimic the ifdef patterns that the MythBuntu people had applied elsewhere, so as to get the SetFormatV4L2 and DoV4L2 functions defined properly while still being able to compile without the V4L1 header. And, woot, it all works now. I posted a bug report to the MythBuntu team, so hopefully that’ll get fixed upstream before the next update to libmythtv hits my box.

In any case, thanks once again to open source and the bazillion people who inadvertently cooperated to ensure that I can time-shift Soul Quest Overdrive. Because, in the end, that’s really what it’s all about, isn’t it?

Do I live with this and split out the algorithm to compute the dependencies for me, or do I try to restructure the generation process so that the dependencies become simple again? Or do I skip the whole dependency-calculation thing and run at it from the other end?

Such are the problems of the Steven.

Given: a build that produces many SWFs, a source tree with lots of dependencies, not all of which are obvious, a revision control system from which to get a history of your source. You wish to write a script that correctly predicts which SWFs will change between two revisions of the source. You want to test that script by actually comparing the binaries to see if the predictions were correct.

The first thing you’ll note is that doing a binary compare of the SWFs, or a text compare of the swfdumps, does not answer the question. There’s a timestamp, and if debugging is on, there’s a debugger password. But much worse, internal ID numbers and ordering of classes and chunks of code changes semi-randomly. Do two builds of precisely the same source, and unless you write a lot of fancy code, you can’t tell whether the SWFs are really the same.

So, go grab the Flex SDK source and modify it with the following two bits of sed:

sed -i 's/\bHashSet/LinkedHashSet/g' `find -name \*.java`
sed -i 's/\bHashMap/LinkedHashMap/g' `find -name \*.java`

In other words, replace the Java collections which have non-deterministic iterators with equivalents that have deterministic iterators.

Your new compiler produces pretty deterministic output. You still have to deal with the timestamp and debug password. One way to do that is to

swfdump -abc

the binaries and compare the dumps textually, ignoring those few lines that always change. Another approach is to modify SimpleMovie.java to remove the code conditional on configuration.generateDebugTags(), and set the last argument in the ProductInfo constructor (the timestamp) to 0. Then the binaries should be bitwise identical.

Maybe some day I’ll be able to add to my resume that I have great expertise in understanding and eliciting people’s aspirations.

I suppose that if you dig deep, you’ll find that most of the data in the world is crazy in this sense. So despite my distaste for it, I’ll probably never get away from it. Still, I need to whine about it occasionally. But in addition to whining about it, I’ll present a little practical advice on dealing with it.

The first and most obvious reality in dealing with crazy input is that some heuristic methods will have to be used. For example, in some HTML input, I had to deal with this little number, which, in some documents, occurred between every two paragraphs (well, divs, because these documents don’t use p tags much at all):

<div style="margin-top: 6pt; font-size: 1pt">&nbsp;</div>

These aren’t really paragraphs. Even if you call a div a paragraph, they’re still not paragraphs; they’re just there for spacing. So I have a rule that says something like “If a paragraph is preceded by a paragraph that is essentially empty and has a font size less than 5, remove the empty paragraph and change the spacing on the current one to get the same spacing effect”.

That’s not the best/worst example, but it gets the point across. Heuristics will be necessary. That implies two things: the overall code structure must be adaptable to the addition of heuristics, and I’ll need a bestiary to test the code.

It’s part of the software developer’s mindset to try to neatly partition the entire universe into non-overlapping subsets, then write chunks of code to deal with each partition separately. The introduction of heuristics into such a beautiful scheme will cause some pain. In the beautiful world, I’d have code that says “it’s a paragraph, let’s do the paragraph thing with it”. In a world laden with heuristics, I have code that says “it’s a paragraph, but let’s see if it’s _really_ a paragraph, then we’ll either do the paragraph thing or do some wildly different thing”. I guess it’s less that the code structure has to be adaptable than it’s that my mindset has to be adaptable.

Regardless of the flexibility of my mindset or my code, though, heuristics, by their nature, do not neatly partition the universe. They leave some things out, they overlap, and/or they tangle together in increasingly strange ways. I’ll never remember, when it comes time to add some new code, all the situations that got me to this point or all the ways that things can go wrong.

I’m not yet a full convert to test-driven development, but when dealing with beastly input, I consider a bestiary to be quite necessary. A big set of unit tests, with a perfect specimen of each of the beasts I’ve encountered, each named after the ticket in the ticket-tracking system that brought it to me. I had one project in the past where I should have created a bestiary but didn’t, and that project was one of the worst disasters in my professional life. Another one like that and I would have traded my keyboard in for a shovel and started a new career…

It’s easy to think that this person just did not have it, whatever it takes to be in computer science. Naturally, he should leave because he had no business in the class in the first place. And in a sense, yeah, that’s precisely right, he didn’t have It, and It is quite necessary. But I wonder whether what It was is something that could be gifted by the professor or another student, or if he could have continued with the lectures and exercises for a while before It began to dawn on him, and things would start to crystallize.

I also wonder whether It was, to quote a phrase that popped up unbidden in my head just a while ago, “design balls”. Because I remember a time when I didn’t have my computer program design balls, when I was basically bluffing it, and now I really have It, and I’m not bluffing. I’ll have to do some serious introspection to determine when the magic transition happened, though I can say that luckily it happened well before I was taking formal courses in computer science, so I didn’t have the great difficulty outlined above.

You can build a textbook definition of the design process by stringing together various phrases:

• define need
• derive system outline
• calculate component parameters
• analyze expected system response
• simulate or prototype system
• verify conformance to the specification
• ship design artifacts to manufacturer for implementation

blah blah blah, you know what I mean. Those phrases even make sense to someone who is already past a certain point in their design education. But there’s an unspoken, ummm, let’s say, glue, that holds the pieces of the process together. I don’t know if it’s unspoken because it’s impossible to speak of, as my lame analogy suggests, or whether it’s just that sort of silence that develops around certain conceptual vortexes in a field, for whatever reasons.

And for now, I’m going to give up on speaking about this. Not forever, because now that I’m pondering it, I feel like it’s a really important thing, and perhaps there is hope and help for those who want to have It but don’t yet have It, and maybe It is teachable. Hmmmm.

Update (which was actually posted simultaneously with the original): yup. Yay, open source is just so sweet to me.

In software development, there are two things we routinely do that occasion such feelings. (Ooops, I’ve been told I’ve been using the word ‘feel’ too much.) We use ticket-tracking systems, and we write TODO comments.

Ticket-tracking systems are a way to keep track of work to be done and the evolution of the product. A ticket is a short, detailed description of an action to be taken to improve the product. They come in different types: bug, improvement, feature, task, etc. The description is action-oriented and specific, and assumes the current product as the context (at least the best tickets are like that; maybe I’ll talk a little more later about less-than-best tickets). Here’s a decent-looking example of a ticket. I could go on for quite a while about the good and the bad of ticket-tracking systems… For now, the salient points are:

• a ticket represents some work to be done
• tickets are assigned to someone, to do the work
• as a developer, you spend time daily looking at the list of tickets assigned to you

TODO comments are a simpler manifestation of a similar idea. When writing code, you might see room for improvement in some particular technical aspect, but you don’t have the time to tackle it right now. So you put a little TODO comment in there explaining what you think might be improved. Here’s a little contrived example:

if n == 0:
  print "none"
elif n > 0:
  print "some"
# TODO: what if n is negative?

In this case, I can see that the set of all numbers is not covered by the two conditions, so I’m led to wonder whether I should be handling the case where n is negative. Now, if I really thought that case was going to arise naturally and soon, I’d figure out how to handle it and write the code. But in this example, I’m thinking: it could happen, but I can’t see any reason it should. Let’s say n is the count of apples in the box from the apple-counting machine. No reason for that to be negative. But, ya know, it could be, somehow. So I note that I feel uneasy about leaving that case uncovered. Later, when browsing through the code or debugging it, I might see that comment, and due to an increased understanding of the world or an abundance of spare time or something, I’ll decide to cover the case. Maybe I learned that the apple-counter counts a zucchini as -1 apples, and somehow a zucchini gets in the box occasionally.

Where tickets and TODOs can get depressing is when you look at your list and realize that there are 30 tickets on it, 10 of which are more than a year old, and 23 of which you’ve been systematically ignoring for a long time. Or you run across a TODO and a scenario flashes through your mind, where the improbable thing happens, and the resulting chain of causality ends with a lawsuit in Taiwan.

It might be easy to say “Well, this simply should never happen! These tasks should be dispatched with great haste and with all available resources!”. Certainly one can imagine a process or organization where that is the rule, and buildups of old cruft never happen. But follow me for now into my world: in the places where I’ve worked, there is no such rule, and the cruft does build up. I’ve left jobs with dozens of tickets and TODOs left in my wake, where they either evaporated when I left or are still lurking around somewhere today.

The rule in these places is more like “Good ideas should be captured immediately and dealt with when there’s time”. That’s a perfectly valid rule to use, but it does tend to lead to a situation where there are lists of lots of good ideas left unattended. If you believe in good ideas and you want to perfect your work, then these lists sometimes look pretty grim, numerically speaking. There are always more good ideas than time to implement them, by a laaarge factor.

So, you learn to live with the backlog.

Lately I’ve been working on adapting this big open source server project for use for a client (sorry about the vagueness, but the stuff I’m working on is proprietary). In the best of all worlds, I’d deploy the client’s app on the server and it would just work. And it won’t surprise you to learn that we don’t live in that world. The world we live in has features like:

• the server has an old version of the main framework that the client’s app uses, meaning I have to find the uses of the new features and back them up to older code
• the server project has issued very little documentation. That might make me look for an alternative server, but it happens that there are only two alternatives and this one seems, from various viewpoints, to be the best by far
• there is some problem

That last point might seem even more vague than the others, and believe me, I wish I could be more specific. But ya see, that’s approximately all the information I can get the server to give me about this particular problem. I try to run a tiny piece of code, and it tells me “Error: 500″. I’ve spent hours digging through the pieces of the server; it has load-balancers and caches and proxies and RPC servers talking to database adapters talking to databases, talking in HTTP and a few other protocols; it has components written in Python and Ruby, Java, and C; it has log files in many different places and with different ways to enable logging in each component; etc.

So far, all this digging has led me to the conclusion that yes, there still is a problem. That’s frustrating. But the process leads me to probe into the server components with various techniques, so I’m learning. Learned about ngrep, tonight, for example. Learned how to use the shells provided with the various databases to try to see what’s going on in them. Learned about the wonders of libvirt. Learned about some new network protocols.

At a higher level, I think these experiences teach me about patience, persistence and investigative techniques. Those abilities come in handy, and it seems I still somehow have less than the maximum amount of each of them.

One of my clients reached a big product milestone recently. It’s pretty amazing that I’ve been with this product since its inception until this milestone. I felt the need to reflect on what I’ve worked on in the product in that time.

• code generation for AS3 to Python RPC
• HTML and RTF paste
• spell checking
• highlights and callouts
• XBRL HTML slicing
• equation parsing and evaluation
• slimming Flex module download size
• HTML import
• browser issues with keys and mousewheel
• Google AppEngine/EC2 integration
• parallelization of translation functions
• Undisclosed Big-Deal Project
• PDF export

When I look at each of those bullets, I remember lots of work that I had to do on each, the difficult problems that arose and the solutions to them, the necessity of each function and the contribution toward the overall product. But I also remember the things left undone that could make each function more perfect, the work that others did that I can’t take credit for, that if I pointed out these functions in the app, someone unfamiliar with the job wouldn’t understand the work that it represents, and that in at least one of these projects, all of my code is now dead.

So my sense of accomplishment is a complicated and fragile thing. That’s not a problem; actually, when I say it, I feel like that’s a more mature attitude than one that’s more monolithic. Maybe that’s a partial solution to the question I mentioned above of whether it’s a character flaw to feel a sense of accomplishment: maybe it’s only hubris to feel good about the foreground of one’s accomplishments if one doesn’t also understand the inseparable background against which they are viewed.