Terse Systems

Okay, so. I’m going to assume that everyone has now seen Avatar, and hence I’ll assume you’re all up for massive spoilers, etc.

The movie we all just watched was not the movie we thought we watched. And this is not just some George “hahaha, I destroyed the hopes and dreams of a generation” Lucas screwing with the fanbase… this is lurking beneath the surface of Avatar from the beginning. The existence of the Na’vi.

The Na’vi have no common morphology with the rest of the planet. Sig even comments on it in a Youtube article. Why do they exist? How is it that they speak a recognizable language, and have genes close enough to human that it’s possible to MIX IN HUMAN DNA with the Na’vi? How is it that the Na’vi have built in neural interfacing equipment that can instantly domesticate the larger animals and even predators? Wouldn’t evolution make such a thing impossible?

The answer is that the Na’vi aren’t a natural race. Eywa made them. They’re close enough to human that the humans can communicate with them and think they look cute and cuddly (Ewya may have been slightly confused here), and alien enough that they can survive in the local environment. If that wasn’t enough, Eywa provided with some elevated sudo privileges, so they could take advantage of the local fauna without Eywa being directly involved.

The how is easy. Eywa’s a worldmind capable of transferring human neural networks into Na’vi clones on the second try, and it’s entirely feasible to create a race and set it up with false memories. And something that’s smart enough to create room temperature superconductor in bulk (what? You think Eywa runs on plants alone, when there’s massively complex electromagnetic flux happening around the tree and the Na’vi just happening to be sitting on giant deposits of unobtainium?) will have no problem reading our electromagnetic communications. Eywa’s on Alpha Centauri; it’s been listening in since the first radio broadcasts.

The why is a bit harder to explain. Why would a worldmind play dumb?

Well, probably because it has a very good understanding of what happens to things that look like a threat to humanity. If we had any conception of what Eywa is, we’d be terrified, and we’d probably sterilize the entire planet before we even set foot on it. As it is, Eywa looks harmless. It looks beautiful. It’s not exactly friendly, but it’s the kind of environment that keeps humans focused on the trees instead of on the forest. Eywa can afford to watch and wait until it has to act.

It also makes a hell of a way to see human capabilities up close and personal. Eywa is well capable of doing a vacuum cleaner impression on every single EM communication on the planet. And when Eywa saw a human built Na’vi run out the container and disregard orders, it could lay bets that this was someone stupid and romantic enough to provide Eywa with more insight into the military command structure. The cute floaty things aren’t accidental. They are Eywa’s way of saying “hold up, this guy could be useful.”

And Eywa gains massively out of it. Not only does it have a neural imprint of a woman with a massive amount of scientific knowledge, it also got to see a run through of the military. It had to expose itself to a certain extent to get the humans to back off, but you can be certain that the humans will have Eywa bacteria in their digestive tracts when they reach Earth. I wouldn’t be surprised if Eywa had a very specific form of toxoplasmosis all ready to go if needed. (Indeed, one plot treatment specifically mentioned this possibility.) It doesn’t have space travel down, but as long as it stays quiet, it can build up and infiltrate the military before they get to the point they know what they’re dealing with.

And for those romantic souls thinking this was about Jake Sully… dude. THEY LOST. Jake and the Na’vi didn’t stand a chance against the military, they used up most of what they had, and THEN, as soon as it looked like Eywa might actually lose a valuable resource, it acted. It was perfectly happy to use the Na’vi as cannon fodder if it meant it didn’t have to show a card.

And I, for one, welcome our worldmind overlord. Life under Eywa would not be a bad thing for the vast majority of people, and Eywa would make better use of our technology and infrastructure than we could. Eywa keeps the reins loose for the most part, and doesn’t enforce behaviour as much as “convincingly persuade.” That may be the Na’vi, the charming public face presented by the unknowable, near-omniscient power that is Eywa, but even if the Na’vi are rolled up and tossed out like last week’s pizza once we are no longer a threat (which is unlikely – the Na’vi are too damn useful for offworld activity), then humanity, and Earth as a whole, can rest in peace knowing that something much smarter, tougher and biologically engineered to perfection took us out.

If you want to know what makes you happy, you have to be willing to think hard about what happiness is, and pay attention to what makes you happy.

“Happiness comes in small doses, folks. It’s a cigarette butt, or a chocolate chip cookie, or a five second orgasm. You come, you smoke the butt, you eat the cookie, you go to sleep, wake up and go back to fucking work the next morning, THAT’S IT! End of fucking list!” - Denis Leary

So after much time and experience, here’s the list of things that make me happy.

1) Direct sunlight.
2) 8 hours of sleep.
3) Movement outside.
4) Social interaction.
5) Regular meals.
6) Satisfying work.

The upshot from this list is that I’m not all that complicated. Also, what actually makes me happy may not be what I spend most of your time thinking about. I don’t think about sunlight all that much. But I can tell the difference between when I have it and when I don’t. Biking has a huge effect on my mood. Not eating has a huge effect on my mood. And I’m going to take a leap in logic and say that this list is globally applicable to all humans.

The complicated part of this is social interaction and satisfying work. But even the satisfying work is simpler than you might expect.

But hang on a sec. This list doesn’t just apply to humans. Look at dogs, for example.

Dogs need sunlight. Just ask one.
Dogs need sleep. They get cranky if they don’t.
Dogs need walkies.
Dogs need to meet other dogs and sniff each others butts. And humans.
Dogs need regular meals. And they’ll eat anything you give them.
Dogs need to do something. Pointers need to herd, bloodhounds need to sniff.

You break it down and you’ll conclude that human beings are social animals, and have the same needs as social animals. I used to think that the people who get up at 8 am, eat breakfast and then jog for an hour were obnoxiously happy people who were just naturally gifted. They’re not. They’re happy because doing those things will make you a happy human. Likewise, staying up until 3 am, not getting outside, getting crappy sleep and reading existentialist philosophy will make you pretty damn unhappy. It doesn’t matter what your brain thinks about the activities you’re doing – do these things and you’ll look and act like an unhappy person the next day.

Again, this goes back to how to get the most milk out out of cows or the most work out of programmers.

The Dog Hypothesis: Human beings need walkies.

So here’s how I think you get the most work out of programmers. This is a follow on from part 1.

Morale. Programmers have good morale when they are treated well, and they are given a problem that they know they can solve. Thank programmers whenever they do something. Let them know you not only know how much they work, but that you care. Keep track of morale through weekly one on ones with each and every programmer. Keep track of commitments you make to your programmers (including the verbal ones) and follow through on them. Make it clear that you are looking out for their interests. (Creating a Software Engineering Culture, Chapter 3.)

Money and stock options only have a limited effect on morale, and may have a negative effect (Agile Development, p63). Most often, a gift of money or stock options is a substitute. (Rapid Development, p262). Morale events can be fun, but don’t actually raise morale – they just allow for a different kind of interaction with people. (How to avoid Lame Morale Events).

And there are all kinds of things that can hurt morale. I think the major one is the Broken Window Theory (Pragmatic Programmer, p4). Under the Broken Window Theory, any neglect or rot in a system that is not directly addressed and countered is a drag on morale. People wonder why it is that they have to write good code and do things right, when they’re not allowed to fix the crappy code. Management assertions that “we don’t have time right now” or “we’ll do it later” start to sound empty and hollow as project after project goes by, and the crappy code festers and rots as hack after hack is piled on top of it.

The Psychology of Computer Programming, Chapter 10 deals specifically with Morale and Motivation. Rapid Development, Chapter 11 goes into typical developer motivations. They are both very much worth reading.

Sleep. You can’t make people sleep, and you can’t do much about sleeping arrangements. But you can tell them that you want them to get 8 hours of sleep a night, and you can bring up lack of sleep as an issue. Anyone who looks sleep deprived needs help; either they have been trying to sneak in work late at night, or they’re suffering in other ways. Give them all the help you can. The number of work hours will be an issue there. And if someone loads up on caffeine and junk food late at night… well, I’d point out that there may be a connection (How to Sleep Better).

Exhausted employees are easy to spot. They’re the people who frighten small children and spouses. They are not fit for work. They barely even know they are at work. They should be sent home until they know what they’re doing. Just that simple act of humanity will raise morale.

Focus. The best way to ensure focus is to ensure transparency and feedback. If programmers have a public, physical way to see what needs to be done at a granular level, whether in a todo list on a whiteboard or a series of 3x5 cards on the wall, they can see at a glance what they will be working on not just today, but next week as well. (Agile Development, p97) This works very well for helping out programmers who have a large list, or determining the priorities – you can only have one task on the top of the list, so what you are supposed to do is never in doubt. This is a technique that is used in several agile methodologies, and is known as an information radiator. (Crystal Clear, p32)

Don’t confuse your programmers, or worse, try to multitask them. If you give them two jobs to do at the same time with the same priority, you’re putting them in a situation where they cannot win; no matter what they do, they’ll be working on the wrong thing. And if they try to do them in parallel, they’ll do both jobs more slowly than they would if they did them sequentially. (Joel on Software) (The Multitasking Myth) (Quality Software Management)

Keep the number of goals small in a project. Pick one objective and make it clear that it’s the most important one. (Rapid Development, p257)

But the best thing you can do for programmers is to let them work. Don’t interrupt them. Don’t spring meetings or interviews on them with no warning. Don’t change what they’re working on. Don’t spring last minute high priority projects on them. Don’t file marketing requests to change the font size as priority one critical bugs. Don’t come up and ask when a bug is going to be fixed. Don’t ask them for status updates every five minutes. I’ve seen constant change requests happen at company after company and I can tell you from experience what happens… the programmers roll their eyes, and they stop taking priority changes seriously. Because they know that in an hour’s time, it won’t be a priority any more. (Rapid Development, p259)

If you let your programmers work uninterrupted, something wonderful will happen. There’s a psychological state called flow that has been documented by Mihaly Csikszentmihalyi. In this state, programmers are able to be “in the zone” and become focused to a great extent. Programmers in flow can produce far more code than they would be able to ordinarily. There is a catch though; it takes the average person at least 15 minutes of uninterrupted work to enter flow. If they are interrupted, or even expect to be interrupted, then they’re less lightly to enter flow. (Rapid Development, p506) Some teams implement a policy called “focus time” (Crystal Clear, p33) or the “cone of silence” (Alistair Cockburn) where meetings and interruptions are banned for a portion of the day. Other teams use a red bandanna (Peopleware) or a sign to indicate that they are not to be interrupted, or other methods.

Background. The best way to have programmers with the best background in the problem domain is to cultivate them. I’ve been surprised in the past how little programmers know what the business does. I’ve often thought it would be a useful exercise to have new programmers spend some time with each member of the business team to understand their concerns and priorities. Failing that, it can be a good idea to have documentation (business process management or six sigma documentation) that can bring new programmers up to speed on the organization as a whole. Of course, this only works so far in that it doesn’t track the history of the organization. Legacy code is nettlesome to new programmers, because typically they reflect legacy business processes and legacy business decisions. But ultimately, it comes with time.

Business specific domains, by nature, share little common ground with each other. There are only a couple of books I can recommend here. Domain Driven Design is the single best book I have read about how to effectively model and discuss domains. It is a similar book to Design Patterns, as it not only talks about implementation and common patterns, but it talks about domains as a common language. And Working Effectively with Legacy Code does an excellent job of pointing out useful ways to desnarl and refactor code that is no longer up to snuff.

Experience. Experience comes with work. It doesn’t always come with time. To quote Weinburg, experience is the best teacher, but it doesn’t necessarily teach anything. Experience can be passed on by proxy, through education and mentoring: some of the best experiences I’ve learned from have been the ones other people have had. If you want books that give experience, then Code Complete 2 and Refactoring are the best bets. If you want to read about experience, then Software Craftsmanship and Software Creativity are the best books. But if you want to make gathering experience, then make it available to your programmers. Set aside an education budget. Encourage your employees to attend conferences and seminars. Join a software engineering book club and have programmers pick out reference books for an in-house library. Have regular brown bag sessions and encourage your team to pass techniques around the company. Do this, and you will not only raise the general experience level of your team, but you will raise morale as well. (Rapid Development p257) (Software Craftsmanship, Chapter 19) (Building a Software Engineering Culture, Chapter 4).

Talent. You can hire talented programmers, but I think that’s as much as can be done. I think that talent is not a static quality. I believe talent is a series of mental habits, and that new habits can be learned, just as old habits can be put aside. I believe that some useful habits are solid grasp of systems theory, along with an ability to ask the “right” question. But that’s another essay. And there is another problem: talented people get bored doing things that don’t stretch their capabilities. If you have work that doesn’t require PhDs, you may be better off not hiring them.

Hours Worked. According to decades of economists and management experts: 40 hours. Contrary to popular belief, the standard work week was not invented by the government or the unions. It was pioneered by Henry Ford in 1926. More than 40 hours a week, and the factories didn’t produce as much money; the temporary increase in productivity was more than offset by industrial accidents and mistakes, and after two weeks there was less productivity. (Why Crunch Mode Doesn’t Work) (Can People Really Program 80 Hours a Week?) (When Should You Start Project Overtime?)

This is counter-intuitive, so I’ll say it again: studies prove overtime provides a temporary benefit for a maximum of two weeks, and is worse than useless thereafter. James Shore provides a recent example.

I believe this, not just from the studies, but from my own experience with extended overtime. Programmers will not only do less work, they’ll make mistakes in the work that they do. Then they’ll get irritable and suffer from low morale. Then they’ll burn out completely. I believe (but do not have the studies to prove) that after even after normal work hours are restored, there is a convalescent effect; people will produce less work following the overtime, producing the same amount of work overall. So after the project goes live, they’ll need a long convalescent period before they’re up to snuff, or even worse, they’ll look up from their monitors, take a good hard look at the results of their labor and their (usually meager) rewards, and quit, producing a huge opportunity cost for the company in terms of hiring, maintenance, and reputation. (It’s Not Just Abusive, It’s Stupid)

For every complex problem, there is a solution which is simple, obvious, and wrong. Extended overtime is that solution.

Fine; hours worked have no effect. What about directly applied pressure? What happens if we keep the hours, and if we tell the programmers to work harder and produce more work during those 8 hours?

Surprisingly, nothing. Programmers produce work using their brains; the amount of thoughts a programmer can have is fairly constant. Tom DeMarco and Tim Lister researched this and formulated Lister’s Law: People under time pressure don’t think faster. (Slack, p50) They might be more stressed, but that doesn’t help people think faster; stress impedes complex thought, and pushes the brain to a “flight or flight” response. If you’re stressing your programmers, they’ll be at their desks more. But they’re not going to write any more code.

This advice is all simple and straightforward. I don’t see anything in here that comes as a shock, and even my mother said “Well, that’s obvious, isn’t it? It’s like being a farmer and taking care of your cows. If you want your cows producing the most milk, you make sure they’re treated like cows should be treated.”

So treat your programmers well. Keep track of morale through weekly one on ones. Make it clear you care about the health and welfare of your programmers. Make sure programmers know what they should be working on at all times. Don’t change out work that the programmers are doing or abuse the bug tracking system. Keep interruptions to a minimum to allow for flow. Establish a training and education budget, and establish mentoring and brown bag sessions to transfer experience. Keep to 40 hour work weeks, and forgo direct pressure.

Do all of these things, and you’ll get more work out of your programmers. And you’ll probably have programmers beating down your door to work for you.

EDIT: Also see this LinkedIn question that provides some useful advice. Larger monitors have been mentioned in several studies, but I don’t have the references to hand.

I was recently asked the question: what is the best way to get the most work out of my programmers?

I’ve thought about this for a while. In some ways, it’s the story of my career – every manager I’ve ever had has wrestled with this question. I’ve worked with enough teams and individuals that I think I have a good understanding of what programmers can and can’t do, and how work gets done. And how work doesn’t get done, for that matter. I believe I have an answer, although certainly not the answer.

The first thing to do is to look at the assumptions behind the way the question is phrased. The assumption here is that there are veins of untapped work, hidden inside of programmers somewhere. And the manager’s job is to extract it. I think that this is the wrong place to start from. I believe that most people would rather do a good job than a bad one, and will do the best work they can do given what they have to work with. (Software Creativity 2.0, p22.) The best way to phrase this question would be: “What can I do to help an individual programmer be most productive?”

So, let’s construct a hypothetical employee. What is known to create productivity? Let’s start from the simplest level and work up.

Morale. An employee who feels safe and secure in his position and his ability to raise issues will do more work. You can argue whether this is the cause or the effect, but it’s well recognized that morale is extraordinarily important. Someone who believes in the team, the company and the work he is doing is likely to do more of it. I believe motivation and morale are equivalent, but morale feels like a better word to me.

Sleep. An employee who has slept 8 hours a day will be more productive than one who has slept 5 hours. I have yet to hear anyone argue with this in principle. Studies have been done that compare the effects of sleep deprivation to alcohol intoxication. The outcome: a programmer awake for 21 hours is effectively too drunk to drive. Think about that. There is a second stage of incapacitation, which is exhaustion AKA burnout. Exhausted employees are, by definition, not productive.

Focus. An employee who knows what he is supposed to be working on will be more productive than one who is not sure which programming task takes priority. An employee who knows that he can complete the work he is doing without having his priorities changed will be more productive than one who does not know what’s coming next. Programmers dislike surprises as much as anyone else. But this also covers the programmer’s inherent self-discipline. Given a gap in work, will the programmer ask what he should be working on, or will he create his own scripting language?

Background. An employee who knows the domain, the history and the subtle interplay of forces at work in the codebase will be more productive than a programmer who has no previous background in the work assigned. This factor is often overlooked, but domain expertise is an incredibly valuable asset for a programmer, and anecdotally speaking I’d say it makes an order of magnitude difference in time and quality.

Experience. An experienced programmer – a veteran – is more productive than an inexperienced one. An experienced programmer will understand that this project needs two weeks just to be deployed, that there’s a particular tool that’s perfectly suited to refactoring the database, and when it’s worth building extra flexibility into an area of the system because the business team will almost certainly want customized logic in the next release.

Talent. Some people are just good programmers. It’s not simply intelligence, or interest; I’ve known very smart people who don’t make good programmers, and some of the most interested programmers often did the worst work. A talented programmer may do work that may be simply beyond other programmers, no matter how much experience or background they have.

I don’t think anyone will argue that the above factors matter. If any one of these goes to zero, your programmers will be effectively useless. There’s also another factor, which is more complex.

Hours Worked. A programmer who works 40 hours a week will do more work than one who works 20 hours a week. And a programmer who works no hours a week, will do no work. But I think everyone will agree that a programmer who works 168 hours in a week will not produce 168 hours of work. In fact, I feel fairly safe in saying that a programmer who works 168 hours will produce less usable work than the programmer who works 40 hours, because the programmer’s ability to work will be degraded by lack of sleep long before they stop working.

Obviously, these factors are interconnected. Someone with no experience will typically lack background. Someone with no sleep will have low morale very shortly, and will even lose talent. So let’s see what we can do to increase these variables.

[This is the speech I gave at Kristie’s memorial.]

I met Kristie through OKCupid on March 29th. She messaged me and said that I had the same first name and first initial of last name as the man she was divorcing, and how disturbing that was.

Despite that, we kept writing to each other. We had too many of the same interests not to write. She loved books and despised Ayn Rand. She liked philosophy but thought that most philosophers were not worth the effort. We both had far too much we wanted to read, and far too many books we had already and not enough time to read them all, and no good way to get rid of the books that we had already read.

We wrote to each other almost every night. We talked about the best way to dispose of books. She liked the idea of bookcrossing, but was horrified by the idea of leaving books exposed to the elements, and I thought my idea of leaving them at the library was too pedestrian.

I met her once on the weekend on May 27th, when she was coming in to San Francisco for a haircut. Whereupon we discovered that we had the same hairdresser. We met up, had a double jointed contest. I made jokes about her pugs and she made jokes about my man purse. We went back to my place, she went through my library with a fine tooth comb, we exchanged books, and I gave her some comic books. And then I walked back to 4th street with her, gave her a hug, and let her go shopping for girlie things.

After we met, she sent me an email telling me about the rest of her day, ending it with “So there you have me. Wierd, inconsiderate, and kind of annoying.” And I knew I’d found a kindred soul.

And then work got crazy. People quit, and the deadline came up close. I started working weekends. Kristie had work explode on her as well. And then I finally had a weekend free and was trying to figure out why Kristie hadn’t returned my e-mails. And so I went to her livejournal. And she was dead, three weeks after we first met.

I wondered why I was crying.

I thought that part of it was because it’s a horrible thing to die, in the prime of your life. With no chance to say goodbye and no chance to find out all that you are, and all that you could be. But that wasn’t it.

I only met Kristie once. I wondered how I could miss someone so badly that I had barely met. But Kristie was special. She was so real and so important to me. She cared about TRUTH. She cared about JUSTICE. And when I finally believed that she was dead, I have to confess that one of my thoughts was that there are so many people that the world can spare.

I miss her. Kristie was a friend. A new friend and a real friend. She was someone I could have liked for many years. She was someone I could have shared things with for many years. She was someone I wanted to know better, and I never will now, and that sucks.

But I’m glad that I met her. I’m glad that I got to share something with her. I’m glad that I knew her. And I’m glad to have a little voice in the back of my head that says “Kristie would have liked this.” So thank you.

What is the best way to make decisions? Given a limited amount of time and information, how do we decide on the best course of action? And how do we determine what “the best” is?

Cognitive Science studies how people make decisions in real life. There’s also a branch of cognitive science which tries to form models of decision making. I read Simple Heuristics That Make Us Smart as a way to figure out if there were any shortcuts to decision making. I didn’t find anything globally applicable, but the book is fascinating, and I’ve been meaning to write about it for two years now… so bear with me while I run through this.

Decision making in academic literature is referred to as rational choice theory or rationality. The book goes into some detail about different kinds of rationality: unbounded rationality assumes total knowledge of the situation (as in the “perfect market” economic theory) whereas bounded rationality suggests that the human brain has limits and “uses approximate methods to handle most tasks.” It also suggests that bounded rationality exploits the environment: different heuristics are applied in different environments, and a simple heuristic may work extremely well when applied to a specific environment.

The book divides bounded rationality into two categories: satisficing and ”fast and frugal heuristics.”

Satisficing is a heuristic organized around making an acceptable decision in the shortest possible time. It is literally “decide what you need, then pick the first acceptable solution.” Don’t bother looking at the other options. Don’t even try to find out if there are other options. Pick the first solution that meets your needs.

Satisficing is a good heuristic in time constrained situations. It’s a good strategy for buying a house. But it’s not the best heuristic, because deciding what you need can be complex, and deciding what option is the best one can be complex. Satisficing can lead to sub-optimal solutions. But it’s a great way of avoiding analysis paralysis.

Fast and Frugal Heuristics are the approach that receive the most attention in the book. They are environment specific heuristics. They may not work in some situations, but they will make the right choice more often than not. And they work better in spite of using less information than general decision making.

The best way to describe a frugal heuristic is by an example from the book. When a patient is admitted into a hospital, there are 15 different variables (also known as cues) which could describe whether the patient is at high risk for a heart attack. A doctor could look at every single variable and come up with a decision, but it turns out that only three decisions need to be made:

“Is the minimum systolic blood pressure over the initial 24 hour period more than 91?”
“Is Age > 62.5?”
“Is sinus tachycardia present?”

If the answer to these three questions is yes (or the first answer is no), then the patient is high risk. If not, the patient is low risk. This may seem like an oversimplified example, but this heuristic actually works better in the field than the one which examines all possible variables.

The interesting thing is that this works not because it’s a generalized strategy. It works because the environment is fairly static – patients come in and either get heart attacks or don’t. Given enough time and data, heuristics like these can be derived. However, they can’t be known ahead of time. You know this stuff from experience. It is, to the people who know the heuristics, “common sense”, even if they can’t define it.

The nice thing about heuristics is that they’re practical. Using a heuristic will lead you to making the wrong decision some of the time, but it works better when you have to make a decision, and you know that making a decision is more important than getting it wrong.

There are a number of useful heuristics that appear in the book, and the greatest challenge to writing this post is that I want to write about all of them. Instead, I’ll just present summaries and go over them briefly.

Ignorance based decision making

People know when they know something. There’s even a saying: “Better the devil you know, than the devil you don’t.” A recognized quantity is more likely to be better than an unrecognized quantity. Heuristics exploit recognition to put a greater weight on a cue with a known value. This principle opens the door to a number of heuristics, notably “Take the Best.”

“Take the Best” can be described as follows: All things being equal, pick the movie with Tom Hanks in it. If not, at least go for an actor whose name you recognize over one you don’t. There are other one-reason decision making rules, but I’ll focus on this one because it’s clearly the one the researchers like best.

Take the Best makes an intuitive amount of sense, even if it does have the big weakness to “Take the Best” is that it has to be trained to know what’s a valid cue (you have to know who Tom Hanks is). Or as the book charmingly puts it, “the validity of a cue is defined as the number of correct inferences divided by the number of correct and incorrect inferences made using the cue alone.” In short, you have to know who Tom Hanks is before Take The Best will do you any good.

In a competition pitting heuristics to guess the size of Germany’s largest cities, these three fast and frugal heuristics, using at most 3 cues, were more accurate than heuristics that looked up all 10 of the cues. The effectiveness of the frugal heuristics varied with how much information was known, but was still higher than the non-frugal ones. Sometimes having all the information isn’t a good thing.

In practice, people tend to use a strategy called LEX, a generalization of Take the Best. Lex is defined as “the highest cue value on the cue with the highest validity. If more than one alternative has the same highest cue value, then for these alternatives the cue with the second highest validity is considered, and so on.” If there are two movies with Tom Hanks, then pick the one directed by Ron Howard over the one directed by Michael Bay.

Why does it work?

These class of heuristics work in situations where the information is “non-compensatory” (i.e. a movie with an A-list actor and an A-list director is unlikely to have a D-list cinematographer) and the information is “J-shaped” (which I think corresponds to Zapf’s Law… there are only a few excellent movies each year). In addition, this heuristic works very well in situations with “scarce information” (you don’t know everything about every movie) and “decreasing population” (that is, good movies run longer than bad ones). This covers a large percentage of situations, without tying the heuristic too tightly to a particular situation.

Satisficing in Mate Search

There are situations where a “Take the Best” heuristic is not the best one. While you can choose between several movies, only the determined or foolhardy dates several women at once and compares them against each other.

There’s a puzzle called “the secretary problem” in which the best secretary must be picked from a pool of applicants. However, the applicants appear in random order, and once a secretary has been rejected, she doesn’t come back. The best solution to this problem (and bear in mind that this problem assumes a lot) is to interview the first 37% of the candidates, and then pick the next candidate who is better than the best in the sample size. Following this rule, you will end up with the best applicant 37% of the time.

But the secretary problem misses one small issue: while you are searching for the best match, your match is also looking around for his or her best match. Ironically, there is not much research into the best practices for mutual search. There is some research into the mate search strategies that people actually use, notably the charmingly named “one bounce” rule, where people continually look for the best mate, but stop as soon as the next prospect is less attractive than the current selected one.

And here is where I go off the geek deep end, past even Dating Design Patterns: the book actually goes looking for simple satisficing search heuristics for “biologically realistic mate search problems.” Oh baby.

If you have high date value, and know that you’re a good catch for any prospective mate, then the simplest dating strategy is called “Try a dozen.” The best chance you ever have of meeting the perfect mate is 37% – not very good odds, and that’s assuming you date at least 30% of the available population. The mitigating factor in mate search is that you don’t have to find the absolute best mate: you just have to find a good mate. Instead of taking the next best from 30% of the population, we can sample a smaller size and relax our restrictions a bit for a faster result.

By only dating 14% of the population, you have an 83% chance of finding a mate in the top 10% of the population. If you are willing to settle for a mate in the top 25% of the population, then your numbers get even better: dating 7% of the population will find you a suitable mate over 92% of the time.

Even though this looks statistically inviting, 7% of the population still looks daunting given a large enough population. However, the interesting thing is that the statistical significance of mate quality goes up with the rise in population: to find a mate in the top 25%, only 1% or 2% of the population needs to be checked. So Try a Dozen works for even large populations, although there does come a point where it doesn’t scale.

But there’s a catch. Try a Dozen only works when you have a mate value and know that none of your candidates are going to reject you.

The researchers wrote a dating simulation program, where they could see the best overall strategy for mutual date search. In this simulation, individuals had the same problem that we do: it’s hard to know how attractive you are to prospective mates from the inside. In fact, your dates have a better idea of how much of a catch you are than you do.

The only realistic way to know your own mate value is to go through a learning stage (“adolescence” in the book) where you date a number of candidates and determine your own attractiveness through feedback. If a highly attractive mate proposes to you in adolescence, then you increase your own rating by a value proportionate to the mate’s attractiveness. If a low-rated mate rejects you, then you decrease your own rating proportionately.

There are two measurements of success in this simulation. The first is the number of matches: if too many individuals have unrealistic expectations, not everyone will be matched. The second is the spread between matches: if a high value individual “settles” for a low value individual, then the match is not optimized ideally.

It turns out that with a short adolescence, most of the high valued individuals pair off immediately with the first random person, and the low valued people remain single because they are overconfident. But give the individuals an extended adolescence, and the group equilibrium quickly comes within 10% of optimum. Best of all, this heuristic works only even when the total size of the population is not known, and seems to be asymptotic after checking 20 individuals.

This also seems to make intuitive sense. However, the researchers explicitly state that the model is lacking some of the finer details common to dating. Populations are not fixed. Different age ranges may have different fitness criteria. The distribution of mate values are not as evenly distributed as they were in the simulation, and individuals don’t have a chance to change their mate value based on their adolescent experiences.

The researchers are also careful to explain that mate search is not just a matter of percentages, even if it can be simulated as such. Love, they carefully explain, is essential in making decisions stick: in the heuristic of dating, love is the stopping rule.

Remembering and note taking is a different skill than organization and action items. GTD goes into great depth about how to make sure you know what to do, but doesn’t explain how to structure semi-random data.

When I first started working with computers, I had to remember things. Especially the details: what the Samba network name was, the syntax for setting up FreeS/WAN, and what tools I had tried to configure the network that didn’t work, and why they didn’t. My solution was a boring Mead notebook. I dumped everything in there, and referred back to the notes if I had to work with the system again.

Then, I found out that I was using the notebooks for todo lists. And addresses. This had a couple of bad effects: one, if I wrote a todo list and then wrote some more notes I had to flip back to where I was to find out what I should be doing… and two, there was personal information in there (i.e. do laundry) that really I didn’t want to see when looking through the notebooks later.

So I got another notebook for personal stuff. One notebook for work, one notebook for home. Simple. I’d use the notebook at home for all my personal stuff and that would be it.

Then I started using the personal notebook as a diary. This was actually a good thing (you can learn a lot from reading through old diaries), but it did have the unwelcome side effect that I was terrified to go to Safeways with it.

So I split the personal notebook into a Todo list and a diary. Write essays in one, write tasks in the other. This actually worked until I replaced it with the GTD system I use currently, where diaries are treated just as another inbasket item.

For work, I ran into another problem after a few years. I have several different projects that I’m juggling, as well as my own personal projects. Some of these are at different stages i.e. I’m writing code on one project, reviewing code for another, and looking at product options for another. When all of these got hammered into one notebook, the end result was not great.

So the current system. I buy about 10 notebooks at a time. I write the name of the project on the front of the notebook, and the start date, and use the notebook for the project only. I write down technical information, logs, and caveats. No tasks. When the project is finished, it gets retired, and I can then look back through it for a post-mortem.

This probably isn’t the best system, but it’s the best I’ve found so far. Writing things down in a notebook gives a permanence to thought that I just don’t get when I use scrap paper or a PDA.

I’ve always wondered about the state of the industry. Why 30% of projects fail, why it is that there’s always a huge need for more programmers, and why this has been the situation for around 30 years.

Apparently Bill Gates is calling for the cap on H1B visas to be lifted because there’s such a serious demand. And yet… there are no shortage of people who come to interviews. There are still plenty of people out of work. So what gives? How can there be not enough people and not enough jobs at once?

The blunt truth is that most people looking for a job really suck. I’ve interviewed people who literally couldn’t write a for loop from memory. You would think that this situation was caused by the dot-com boom sucking up all the talent… well, sort of. The boom sucked in everyone. But the bust dropped the least experienced people first. As a result, you now have even more minimally trained people looking for positions.

Joel Spolsky goes into this situation a bit. His rendition of the interview process is dead on, and I’ve seen a number of people get into trouble by picking the resumes with the right buzzwords, instead of one with all the words spelled correctly. There are a ton of people out there, if you’re not picky. Even if you are picky it still not be enough to save you: after looking at 200 D-list developers, a C-list developer starts to look pretty good.

But still. If there are so many people out there, why call for more visas? Why is there so much demand if there’s a limitless supply of mediocre talent?

It has something to do with the amount of work that these people do. Steve McConnell mentions “problem programmers” in a paper citing ranges in programmer productivity. He points out that a number of programmers couldn’t finish the project at all, and theorizes that some programmers may have negative productivity, “because eventually someone else will have to redo the work of programmers who can’t finish their assignments.”

But David Parnas lays it out flat in this interview:

Q: What is the most often-overlooked risk in software engineering?
A: Incompetent programmers. There are estimates that the number of programmers needed in the U.S. exceeds 200,000. This is entirely misleading. It is not a quantity problem; we have a quality problem. One bad programmer can easily create two new jobs a year. Hiring more bad programmers will just increase our perceived need for them. If we had more good programmers, and could easily identify them, we would need fewer, not more.

Parnas’s Law of Hiring: The more incompetent developers you hire, the more you need.

Software Engineering studies tend to put the cost of poorly designed software in terms of “maintainence”. I thought for a while that meant how much work developers had to do to add a new feature to an existing product. But there’s another meaning behind the word, and that’s how many developers need to be hired to fix and keep running the broken software.

This law explains why no matter how many H1B visas companies get, they always seem to need more, and why software projects have been having a “crisis” for over three decades. I wish I could say that H1B visas and offshoring would fix the problem, but from what I can see it’s about cheap labour, and only that. The industry is digging its way into a hole.

I’ve had some time to think about the AYE conference. I’d been expecting it for so long that the actual event felt like a movie – I wandered around for the first hour or two thinking “Am I here now? Is this real?”

And yes, it was. It was better than I’d hoped, and I hope to go to it again next year. But my understanding of the conference changed dramatically once I sat down to dinner and started talking to people. The strength of the conference isn’t in the presentations. It’s in the people attending them. In the first breath, I heard a management consultant, an architect and a couple of QA managers trading war stories and bitching about what passes for “state of the art” in the tech industry. Each of these guys could be the driving force behind a project. Putting all of them in a room together is a recipe for the perfect storm.

This wasn’t an accident: the presentations are structured to take this into account. The first presentation, appropriately entitled “It’s 2004. Why is Risk Management Avant Guarde?” I went to the presentation expecting an answer. Instead, the presentation itself was an open question. About five minutes into the presentation, I realized that Tim Lister was the same person who co-wrote Waltzing with Bears. He related some of his experiences, and pointed out that there is no rhyme or reason to the quality of processes in a project. There can be excellent process in breweries, and lousy process in a software development company. Large corporations seem to be prone to poor process, but that’s only because a large company manages more projects and poor process is the rule, rather than the exception. Then he opened the floor to discussion, and served as a faciliator for the various war stories.

He had some very interesting insights. The main one was that people routinely practice risk management in their daily lives (“do I think I can catch the plane? What happens if I get sick?”) but completely fail to do it in business. You can take some very intelligent people, stick them in a room and they become completely incapable of managing risk.

I know this happens, because I’ve seen it before. I was told that at a high enough level, managing an organization becomes less about risk management and more about blame management. As you get higher and higher, it becomes easier to manipulate perceptions and image than it does to manage the facts on the ground, and so the people who try to (perhaps naively) manage the facts alone end up looking significantly worse than the people who devote themselves to looking good. Since risk management is about accepting the possiblity of failure, anyone who tries to kill an obviously broken project is taking the responsibility for it, and therefore the blame. Really good practitioners will make sure that responsibility for the project is spread around everyone at the same level, meaning that anyone who moves to kill it will give the others an opportunity to scapegoat the individual and distance themselves. Tim sensibily said that that sounded like a completely dysfunctional organization and life is too short to work for them, when there are perfectly good companies. (I think asking for a non dysfunctional organization is like asking for a non dysfunctional family, but presumably they must exist somewhere. At one point, he got asked for a list.)

At times, Tim’s responses were so brutal (although perfectly consistent with his book) that I wondered about his relationships with clients. The answer is simple: he gets brought in to either fix something that everyone knows is broken, or he serves as official excutioner of the project. Either way, his prime targets haven’t read his book, and probably never will.

The second session was with Esther Derby. She gave a talk on making effective decisions. This was a little less successful than the first session, as it was mostly geared towards managers and didn’t take technical decisions into account (where achieving consensus is typically about finding a solution that can’t be shot down by anyone and is only as ugly as it has to be). At one point we were asked to form a team and make a decision on a given subject, which led to our first decision which was… what decision making method we were going to use.

I (finally!) met with Dave Hoover and his wife Traci. In a conference full of strangers, it was nice to meet someone I knew, even only online. I had a great time, and even dug myself into a few holes with Traci’s help.

The next morning was exhausting. After a while, the constant presence of grizzled twenty year veterans became too much and every conversation was like drinking from a firehose. I attended a presentation on the Satir System that made no sense to me, and made me doubt what I’d read about it. I understand the human need for symbols, but symbols by their very nature are symbolic, and handing someone a stick doesn’t make them any more courageous, any more than handing someone a cross makes them a Christian.

The second presentation was about non-quantiative metrics. This presentation also didn’t make any sense, until Johanna Rothman sat down next to me. She explained that an NQM is based around everyday observation, i.e. noticing the presence of pizza boxes and takeout is a good sign of extended overtime. I understood the idea, but still don’t get why the “NQM” word, a concern that was brought up by several people and led to a long and boring argument about phraseology and semantics. The most useful metric we came up with was “The number of times the team goes out drinking.” Johanna and I talked after the presentation, and I was struck by how clear Johanna is – she says what she knows and limits herself to only what she can clearly explain to others.

After two days, I thought I knew what to expect, but the third day was different again. The first presentation was “Effective Iterative Development”, but it turned out that (either by accident or by design) one team consisted of what one man referred to as “agilistas” – devotees of Agile Development – and the other team consisted of BDUF people. I was in the BDUF team, and we spent the first 10 minutes trying to understand what a SCRUM leader was, who was doing what, and trying out different design ideas. Our team hadn’t gelled immediately, and by the end of the session we did not have a working prototype, although we had worked out the design. Meanwhile, the other team had cobbled together a prototype that looked horrible, but was at least functional.

At the end of that session, we got owned. I got owned in particular for trying to defend our approach without addressing the client’s concerns first. And when we went on break, we reflected on just how owned we had been. And THAT was when the team gelled.

When we came back in, we had the design worked out and we each worked on modular blocks. In the next ten minutes, we had something that not only met the requirements, but exceeded them and was scalable and elegant. The other team hacked on theirs a bit, and had something that met the requirements. But they had to repeatedly test it and hack it, and there was no division of labour, meaning resource contention and scheduling conflicts. In addition, they had to work out design at the same time, while our team had the design issues done and could focus on implementation.

And the finished products were not as interesting as the discussion afterwards. There was a general consensus that both approaches were valid, and the agile team had a happier client for most of the project. The counter argument was that with the future overhead of maintainence and enhancements, the agile team would have had an unhappier client as refactoring was done to support the additional demands on the system. A very enlightening discussion at times approached a philosophical debate, it was clear that it wasn’t going to be definitively settled by either side, and that both approaches had their strengths and weaknesses.

The final session was with Jerry Weinburg and Johanna Rothman. It was on turning rules into guides. We were asked to give examples of a rule – something we would never do under normal circumstances. One example was “Don’t stick your finger in your eye.” Not all rules are bad. We grouped into teams to make examples of our own rules, and had difficulty until Johanna came around and explained the principle. Ask what it would be like if it happened. If you can’t imagine what your life would be like or what would happen if the rule were violated, it’s a survival rule.

So my survival rule was be “Make rational decisions.” This is a survival rule because I have difficulty imagining what life would be like if I didn’t make logical decisions. Another man’s survival rule was “Never allow my integrity to be besmirched.” And because these are survival rules, they are MUST rules. I MUST ALWAYS make rational decisions, as opposed to “I can sometimes make rational decisions.”

But the decision to make rational decisions cannot itself be a rational decision. It’s a self-contradictory statement. Not only that, it’s not even an effective decision making strategy. Unbounded rationality satisficing always comes second to fast and loose heuristics, because it’s too expensive to determine whether a decision is rational or not, and there’s no way of telling whether the determination is rational either. If you want to make lots of rational decisions, statistically you’re better off picking a method that will accept irrational decisions being made from time to time.

As a result, I have a statement that makes my head hurt. To make rational decisions, I have to make irrational decisions. If I’m sure that all my decisions are rational, that’s a sign that my decision making is irrational. Being convinced I can make rational decisions all the time is a sure sign that I’m being irrational, the same way that being convinced of your sanity is a sure sign that you’re crazy.

I spent the next session and most of the next two days stomping around the place and mumbling under my breath. I still find it a very hard concept to grasp. But I know enough not to poke at it too much now.

I haven’t written much about programming lately.

The main reason is because every time I start writing, I feel that I’m Missing The Point. I don’t know what point I’m missing, but I have the sense that it’s obvious. Or it should be. And writing about a code optimization technique is just a pointless hack.

I think part of it could be that I’m just getting old. I remember when CORBA came out, I was so excited. And then EJB (clearly more of an answer to CORBA than it was to any kind of persistence solution). And now there’s SOAP. The common factor in all of these technologies is that they’re not simple, and the implementations are hard to get right. And then they don’t work with each other. The users complain back to the vendors, who go back to the spec committee. A new spec is written, but there are already fixes put in place which must be worked around. Versioning becomes an issue. Newer specs try to define how to deal with older versions of the spec. And so on. The organizations which avoided this problem did so by defining one implementation. If you use X’s RPC model or RMI (or most Apple stuff), you only use one implementation, so it All Just Works. But every interoperable system has to deal with incredible amounts of crap, and generally speaking they don’t do a good job of shielding that crap from the user.

Not only that, but these systems typically don’t realize when they don’t work. There’s no way to open the hood up on your typical implementation and disable or custom write a small section of code. In most cases, the public API of the system will be well documented and well written, but in my experience the internal interfaces of most implementations (especially the first few versions) are confused, and even mislabeled. These systems don’t fail gracefully, and they can’t be fixed.

And really, fixing bugs is what I spend most of my time doing. I like to think of myself as a programmer, but maybe a more accurate term would be “bug-avoider.” I can count out my hours in terms of bugs avoided while writing, and bugs fixed after writing. When I first started coding, I ran into every single damn bug head on, and I had no ability to see bugs on the page. I stared at the code, but I still couldn’t see bugs. I knew there were bugs in there. But I was missing something I was staring right at. So I thought “if I’m always going to write bugs into my code, I may as well write code so it’s easy to take the bugs out.”

And that’s the problem here. There’s no tolerance for failure built into the specifications. There’s no acknowledgement in the implementations that parts of the spec will have been misunderstood and may need to be tweaked by the user. There’s no lower level debugging API available, no diagnostics and no logging. It’s not that the implementations are at fault – no-one said that they had to do this stuff. It’s the specification’s fault for not defining a diagnostic API along with the main spec.

This is something the industry does all the time. It consistently gives users what they ask for, instead of what they want. Products are measured by their features instead of by their bug-avoidance factor. But I’d choose a more limited product with a solid test suite and solid logging and diagnostics over a glossy black box with no access to the internals any day.

I wonder sometimes if I’m living in the same industry that I read about in the periodicals and on the blogs. My problems are not going to be fixed by a new scripting language or programming trick. They are fun to read, but they are not on the critical path that helps me get my job done. And I almost never hear about bug removal techniques, theorem provers or the importance of writing maintainable code. Or a zen discussion on what “maintainence” really is, or how “maintainence-nature” can be revealed. I want more books like TCP/IP Illustrated: The Implementation and Tex: The Program. I want a book about how to recognize broken code and another book about how to fix it.

That’s part of the point, but it’s not all of it. I feel like the answers I’m looking for are just shadows cast by a larger question. I’m going to the AYE conference tomorrow in hopes of enlightenment, but until I can define the question properly, I’m going to be one confused geek.