Terse Systems

I moved all my note taking into Evernote, and I’m using it for everything – diaries, documents, clippings, etc. Much simpler and more powerful than using three different systems.

I had to write some Ruby scripts to get everything imported: they are available at http://github.com/wsargent/evernote-import. Imports from plain text files and from Memiary XML format.

I started a job at Grockit a couple of weeks ago. This has been my first chance to see what an agile shop looks like, and it’s an eye opener.

You show up at 9:15 am every morning. There’s a daily stand up meeting (truly stand up, everyone is around a circle), and then breakfast, which is served in house; people sit down and chat about stuff they’re doing, but it’s less formal than the standup is. (I don’t have a good handle on how retrospectives or user story planning work, but from what I can tell there’s an overall direction that is set during releases, and then the business owner participates in standups to ensures that everyone’s on the right page. Spot demos also appear to be frequent, although I don’t know how much of this is a conscious strategy.)

Following the standup, there’s a secondary huddle where engineers look at the list of user stories open, and pick off the stories that they want to tackle. Pairing is done by the tech lead rather than the individual engineers, and there’s an emphasis on switching pairs around. Tasks get written down on the whiteboard, photos with names are assigned to them for the day, and then it’s time to find a free workstation and move in.

Pair programming is the natural order of things at Grockit. The workstations (Mac Pros with 30 inch monitors) are all set up in exactly the same way (IntelliJ IDEA with Ruby Plugin, QuickSilver) and have two keyboards and mice attached. There are two engineers per desk. The first thing you do is bring up Pivotal Tracker, find the user story that you’re working on, and then mark it as “In Progress.” The user stories are typically single sentences such as “Put the top players per week on this page (refer to mockup)” and have points assigned to them by the engineering team beforehand. Then you discuss a couple of ideas, make sure you know what the state of play is, and start coding.

Only one person codes at a time. At this stage, I don’t know enough about the application to be effective on my own, so I’m mostly monitoring and following what my partner is doing. It’s an involving process to watch someone else code; you’re not just looking at the physical processes involved, but also the intent – usually I can catch bugs a couple of seconds after they’ve been typed, although I usually give feedback after we’ve reached a pause and I can be sure I’m not interrupting my partner’s flow. Actual disagreements are rare, partly because no-one is attached to an idea yet, partly because everyone is “on the stage” – you know that whatever you’re typing is going to be peer reviewed almost immediately, and it’s important to write something that’s going to be clear and acceptable to the partner at the time. I think that mixing pairs every day is also a factor here. Any time someone’s had a problem with code, it’s been new code that is unfamiliar to one person and not the other, so there’s a fair amount of learning that goes on each day.

It’s also draining. Reading code is almost as intensive as writing code, and there’s a ton of moving pieces, plus a couple of custom built frameworks that I’m unfamiliar with. More often than not, when someone asks me to take over or asks for suggestions, I have the odd experience of saying to refactor code using features that I’m not consciously aware of. I can say what to do at the time, and how to rework it, but when I get up from the computer, I have no clue what I just said or even what it meant. It’s a nice ability to have, but it’d be nicer if I actually knew how it worked.

The other reason it’s draining is because either you’re typing or you’re reading. There’s no downtime or distractions while you’re pairing. The “why” of this is interesting.

Grockit has email, but it’s mostly used for company broadcasts, housekeeping and customer feedback. It is not used intra-company, and this is a deliberate choice. Nearly everyone is in the same room. If you want to schedule something or start a group discussion, you mention it at startup and pull everyone in later. Likewise, there’s no instant messaging. No IRC channel. Nothing. So, no distractions. This took a while to get used to – I had to start scheduling and communicating with friends in one large block once I got home, and some people were even worried that I’d vanished completely.

So. You work until 1:30 pm, and then the lunch bell rings. Lunch is served in house, and is delicious organic vegetarian. After lunch, there’s a break where some people check their email on personal laptops, or play Rock Band, or run errands. People usually head back into the office at 2:30 pm, and then it’s pairing time again.

At this point, you’ve got most of the code written and you’re reworking and adding new tests to verify that what you’ve got actually works and that you can check it in. (Or the TeamCity build has broken, which results in immediate and productive feedback from the team.)

The test suite at Grockit is beautiful. Most of it is integration tests, with some mocks in place where they want to see specific exceptions. It makes heavy use of nested describe and contexts, and even has a very elegant use of should_behaves_like in the order processing logic that would have otherwise taken some complex metaprogramming of specs to do. There’s a strong bias to make tests useful and not repeat themselves, so tests usually only happen at one level. If the code can only be adequately tested using Selenium, for example, then there’s no point in writing unit tests for each method that would also test the happy flow.

Having said that, most of the tests involve business logic or concepts that I have to ignore because they’re not related to the problem at hand. Having to write or refactor tests while simultaneously not looking “over there” is mentally taxing, and then there’s the extra juggling involved with the mocking framework, selenium, and the assumptions made in the code itself. This is usually the point in time where my brain gets full. I have to pick up the threads of where we left off before lunch, so either I have to rummage around before I feel I know what’s happening again, or I just get a blinding headache trying to jump several steps ahead. Apparently, this is a common phenomenon during the first three weeks of coding, so I’m hoping it gets better soon.

At some point, the main user story is finished, tested, demoed, and handed it off to be accepted or rejected. Typically we get this wrapped up by 4:30 pm, and then pick off a couple of smaller user stories that involve front end work if they’re not too complex. This goes on until 6 pm, at which point the work day is over and everyone goes home.

And I mean, everyone goes home. There was one user story that I thought we could do, but was told “We can’t finish that; we only have 10 minutes left.” That’s discipline.

To recap: you show up at 9:15 am, do good solid work for eight hours, and then go home. You do not take your work home with you, or work on the weekends. User stories and estimates are scheduled so that all the work you can do is done in eight hours. Velocity is automatically tracked and measured using Pivotal Tracker on an eight hour basis. The hours you work is eight, and that’s the time you need to do a good job, with covering tests and code that checks the edge cases.

I like it. It’s not easy, but it’s so, so worth it to me.

I went to Patagonia for my vacation this year, cleverly timed for SXSW.

It was harder than I expected to unplug. For the first week, I had iPhone twitch (I’d left the phone deliberately). The second week it got easier as we got into longer and tougher hikes.

The food was good. The hiking was gorgeous, and I lucked out on my hiking companions, Lois and Kent (yes, really), who were and are charming people and an incredibly cute couple.

Photos are here: http://picasaweb.google.com/will.sargent/Patagonia

Check out the La Bomba Del Tiempo videos especially; 16 drummers all working together, beating up the crowd into a frenzy.

I’d tell you the blow by blow, but honestly going through holiday snaps is something best done on demand. If you’re interested in anything, send me an email and I’ll fill you in.

I’ve been going through an inordinate number of machines lately. I’m down one Seasonic power supply, two motherboards (one Megabyte, one MSI) and the Thinkpad came back from lenovo with the original problem solved, but with half the screen on the fritz.

This isn’t new or unusual; most of the machines I’ve put together have only lasted a couple of years or so. Consumer hardware just isn’t built for reliability; hard drives being the worst offenders. As a result, most of the information I really care about is based off-site. The two biggest exceptions are the iTunes library, and my financial data.

The standard solution is to have a backup hard drive. The problem here is that backup hard drives also go bad. I’ve run into this a couple of times with the Linkstation and the Western Digital External. There’s not much you can to do test this except restore from backup every once in a while and see if it actually worked (which is about as fun as it sounds).

Another possible solution is to use offsite backup such as Mozy or Amazon S3. I read Zawodny’s experiment with it and started using s3sync. But.

1) Uploading 30GB of music took a week.
2) Most, if not all, of that data was corrupted to the point of unintelligibility.

I only found this out when I thought I was missing some data, of course. Downloading 30GB of data is also a pain (and frankly, I didn’t expect so many errors – every single track sounded like it was playing underwater at half speed.

So. It turns out the best solution is to make sure you have an inordinate number of machines, and have them synced off the master. I may not be able to rely on one machine, or on a backup hard drive, but I can have music (or any encrypted data) put onto available laptops – and I know that data is good, because I play it from there. This is a looser implementation of jwz’s backup strategy, but it’s good enough for me.

Thinking about this a bit more, it’s surprising that backup technology is as limited as it is. Creating an encrypted bittorrent and sharing it amongst a pool would be an excellent way of ensuring redundancy and error correction (is this what Tor does?), but you may not even need to go that far; every time you do a backup, encrypt it, chop it up into yenc blocks, and dump it onto Usenet. A thousand servers will pass it around and make your data retrievable for all time.

The new company blog is up, and I can finally talk about some of the stuff I’ve been doing for months. Stealth mode, you can keep.

It is out of character for me to gush. But seriously, Topspin is the first company I’ve ever been in where I look forward to the meetings. Things just keep getting better every sprint. Our customers are literally rock stars. I’ve been a huge NIN fan since I was 18, and just getting the opportunity to help out was beyond awesome. It’s in Potero Hill right next to the Whole Foods, I bike to work every day, and every day there’s some DJing tunes through the Airport Express.

Oh, and the Billboard cover was nice.

Okay, I understand why Evernote is so cool now.

Take your digital camera and take photos of all your handwritten notes. Import them to the PC. Rotate them so that they’re the right way up. Drag them into Evernote. Wait for it to finish indexing. Presto. You now have all your notes in digital format no matter which computer you’re on, and you have them indexed and searchable.

Today, my brother and I found a man lying in the street on Van Ness and Market. His eyes were open but he wasn’t moving, and he was clearly unconscious. I checked he had a pulse and was breathing, phoned 911, and tried to figure out what the hell happened.

Pupils weren’t dilated. He had blood in his mouth, and it looked like he’d bitten his tongue. The back of his head was bloody. His heart rate was elevated, and his breathing was ragged – it sounded like his tongue was rattling in the back of his throat every time he breathed. It looked like he’d just been walking down the street, bit his tongue really hard and then fell straight back onto the pavement.

He started to move around a little bit after four minutes, and after five could say ‘police’ and ‘wanna get up’. I didn’t think it was a good idea for him to move until the paramedics came by, but I wasn’t going to hold him down either. It turned out he couldn’t really get up by himself, so we supported him until the paramedics came by.

I’m very glad he didn’t stop breathing. I was not looking forward to giving him the breath of life.

An hour later, we successfully contested a parking ticket before a hearing officer.

A little over a week ago, I made the appropriate sacrifices, made a backup to an external machine, setup a new computer (after figuring out why it wouldn’t POST with a USB keyboard plugged into it) and then transferred the hard drives of my old computer into the new one.

Long story short, the hard drive with all my important data died, and the backup contains some hairball that causes the backup server to freeze when the data is transferred. Oh, and my laptop died as well. The same day.

Ever get the feeling the ground has just crumbled under your feet?

I lost some stuff. Most annoyingly, I lost all my software registration keys. This is good in a way – it means I can try mint.com without having to worry about my old data. And I’ve found that I tend to use online services such as Google Docs and Google Bookmarks over any local solution, so in some cases there is no need to backup at all.

I spent most of the beginning of the week reinstalling my toolkit: Perl, Java, Ruby, RSIGuard, jEdit, etc. And making sure this can never happen again. I have everything local (and this server) backed up onto Amazon S3, using s3sync. I also have a weekly backup of the database onto S3, and I’ve already restored it to check it’s still good.

Of course, it’s going to take me three full days to upload my MP3s onto S3, but I think it’s going to be worth it. And eventually, iTunes will get smart enough that I’ll never need to download them again.

Rewrote the blog in Ruby on Rails. Thanks, Aaron. Comments are now back online, after I had to shut them off due to comment spam. The blog now runs all comments through Akismet.

I don’t really have much to say about Ruby or Rails that hasn’t been said by many people before; I just have my experience of it, which I think is more valuable than the technical minutiae.

I think the biggest strength of Rails (and Ruby for that matter) isn’t the framework. It’s the community.

The framework solves the problem it’s supposed to solve. The community is all aligned around an agile mindset. As such, there’s not the “There’s more than one way to do it” philosophy of Perl. Instead, the philosophy is “How do we make this simple, flexible and easy to build on?” When you’re working along the same assumptions, Rails makes it very, very easy to do what you want. The community supports that, not just with tutorials, but with podcasts, screencasts, and books.

However, if you’re doing something that isn’t in that set of assumptions – if you have legacy data or non-Railsy database schema and associations – you’re off the map. In a way, this was good because I got to figure out how Rails really works, but it was certainly a dunk in cold water after having so much help available.

This has been something I’ve wanted to do for years. Gave up some features for it; no automatic java syntax highlighting, no full text search, and no filtering by categories. But I’ve got the bones, and I can work on the flesh later. Now, sleep.

A year and a half ago, I read Hardcore Zen.

I had some understanding of Zen (or so I thought) from reading Hofstader and Pirsig, a short reading through Watts. To me, Zen was about the destruction of ideas; an deconstructionist, almost dada-ist religion where thoughts were meaningless, desire was shunned and even the religion itself “could only be learned by forgetting it.” I’d hear stories of people going weeks without speaking in a retreat, trying to answer unanswerable questions, staring into a candle-flame, and trying to eliminate their very idea of self. Didn’t sound very fun, or practical, or even useful.

I’m not religious. I barely care enough to be an atheist. So why was I reading the book?

Mostly because of the cover. “Question authority. Question society. Question reality. Question yourself.” And then, in a smaller font: “This is Zen for people who don’t give a rat’s ass about Zen.”

Or maybe it was the inside quote: “I have no time for lies or fantasy and neither should you. Enjoy or die.” – Johnny Rotten.

Whoever this guy was, he was not a hippie. And when I started to read, the author took the attitude I had and neatly reversed it; embracing sceptical thinking, pointing out the holes in his experience and his distinct lack of Zen Master enlightenment. And his refreshingly blunt, clear opinions on organized religion, including Zen masters and koans. I liked him. Whoever he was, he thought like I did. I’ve never liked being given a pat answer or being told “Don’t look too deep. Don’t ask too many hard questions. Don’t ask why.”

So that’s why I read Hardcore Zen. And the interesting thing about having read Hardcore Zen is that it didn’t tell me anything I didn’t already know. I’ve had people return it, saying that it just says what’s obvious anyway. And that’s why I like it so much. Zen, at least in some minds, is not about transcendence or enlightenment, or nirvana. It’s about pointing out what’s in front of you, again and again. Not to take it for granted. Not to assume it. Zen isn’t about staring off into the distance. Zen is about paying attention.

As a consequence, Zen has many things to say about how the brain works. Cognitive Science can tell you how we reshape the memories of the past to fit how we see the present. How we then dream about the future instead of seeing the present as it is. That how we see ourselves acting in theory is not actually how we act in practice. Zen has been saying this for thousands of years.

Zen holds a light up to how inconsistent human beings are, and how little we understand about ourselves. And Warner goes into detail about how this applies in practice, to him in particular. A good portion of the book details just how much damage “paying attention” can do to your carefully groomed ego. Warner starts off playing in a punk band, and realizes how much punks depend on authority (police) to allow them to rebel against it. He sees punk turn from being a way to express individuality to a set conformity with its own dress code. He talks about people who spend day after day trying to solve all the world’s ills… but then come home and treat themselves and their friends like crap. And even as Warner tries to dismantle the concept of authority, he has to struggle with the possibility of being an authority figure in his own right; a punk rock guitarist turned Zen Master.

But more than that, Warner realizes just how much of an asshole he can be. He has to face up to everything he’s done, and everything he’s still doing. He starts to see his emotions and desires more clearly, and they’re not always fun to see. He quotes his brother-in-law as saying “It’s impossible not to feel angry when you are facing the gale-force winds of your emotions whipping across your body.” But he says another idea of anger is “sitting in the bathtub frantically thrashing around and throwing handfuls of water into the air while simultaneously wondering why the hell your head and face keep getting wet.” Being angry is intoxicating, but who and what is causing this anger? It’s not coming from the situation. It’s the reaction to the situation that is anger. It’s coming from inside.

And finally, he talks about life as it is. That reality is ultimately waking up in the morning, pulling your sorry ass out of bed, and trying to figure out which limbs do what when you’re brushing your teeth and getting dressed for work. That enlightenment is as much about being okay with being stuck in traffic as it is about seeing the face of God.

Hardcore Zen doesn’t tell you anything you don’t already know. But to me, at least, it reminds me that I know it. And maybe, just maybe, with sustained effort and careful attention, I can be less of an asshole.