Terse Systems

I moved all my note taking into Evernote, and I’m using it for everything – diaries, documents, clippings, etc. Much simpler and more powerful than using three different systems.

I had to write some Ruby scripts to get everything imported: they are available at http://github.com/wsargent/evernote-import. Imports from plain text files and from Memiary XML format.

I started a job at Grockit a couple of weeks ago. This has been my first chance to see what an agile shop looks like, and it’s an eye opener.

You show up at 9:15 am every morning. There’s a daily stand up meeting (truly stand up, everyone is around a circle), and then breakfast, which is served in house; people sit down and chat about stuff they’re doing, but it’s less formal than the standup is. (I don’t have a good handle on how retrospectives or user story planning work, but from what I can tell there’s an overall direction that is set during releases, and then the business owner participates in standups to ensures that everyone’s on the right page. Spot demos also appear to be frequent, although I don’t know how much of this is a conscious strategy.)

Following the standup, there’s a secondary huddle where engineers look at the list of user stories open, and pick off the stories that they want to tackle. Pairing is done by the tech lead rather than the individual engineers, and there’s an emphasis on switching pairs around. Tasks get written down on the whiteboard, photos with names are assigned to them for the day, and then it’s time to find a free workstation and move in.

Pair programming is the natural order of things at Grockit. The workstations (Mac Pros with 30 inch monitors) are all set up in exactly the same way (IntelliJ IDEA with Ruby Plugin, QuickSilver) and have two keyboards and mice attached. There are two engineers per desk. The first thing you do is bring up Pivotal Tracker, find the user story that you’re working on, and then mark it as “In Progress.” The user stories are typically single sentences such as “Put the top players per week on this page (refer to mockup)” and have points assigned to them by the engineering team beforehand. Then you discuss a couple of ideas, make sure you know what the state of play is, and start coding.

Only one person codes at a time. At this stage, I don’t know enough about the application to be effective on my own, so I’m mostly monitoring and following what my partner is doing. It’s an involving process to watch someone else code; you’re not just looking at the physical processes involved, but also the intent – usually I can catch bugs a couple of seconds after they’ve been typed, although I usually give feedback after we’ve reached a pause and I can be sure I’m not interrupting my partner’s flow. Actual disagreements are rare, partly because no-one is attached to an idea yet, partly because everyone is “on the stage” – you know that whatever you’re typing is going to be peer reviewed almost immediately, and it’s important to write something that’s going to be clear and acceptable to the partner at the time. I think that mixing pairs every day is also a factor here. Any time someone’s had a problem with code, it’s been new code that is unfamiliar to one person and not the other, so there’s a fair amount of learning that goes on each day.

It’s also draining. Reading code is almost as intensive as writing code, and there’s a ton of moving pieces, plus a couple of custom built frameworks that I’m unfamiliar with. More often than not, when someone asks me to take over or asks for suggestions, I have the odd experience of saying to refactor code using features that I’m not consciously aware of. I can say what to do at the time, and how to rework it, but when I get up from the computer, I have no clue what I just said or even what it meant. It’s a nice ability to have, but it’d be nicer if I actually knew how it worked.

The other reason it’s draining is because either you’re typing or you’re reading. There’s no downtime or distractions while you’re pairing. The “why” of this is interesting.

Grockit has email, but it’s mostly used for company broadcasts, housekeeping and customer feedback. It is not used intra-company, and this is a deliberate choice. Nearly everyone is in the same room. If you want to schedule something or start a group discussion, you mention it at startup and pull everyone in later. Likewise, there’s no instant messaging. No IRC channel. Nothing. So, no distractions. This took a while to get used to – I had to start scheduling and communicating with friends in one large block once I got home, and some people were even worried that I’d vanished completely.

So. You work until 1:30 pm, and then the lunch bell rings. Lunch is served in house, and is delicious organic vegetarian. After lunch, there’s a break where some people check their email on personal laptops, or play Rock Band, or run errands. People usually head back into the office at 2:30 pm, and then it’s pairing time again.

At this point, you’ve got most of the code written and you’re reworking and adding new tests to verify that what you’ve got actually works and that you can check it in. (Or the TeamCity build has broken, which results in immediate and productive feedback from the team.)

The test suite at Grockit is beautiful. Most of it is integration tests, with some mocks in place where they want to see specific exceptions. It makes heavy use of nested describe and contexts, and even has a very elegant use of should_behaves_like in the order processing logic that would have otherwise taken some complex metaprogramming of specs to do. There’s a strong bias to make tests useful and not repeat themselves, so tests usually only happen at one level. If the code can only be adequately tested using Selenium, for example, then there’s no point in writing unit tests for each method that would also test the happy flow.

Having said that, most of the tests involve business logic or concepts that I have to ignore because they’re not related to the problem at hand. Having to write or refactor tests while simultaneously not looking “over there” is mentally taxing, and then there’s the extra juggling involved with the mocking framework, selenium, and the assumptions made in the code itself. This is usually the point in time where my brain gets full. I have to pick up the threads of where we left off before lunch, so either I have to rummage around before I feel I know what’s happening again, or I just get a blinding headache trying to jump several steps ahead. Apparently, this is a common phenomenon during the first three weeks of coding, so I’m hoping it gets better soon.

At some point, the main user story is finished, tested, demoed, and handed it off to be accepted or rejected. Typically we get this wrapped up by 4:30 pm, and then pick off a couple of smaller user stories that involve front end work if they’re not too complex. This goes on until 6 pm, at which point the work day is over and everyone goes home.

And I mean, everyone goes home. There was one user story that I thought we could do, but was told “We can’t finish that; we only have 10 minutes left.” That’s discipline.

To recap: you show up at 9:15 am, do good solid work for eight hours, and then go home. You do not take your work home with you, or work on the weekends. User stories and estimates are scheduled so that all the work you can do is done in eight hours. Velocity is automatically tracked and measured using Pivotal Tracker on an eight hour basis. The hours you work is eight, and that’s the time you need to do a good job, with covering tests and code that checks the edge cases.

I like it. It’s not easy, but it’s so, so worth it to me.

I know enough about EasyMock now to feel very comfortable with it. I’ve even got an acronym for writing unit tests with EasyMock – SEREVA.

This breaks down to the following stages:

Setup the object under test with any required mocks.
Expect some methods on the mock to be called.
Replay the mock once all the expectations have been made.
Execute the object’s method when under test.
Verify the mock has had all its expectations met.
Assert the output of the method.

This basic framework takes care of most of the grunt unit tests.

But there’s more to mocking than that. There’s one particular gotcha I know about with EasyMock, and then there’s the horrible, nasty, no-good edge cases that you run into. Let’s start off.

Custom Argument Matching with Multiple Parameters

Let’s start off with the gotcha first. Say you’ve got a method like the following:

List findCarsByYearAndColor(Integer year, String color);

Say that you want to match year, but not color. You can’t say:

expect(carService.findCarsByYearAndColor(year, anyObject()).andReturn(null);

If you want to apply a custom argument matcher to either argument, you must apply it to both. So instead, you need to do this:

expect(carService.findCarsByYearAndColor(same(year), anyObject()).andReturn(null);

This is why same() exists.

Static and Final Methods

EasyMock does not allow you to mock static or final methods. EasyMock ClassExtension does not allow you to mock private methods or static classes. You can argue that such things should not exist, but we live an imperfect world where people don’t always think of our needs.

This means that dealing with final classes like java.net.URL or java.io.File are fiddly. But not impossible. There’s more than one way to skin this particular cat.

The by hand way is to use reflection: “final” is not final anymore. Heinz demonstrates how you can change final fields through several different versions of the JVM. This is a somewhat disturbing read, as it plays merry hell with my conception of what should and shouldn’t be possible in Java. But it’s there if you need it.

The snazzy way is to use JMockit, which is a small toolkit specifically designed to subvert the JVM. jMockit depends on the java.lang.instrument library, which is only available in JDK 1.5, but it provides functionality you can’t find anywhere else.

The alternative method is to say “if your code is hard to test, then your code is badly written.” While there may be a correlation, I think that what is ‘badly written code’ can only really be accurately be judged by a human being.

Private, Default or Protected Methods

jMockit is all over this.

Native methods

Don’t know.

Static or Final Classes

This is handled as above by reflection, or by jMockit.

Inner Classes

Don’t know. I suspect that anonymous inner classes are very tough. I believe that static inner classes should be accessible through jMockit, but haven’t checked.

Constructors

EasyMock 2.3 allows you to define a partial mock that will call a specified constructor. That’s good if you have something that needs to be passed into the object under test. So you can do:

public static abstract class A {
    public String s;

    protected A(String s) {
        this.s = s;
    }

    protected abstract int someMethod();
}

public void testFoo()
{
    Constructor cstr = A.class.getDeclaredConstructor(String.class);
    ConstructorArgs constructorArgs = new ConstructorArgs(cstr, "test");
    A a = createMock(A.class, constructorArgs, new Method[0]);
      
    foo.setA(a);
    expect(a.someMethod()).andReturn(0);
    replay(a);
    foo.execute();
}

If the object under testing needs to have a constructor mocked out, i.e.

public class Foo
{
    public Foo() { new File("c:\\boot.ini").delete() }

    public void methodToTest() { }
}

Then jMockit will replace non-default constructors. I don’t know of anything that will replace a default constructor.

Enums

As far as I can tell, there is no facility in EasyMock, EasyMock ClassExtension to mock out enum logic, as it is defined as final by the JVM. jMockit should be used here. And that’s it. jMockit looks like it’s branched out into integration tests as well, notably Hibernate. I know that Unitils is a competitor here, but haven’t played with either of them that much.

I’ve stopped using Ecco Pro and Shadowplan, and I’m now using Toodledo.

I looked some other online systems, but Toodledo was the only one that had both state (well, folders) AND context. Since Toodledo is a web based task system was that I’d be unable to add to it when I wasn’t in front of a computer. However, with the integrations to Twitter and Jott, I can SMS or even phone up and leave messages.

Once I found out there was an API as well, it was really only a matter of time before I’d find myself writing an adapter for it. And a command line client, in case you want to access it from a shell.

So here’s my first Ruby gem.

http://toodledo.rubyforge.org

If you’re on a mac, you should be able to type ‘sudo gem install toodledo’ and then ‘toodledo setup’ and that should walk you through most of it.

Flexmock does mocking of ActiveRecord objects if you ask it nicely, but it doesn’t say much about associations.

It turns out that ActiveRecord does the same thing that Hibernate does and uses an internal proxy. So if you want a post containing a single image as an association, do this:

# Create a single image.
image = flexmock(:model, Image) { |mock|
  mock.should_receive(:url).and_return('/post/10/blah.jpg')
  mock.should_receive(:name).and_return('Blah Description')      
}

# It's an association proxy pretending to be an array.
images = flexmock(ActiveRecord::Associations::HasManyAssociation) { |mock|
  mock.should_receive(:find_by_name).and_return(image)
}

# The post itself.
post = flexmock(:model, Post) { |mock|
  mock.should_receive(:object_id).and_return('10')
  mock.should_receive(:images).and_return(images)
}

This is a Law of Demeter violation, so the better solution may be to have a get_images method.

More info here:

Misunderstanding the Law of Demeter Demeter’s Revenger Loose Coupling takes Tight Logic

I think I understand why the unit tests in Rails were confusing me. I had assuming that unit test fixtures ran YAML into some kind of in-memory activerecord object. But in reality, fixtures will put data into a test database and then unit tests will interact with it.

As far as I understand it:

1) An integration test is one where the test code goes outside the realm of the class and has a dependency on an external entity.
2) Databases are external entities.
3) Fixtures will write to the test database.

Therefore:

Any test containing a fixture is an integration test, even if it’s in test/unit and extends Test::Unit::TestCase.

Furthermore:

Unit tests are tests that can be run in isolation, in any order, without any external dependencies.

Whew. Had to get that out of my system.

Not that I don’t understand the rationale; or that using fixtures could be extremely handy. But a spade is a spade.

As usual, as soon as I realized the underlying issue, there was no shortage of help, Jay Fields being the most helpful.

A problem I have learning Ruby on Rails is that there is such an active community base that I can’t simply pick up what I need to know without getting sucked in.

For example. I want more integration tests. So I need fixtures to plop data into the test database. Fixtures are written in YAML.

Well, all my data’s in the development database. I suppose I can hunt through Google. Ah. I can export it to YAML.

And there’s a plugin that will allow YAML to be created with fewer foreign keys. Okay. I can use that then.

And the fixtures aren’t run in dependent order. Okay.

Oh, there’s another way to do do that. Okay. I’ll use that instead.

And Rails doesn’t tear down fixture data. Okay…

By the time I’ve read through everything in Google, I’ve forgotten what the heck I was trying to do in the first place. It doesn’t help that some of the blog posts date back to the point where it’s part of the core Rails package now – I have no ability to distinguish between old and new.

So. Start at the beginning again. I want more integration tests. Step 1: Ignore Google.

Finally scratched a long standing itch today and wrote something to break up Log4J configuration and enable parameter substitution. You can download it here: layered-configurator.zip

LayeredConfigurator will go through a properties files with references to DOM or Property configurator references (files or URLs), and will call the appropriate configurator in the order they are defined.

This doesn’t mean much in theory, so let’s explain why this package exists, and give an example.

Log4J lacks a way to spread configuration over several files. You have one configuration file, either log4j.properties or log4j.xml, and that’s it.

In projects where you have several environments to keep track of, you may want to have some loggers set to DEBUG in the development environment, but set to WARN on the production environment. However, you usually want to keep all the appenders and logging infrastructure the same.

Because Log4J defines appenders and loggers in the same file, you either have to define several almost identical files with the appropriate changes for the environment, or you have to have an Ant script that goes through and replaces tokens for the appropriate environment.

The file syntax for the configurators is simple. Here’s an example log4j-config.properties file:

# The complicated appenders are rendered in XML.
base-appenders.xml

# The logging levels can be defined in properties, and can use system properties
# as parameters. (in this case we assume -Dmy.environment=dev is defined)
${my.environment}-loggers.properties

The class is called LayeredConfigurator because it’s meant to work in layers. The configurators are not reset, so all the settings from the previous configurator will still apply unless you override them.

When you define -Dmy.environment=dev, then your development settings will be loaded. When you define -Dmy.environment=prod, then your production settings will be loaded. Either way, your binary distribution is exactly the same, with only the environment specific properties

To enable this class, you must start the JVM with the following system properties:

 -Dlog4j.configuratorClass=com.tersesystems.log4j.LayeredConfigurator
 -Dlog4j.configuration=log4j-config.properties

And that’s it. Pretty simple solution for something that’s bugged me for at least three years now.

Right now, I know three people who are burnt out. Almost everyone I’ve worked with has worked to the point of burnout, and I’ve done it myself, both at home and at work.

Burnout is not unusual. It’s a physical exhaustion brought on by protracted, unrelieved stress. If the human body is under stress for long periods of time, every day, then eventually the body will break down, and take the mind with it. The precise mechanism is not known; it’s assumed to be a breakdown in the endocrine system that manages stress.

What distinguishes burnout from just plain stress is the inability to care; not because you are bored or apathetic, but because caring requires an insurmountable effort. People who are burnt out are disengaged. Going through the motions. Blunted. And unable to relax or breathe; any stress or unexpected activity will set them twitching. Because it’s brought on by prolonged stress over a period of weeks or months, it’s not an exhaustion that is cured by a couple of nights of good sleep. Because it’s physical, it’s not something that focus or willpower will fix. Trying to work through burnout will only make it worse. And denying burnout exists because you don’t have room in it for your life… well. I’ll tell you about a man I know who refused to admit to physical limitations. He worked until he collapsed on the floor of the server room and had to be taken to a hospital bed.

Of course, if you’re burnt out already, you are not in a position to fully appreciate it. In fact, one of the ironies in burnout is that the more burnt you are, the less able you are to effectively manage yourself and your stress levels.

The interesting thing about burnout is that it’s not brought on by prolonged periods of work. Work in itself is fine – it’s stress which is the problem. And what causes stress is not work.

I remember a study that measured the amount of stress that an executive and a secretary went through in a day by measuring their cortisol levels. What they found was that even though the executive was making decisions and working moment to moment more than the secretary was, the secretary was under more stress. Why? Because although the executive was doing more, he had more control over his decisions and his environment. The secretary was interrupted on a regular basis, was multitasking far more than the executive, and had less control over what was happening at work. The same results have been found in lab rats: what gives rats the most stress is not receiving electric shocks, but the uncertainty of possibly being given an electric shock. What causes stress is lack of control, and more basically a lack of respect.

That’s why Christina Maslach defines burnout as:

“The index of the dislocation between what people are and what they have to do. It represents an erosion in values, dignity, spirit, and will—an erosion of the human soul. It is a malady that spreads gradually and continuously over time, putting people into a downward spiral from which it’s hard to recover.”

In some ways, burnout is the end result of a reaction to stress. When people used to doing a good job feel out of control or lacking in respect, then they’ll work harder or try to take on their environment. They might feel in control or respected while they are working long hours or dealing with a stressful situation, but that will only last until they use up their reserves.

So what do you do about burnout?

1) Just about every article I’ve read says that you should consider quitting your job. I understand if this isn’t an option.

2) Learn how to say no at work. Let go of things that aren’t your responsibility.

3) Go home. It’s a marathon, not a sprint. You will only be able to do so much work in a day, no matter how much work you put in.

4) Relax. Accept that there will be time when you get home and don’t feel like doing anything. Lie there and practice blinking.

5) Meditate. Meditation’s a little strange at first, but studies prove it has a positive impact on stress and mood. Here’s how it works: you sit down, crosslegged, on a cushion. You set a timer for ten minutes. You count your breath on every exhale, from one to four. When you pass four, you count from one again. And you don’t try to think about anything else. You may notice thoughts passing through your brain, but you focus on counting breaths. For ten minutes.

There’s more to it than that, of course. But that’s really it – the idea is to be able to focus on one thing without getting distracted. (Or thinking of a pink elephant. Oops.)

5) Delegate. Whil Hentzen recommends getting an administrative assistant. He goes into it in some detail, but the basic idea is that there are any number of small, non-technical tasks in the course of the day that could be passed off to someone specifically for doing those jobs. I’ve often thought this would be a great idea; my parents work in film and video work, and it’s common practice to have someone who can fill in when needed.

6) Vacation. I’ve heard various people refer to this as “unplugging.” I’m not sure I buy this route; it seems to be a short term solution that doesn’t really address the problem.

The best thing you can do is be aware that it is real, it does exist, and that, left untreated, it can cause problems that are not easily solved. The New York Post has an excellent article on burnout, although it tends to meander towards the end.There are a couple of articles that really nail the experience of burnout:It’s Not Just You: Exploring the causes of worker burnout (which has a link to What Makes the Job Tough? The Influence of Organizational Respect on Burnout in Human Services, a PDF that is the basis of the article) and Job Burnout. I highly recommend you read them.

I’ve been going to AYE for years now. The last time I went, Charles Adams asked me “Will, every time you come here, you seem so frustrated.” I don’t disagree. For me, AYE is frustrating for many different reasons. But part of it is just that AYE is by its nature, ahead of the curve.

People in AYE take for granted concepts and methodologies which have not filtered out into the wider world. And much of the discussion is not about getting the framework up. It’s about incremental improvements on what is already known.

Over several years, I’ve made progressively closer steps to agile. I’ve written unit tests, mock objects and unit testing frameworks. I’ve written integration tests and built integration test frameworks, and can talk about data driven frameworks. I’ve created repeatable, automated build and deployment processes and have continuous integration with built in code analysis and code coverage.

But I’ve never worked in an agile project using iterations, or a company comfortable with agile processes. The closest I’ve ever come has been the promise of a discussion where agile would be discussed as a serious option. Someday.

To me, AYE, or BayXP, or any number of blogs, sound like discussions on the best way to fly using the wings on your back. If you have wings, flying is easy. If you don’t have wings… how do you get them?

I’m not trying to be funny here. I think that agile processes are an honest attempt to redefine the practice of software engineering. But there is a divide between the have and have-nots. And trying to learn what I can in a field in which I can’t practice it is… well… it’s…

It’s frustrating.