Terse Systems

As of July 25th, there’s a new lighting configurator on Restoration Hardware, using Macromedia Flex. This was a multi-displinary effort on behalf of many people. The business team at Restoration Hardware came up with the initial idea in March; they wanted to “mix-and-match” different shades with different bases.

We could have done it in DHTML, or using a browser-based plugin such as a Java applet or Flash. Given the need for a smooth customer experience and reliance on development tools, no-one was really keen to use DHTML. Java applets relied too much on deep GUI skills (disregarding things like thinlets), so that left Flash. Flash seemed like a natural fit – it’s well known in the designer community, there’s a good infrastructure of tools and support, and it is literally built to look nice.

But Flash had a couple of problems. It was built around movies, and we wanted to write a GUI. So we looked at Flex.

I took a long hard look at Flex before I decided it was a good thing. I wanted to make sure that it was a real application and not a marketing gimmick. I didn’t trust the blogs to come up with the right answers, so I poked at it myself until I was fairly sure that it didn’t stress the server, didn’t stress the client, provided features in the context of a conceptual framework and had all the obvious bugs worked out from the first release. I looked at Laszlo and another tool, but it was obvious that they didn’t have the depth that Flex did.

For Java people, Flex has a number of similarities to javascript, J2EE and Swing models. Like JSP, MXML is an easy to use format that is converted on the fly to a lower level programming language (Actionscript), and then compiled to bytecode. In MXML, like Javascript, a component can fire off events and methods are called in response to those events. Best of all, Flex contains the idea of a dataProvider which provides the classic Swing separation of model and view.

In short, Flex is not a completely new paradigm. It’s a synthesis of already known ideas. I consider this a strength. It is reassuring to see something that you’ve seen before, and it’s good to know architects of this system use the same mental shorthand. There are fewer surprises. So we picked Flex, and then worked on making it do what we wanted it to do.

The short version of the story is that we all worked very very hard, the project was a success, and the lighting configurator was Macromedia’s Site of the Day. The long version will probably turn up sooner or later, but here’s the technical skinny before I get too familiar with Flex and start drinking the kool-aid.

I worked through a number of prototypes during the design phase to be sure that I had a good understanding of the relationship between MXML and Actionscript.

For the most part, I didn’t have a problem with anything in the actual framework itself. The new stuff is good. My biggest issues in Flex were to do with Actionscript and the Flash MX framework underlying Flex. Bear in mind that I approached Flex as a complete newbie, with a good theoretical understanding of Javascript and the iterationtwo and Moock books. As a result, some shared understanding was missing.

My biggest beef has to be with Actionscript. I realize that I’m approaching this from an enterprise Java perspective, but I can see good choices made on top of bad ones on top of good ones. It feels like Perl worked over by the Ada committee and then handed off to Gosling. For example, I understand the need for strong typing, and the rationale for mixing strong typing with weak. But Actionscript specifies a type using postfix notation:

var s:String = "foo";

I can see absolutely no advantage to this notation. Except for maybe copying it into UML diagrams.

There are exceptions in Actionscript. But they’re all unchecked. And they’re inconsistently used. This is almost worse than no exceptions at all.

There’s a difference between null and undefined. I believe this is from Javascript, but I dislike the idea that there is an undefined value at all. Wasn’t null magic enough?

Every untyped object in Actionscript is an associative array. This means that you can say:

var o = new Object();
o.name = "test";
o.age = 25;

And it gets stored with the keys of “name” and “age”, respectively.

This is good in one way, because it means that with a single debug call, you can get to all the properties:

public function toString(o:Object) : String
{


var msg:String = "Object:";
for(var prop in o) {
  msg += " ";
  msg += (prop + " = " + o[prop]);
}   
return msg;


} 

The bad news is that if you get an event, you have to poke around with the above method before you know what it is and what you’re supposed to do about it.

if an array is passed through as a parameter and there’s only one element in the array, then the parameter appears as a single element instead of an array containing a single element. Why? It’s a “feature.” This is a feature the same way it’s a feature to have a single tube for both breathing and eating: it sounds good right up until you breathe a peanut and choke to death. Seriously, it took me almost a full day to figure this one out. What happens when the array is empty? I never tried. It probably represents it as null.

The final, inexcusable, really bad thing in Actionscript: no NullPointerException. If you call a method on a null object, the result is null. You have no way of distinguishing except by an equality test. This would be acceptable if there were a lint for Actionscript, but from what I can tell this is just The Way Things Are. There’s not even a ‘use strict’ option for us language fascists.

Then, once you get past the Actionscript design decisions, you run into issues with the Flash browser. The documentation says that Flex supports CSS. This is true to an extent, but it is limited to what the Flash plugin will support. CSS1 supports the text-decoration attribute, but you cannot use the strikethrough option because the client can’t display it. Macromedia support supplied a custom component that overrode the component draw method with a strikethrough as a workaround, and we were good to go with an hour.

We also ran into a bug with mx:Text not displaying HTML correctly. Resource Interactive found that htmlText cannot be used with embedded fonts (we were using Minion) and the workaround was to use a font tag:

<font style="Minion">{ my.text }</font>

which was inelegant, but it worked. CSS divs and styles do nothing with embedded fonts. It has to be old school.

The only issue I ran into directly with Flex was the difference between Flex 1.0 and Flex 1.5. The iterationtwo book is good, but it provides code examples which are already out of date. That is, the book suggests

public var _remoteClass = "com.rh.flexlighting.proxy.ShadeSKU";

when it should be using:

public static var regClass = Object.registerClass("com.rh.flexlighting.proxy.ShadeSKU",


  com.restorationhardware.lightingapp.vo.ShadeSku);

Even with the minor version drift, the iterationtwo book is mandatory. You simply will not get a good idea of Flex without it – the Macromedia documentation is good, but far too linear and abstract to get a handle on. Iterationtwo has worked on this stuff in real-world projects, and they understand very clearly what questions developers ask and in what order.

That said, Flex is great. It’s easy to use. The communication between client and server is done with POJOs, is easier than RMI, can be debugged over the wire using Flex Builder, and can be configured through a single flex-config.xml file. There’s a <production-mode> tag which will instantly flip off all the debugging flags and start aggressively caching and optimizing the SWF files for you. The entire Flex application is bundled up as a single J2EE web application, and behaves appropriately. We needed to make a couple of changes to our build system to deploy flex-config files for different environments, but it worked on the second try with no surprises.

I think I’ve written about this before, but really Flex is overdue. Something has to replace HTML (or evolve it), and I don’t see another decent competitor right now: Avalon is vaporware, Laszlo doesn’t have the infrastructure, Mozilla has a bad habit of changing APIs faster than the books can come out, and everyone gave up on Java applets in 1999 and are shy about going back.

Flex doesn’t make any shortcuts. It’s an extensible, complete XUL framework that insulates you from the Flash underpinnings without cutting off any functionality or future expansion. There’s a very strong feeling from the code that the Flex architect want to get this right, and for the most part they’ve pulled it off. It’s actually fun to use.

For the last project I had, we depended on data from the business users. There wasn’t very much data, but it was very variable and being revised in an excel spreadsheet on a daily basis. Not only that, but it was changing in a structural basis as well – new rows and columns would be added, links between workbooks were used. We couldn’t use a snapshot, because the snapshot would be obsolete in short order, and the business users couldn’t use a database that would restrict them.

In the past, the business users had transcribed the excel data that they had one row at a time into the database through a web or Swing UI. We needed something faster this time around. We needed a way to import the excel spreadsheet directly.

This idea had appeal on an intuitive level. The business users could work in the structure they were most experienced at, and we could get fresh data almost as soon as they’d revised it. We knew that there were some data validation issues where values had to be restricted to a list, but we thought that Excel’s validation features would cope with that.

I looked into reading Excel in its XML dialect, but quickly discarded the idea after eyeballing the XML Excel generated itself. Instead, I looked for a library which would do the work of conversion for me, and present an easy to use API.

There was only one freely-available Java library that handled Excel spreadsheets, and that was POI. POI is a library written specifically to handle Microsoft Office formats, and their solution for Excel was called HSSF (Horrible SpreadSheet Format).

On the database side of things, we had an advantage in that we were using ATG’s persistence solution, the SQL repository (or Data Anywhere Architecture, as marketing calls it). Because the SQL repository can be fed data imported through XML, we could convert the Excel spreadsheet to a file in XML format, and then run startSQLRepository to import data.

The documentation for HSSF was eclectic, but servicable. Once I’d found the document to read an Excel document, I could iterate through the rows, and call row.getCellAt(0) to get the first cell in the row.

Once I’d done this, I copied the data from the row to a JavaBean object, passed a list of those object to the export class, and generated XML from them.

There were a number of issues that popped up in implementing this solution. The first one was that POI is abandonware. The last update was from 2004. Using Excel XP with some features (validation of specific cells, for example) cause NullPointerExceptions when it tries to parse it. The solution here was to not use those features.

Another issue was that the Excel data was linked between different workbooks, but was not linked tightly together. Because all the fields were text, any text that was not specifically validated had to match between workbooks exactly. Identifying data mismatches between workbooks was a time consuming process, and I didn’t think of a good way to automate it.

Finally, as new data was added, the columns would be moved around or added to as the business users tweaked it. Sometimes I would be notified of this change, and sometimes I would find out by having the script fail.

Importing data directly from an Excel spreadsheet is a good solution in many circumstances. However, it’s probably a good idea to have a ‘business’ that can be quickly modified, and a ‘data’ spreadsheet with immutable columns specifically for data export. Given a static schema, it’s easy to use and extend, and is much easier to modify than tweaking XML directly.

Okay. I bought Enterprise Architect because it made UML diagrams easy. Or at least easier. (Visual Thought is nice, but don’t try doing interaction diagrams with it.)

Enterprise Architect does UML very well. The UI is slick and intuitive. But it also has a bunch of useful features that weren’t obvious at first glance. In fact, it took a good deal of playing around with EA before I even understood everything it did.

Forward and reverse engineering works out of the box, with no hacks necessary. I mock up a class design, generate the classes from UML, then go in and write all the stubs in Eclipse. Then I go back to Enterprise Architect, synchronize the classes with the codebase and I’m done. If you have any dependencies on external classes (and who doesn’t) then you can import classes directly from the JAR file.

It has integrated database support. I can create table diagrams with foreign key references and autogenerate the DDL. I can even reverse engineer tables by hooking Enterprise Architect to an ODBC data source. This makes DBAs happy.

It has MDA transforms. This feature sounds pretty but useless, until given an example: autogeneration of getters and setters given the attributes of a class. Create DDL with a handful of attributes.

But this is not even getting to the one feature that truly makes Enterprise Architect a killer app: autogenerated documentation.

You generate documentation by pressing F8 on a package. There are actually two document generators – one of them referred to as “legacy.” Ignore that one. You want the ”New RTF style template editor” which actually works right. This will create RTF documents containing the diagrams, the notes for each element, and any scribbles that get attached to the connection. (The example text in the base templates is very obviously fake, so you have to click “Manage Templates” and create your own with company headers and suitably generic intros.) It’s massively useful, even in its raw form.

If you don’t want to use the “raw” version, you can create your own Word document and link to the diagrams in the RTF document. This solves two problems: the UML diagrams in the tech design document never get out of date, and you don’t have to have masses of JPG files littering your directory.

Enterprise Architect also seems to have features for requirements and testing, although I haven’t been able to figure how useful those are yet. In addition, there are a number of features I would like to try because I can’t figure out how the heck they work. Part of the problem is that I am a UML newbie and haven’t figured out what a UML profile or a UML pattern is. But part of the problem here is the interface.

Generally speaking, the nicer the feature, the more kludgy the interface. EA has more dialog boxes and context-free buttons than it knows what to do with, which is to be expected in a technically-oriented tool but requires (REQUIRES) that I click help button on every single dialog box so I can figure out what the heck it does. It gets worse the more esoteric you get; for example, it’s possible to create your own language to generate from UML diagrams, except that it’s just not worth it. (And Actionscript support is coming in the next version anyway.)

Still, it’s really, really good. The central UI (as long as I avoid any modelling features) is seamless enough that I can actually develop and diagram at the same time, tweaking one or the other to stay in sync without losing the thread of what I’m doing. There’s a 30 day trial, and it’s damn cheap for what it offers. Go try it.

IF you ever get one of these exceptions AND you haven’t been playing around with the ACC AND you have more than one server running against the database, then check the cache-mode of the item-descriptor. There’s a good chance it has been left as simple when it should be locked, distributed or disabled.

Okay. I asked earlier about the benefits of a strongly typed manager class. The general consensus was that:

1) Weakly typed repository items (where you have to call getPropertyValue(“price”) and cast to the appropriate type) suck.

2) Strongly typed objects (where there’s something like getPrice() that has more logic and intelligence associated with it) do not suck, but the code involved with managing complex objects can get so insanely complicated that the possiblity of bugs is actually higher than just using repository items.

3) In addition, using strong type can involve a lot of boring manual typing unless a tool is used to autogenerate the type wrappers.

From this input, I can think of some good use cases for strongly typed patterns:

1) Where the expectations for the manager are well defined. In this circumstance, there is only one thing that the manager can do and it is not expected to do anything else. Relationships between items are not expected to change and the interface is effectively frozen.

2) The manager is not frozen and may be extended, but the interface is controlled by an in-house team of engineers. In this case, the manager is not bound to an external framework, and the engineers can modify the manager to fit their needs exactly.

3) The manager is expected to be extended, and will be used as part of an external framework. However, the engineers designing the manager are all geniuses, have designed it to be easy to extend, and can cover every possible use case.

However, anything that can ease the pain of creating type-safe wrappers is a plus, so I’ve poked through the “repository-to-java” code and updated the Car project to use it. The ant build file will run out of the box and I’ve included comments on what worked and what didn’t.

Maintaining a seperation of concerns between the view and the manager is still important to me, and so I’ve kept the Car interface around even though strictly speaking I could use the autogenerated CarFacade interface. It’s easy enough to copy the method definitions from one class to the other.

The only thing that bugs me about the current solution is the EJB implementation methods and exceptions. I really only want to see CarExceptions, and I don’t like it when implementation details leak.

Anyway. The autogenerated code works perfectly, even though there’s a few bits I still don’t understand (like the role of CarFacadeHome and the method delegation through the wrappers.) I’ve marked those bits out so people can possibly use them for something. The configuration.xml file is complex, but the DTD goes a long way to making it understandable. See what you think.

Well, I was going to write a big response to Repository Patterns. Somewhere along the line, it turned into its own project. Read it here.

It’s pretty good. It’s got the main concepts down, plus it’s got OperationDemarcation. OperationDemarcation is the new bacon. I wish I’d thought of it years ago. Plus a few utility classes like UserMessages, BeanUtils, and real unit tests instead of the toy examples on ATG mockobjects.

I would like to give a shout out to Enterprise Architect. EA has done the impossible: it makes creating UML diagrams fun. It’s also at least 25 times cheaper than Together/J or Rational Rose. And it doesn’t crash like Rational Rose either.

The javadoc comments provide more detail about the purpose of each class. Note that the benefit of the manager class is not that you can replace the repository entirely, but that you have the flexibility to decide when and how to use it. You might want to cache some data, make some remote calls somewhere else, consult an authentication service… who knows. Putting a layer in from the beginning ensures that any changes to the repository don’t have to propagate to other areas of the application.

Let me know what you think, and I’ll update the project as needed.

This link, here. Click it only if you’re not doing anything for next two hours.

For some years, I’ve been using a pattern to abstract away access to a repository and make it more manageable. Here’s how I do it. I understand that this looks like (and probably is) DAO, but not everyone has heard of it, and I think that a concrete example is more helpful than a three letter acronym.

So. You have some custom information that you need to store in a repository. You know that you’re going to need at least one new item-descriptor, and you’re going to have to expose that data to the rest of the system.

The item descriptor looks like this:

<item-descriptor name="car">
  <property name="displayName" column-name="name" data-type="string"/>
  <property name="color" column-name="color" data-type="string"/>
  <property name="price" column-name="price" data-type="double"/>
</item-descriptor>

And you want to be able to search for cars, create new cars, modify existing ones, etc.

First thing to do is to create a JavaBean for the item descriptor:

public class Car {
  protected String mId;
  protected String mDisplayName;
  protected String mColor;
  protected Double mPrice;


  public Car() {

  }

  // get and set methods
}

Note that the variables are protected rather than private. I do this because someone after me may need access to those variables in a subclass. I don’t advocate it, but it’s up to them to make the call if it’s necessary. There is also no link directly to the repository item.

And then create a manager which interacts with these Car objects, and throws exceptions on the same level.

public interface CarManager {
   public Car getCar(String pId) throws CarException;
   public void deleteCar(String pId) throws CarException;
   public void updateCar(String pId, Car pCar) throws CarException;


   // Known searches
   public Car[] getCarsByColor(String pColor) throw CarException;
   public Car[] getCarsByPrice(Double pPrice) throw CarException;
}

One advantage of explicitly defining an interface is that clients can write against the interface even when the implementation is still incomplete. This means that you can write a mock object that implements that interface and hook your business logic to it for unit tests.

After defining the interface, you create the manager to translate between repository items and domain objects:

public class CarManagerImpl implements CarManager
{


public Car getCar(String pId) throws CarException 
{                                       
    if (isLoggingDebug()) 
    {
        String msg = "getCar: pId = {0}";
        Object[] params = { pId };
        msg = MessageFormat.format(msg, params);
        logDebug(msg);
    }

    if (StringUtils.isEmpty(pId)) 
    {
        throw new CarException("null pId");
    }

    String repositoryId = pId;
    try
    {
        boolean rollback = true;
        TransactionManager tm = getTransactionManager();
        TransactionDemarcation td = new TransactionDemarcation();
        // Make sure that a transaction exists before we do anything.
        td.begin(tm, TransactionDemarcation.MANDATORY);
        try
        {
            Repository r = getRepository();
            RepositoryItem item = r.getItem(repositoryId, CAR_ITEM_DESC_NAME);
            if (item == null) 
            {
                return null;
            }

            Car car = new Car();
            car.setId(repositoryId);
            copyProperties(item, car);

            rollback = false;
            return car;
        } finally 
        {
            td.end(rollback);
        }
    } catch (RepositoryException re)
    {
        throw new CarException(re);
    } catch (TransactionDemarcationException tde) 
    {
        throw new CarException(tde);
    }


   }
}

There’s a number of things going on in the code above. First, the manager takes responsibility for logging debug information. Yes, you can turn on debugging in an item-descriptor directly, but that will spit out debugs for any access of that item-descriptor. Logging on the manager level allows for more direct and customized control.

Second, the manager handles transaction management. It may be the case that a transaction already exists before this method is called, but that’s not the important bit. The important bit is that an existing transaction gets rolled back if this method fails. I can’t stress this enough: almost nothing is as frustrating as a method that both throws an exception but still commits bad data to the repository.

Third, the manager does not use RepositoryException. Instead, exceptions are nested and thrown so that the client may determine how to deal with the error. I used to explicitly log errors in the manager, but that got tedious as the applications scaled up and every single component logged the same error at different levels. So now I have a simple rule: if you catch an exception and don’t rethrow it, you are responsible for logging it. This typically means that the form handler or droplet at the UI end of the chain catches the exception, logs it to console and adds a form exception for user display.

So this handles the basic case. Let’s see what happens with modification:

public void updateCar(String pId, Car pCar) throws CarException {


// logging code
// Check input for nulls
// transaction code wrapper

MutableRepository mr = (MutableRepository) getRepository();
MutableRepositoryItem mutItem = mr.getItemForUpdate(pId, CAR);

// Copies all the public properties from pCar to the mutable repository item
copyProperties(pCar, mutItem);

mr.updateItem(mutItem);


}

Much like you’d expect, except for a couple of points: I explicitly use an id for the update. I could put the id in the Car object, but that would get confusing as we are using the Car as a bag of data here for application, not query. The “copyProperties” code is actually not hard: you can leverage the DynamicBeans API with an array of public properties to get and set property values from one to the other. However, this is a blind copy. You can’t selectively change a property value with this method. Usually this isn’t a problem (in forms which display all properties at once), but it’s always possible to use a key-value approach or allow for more selective updates.

The advantage of updateCar(Car) is that the Car can contain as much data as needed. If you do updateCar(String pId, String pColor, Double pPrice), then you have to change the interface every time you add a property to the item-descriptor.

Finally, there’s the searching. RQL statements are very useful in this context, as they can be defined in the component properties, and it’s much less work to get data in and out of them.

public Car[] getCarsByColor(String pColor) throws CarException {
   // logging debug code


// Check input for nulls
// assume this happens in a transaction code wrapper..


   // this is set in the properties file as “carsByColor=color = ?0”
   RqlStatement carsByColor = getCarsByColor();
   Object[] params = { pColor };
   Repository r = getRepository();
   RelationalView view = r.getView(CAR);
   RepositoryItem[] items = carsByColor.executeQuery(view, params);

   // always pass a zero length array, as this makes client access less fiddly…
   if (items == null) {

   if (isLoggingDebug()) { ... }
   return new Car[0];


   }

   Car[] cars = convertItemsToCars(items);
   return cars;
}

There are a number of complexities that can arise with this pattern. I don’t cover what happens when you have items that reference other repository items. I don’t cover the memory bloat that this pattern can cause with large repositories. And I don’t cover the ‘stale data’ problem that happens if you keep references to the Car object. All of these problems are solvable, but the best solution depends on the circumstances.

I knew I was in trouble when I tried to Tivo the meeting.

I got the Tivo just so I could watch The Daily Show without being in front of the TV at 11 pm. But I found that Tivo enabled the manipulation of time in a program to the point where I no longer thought of a TV program as a contigous sequence of events – I’d flip through to the end, speed through the middle, repeat interesting bits as needed. Instead of channel-flipping, I was time flipping.

Reality doesn’t measure up to Tivo. You can’t pause. You can’t review. You can’t fast forward. And worst of all, you can’t save. Reality doesn’t even have subtitles you can scan through. All you have left when you’ve finished a meeting is the notes, some doodles, and a vague memory that you said a bunch of technical stuff. I want to go back and check everything I meant to. It bites.

I went through this before when I discovered e-mail. Suddenly, reality was missing a backspace key. And when I discovered IM? I wanted to phone up people in different states, scream “aitch aitch tee pee colon dash dash! ell o ell!” And then hang up on them.

This is the problem with my mediated experiences. They leak. At least I didn’t make that bee-boop sound, that would have been embarrassing.

Remembering and note taking is a different skill than organization and action items. GTD goes into great depth about how to make sure you know what to do, but doesn’t explain how to structure semi-random data.

When I first started working with computers, I had to remember things. Especially the details: what the Samba network name was, the syntax for setting up FreeS/WAN, and what tools I had tried to configure the network that didn’t work, and why they didn’t. My solution was a boring Mead notebook. I dumped everything in there, and referred back to the notes if I had to work with the system again.

Then, I found out that I was using the notebooks for todo lists. And addresses. This had a couple of bad effects: one, if I wrote a todo list and then wrote some more notes I had to flip back to where I was to find out what I should be doing… and two, there was personal information in there (i.e. do laundry) that really I didn’t want to see when looking through the notebooks later.

So I got another notebook for personal stuff. One notebook for work, one notebook for home. Simple. I’d use the notebook at home for all my personal stuff and that would be it.

Then I started using the personal notebook as a diary. This was actually a good thing (you can learn a lot from reading through old diaries), but it did have the unwelcome side effect that I was terrified to go to Safeways with it.

So I split the personal notebook into a Todo list and a diary. Write essays in one, write tasks in the other. This actually worked until I replaced it with the GTD system I use currently, where diaries are treated just as another inbasket item.

For work, I ran into another problem after a few years. I have several different projects that I’m juggling, as well as my own personal projects. Some of these are at different stages i.e. I’m writing code on one project, reviewing code for another, and looking at product options for another. When all of these got hammered into one notebook, the end result was not great.

So the current system. I buy about 10 notebooks at a time. I write the name of the project on the front of the notebook, and the start date, and use the notebook for the project only. I write down technical information, logs, and caveats. No tasks. When the project is finished, it gets retired, and I can then look back through it for a post-mortem.

This probably isn’t the best system, but it’s the best I’ve found so far. Writing things down in a notebook gives a permanence to thought that I just don’t get when I use scrap paper or a PDA.