Terse Systems

There’s a new ATG Filebrowser available at datafaber. This is a very useful debugging utility, because now you can edit your JSP pages through the same interface that you view JSP pages. This can be a big help when you’re working on a foreign dev machine, or when you’re just tired off bouncing between apps. So go get it.

I’ve been playing around with the ATG Eclipse plugin. I like the Repository editor, but it can’t cope with more than two layers of XML file combination, so it’s not useful for modifying large scale catalogs. But it is great for proof of concept repositories and prototypes. And I like that I don’t have to run through a build script when using an ATG project in Eclipse.

The side effect of using the plugin is that I’ve been creating a whole bunch of disposable ATG modules. I’m fairly good at cranking these out now, but they’re a hassle in bulk. So I wrote an ant script that creates ATG modules from a template. This will setup the EAR, WAR, and libs for you automatically. Click here for the file.

There’s also some stuff in there about ATG versioned modules. I remember that there was a big annoying problem with CampaignOptimizer (otherwise known as A/B Testing). The details are in build.properties, but I don’t remember precisely.

It’s worth noting that versioned modules could have some advantages over regular modules for large scale enterprise builds. Notably, versioned modules can contain scripts that can automate deployment. Jason Goth goes over this in his presentation at the ATG Open. Relevant bits are as follows:

Versioned Modules
- Each versioned module contains an install script(s).
- The AACom module is a “shell”
– Creates a dependency on the actual version of the site
– The MANIFEST.MF contains just the following:
ATG-Required: AACom-1_36

Versioned modules allow completely automated deployments.
- The deployment script:
– Copies the module files
– Stops a portion of the servers
– Runs the install script(s)
– Starts Dynamo
– Repeat process for remaining servers
- Written in Perl

The deployment script allows us to:
– Upgrade without manual intervention
– Avoid inconsistencies during the restart
– Handle failed file transfers
- Don’t start those servers
– Rollback in case of failure

This is taken from the powerpoint slides. I have no idea exactly what he meant by much of this – since ATG versioned modules are completely undocumented, much of what I already know is trial and error. Still, it’s a good starting point.

I wrote my blog from the ground up. It seemed like an interesting technical exercise at the time. Now I have actual content and people who comment, I’m in a much better position to appreciate the benefits and drawbacks.

Plus: I know exactly what’s going on. It does exactly what I want.
Minus: If something doesn’t work, I can’t draw on a user community to fix it for me.
Plus: I can use the blog as a proving ground for new technology.
Minus: Spammers also learn new technology.

I thought I was very clever for writing my own blog, because spammers were oriented against WordPress and Blogger and wouldn’t bother with as small a target as one custom written blog. I failed to take into account that spammer technology would advance to the point where they would figure out how to act like users and hit a comment button without regard to the back end technology.

I tried cleaning out the spam manually. This is like trying to stop the tide with a bucket. The problem is to come up with a comment submission scheme that spammers cannot hack, and that I need to think about.

In the meantime, I’ve disabled comments altogether. And in lieu of deleting the comments by hand, I’ve used this problem as an opportunity to think about how to do ad-hoc deletes from the repository.

Hence the BeanshellAdminService. This little beauty runs in the AdminServer, and allows me to run beanshell scripts on the server. It’s so convenient that I can’t believe I didn’t think of it years ago. For development, it lets you do just about anything.

I’ve uploaded it to the usual place. This is useful for your normal services as well. You can override the admin servlet to set your own parameters for the interpreter, so if there’s a function that you want to pull in or some parameters that you want to set, you can set them with an anonymous subclass. Best to do that from another module though, so you don’t inadvertently deploy scriptable code to production…

The named query feature in ATG 7.0 is very useful. However, it’s actually confusing to use, because it’s not clear how to use it.

You start off by creating the query in the XML:

  <item-descriptor name="person">


 &lt;named-query&gt;
    &lt;rql-query query-name="myQuery"&gt;
       &lt;rql&gt;name = ?0.name AND age = ?1.age&lt;/rql&gt;
    &lt;/rql-query&gt;
 &lt;/named-query&gt;


  </item-descriptor>

And this exposes a NamedQueryView to the repository. However, it doesn’t show how you execute the query with parameters, which is what you used to be able to do with rqlStatement.executeQuery(view, params). This is actually covered in the section before, concerning Parameterized Queries… but you need to put the two chapters together before it makes sense.

RepositoryItem steve, bill; // assume these get initialized correctly
Repository r = getRepository();
RepositoryView view = r.getView("foo");
NamedQueryView namedQueryView = (NamedQueryView) view;
ParameterSupportView parameterSupportView = (ParameterSupportView) view;
Query myQuery = namedQueryView.getNamedQuery("myQuery");
Object[] parameters = { steve, bill };
// Returns an array of person items.
RepositoryItem[] items = parameterSupportView.executeQuery(myQuery, parameters);

And there you go.

The ATG security framework has long been a source of fascination to me. Mostly because I couldn’t figure the damn thing out.

The canonical reference is here, but there’s one vital piece of information missing. It doesn’t tell you where you get a secured object from.

I broke down tonight and (using this opportunity to try out Enterprise Architect 6.1) mapped it out.

Security in ATG can be broken down into three areas. There’s user and persona related information. There’s access right information. And finally there’s the “secured object” which says who can do what.

There are the rights:

And finally there are the secured objects. The code sample for determining the rights of a secured object is:

SecurityContext context = securedObject.getSecurityContext()
SecurityPolicy policy = context.getSecurityPolicy();
boolean hasAccess = policy.hasAccess(securedObject, user, StandardAccessRights.EXECUTE);

Most of the security framework assumes secured repository items. I had thought that you could use AccessConfiguration, but those classes are not only undocumented, but they’re not functional. The only way I’ve found to get good secured objects is to use /atg/webservice/security/NucleusSecurityManager, mentioned in the Integration Framework guide. This uses a repository for the secured objects, but since it’s a part of DAS and there’s an Admin interface that allows you to add items yourself, it’s not too much hassle.

So. If you want to use restrict access secured object, create a new user in the ACC. Then add the user to a group. Then bring up the NucleusSecurityManager and add a new functional id to it with the ACL of the group. After that, you just get the SecurityObject from the SecurityManager, and run through it as usual.

ATG has been busy lately. The big news has to be the new ATG Eclipse plugin (remote plugin site is http://www.atg.com/eclipse). It includes a Repository Editor that looks like this:

Right click, expand, and override properties. It works with XML file combination (so it only defines the changes you make), and make viewing and editing repositories a snap.

There also some functionality that I’d somehow missed in the shuffle: repositories can now support many to many tables. It used to be (back in prehistory) that they didn’t support them directly because adding an item to a list automatically added it on the other end as well, and the caches got confused. Now bidirectional links just works, without requiring a link table in the middle to disambiguate them.

I both hate and love little tweaks like this, because they make my life easier when I know about them, and make me feel like an idiot when I miss them. And since ATG doesn’t come out with a “new in this release” feature list, the only recourse I have is to read through the manual and classes in every version. It builds character, but it gets a little old.

Still. Named queries support in ATG repositories. This is not a vastly new feature, but I don’t remember it supporting stored procedures or SQL queries before. I’m in favor of named queries, because it keeps all the query logic together. It’s interesting how many layers queries have gone through. I’ve gone from queries in UI to queries in DAO/manager components, and now the queries are completely outsourced back to the repository. Eventually there will be a dedicated query engine that we’ll communicate with and we’ll have to make queries to get the queries out.

There is a “removeNullValues” attribute for multi-valued collections which will make sure that any null elements will not show up when the collection is called. This is actually important, because there are a number of places in DCS where an order has to be reconstituted and will not value the relationships properly if a pipeline processor fails, leaving it null. This kind of thing causes all kinds of headaches, because no-one ever checks for null elements when iterating through a collection. Ever. So it dies with NPE and doesn’t tell you what didn’t work. So thank goodness for that.

I read about the session backup for repository items feature. I don’t know how useful this really is, but I’m sure someone must want it for something. I also found it’s possible to notify a repository through RMI or SQL JMS that it should invalidate its caches through the GSAInvalidationService. The distributed cache concurrency through SQL JMS is also new.

Added to this, I found some really old functionality in the middle of comparing ATG Nucleus to Spring. It turns out that Nucleus has been able to able to expose JNDI through a component reference (atg.nucleus.JNDIReference), and can even call static methods directly (atg.nucleus.StaticMethodReference). There’s even a system for component aliases built in with atg.nucleus.GenericReference, which explains why I always got confused when looking at price lists and secure price lists (they’re actually the same component).

Other than that… well, I’m still investigating atg.repository.test. I’ll see if I can reuse anything there over the mock repository stuff. It would be nice to be able to define a repository, run some integration tests against it, and then start the server…

For the most part I haven’t found the ATG Eclipse plugin to be of much use since it assumes that you are doing all your development from inside a deployed ATG module. However, I have found one useful thing it does: it incorporates a component browser that you can use on the pre-existing ATG modules to poke through the default settings.

Add the ATG Eclipse plugin from http://www.atg.com/eclipse to the Update Manager.

Click on New / Other / Java / ATG Wizards / Existing ATG Module:

Remove all the source directories when creating the project, and you’ll get an ATG module you can see in the ATG Component Browser view.

Click on a component, and you can see the settings of the component, plus all the different properties file that layer the component.

And that’s it.

I was confused about all the different frameworks, until I realized that many of them only attempt a small piece of the domain. I came back to it again once I felt confident I’d gotten past the hype. I haven’t examined every solution closely, but so as far as I understand it, the current state of the art looks like this:

If only DAS had been open-sourced 10 years ago. It could have been so different.

Currently playing around with AppFuse, the short way of saying for “I’ve got a sample application using all of this stuff integrated together, and it actually does work, if you use a code generation framework, use Tomcat, and don’t poke around with it too much.” Masses of documentation, but still a big pile of libraries. No conceptual integrity. And glue code to stick everything together. Argh.

Still, much better than last time I looked. They’re getting there, but it’s a long way to go yet.

For the last project I had, we depended on data from the business users. There wasn’t very much data, but it was very variable and being revised in an excel spreadsheet on a daily basis. Not only that, but it was changing in a structural basis as well – new rows and columns would be added, links between workbooks were used. We couldn’t use a snapshot, because the snapshot would be obsolete in short order, and the business users couldn’t use a database that would restrict them.

In the past, the business users had transcribed the excel data that they had one row at a time into the database through a web or Swing UI. We needed something faster this time around. We needed a way to import the excel spreadsheet directly.

This idea had appeal on an intuitive level. The business users could work in the structure they were most experienced at, and we could get fresh data almost as soon as they’d revised it. We knew that there were some data validation issues where values had to be restricted to a list, but we thought that Excel’s validation features would cope with that.

I looked into reading Excel in its XML dialect, but quickly discarded the idea after eyeballing the XML Excel generated itself. Instead, I looked for a library which would do the work of conversion for me, and present an easy to use API.

There was only one freely-available Java library that handled Excel spreadsheets, and that was POI. POI is a library written specifically to handle Microsoft Office formats, and their solution for Excel was called HSSF (Horrible SpreadSheet Format).

On the database side of things, we had an advantage in that we were using ATG’s persistence solution, the SQL repository (or Data Anywhere Architecture, as marketing calls it). Because the SQL repository can be fed data imported through XML, we could convert the Excel spreadsheet to a file in XML format, and then run startSQLRepository to import data.

The documentation for HSSF was eclectic, but servicable. Once I’d found the document to read an Excel document, I could iterate through the rows, and call row.getCellAt(0) to get the first cell in the row.

Once I’d done this, I copied the data from the row to a JavaBean object, passed a list of those object to the export class, and generated XML from them.

There were a number of issues that popped up in implementing this solution. The first one was that POI is abandonware. The last update was from 2004. Using Excel XP with some features (validation of specific cells, for example) cause NullPointerExceptions when it tries to parse it. The solution here was to not use those features.

Another issue was that the Excel data was linked between different workbooks, but was not linked tightly together. Because all the fields were text, any text that was not specifically validated had to match between workbooks exactly. Identifying data mismatches between workbooks was a time consuming process, and I didn’t think of a good way to automate it.

Finally, as new data was added, the columns would be moved around or added to as the business users tweaked it. Sometimes I would be notified of this change, and sometimes I would find out by having the script fail.

Importing data directly from an Excel spreadsheet is a good solution in many circumstances. However, it’s probably a good idea to have a ‘business’ that can be quickly modified, and a ‘data’ spreadsheet with immutable columns specifically for data export. Given a static schema, it’s easy to use and extend, and is much easier to modify than tweaking XML directly.

IF you ever get one of these exceptions AND you haven’t been playing around with the ACC AND you have more than one server running against the database, then check the cache-mode of the item-descriptor. There’s a good chance it has been left as simple when it should be locked, distributed or disabled.