Command Line Interface (CLI)

We have been testing the developer preview of AT Version 2.0 Update 6, which now features support for the command line interface. Based on work from Cyrus Farajpour at USC, the new version and corresponding ATCLI plug-in developed by Cyrus and Nathan Stevens allow for development of handy command line instructions such as batch export of resources--something we and perhaps others with lots of finding aids are excited about. After initial testing, the plug-in has been modified to address memory consumption issues (a new session needs to be open for each record otherwise the records get cached and used up memory) and allows for specification of output directory.

Our first batch export test of our 3000+ resources was performed on my desktop (2.99 GHz processor, 2 GB RAM) under normal workload. It took approximately 96 hours for the process to complete! Given these results, we've tweeked our approach quite a bit, running it from a better computer and adjusting the amount of memory assigned to the atcli.lax file (see AT's FAQ's for instructions). This seems to help a bit. The main problem though for us is the sheer number and size of our finding aids. No matter what we do, it's going to take some time. In addition, we have several finding aids that don't need to get exported. Ideally, we'd like to be able to select which finding aids to export (i.e. those not marked internal only or those updated since a given date). Perhaps further modifications to the the plug-in will allow such customization.

For the time being, we'd like to thank Cyrus and Nathan for developing this key tool and look forward to working with others to improve it's design and functionality.

UPDATE (6/23/2010)
Good news. Cyrus has modified the command line plug-in to incorporate parameters, allowing you to specify which subset of resources/EAD you want to export from the AT. The parameters include: findingAidStatus, author, eadFaUniqueIdentifier, resourceIdentifier1, resourceIdentifier2, resourceIdentifier3, resourceIdentifier4, internalOnly, and lastUpdated. Plus, you can use any combination of these to easily fine-tune the export process as necessary. For more information on setting up and using the plug-in check out his github site: http://github.com/smoil/AT-CLI-Export-Plugin

Collaborative AT Instances: Pros & Cons

This post examines the pros and cons of consortial or collaborative AT instances. My comments are based on my experience administering the Yale University Library Collections Collaborative AT project and MSSA's AT development.

Pros

1. The central benefit of a consortial/collaborative AT instance is the consolidation of systems, procedures, practices, and resources. Having one system, one set of procedures, one course of training across multiple repositories not only conserves resources, but also greatly facilitates consistency and efficiency across the institution. Such a configuration inherently allows for enhanced understanding of each other’s collections, and provides faster and more consistent access to collection information, as well as the possibility of—from one location—getting an overview of the special collections holdings across diverse repositories. In Manuscripts and Archives alone, implementation of the AT will result in the consolidation of numerous databases, centralizing collection information and reducing ongoing systems maintenance for these un-integrated databases.
   
   
2. Centralizing collection and other archival information leads to increased security, as potentially sensitive information will no longer be scattered across databases, electronic office files, and often, paper logs.

Cons

1. The primary challenge facing collaborative/consortial instances is that the AT does not scale well for large, complex data sets and hence causes noticeable performance issues. Based on an acknowledged design flaw, the AT will impede performance/functionality at a certain point (see my previous post on Resource loading). As a result, especially for large institutions with multiple repositories or, more importantly, with extensive legacy data, consolidating multiple systems in a single AT instance will result in a slow performing system. Unfortunately, there is currently no plan to alter the design of the AT to address this issue in the 2.0 release, but the potential exists for a third round of grant support (to merge the AT with Archon) that would allow for such. The only alternatives at this point are for one of us institutions to pay a consultant to do the redesign work, which might be costly, or to develop lookup tools/plug-ins that query the database without having to say build an entire Resource.
   
   
2. Sustainability is another major issue. With two major releases left, development of the AT is coming to a close. Unfortunately this is happening just as many of us are finally getting around to evaluating and adopting the AT. Thankfully though with version 1.5.9 the AT team introduced plug-in functionality as a viable means for the community to customize and develop the AT. In addition, it's exciting to hear that the opportunity exists for a third round of grant support to merge the AT with Archon. Beyond greatly expanding the functionality and usability of AT/Archon, this would allow for finishing any existing development commitments still on the table when the AT development ends.
   


3. A third issue for consortial/collaborative instances is that any modifications/customization done with the AT by the superuser, e.g. modifying default values, lookup lists, user-defined fields, etc., apply to the instance as a whole and cannot be applied to just a single repository. Extensive modification of lookup lists, user-defined fields, and default values may thus result in a cluttered, hard to use interface or may even impede efficiency and performance. Plug-ins may be a solution to this problem though as they can be used to alter appearance and workflow.
   


4. A fourth issue is that the only way to restore an instance (e.g. after a crash or failure) is to import an entire MySQL dump. Hence if one repository or individual screws something up, you'll have to go back to the last backup of the entire database, which may cause lots of data to be re-entered.
   


5. A fifth challenge is the difficulty of migrating legacy data, especially instance or box information (e.g. location, box type, barcode, etc.) into the AT. Migration can also be difficult for those institutions that do not have EAD 2002 or who lack the expertise to export, format and import their legacy data. For those institutions like us with an significant amount of complex legacy data the only real option is to hire a consultant and develop a custom import process/plug-in.
   


6. A sixth shortcoming of the AT is it's inability to support the full EAD tag set, meaning that addition tools (e.g. stylesheets) or systems may be necessary to fully manage an institution's finding aids. On the bright side, especially for smaller institutions or those who lack a finding aid display/delivery system, the proposed AT-Archon merger might address the issue of full EAD support.
   
   
7. A seventh issue is the inability to batch edit data in the AT. For those with a little know-how though a MySQL database administration tool such as Navicat can be used to query and update data in the AT's MySQL tables. This is definitely beyond the capability of the typical user so you may want to address this inability via a plug-in.

All told, it might seem that AT has more than its fair share of cons. This is not what I want to get across. True, there are some issues to take into consideration especially when creating a collaborative instance, but as MSSA is easily the largest user of the AT, none of what we've encountered thus far is a real deal-breaker. True, we'd like it to perform better, but there are options at this point to at least sidestep this issue until the AT/Archon redesign. And with plug-in functionality now available and soon to be expanded, the means for addressing the AT's shortcomings is now in the hands of the community. We just need to step up.

AT Issues: Resource Handling

We have run into a significant performance issue at the half-way juncture in our legacy data migration efforts. We have found that Resources are taking much too long to load--small ones are taking at least 15 seconds; medium resources are taking up to 3 minutes; our largest ones are taking upwards of 5 minutes. Again, since we're only half-finished migrating our legacy data, the problem is likely to get much worse, especially as we begin to create/describe the thousands of digital objects we've been waiting to use the AT for.

We think the reason for the problem is how the AT loads Resources, i.e. recursively issuing SQL commands for each component, rather than issuing a single command for all resource components at once. Hence, given a large, complex finding aid (e.g. one with several levels of nested components and associated instance and location information), the AT grinds to a stand-still as it tries to systematically open all of the hierarchy. The main issue here appears to be the number and depth of nested components. Resources with only one or two levels of nested components do seem to open right up, the exception being two-level resources that have larger numbers of sub-components. However, resources that have three or more levels of nested hierarchy always open slowly, even if they don't have many actual associated instances.

We have tried to address the problem by maximizing the memory assigned to the client (see AT Wiki), boosting the database server's use of system memory, increasing the key buffer size, and engaging Memory Lock to discourage swapping--all to no avail. These steps however did boost performance of the MySQL database administration tool Navicat, which we use to access the MySQL database. This therefore points to the problem as lying in the AT and its Java encoding. This hypothesis was further supported when we set up a slow query log (i.e. anything over 2 seconds) on the database and tested resource loading, finding that no such queries were logged. As a result, all of our efforts seem to suggest that while we can indeed speed up operation of the MySQL database, we cannot speed up the performance of the Java that hits or interacts with the database. Since this however lies in the core of the AT code, it would have to be addressed either by the AT development team and put into the next/final release, or developed externally and then somehow offered back to the AT community. We are not sure yet how to proceed. An alternative that we and perhaps others might be happy with would be for only the top-level components (i.e. c01s) be built and displayed within the main Resources window, with further sub-components built when clicked. After all, you don't need to see/retrieve the whole Resource to make an edit to a specific part.

One of the questions this problem has raised for us is why hasn't anyone noticed/reported this performance issue? Was this borne out in scale testing? Surely there are other repositories who have loaded large finding aids, or who are working with complex resources or digital objects. Are they not using/editing these Resources and therefore not noticing the lag in performance? Perhaps they haven't ingested the amount of instance and location information that we have? Or, is there an upper data limit to the AT's functionality? That seems short-sighted. Regardless, others are bound to run into the problem sooner or later as they begin to ingest more and more data into the AT. As a result, we hope that progress can be made to address this critical problem.

AT Issues: Large Finding Aids

Manuscripts and Archives encountered several performance issues when loading our finding aids into the AT. First, given the large number of finding aids in question (2500+), we encountered load time issues. Our first attempt to batch import our finding aids lasted the better part of a weekend, ultimately crashing with the dreaded java heap space error. Adding insult to injury, no log was generated to indicate the cause of the crash or the status of import (e.g. which files were successfully imported). Our initial diagnosis pointed to our setup. We had the database installed our one of our local, aging servers and ran the import via a client on a remote workstation. To address the issue of load time we changed our setup and moved the database to our fastest machine, a Mac with 8GB of memory and multiple processors, installed the AT client on it, and saved copies of our finding aids to it for easy import.

Our second attempt, which involved approximately 1800 finding aids, was much much faster but still crashed. The likely culprits this time were large finding aids and memory leak. We found that large finding aids (3MB+) crashed the import process when included as part of a batch import. In addition, we found a so called memory leak (i.e. successive loss of processing memory with each imported finding aid), which greatly slowed the process over time and contributed to the crash. As a result, we separated out large finding aids and imported them individually, as well as creating smaller batches of finding aids (both with respect to total number and total size) to import in stages. Just to give you some idea of the time required to import larger finding aids, we found that using a remote client to import up to a 2 MB file averaged 20-30 minutes; 2-3 MB took 30-60 minutes; 5-6 MB took 90-120 minutes.

These strategies proved effective, allowing us to import all but the largest of our finding aids (8-12 MB each), which we are currently working on. Because these present problems for our finding aid delivery system as well, one option is to split them up into multiple files, each around 2 MB. The only problem with this option is dealing with/maintaining multiple files.

For other institutions with similar numbers and sizes of finding aids these strategies may be of help to you.

AT Issues: Large Collections

Two issues that we have encountered during testing are load time and slow performance. Given MSSA's 100,000+ instances, 2600+ resources, 7000+ accessions, and multiple locations, we've encountered lengthy start-up times and slow performance on even just a single client, let alone the dozen or so clients that will eventually be using the AT at a given time. Since we have not fully deployed the AT at this time, so we cannot yet accurately test how the AT will perform when running across several clients. Two things we will be doing to speed up performance is to move the AT to a dedicated server and to simplify our location assignments. Concerning the latter, in a previous iteration involving writing location information to the tables directly we created an enormous locations index, including many duplicate values, which significantly slowed client performance.

A third performance issue encountered concerns report generation. Again given the amount of data we have in the AT, running reports across our resources and accessions can take a considerable amount of time. Rather than running reports in the AT, which also aren't customizable, we use the database administration tool Navicat for MySQL to query the AT tables. Aside from being much faster, it's customizable (provided you know a little MySQL) and allows for batch editing.