Command Line Interface (CLI)

We have been testing the developer preview of AT Version 2.0 Update 6, which now features support for the command line interface. Based on work from Cyrus Farajpour at USC, the new version and corresponding ATCLI plug-in developed by Cyrus and Nathan Stevens allow for development of handy command line instructions such as batch export of resources--something we and perhaps others with lots of finding aids are excited about. After initial testing, the plug-in has been modified to address memory consumption issues (a new session needs to be open for each record otherwise the records get cached and used up memory) and allows for specification of output directory.

Our first batch export test of our 3000+ resources was performed on my desktop (2.99 GHz processor, 2 GB RAM) under normal workload. It took approximately 96 hours for the process to complete! Given these results, we've tweeked our approach quite a bit, running it from a better computer and adjusting the amount of memory assigned to the atcli.lax file (see AT's FAQ's for instructions). This seems to help a bit. The main problem though for us is the sheer number and size of our finding aids. No matter what we do, it's going to take some time. In addition, we have several finding aids that don't need to get exported. Ideally, we'd like to be able to select which finding aids to export (i.e. those not marked internal only or those updated since a given date). Perhaps further modifications to the the plug-in will allow such customization.

For the time being, we'd like to thank Cyrus and Nathan for developing this key tool and look forward to working with others to improve it's design and functionality.

UPDATE (6/23/2010)
Good news. Cyrus has modified the command line plug-in to incorporate parameters, allowing you to specify which subset of resources/EAD you want to export from the AT. The parameters include: findingAidStatus, author, eadFaUniqueIdentifier, resourceIdentifier1, resourceIdentifier2, resourceIdentifier3, resourceIdentifier4, internalOnly, and lastUpdated. Plus, you can use any combination of these to easily fine-tune the export process as necessary. For more information on setting up and using the plug-in check out his github site: http://github.com/smoil/AT-CLI-Export-Plugin

Yale AT plug-in development: status report

This post provides an update on MSSA's plug-in development efforts.

1. Revised Analog Instance module/view
   
   To increase the AT's collections management functionality and usability MSSA has asked for user-defined fields (two strings, two booleans) to be added to the Analog Instance table/module in version 1.7. We will use these fields to facilitate interaction with our ILS (Voyager), using them to capture Voyager Bib and Holdings numbers, box type, and flags for item-level restrictions and tracking of exported items. Here is a screenshot of the revised Analog Instance view our plug-in presents when an individual instance is clicked in the main Resource module.


Status: design finished; user-defined fields will be present in version 1.7 release.


2. Assign Container Information plug-in
   
   A second functional improvement MSSA is contributing is enhanced batch container information assignment via an Assign Container Information plug-in.

Much more efficient than data entry via RDE, which requires assignment one instance at a time, the Assign Container Information plug-in allows batch box type, restriction flag, and ILS export assignment to selected instances all at once.


Additionally, we've designed a Rapid Barcode Entry module to allow use of barcode scanners to quickly wand in barcodes for containers.


The Rapid Barcode Entry interface allows you to either select a specific container or an arbitrary number of containers. If you select one container then it will be the starting point, allowing you to progress numerically through the boxes until finished. If several containers are selected, then that will be the group that the interface will assign barcodes to. Setting your barcode scanner to include a return/break will allow you to input barcodes as fast as you can scan them.

Status: design finished; should be ready for use with release of version 1.7.


3. EAD export plug-in
   
   Yale EAD instances validate against a Yale-specific RNG schema informed by Yale's EAD best practices guidelines. Because this schema differs from the EAD 2002 schema, we need to develop an EAD export plug-in to modify data in the AT to validate against the Yale schema for easy ingest into our finding aids database. Such a plug-in will allow others to modify their data to meet their various output needs.
   
   Status: not yet started.
   
   
4. Partial EAD import plug-in
   
   Because MSSA accessions several hundred additions to collections each year, many coming in with inventories and EAD ready to paste into a finding, deleting and re-importing finding aids (and corresponding instance info)--the only way currently to input EAD aside from direct entry into the AT, is impractical for us. Hence, we will be developing a partial EAD importer, allowing import of new addition/accession EAD (e.g. ...) to append to an existing AT resource.
   
   Status: not yet started.
   
   
5. Lookup/read-only plug-in
   
   To facilitate collections management and public services activities in MSSA, we have need to develop a quick look-up or browse plug-in that will allow us to select specific data fields (e.g. accession number) to retrieve certain information (e.g. barcode) without the need to build the entire resource, accession, or digital object record.
   
   Status: not yet started.



6. Export to ILS plug-in
   
   To facilitate easy export from the AT to an ILS we will be developing a text file export plug-in. The plug-in will allow you to format data entered into the AT for easy export/ingest into an ILS. Because we are one of I believe two institutions with a Voyager import API, in our case the plug-in will export the text file to an intermediary application that will in turn ingest the AT data into Voyager.
   
Status: not yet started.



7. Revised Digital Object Instance module/view
   
   To facilitate batch digital object creation we plan to modify the Digital Object Instance module in a similar fashion as we have done with analog instances, allowing bulk creation and metadata assignment. Specifically, we want the ability to create and assign metadata for an arbitrary number of items via a '+ n' button, which allows you to enter the number of items you want to create with assigned values.
   
   Status: not yet started.

Collaborative AT Instances: Pros & Cons

This post examines the pros and cons of consortial or collaborative AT instances. My comments are based on my experience administering the Yale University Library Collections Collaborative AT project and MSSA's AT development.

Pros

1. The central benefit of a consortial/collaborative AT instance is the consolidation of systems, procedures, practices, and resources. Having one system, one set of procedures, one course of training across multiple repositories not only conserves resources, but also greatly facilitates consistency and efficiency across the institution. Such a configuration inherently allows for enhanced understanding of each other’s collections, and provides faster and more consistent access to collection information, as well as the possibility of—from one location—getting an overview of the special collections holdings across diverse repositories. In Manuscripts and Archives alone, implementation of the AT will result in the consolidation of numerous databases, centralizing collection information and reducing ongoing systems maintenance for these un-integrated databases.
   
   
2. Centralizing collection and other archival information leads to increased security, as potentially sensitive information will no longer be scattered across databases, electronic office files, and often, paper logs.

Cons

1. The primary challenge facing collaborative/consortial instances is that the AT does not scale well for large, complex data sets and hence causes noticeable performance issues. Based on an acknowledged design flaw, the AT will impede performance/functionality at a certain point (see my previous post on Resource loading). As a result, especially for large institutions with multiple repositories or, more importantly, with extensive legacy data, consolidating multiple systems in a single AT instance will result in a slow performing system. Unfortunately, there is currently no plan to alter the design of the AT to address this issue in the 2.0 release, but the potential exists for a third round of grant support (to merge the AT with Archon) that would allow for such. The only alternatives at this point are for one of us institutions to pay a consultant to do the redesign work, which might be costly, or to develop lookup tools/plug-ins that query the database without having to say build an entire Resource.
   
   
2. Sustainability is another major issue. With two major releases left, development of the AT is coming to a close. Unfortunately this is happening just as many of us are finally getting around to evaluating and adopting the AT. Thankfully though with version 1.5.9 the AT team introduced plug-in functionality as a viable means for the community to customize and develop the AT. In addition, it's exciting to hear that the opportunity exists for a third round of grant support to merge the AT with Archon. Beyond greatly expanding the functionality and usability of AT/Archon, this would allow for finishing any existing development commitments still on the table when the AT development ends.
   


3. A third issue for consortial/collaborative instances is that any modifications/customization done with the AT by the superuser, e.g. modifying default values, lookup lists, user-defined fields, etc., apply to the instance as a whole and cannot be applied to just a single repository. Extensive modification of lookup lists, user-defined fields, and default values may thus result in a cluttered, hard to use interface or may even impede efficiency and performance. Plug-ins may be a solution to this problem though as they can be used to alter appearance and workflow.
   


4. A fourth issue is that the only way to restore an instance (e.g. after a crash or failure) is to import an entire MySQL dump. Hence if one repository or individual screws something up, you'll have to go back to the last backup of the entire database, which may cause lots of data to be re-entered.
   


5. A fifth challenge is the difficulty of migrating legacy data, especially instance or box information (e.g. location, box type, barcode, etc.) into the AT. Migration can also be difficult for those institutions that do not have EAD 2002 or who lack the expertise to export, format and import their legacy data. For those institutions like us with an significant amount of complex legacy data the only real option is to hire a consultant and develop a custom import process/plug-in.
   


6. A sixth shortcoming of the AT is it's inability to support the full EAD tag set, meaning that addition tools (e.g. stylesheets) or systems may be necessary to fully manage an institution's finding aids. On the bright side, especially for smaller institutions or those who lack a finding aid display/delivery system, the proposed AT-Archon merger might address the issue of full EAD support.
   
   
7. A seventh issue is the inability to batch edit data in the AT. For those with a little know-how though a MySQL database administration tool such as Navicat can be used to query and update data in the AT's MySQL tables. This is definitely beyond the capability of the typical user so you may want to address this inability via a plug-in.

All told, it might seem that AT has more than its fair share of cons. This is not what I want to get across. True, there are some issues to take into consideration especially when creating a collaborative instance, but as MSSA is easily the largest user of the AT, none of what we've encountered thus far is a real deal-breaker. True, we'd like it to perform better, but there are options at this point to at least sidestep this issue until the AT/Archon redesign. And with plug-in functionality now available and soon to be expanded, the means for addressing the AT's shortcomings is now in the hands of the community. We just need to step up.

AT 1.5.9 Plugin Functionality

With the release of 1.5.9 and plugin functionality the AT team has introduced a much-needed mechanism for customizing and expanding functionality in an extensible fashion. In keeping with current software development trends centered around a core system that can be modified or expanded via external applications, the beauty of AT plugin functionality is that all customizations or data imported via plugins is not affected by future AT upgrades as the plugin code lies outside of the main AT source code, stored in a separate plugin folder.

Plugins will thus offer repositories the capability to do many of the things that have perhaps up until this point kept them from fully adopting the AT. This includes creation of custom importers, export modules, record editors, and display/workflow screens. We in MSSA plan to take full advantage of plugin functionality to create a custom import mechanism for our marrying our existing container information (e.g. box type, barcode, VoyagerBib, and VoygerHolding) to instance records in the AT, a custom exporter to send data from the AT to Voyager, and a custom import mechanism for importing EAD or partial EAD for an existing resource record (e.g. new accession).

As with the AT itself, plugin development requires knowledge of Java programming. For more information on developing plugins check out the Plugin Development Guide (.pdf). A list of current plugins can be found here: http://archiviststoolkit.org/forDevelopers/plugins.shtml.

Sustainability & the AT

Perhaps the biggest question facing the AT is its sustainability. With only one or two releases left, the AT is soon approaching the end of development. What does the future hold?

At SAA in San Francisco last year, the AT group reported that it had begun to work with a business consultant to formulate a business plan for the AT after Phase 2 of development ends in 2009. Options at that point included institutional affiliation, subscription-based service, or the pursuit of another Mellon grant. Also briefly discussed was the related need to develop a community governance model for guiding the direction and future development of the AT. Although initial steps have been made to address the governance model thanks in part to the SAA AT roundtable, little has been said yet as to the AT business plan. This post will explore the pros and cons of the three options on the table at this point.

Institutional affiliation/hosting is one option for the AT. The main challenge of this approach is the reality that few institutions are capable of taking on this responsibility, not only due to limited technical expertise and infrastructure, but also because the current economic situation is precluding many of us from taking on outside projects. Without a commitment of significant resources it seems likely that this model will only allow for ongoing support and not any additional development. Given some of the needs addressed in the AT user survey, ATUG-listserv postings, and other forums, however, this option seems in the end the least beneficial to the archival community as a whole.

A second option is subscription or fee-based AT support and development, either by individual consultants or a single software firm. Although this option would incur costs on individual institutions, this approach does allow for continued development, especially for institution-specific needs. The central challenge for this approach is that not all institutions have the resources to commit to development and so might not have a say in the future direction of the AT. Bigger institutions with bigger budgets would drive the agenda. To prevent this disparity an effective governance model would have to evolve to lobby for community interests and manage ongoing development for all interested parties.

The third option is to pursue another round of Mellon grant support. This would obviously allow for continued development of specific needs voiced by the archival community in the AT user survey and other requests posted to the ATUG-l. Given the demands on the project staff, our current economic situation, and the little traction this option seems to have garnered up to this point, it seems unlikely that this will happen.

So where does that leave us? The AT project is coming to end, sooner perhaps than we would like. Sure we'd all like a few more bells and whistles before then, but whatever business model is ultimately adopted, the AT will at least be supported for the next few years. Will Archon, ICA-AtoM, or other products, open-source or proprietary, sufficiently evolve? It's hard to tell at this point. The one thing we at least in MSSA know is that we will be much, much, MUCH! better off to have undertaken the work to get into the AT. No amount of my blogging can sell this enough. I'm sure many institutions would agree. And we thank the AT for that.

Customizing the Toolkit: Instance Creation & Voyager Export

For some time now we have been looking to replace our collections management and accessioning systems, which are obsolete both in platform and functionality. When the Toolkit was being developed we made the decision that since it would not fully meet our needs, we would not adopt it. A few years later, however, committed to consolidating our systems into an open-source alternative, we have reversed course and decided to move to the AT. Since the latest version (1.5) still does not fully capture our existing systems functionality we have contracted out the modification of the Toolkit to meet two specific needs: 1) improved container (e.g. Instance) creation; and 2) Voyager export functionality.

Container/Instance creation

Although version 1.5 of the AT comes with customizable Rapid Data Entry (RDE) Screens and “sticky” values for improved data (e.g. Instance) entry, it is still time consuming to enter information for large collections. Here is a screenshot of a RDE I created for Instance creation:



AT 1.5 allows for the creation of one item at a time with no option for batch creation and no automatic assignment of successive box numbers. Here is a screenshot of our current system:



This system features the capability of populating successive items/containers, either next or n-multiple lines. It also allows for assignment of location info, which the AT does in a separate module

What we are building might look something like the following:



Voyager export functionality

To easily export collection information from the AT to our ILS (Voyager) we need to add the following fields to each instance: 1) VoyagerBib; 2) VoyagerHolding; 3) Date (last changed); and 4) Restriction. This will require a redesign of the Resources and Instances module to include these new fields, as well as the incorporation of a script (triggered by an Export to Voyager button) to process newly entered data in the proper format for export into our ILS. We already use a similar script to do this work with our current system, so it shouldn’t be much work to modify it for use with the AT.

Although it is unlikely we will complete all of our work for the next release of the AT, we hope to finish in time for incorporation into the next AT release.

Making the Jump: Standardization of Practice

In addition to the important and lengthy work spent standardizing our collections management data, Manuscripts & Archives also had to analyze its practices and procedures to determine how the AT could handle them.

1. Accessioning

For quite some time now we've handled accessioning differently on the University Archives side than on the manuscript side. This has resulted in two widely different processes, and two sets of disparate accession info. To remedy this situation we developed an accession checklist and procedures common to both University Archives and manuscripts, as well as instructions and tutorials for accessioning using the AT.

2. Description

As with accessioning, description has also differed between University Archives and manuscripts. With the welcome addition of Bill Landis as head of arrangement and description, and considerable time spent on the University Archives side revising its processes, this disparity has been addressed. Also helpful has been the creation of descriptive standards for handing a/v materials and electronic records, as well as the creation of finding aid templates and EAD best practices guidelines.

3. Collections management

This was less of a problem as collections management was handled well by one person for quite a while. The only problems we encountered were from incomplete data and improper use of the system by clerical staff (e.g. entering temporary holdings, capturing accession or descriptive info, or entering several different data elments of in a single note field). To address these problems we had to map out export of the problematic data elements into appropriate AT fields (where possible) and plan for post-import clean-up in the AT.

Deploying the AT Across Multiple Yale Repositories: Background

In 2008, four Yale special collections repositories (Arts Library, Divinity Library, Manuscripts and Archives, and the Music Library) participated in a project to 1) install and test the AT as an open source collections management system, and 2) examine the feasibility of establishing a Yale way of managing and tracking collections across the Yale Library system. This project was one of a number of projects that were undertaken as part of the Mellon Foundation Collections Collaborative at Yale University.

Focus

The specific focus of the project was to use of the Toolkit for accessioning. Although the Toolkit can do much more, we limited our focus in this project to accessioning primarily because, with the exception of MSSA, the other participants have rudimentary systems in place for recording and managing accession information. In the past, this has been done primarily on paper. The Toolkit, however, facilitates easy capture, management and searching of collection information that is vital to the day-to-day operations of repositories. This allows participants to utilize the same system and terminology to enhance understanding of others’ collections, provide faster and more consistent access to collection information.

Work Summary

The principal investigator first met with the participants to examine existing collections management tools, record-keeping practices, and discuss needs and expectations. These sessions provided the opportunity to specify collections management practices the Toolkit does not accommodate, distinguish between software issues and points where the staff could or should be persuaded to do things differently, and explore the feasibility of developing a “Yale” way of managing special collections. In addition, with little in the way of legacy systems and practices for accessioning materials (Manuscripts and Archives excluded), it was determined that adoption of the Toolkit would be easy to implement and much welcomed.

Following installation, project staff instructed participants in use of the Toolkit and discussed issues concerning implementation and conversion of legacy data. Project staff then gave participants several weeks to use the Toolkit before following-up with a focus group to examine participants’ experiences, issues, questions, and needs. Important outcomes of the focus-group included the expressed need for common practice and use (e.g. best practices guidelines), improved documentation (especially concerning required fields, terminology, and reports), and identification of concerns regarding future administration and tie-in to other systems (i.e. Finding Aid Creation Tool).

Products

To support and further Yale’s use of the Toolkit, a variety of products were (and continue to be) created. These include:

a. Website <http://www.library.yale.edu/mssa/at/>

b. Wiki <https://yaleat.pbwiki.com/>

c. Guided instructions and tutorials for 14 separate Toolkit features and processes.

d. Expanded data dictionary (.xls) [in progress]

e. Best practice guidelines for accessioning

Staff also reported to the Toolkit developers participant experiences and provided recommendations for potential incorporation in future AT releases.

Conclusions & Recommendations

With little in place to accession and track collection material, participants have enthusiastically adopted the Toolkit. Given varieties in local practice/needs however and the amount of information the Toolkit allows you to capture, it is recommended that best practices for accessioning be undertaken to standardize its use. Additionally, ongoing central support and administration will need to be formalized, including bringing in additional special collection repositories. Particularly important here, especially for larger repositories with established record-keeping systems, will be helping repositories map and migrate their legacy data. With future releases and expanded Toolkit functionality, efforts will likely need to be made to further integrate into the AT more and more legacy systems across Yale libraries and special collections.