University of Wisconsin-Madison

Data Movement and SPADE

This briefly describes the Data Movement and Archival Subsystem (Data Movement), a suite of software which controls the transfer of data from the South Pole to the Northern Hemisphere data warehouse, as well as the archival of data on tape at the South Pole. Data Movement is comprised of a set of Java applications. These applications run on several computers in the South Pole LAN, and interact with Java counterparts in the Northern Hemisphere. The primary application is called SPADE, for South Pole Archival and Data Exchange. Another application at the South Pole, called TapeServer, runs on each host which serves a tape drive for use by SPADE.

The above diagram summarizes the functions of SPADE at the South Pole. The tasks which it performs can be grouped roughly into four categories: operational (real-time) tasks, verification tasks, monitoring tasks, and administration tasks related to managing the Data Movement subsystem itself.
The above diagram summarizes the functions of SPADE at the South Pole. The tasks which it performs can be grouped roughly into four categories: operational (real-time) tasks, verification tasks, monitoring tasks, and administration tasks related to managing the Data Movement subsystem itself.

A. Operational Functions

File pairs are fetched by Data Movement via secure copy (scp) commands from the upstream Process and Filter subsystem and other clients. SPADE looks for and fetches files in pairs – one file containing the (usually binary) data, and the other file serving as a flag to indicate that the pair is ready for pickup. After receiving the data files locally, they are deleted from the source host. The data is prepared for transfer and sent to the Northern Hemisphere by one of three transfer methods: high-volume bulk transfer over TDRSS satellite, low-volume transfer by scp, or transfer as email attachment.

Information about data files to be fetched and transferred by Data Movement must be recorded in advance, as a one-time task, in the Data Movement "file registry" database. The information includes details of the host and directory where files are found, and a filename prefix which is used as a template to locate the files when they appear. The priority of the data is also registered, which helps determine how the data is handled by Data Movement. The priority scheme determines the transfer method as follows:

Priority Transfer Method Comments
0 Tape archive made, but files not transfered Usual priority level for raw unfiltered datasets
1 Bulk satellite transfer High-volume filtered datasets
2 TCP/IP network transfer (scp) Medium priority low-volume data
3 Email attachment High priority low-volume data

The priority also determines which tape archive is used for any given set of data. Raw unfiltered data is recorded on one set of tapes, and all other data – filtered data from the detector, monitoring, calibration, or any other ad hoc data – on another set of tapes. Multiple tape drives are used, and potentially more than one copy of a given dataset can be made. SPADE creates tape archives in parallel with transferring the data, so as to avoid delaying data transfer by the time required to tape it.

The operational tasks of SPADE include two additional important functions. The first of these is that SPADE optionally makes a copy of data files it handles at a location on the South Pole LAN. This allows South Pole personnel to examine data soon after it is collected, to facilitate monitoring and calibration of the detector. Whether or not a local copy of data is made and the location for the local copy is controlled by settings in the Data Movement file registry.

Secondly, SPADE begins the process of cataloging IceCube data by producing a metadata file for each data file it receives. This XML file follows a metadata specification called "DIF Plus". The "DIF" portion of the XML is metadata in Directory Interchange Format (DIF). DIF is the format required by the Office of Polar Programs for all experiments funded by the National Science Foundation, and is required for cataloging data in the Antarctic Master Directory. The complete DIF specification is lengthy and contains many optional fields which can be filled in once the data reaches the data warehouse. SPADE starts this process by creating a "skinny DIF" – one which specifies only the minimum number of required fields. The "Plus" portion of the XML is a collection of fields that have been defined specifically for IceCube, to aid in the storage of data in the data warehouse and later searching of the contents of the warehouse. SPADE tars and gzips the data and "DIF Plus" metadata file, transferring them together to the Northern Hemisphere.

B. Verification functions

In the Northern Hemisphere, another Java application (called Ingest) detects and collects incoming data from all three transfer methods. This application creates and sends emails back to SPADE at the South Pole. The emails contain a checksum of the transferred data as it was received in the north, which SPADE compares against its saved value to confirm that files have been transferred correctly. If the data was received without error, SPADE finalizes the database record for those files and deletes them from disk.

If, however, there appears to have been a problem in transmission, SPADE resends the data up to two more times. Resending data could be prompted either by an invalid checksum reported from the north, or by no verification email being received at all.

Note that this verification scheme applies to filtered data being transferred to the north. Raw unfiltered data files, which are archived but not transferred, are retained on disk for some period of time (tentatively, one week) before SPADE automatically deletes them. This allows time for possible re-analysis or filtering of raw data without having to read it from tape archives.

C. Monitoring functions

SPADE performs routine monitoring of network connectivity between the South Pole and the Northern Hemisphere, with a check made every few minutes. Statistics saved in the database includes the average packet transit time to a known Northern Hemisphere host, the average number of network hops taken over multiple traceroute commands, the packet loss percentage calculated over multiple ping commands, and the satellite network used for each connection attempt.

The application also monitors its data throughput, captured in windows of 5-10 minutes (this value is configurable). For each time window, SPADE tracks the number of files and total bytes fetched by the system, the corresponding numbers of files and bytes transferred out of the operational block, and the number of files and bytes archived to tape. Daily and/or weekly totals of files and bytes fetched, processed, and taped may be produced, depending on user demand.

D. Administrative functions

The administrative capabilities of Data Movement are concerned with configuration and operation of the software. These include:

  • A programmatic interface to Experiment Control to start and stop the Data Movement subsystem, as well as to temporarily suspend and resume processing;
  • A web interface to South Pole operators to display and configure system configuration parameters, and view and modify the contents of the Data Movement file registry;
  • A web interface for display of system status, including notifications of errors detected by SPADE, and monitoring information such as network connectivity and data throughput.
  • Historical information on Data Movement operations in two forms: database tables and file logs. Concise information on data file movements will be saved in the database as a permanent record, sufficient to reconstruct the path of data through the system. More verbose, "user-friendly" informational messages created by SPADE as it runs will also be saved to a log file.