Archiving medical image files presents some daunting challenges. Files are stored in different applications and are incredibly large, averaging 40 MB. The rate of this growth is massively expanding as well. For example, an orthopedic office with four doctors can generate a terabyte of data every six months. It is further estimated that all archives will triple in size every two years.
Medical professionals need fast access to these ever-increasing files, especially in trauma situations and other medical emergencies. In these environments, the speed of retrieving and the reliability of protecting data are critical. In addition, medical employees need the capability to collaborate and consult with specialists in other locations.
Furthermore, hospitals and imaging centers of all sizes face an increasing number of regulatory requirements mandating, among other things, the retention of medical images for long periods of time. For a system administrator, it's a management burden and a budget challenge to keep archive capacity sufficient for storage needs.
Limitations of conventional backup
Traditional methods of medical archiving have a number of inherent limitations, including shrinking backup windows, frequent restore failures, lack of file-level retrieval or end-user management capabilities, and the need for a system administrator to perform manual operations.
Until recently, storing data to magnetic disk and backing up to tape for disaster recovery purposes have worked well for small- and medium-size medical image archives. However, as the volume of stored data has increased, the time required to complete a full backup has grown as well, making this approach more and more difficult to implement. For example, it takes 13.8 hours to perform a full backup of 1 TB of data at an average transfer rate of 20 MB/sec, and more than five days for a full backup of 10 TB.
The decreasing cost of magnetic disk capacity has made the use of redundant arrays of inexpensive disks (RAID) replication for data protection increasingly popular. While this approach avoids the need for a backup window, it has two important limitations: It does not protect against accidental data deletion, and -- given that the primary and replica data should be geographically sited so that major disasters will not destroy both datasets -- it requires that the replica RAID be in a different location. Few small medical facilities can afford the high-speed link between locations that this requires, and most opt for either tape or optical disk for their offsite storage.
Limitations of optical disk systems
Optical disk systems have been popular for medical image archiving for reasons of longevity and write-once-read-many (WORM) capabilities but suffer from problems of poor reliability, low data transfer rates, relatively low capacity per disk, and high cost.
Poor reliability has been associated with CD and DVD media formats because the optical disks themselves are unprotected within a cartridge, making them prone to damage and data loss. This can be overcome at a higher cost using formats such as magnetic optical (MO), ultra density optical (UDO), and pro-data disk (PDD), which employ a protective cartridge for the optical disk.
Today, optical disk systems suffer from low data transfer rates. Even the latest formats have transfer rates that are very low relative to speeds that are easily achieved with magnetic disk and tape. For example, it is difficult to achieve more than 2 MB/sec when writing to the inner diameter of a UDO disk, whereas tape formats and hard disks routinely write at 30 MB/sec or faster.
The highest capacity optical disk formats available today are UDO and PDD. UDO offers 15 GB per side on a double-sided disk, and PDD provides 23 GB on a single-sided disk. These are relatively low compared to the latest magnetic disks (400 GB) and tape formats such as linear tape open (LTO)-3, which offers a capacity of 400 GB, and super advanced intelligent tape (SAIT), which offers 500 GB.
Consequently, an optical disk system requires many more units of media compared to a disk tape system. This is less of a problem for storage requirements below a terabyte; however, it is problematic for multiple terabytes. For example, a data store based on PDD disks, including a duplicate copy for disaster recovery, requires more than 400 units of media for just 5 TB. This adds to the difficulty of media management and makes for bulky and expensive optical library systems.
What to look for in medical archiving and other data storage systems
For clinics and small- to mid-size hospitals that cannot afford to invest in multiterabyte installations, systems are available that are targeted toward these organizations. Purchasers of these systems should consider several factors when comparing data storage products for medical data archiving, as well as backup and recovery:
Easy setup
For starters, when looking for the right server in a system, it's important to go with an option in which installation and implementation are easy to set up. Often small- to mid-size facilities do not have the time or staff capacity to handle technically complex, less user-friendly server and software products.
Hierarchical storage management and demigration
Look for storage setups that come with easy-to-use graphical user interfaces that allow system administrators to define policies that determine where data files are physically stored. These policies should support hierarchical storage management (HSM) and automatic tape cartridge replication so users can take advantage of both magnetic disk and tape attributes.
Some archive servers provide policy-based HSM on the basis of file demigration. The use of file demigration, rather than classic file migration, means that multiple instances of a file are generated, providing enhanced data protection benefits.
Classic HSM can migrate infrequently used files from magnetic disk to a lower-cost medium, leaving a pointer on the magnetic disk. This approach is less relevant today because of the low cost of disk storage. In addition, it creates problems because the data on magnetic disk must still be backed up.
HSM on the basis of demigration takes a different approach that allows integration of data protection, replacing conventional backup. With the right software, the HSM and data protection can be driven by administrator-defined policies.
On data retrieval, the file is always sought from the fastest physical storage device available. If it is stored on magnetic disk and tape, the file will be read from disk. If the file is available only on tape, it should be easily retrieved from tape.
When choosing the right software, make sure the following on-, near-, and offline levels of file storage hierarchy are supported:
- Online with one copy on RAID and at least one copy on tape
- Near-line with at least one copy on tape within the library
- Offline with one or more copies on tape, all of which have been exported from the tape library
Software is also available in which demigrating (removing) a file from RAID moves it from online to near-line, freeing up space on the magnetic disk. The rules for demigration should be defined by the administrator and based on the time elapsed since a file was written or last read. For example, a file could be demigrated seven days after it was written, if it has not been read in the last 24 hours. Or, if it is then read from tape, it could be restored to RAID and retained online until 24 hours have elapsed without it being read.
When a file is demigrated from RAID and is on either near-line or offline tape, an attribute should be able to be set that indicates that the file is no longer online. This changes network timeout periods to allow retrieval of the file from media with long access times. With some software, it also changes the appearance of a file within Microsoft Windows Explorer and a small clock can be added to the bottom left of the file icon. Make sure administrators are offered the capability to set long access time attributes.
Standard file systems and continuous backup
Other key functions to look for are standard file systems in which a server can work with standard imaging and medical applications without modification. This is frequently offered as a RAID and tape library, presented to users as a single rewritable logical drive letter that also makes sharing the server over a network easy.
Systems offering continuous backup are always recommended when files are continuously written to disk and tape, as they can immediately generate two physical copies of the data. Some approaches on the market totally eliminate the need for a backup window. As noted previously, backup windows are a major limitation in handling medical archiving.
Some archive server products replace conventional backup and automatically produce multiple instances of files and metadata, as defined by administrator policies.
Multiple instances of a file can be created in no more than two steps. First, at the time of writing to the archive, a file can be written to both magnetic disk and tape, providing a continuous backup of files that overcomes the two major limitations of conventional backup: no backup window and during recovery all data is restored. Again, past systems witnessed lost-file system changes on the last backup. Today this is no longer an obstacle. Second, make sure files can be written to additional tape cartridges according to tape replication policies defined by the administrator.
This automatic replication of tape cartridges can be implemented in a way that supports rotation of tape cartridges between an offsite location and the tape library. Unlike replications of the past, there is no longer a need to wait until a tape cartridge becomes full before it can be replicated. Further, when a tape cartridge replica is reintroduced into the library after being retained off-site, make sure you are working with a system that will automatically bring it up-to-date.
WORM and data recovery
Systems with automatic update of tape cartridge replicas can offer additional data protection provided by the automatic generation of tape cartridge replicas. Note that WORM tape support provides a full audit trail of file systems and, when used, a system can meet the most stringent regulatory compliance requirements for digital document retention.
In addition, remember that even when using WORM tape cartridges, files written to the logical drive should be able to be deleted or modified as though they are being written to a standard rewritable magnetic disk. And make sure all versions and all deleted files can be retained on tape and accessed remotely.
Unlike conventional backup, it is easy to test if files have really been written to tape. An individual file can be quickly retrieved from a tape cartridge and opened if using utilities that provide confidence that the system is functioning correctly.
A server that can simultaneously manage both WORM and rewritable tape cartridges in the same tape library is no longer the ideal. Policies that are set by an administrator can define which files are written to which type of tape cartridge. Since WORM is both nonerasable and nonrewritable, WORM tape cannot be reformatted, meaning that after data is written to the tape, it cannot be changed.
However, further data can be appended if the software can manage this nonerasable medium in such a way that files can be deleted within the file system in the normal way. In these cases, the deleted files can be hidden from the normal file system interface and can be retrieved using utilities that can identify and restore all old files.
The capability to identify and retrieve all old and deleted versions is vital when using WORM, and when legal compliance is an issue. It is imperative that any system purchased meets all legal compliance tests and best business practice guidelines, including the ability to meet U.S. HIPAA data authentication requirements.
Additional features to consider
Other features that are available today and should be considered for medical data archiving, backup, and recovery systems are an offering of multiple tape sets and dynamic expansion of tape sets. This enables the administrator to group related files together on the same set of tapes and dynamically expands tape sets, allowing administrators to continuously meet on-, near-, and offline capacity demands.
Further, ask for cartridge-spanning capabilities that allow systems to support storage of large files across multiple tape cartridges. In addition, make sure the system is protected by a safe, robust security model, and be sure all systems work only with standard tape formats or an open standard format that allows the tape cartridges to be read using standard third-party utilities. And finally, another feature to consider is a software-based management console that makes policy entering simple for administrators and includes comprehensive diagnostic functionality, hardware monitoring features, and remote support.
It's never easy picking the perfect solution for data archiving, backup, and recovery, especially in medical environments where these decisions are often based on data that is central to the timely well-being of your customers. With the proper knowledge of what is currently available on the market balanced against the needs of your facility, it will be easy to find a solution that is right for any organization.
By Bob Schaefer
AuntMinnie.com contributing writer
January 5, 2006
Bob Schaefer is chief executive officer at Breece Hill, a Louisville, CO-based information storage and retrieval firm for small- to medium-size businesses. The company can be contacted at 303-664-8200 or via its Web site.
Related Reading
Digital library can ease transition to digital mammo, October 26, 2005
Analog workflow analysis guides RIS/PACS implementation, September 16, 2005
Information life cycle management: A prescription for the healthcare industry, December 31, 2004
Digital archiving ubiquitous in U.S. cath labs, September 7, 2004
MiniPACS networks bring advantages, integration challenges, April 29, 2004
Copyright © 2006 Breece Hill
