Overview of Snapshot (Snapshot) technology development

absrtact : The traditional data backup technology has the problem of backup window, recovery time objective RTO and recovery time RPO too long, which can not meet the requirement of data protection for enterprise critical business, thus the data snapshot technology is produced. In this paper, the concept of snapshot technology, characteristics, implementation technology and the status of the development of a general exposition, and the future development of the prospect.

keywords : Snapshot, Backup, copy, mirror, copy on write, pointer remap

Author Profile : Liu, research direction for network storage, data mining and distributed computing; graduated from CAS, currently working for Symantec @symantec, engaged in storage software development. Email:aigui.liu@gmail.com

Note: the author's knowledge and experience level is limited, if there are errors or inappropriate, please criticize.

Download in PDF format: http://download.csdn.net/source/1613486

 

First, Introduction

With the continuous development of computer and network technology, the level of information technology has been improved. After the 21st century of human access to the information society, such as digital communications, digital multimedia, e-commerce, search engines, digital libraries, weather forecasts, geological exploration, scientific research, such as the emergence of massive data-based applications, a variety of information presented explosive growth trend, storage becomes the center of Information Technology. Applications to the storage system requirements continue to improve, storage capacity is constantly upgrading, from gigabyte to terabyte, Petabyte, ExaByte, the more significant. Turing Award winner Jim Gray proposes a new rule of thumb: the amount of data produced every 18 months in a networked environment equals the amount of data ever recorded. At the same time, the modern enterprise's dependence on the computer is greatly enhanced, the information data becomes the foundation that the enterprise relies on, the data corruption or the loss will bring the huge loss to the enterprise. Due to the failure of hackers, viruses, hardware and other natural disasters such as fire, earthquake, and so on, the system and data information is destroyed and even destroyed, if not in time to recover, will cause huge losses to the enterprise, so backup disaster recovery technology is particularly important. In particular, the catastrophic consequences of events such as 9.11 have led to a greater awareness of the value and significance of data and the increasing importance of data protection.

In the past more than 20 years, although the computer technology has made great progress, but the data backup technology has not progressed much. The cost and cost of data backup operations are still high, and they consume a lot of time and system resources, and the recovery time and recovery point objectives for data backup are relatively long. Traditionally, data replication, backup, and recovery techniques have been used to protect important data and to regularly back up or replicate data. Because the data backup process affects application performance and is time-consuming, data backups are usually scheduled to occur when the system load is low (such as at night). In addition, in order to conserve storage space, it is common to combine full-volume and incremental backup technology.

Obviously, there is a significant shortage of such data backup methods, that is, backup window problems. During the data backup period, the enterprise business needs to temporarily stop the external provision of services. As the volume of enterprise data and data growth accelerates, this window may become more and more lengthy, which is unacceptable for critical business systems. such as banks, telecommunications and other institutions, information systems require 24x7 uninterrupted operation, short-term downtime or the loss of a small amount of data can lead to huge losses. Therefore, the Data Backup window needs to be as narrow as possible, or even reduced to zero, data snapshot (Snapshot), Continuous data protection (CDP, Continuous-Protection) and other technologies, is to meet such needs and the emergence of data protection technology.

Second, the concept of snapshot

A snapshot (Snapshot) is an image of a dataset at a particular point in time, also known as an instant copy, which is a fully usable copy of the DataSet. Storage Networking Industry association The SNIA definition of a snapshot is [1]: A fully available copy of the specified data collection that includes an image of the corresponding data at a point in time (the point at which the copy began). A snapshot can be a copy of the data it represents (duplicate), or it can be a replica of the data (replicate).

Snapshots are widely used, for example, as a source of backup, as a source of data mining, as a checkpoint for saving application state, or even as a means of simple data replication. There are many ways to create a snapshot, according to the definition of SNIa [2], the snapshot technology is mainly divided into mirror separation (split mirror), change block (changed block), concurrency (concurrent) three major categories. The latter two are typically implemented using pointer remapping (pointer remapping) and copy on write technology. The flexibility of the changed block approach and the efficient use of storage space make it a mainstream snapshot technology.

The first snapshot is image separation. Building a data mirror before an instant copy, when a full copy of the image is available, it is possible to create an instant copy by "detaching" the mirror instantaneously. The advantage of this technique is that it is fast and no additional work is required to create a snapshot. But the disadvantage is also very obvious, first of all it is not flexible, can not take snapshots at any time; second, it requires a mirrored volume with the same capacity as the data volume; Furthermore, continuous mirroring of data changes affects the overall performance of the storage system.

The second type of snapshot is the change block [3]. After the snapshot is created successfully, the source and destination share the same copy of the physical data until the data is written, and the source or destination is written to the new storage space. A shared data unit is a block, sector, fan path, or other granularity level. In order to record and track block changes and copy information, a bitmap (bitmap) is needed to determine where the actual copied data is located and to determine whether to fetch data from the source or destination.

The third type of snapshot is concurrency [3]. It is very similar to changing blocks, but it is always physically copying data. When an instant copy is executed, no data is copied. Instead, it creates a bitmap that records the replication of the data and makes real data physical copies in the background.

Three, different storage levels of snapshot implementation

"Any problem in computer science can be solved by increasing the level of indirection,"-butler lampson,1992, an annual Turing Award winner. For snapshot technology, we can implement it at different levels in the storage system, in a disk array, file system, Volume Manager, NAS system, or backup system.

Figure 1 Storage System stack and snapshot implementation

The storage stack consists of a set of hardware and software components that provide physical storage media for applications running on the host operating system, as shown in Figure 1. Snapshots can be implemented in many different ways, or at different levels in the storage stack, broadly divided into software and hardware layers, and can be divided into controller-based snapshots and host-based snapshots of the two categories [4].

Controller-based snapshots, implemented in the storage device layer or hardware layer, are managed by the storage-system hardware provider and integrated into the disk array. This snapshot is done at the LUN level (block level), independent of the operating system and the file system. Host-based snapshots, implemented between device drivers and file system levels, are typically performed by a file system, Volume Manager, or third-party software. This snapshot does not rely on storage hardware, but it relies on file system and volume management software. This snapshot acts on a logical view of the data, unlike a controller-based snapshot, which acts with physical data.

In each of the above storage tiers, the physical storage tier and volume manager are the two components that are most suitable for implementing snapshots, and they make it easy to take advantage of physical storage, which is the current mainstream level of implementation. Implementing snapshots at the file system level is a viable option, but applications such as databases directly choose to use logical volumes to implement snapshots because they cannot be managed by snapshot technology at the file system level. In general, there is no need to implement snapshots at the application layer, which can be implemented using the underlying file system or volume Manager interface for backup mechanisms, but requires a temporary pause to ensure snapshot data consistency. In general, software-layer-based snapshots are easy to operate and provide better recovery granularity, while hardware-layer-based snapshots tend to be more performance-and fault-tolerant.

Currently, storage vendors provide storage products that implement snapshots at different levels of storage. Snapshots implemented in the storage subsystem are IBM Totalstorage Disk Systems, NetApp NAS, EMC Symmetrix, implemented in the virtualization layer with the IBM Total Storage SAN Volume Controller, in the volume Volume Manager, Linux LVM, IBM Tivoli Storage manager LVSA, microsoft®windows®2003 VSS system Provider, in the file system Aix JFS2, IBM totalstorage SAN file system, IBM General Parallel file system, IBM N series, NETAPP filers, Veritas file S Ystem.

Iv. methods and techniques of snapshot implementation

Snapshot technology enables instant image of data, and snapshot imagery can support online backup. A full snapshot is a complete read-only copy of all data, and in order to reduce the amount of storage space used by the snapshot, a copy-on-write (COW, copy-on-write) and write redirect (Row,redirect on Write) snapshot technology is proposed. In addition, there are some other methods of snapshot technology, such as log, continuous data protection, etc., can improve the performance of the snapshot.

1. Image separation (split Mirror)

Image splitting snapshot technology before the snapshot point in time, you must first create and maintain a full physical mirrored volume for the source data volume: Two copies of the same data are stored on a mirrored pair consisting of the source data volume and the mirrored volume, respectively. When the snapshot time point arrives, the mirroring operation is stopped, the mirrored volume is converted to a snapshot volume, and a snapshot of the data is obtained. Once the snapshot volume has been applied, such as a data backup, it will resynchronize with the source data volume and become the mirrored volume again. For a source data volume to hold multiple consecutive point-in-time snapshots, multiple mirrored volumes must be created beforehand, and when the first mirrored volume is converted to a snapshot volume as a data backup, the initially created second mirrored volume is immediately synchronized with the source data volume, and the source data volume becomes the new mirrored pair. The mirror split snapshot operation takes a very short time, only the time it takes to disconnect a mirrored volume pair, typically only a few milliseconds, so that small backup windows have little impact on the upper application, but the snapshot technology lacks the flexibility to take snapshots of arbitrary data volumes at any point in time. In addition, it requires one or more mirrored volumes with the same capacity as the source data volume, while synchronizing mirroring also reduces the overall performance of the storage system. [5]

2. Copy-on-write (COW, copy on write)

Figure 2 Copying a snapshot when writing

Copy-on-write snapshots use pre-allocated snapshot space for snapshot creation, and no physical data replication occurs after a snapshot point in time, copying only the metadata of the physical location of the original data. Therefore, snapshot creation is very fast and can be done instantaneously. The snapshot copy then tracks the data changes of the original volume (that is, the original volume write operation), and once the original volume data block is first updated, the original volume data block is read out and written to the snapshot volume, and the original volume is overwritten with the new data block (Figure 2). Copy when writing, hence the name.

This snapshot technique creates a snapshot volume when a snapshot is created, but only allocates a relatively small amount of storage space to hold the updated data from the source data volume after the snapshot point in time. Each source data volume has a data pointer table, and each record holds a pointer to the corresponding data block. When you create a snapshot, the storage subsystem establishes a copy of the pointer table for the source data volume as the Data pointer table for the snapshot volume. When the snapshot point ends, the snapshot establishes a logical copy that is accessible to the upper application, and the snapshot volume shares the same physical data with the source data volume through the respective pointer tables. When a snapshot is created, a write-time replication technique is used to ensure the integrity of the snapshot operations when a data in the source data volume is about to be updated. Access to the data in the snapshot volume, by querying the data pointer table, according to the corresponding data block pointer to determine the physical storage location of the data accessed.

Write-time replication technology ensures that the replication operation occurs before the update operation, so that data updates after the snapshot point in time do not appear on the snapshot volume, ensuring the integrity of the snapshot operations. Copy-on-write snapshots do not consume any storage resources or affect system performance until the snapshot point in time, and it is very flexible to use and can take snapshots of any data volume at any point in time. The length of the backup window generated at the snapshot point in time is linearly proportional to the capacity of the source data volume, typically a few seconds, with little impact on the application, but the storage space allocated for the snapshot volume is significantly reduced; the copy operation occurs only when the source data volume is updated, so the system overhead is very small. However, because the snapshot volume only holds the data that the source data volume is updated, this snapshot technology cannot get a complete physical copy, it is powerless to run into an application that requires a full physical copy, and if the amount of updated data exceeds the reserved space, the snapshot will be invalidated.

3. Pointer remapping (Pointer remapping)

Figure 3 pointer remapping snapshot

This implementation is very similar to write-time replication, except that the first write operation on the original data volume is redirected to the reserved snapshot space. This snapshot maintains pointers and copy data that point to all source data. When the data is rewritten, a new location is selected for the updated data, and a pointer to that data is remapped, pointing to the updated data. If the copy is read-only, then pointers to that data are not modified at all. The redirect write improves snapshot I/O performance by writing the new data directly to the snapshot volume, while updating the bitmap mapping pointers, while the write-time copy requires a read and two write operations, the raw volume data block is read into and written to the snapshot volume, and the updated data is written to the original volume.

It is not difficult to find that the snapshot volume holds the original copy, while the original volume holds the snapshot copy. This causes the data in the snapshot volume to be synchronized to the original volume before the snapshot is deleted, and when multiple snapshots are created, the access to the original data, the tracking of the snapshot volume and the raw volume data, and the deletion of the snapshot become extremely complex. Additionally, the snapshot replica relies on the original copy, and the original copy dataset quickly becomes fragmented.

4. log file Schema (log-structured files architecture)

This form of snapshot technology utilizes log files to record the write operations of raw data volumes. All writes to the original data volume are recorded in the log system, which is equivalent to a snapshot being generated every time the data changes. Therefore, this is very similar to database system transactions or file system logs, where you can recover data from logs or rollback transactions to any reasonable state as needed. Strictly speaking, this method can not be called a snapshot, but it can achieve the goal of the snapshot, many file systems implemented this function, such as ZFS, JFS, EXT3, NTFS and so on.

5. Clone snapshot (copy on write with background copy)

The snapshot mentioned earlier does not basically produce a full snapshot copy, which does not meet the business requirements of a complete physical data copy. Cloning a snapshot produces a mirrored snapshot that is consistent with the source data volume, leveraging the advantages of both write-time and mirrored separation of two snapshot technologies. At a snapshot point in time, it uses a copy-on-write method to quickly generate a snapshot copy, and then starts a copy process in the background to perform block-level data copy tasks for the source data volume to the snapshot volume. Once the replication is complete, a clone snapshot can be obtained by mirroring the detach technique. Cloning a snapshot also inherits the drawback of mirroring the split snapshot, which, in addition to requiring a snapshot volume equal to the capacity of the source data volume, also affects the overall performance of the storage system to some extent.

6. Continuous data protection (continuous protection)

There are common shortcomings in the above snapshot technologies, that is, you cannot create any number of snapshots at any point. Although the log-type snapshot does not have the above shortcomings, it relies on the specific file system and cannot be applied directly to applications that use different file systems, and is powerless for non-file-based data applications.

Continuous data protection [6], also known as continuous backup, automatically continuously captures the changes in the data blocks of the source data volume and records these block versions continuously and completely. Each block change is recorded, generating an instantaneous snapshot, which is different from other snapshot technologies that create snapshots at the snapshot point in time. Because the write operation is saved by the record, it is able to dynamically access the data state at any point in time, provides fine-grained data recovery, can realize instantaneous and immediate recovery, and effectively close the recovery point target. The advantage of data block-level continuous data protection technology is that the coupling with the application is relatively loose, the performance and efficiency is higher, the system runs continuously and there is no snapshot window problem. Its disadvantage is that the storage space requirements are relatively high, which is to limit the data block-level continuous data protection technology widely used in the root cause.

Table 1[4] from different angles to the above several snapshot techniques are analyzed and compared.

	Image separation	Pointer remapping	Copy when writing	Log file	Clone a Snapshot	Continuous data protection
Whether the snapshot depends on the source data volume.	NO The image contains a complete copy of the data	YES Unchanged data is accessed from the source data volume	YES Unchanged data is accessed from the source data volume	YES Unchanged data is accessed from the source data volume	Only Only when a background copy is not completed	YES In addition to containing the source data copy implementation
Space efficiency	NO Require the same capacity storage space for the source data volume	YES Most of the cases require varying data storage space	YES Most of the cases require varying data storage space	YES Requires varying data storage space	NO Require the same capacity storage space for the source data volume	YES Storage space requirements depend on the data and frequency of the stored change data
Source data volume system CPU and I/O load	Low/high Low image separation, high data synchronization before separation	High/none Software snapshot high, hardware-based snapshot No	High/none Software snapshot high, hardware-based snapshot No	High High log time for write operations	Low Typically performed by the storage sub-hardware	Specific implementation of the relevant
Source data Volume Write load	NONE Write load occurs prior to separation	NONE Write direct redirect to new block	High First write generates additional write load	High The write operation must be log	High Write load generated by first write before copy completion	High Each write operation causes the corresponding write operation
Logical data Error protection mechanism	YES The data must be copied from the mirrored volume, the changes are not recorded, the speed is slow	YES Data changes can be rolled back or synchronized to the source data source	YES Data changes can be rolled back or synchronized to the source data source	YES Data changes can be rolled back	YES Snapshots can be reversed, because only changing blocks of data are replicated, faster	YES Data changes can be synchronized to the original copy of the data
Source data Volume physical media failure protection mechanism	YES Mirrored volumes are full replicas	NONE Valid source data volume must exist	NONE Valid source data volume must exist	NONE Valid source data volume must exist	YES Full protection After background copy is complete	Specific implementation of the relevant

Table 1 Snapshot technology at a glance and comparison

V. Case STUDIES

Currently, snapshots are supported by almost all storage-vendor storage products, which is the de facto industry standard for products that do not have snapshot capabilities to compete in the marketplace. Different manufacturers of snapshot implementations vary, each with technical advantages, there are shortcomings, users can be based on the characteristics of the application data to choose different programs. The following is a brief introduction to the snapshot technology of current major storage vendors.

1. EMC TimeFinder

EMC TimeFinder [7][8] provides local storage replication to increase application availability and data recovery speed. Leverage EMC Symmetrix systems to deliver high deployment flexibility and scale-out scalability to meet any service-level requirements. TimeFinder helps businesses perform backups, load data warehouses, and easily provide data for application testing and development without downtime.

TimeFinder only supports mirrored detach snapshots at the earliest, followed by additional write-time replication snapshot support. As a result, it can support both of these snapshot technologies at the moment. TimeFinder mirroring separates snapshots, supports up to 16 copies per production device, TimeFinder copy snapshots on write, and supports up to 128 replicas per production device.

Timderfinder snapshots are implemented at the storage subsystem level, independent of host, operating system, application, and database systems, which can help manage backup windows, enable instantaneous data replication, and effectively reduce the impact of backups on application and host performance.

2. IBM FlashCopy

FlashCopy[4] is a volume-based snapshot technology that is primarily used for point-in-time replication of logical volumes. Flashcopy is a clone snapshot that leverages the advantages of two snapshot technologies for copy-on-write and mirror-based separations. Currently, Flashcopy's background copy operation is optional, which is to support copy-on-write snapshots (that is, do not select replication) to help users make efficient use of storage space. Flashcopy is primarily used in IBM disk array systems, especially in mid-to high-end enterprise application systems.

Select the replication case, which is the standard flashcopy. First, the storage subsystem allocates a snapshot volume with the same capacity as the original volume, and then uses the write-time replication technology to create the snapshot volume and create a bitmap that tracks the replication of the data, a process that is done instantaneously. After that, the background copy process is started, performing a block-level copy of the data from the original volume to the snapshot volume. During replication, data updates to the original volume are processed according to the write-time replication technology, where the data is copied to the snapshot volume before the original volume is updated and the bitmap information is updated. Once replication is complete, a fully available snapshot copy is obtained using the mirror separation technology, which can be used independently for related applications.

3. HDS ShadowImage

ShowImage[9] is an image-separated snapshot scheme based on the storage subsystem that provides instant, non-stop information sharing and access, helps provide decision support, completes testing and development, or optimizes tape backup operations. In addition, ShadowImage enables backups to be made at the same time as production, thereby increasing the availability of revenue-generating applications. Disk-based replication of the ShadowImage software provides very rapid recovery after data corruption.

ShowImage is the data replication technology inside the storage system, the mirroring function of the disk is transparent to the host system, the asynchronous data replication technology does not increase the host I/O response time, and can define the mirror copy relationship between the production data logical volume and the backup logical volume. Asynchronously maintains data synchronization of 2 logical volumes in real time, producing one or more backup mirrored volumes that are identical to the production volume P-vol S-vol, and the data in the backup mirrored volume is identical to the data in the production volume. HDS ShadowImage-generated mirrored volumes can be provided to customers for other application hosts for auditing, analysis, and also for backup recovery of production volumes.

4. NETAPP Snapshot

NetApp Snapshot Technology is implemented based on its WAFL (Write Anywhere file Layout) filesystem. WAFL file system is characterized by: all data blocks (except volume information) can be written to any location, there is no overwrite write operation, because all write operations are written on the new block. A snapshot created by NetApp is a static, read-only WAFL volume view that enables multiple versions of files, directory hierarchies, LUNs, and application data. Snapshot can be created online, with up to 255 per WAFL volume, without impact on application operation and performance. Also, these snapshots are visible to the user and can be user-driven for data recovery.

Figure 4 shows how snapshot works, which is basically the same as a write redirect snapshot. WAFL file system can be understood as a data block tree structure, and its root data structure describes the Inode file information. This inode file information contains a description of all the inode in the file system, which contains metadata information such as free block graphs and idle inode graphs. The figure (a) is visualized as a general picture of the entire file system, and the upper part shows the root data structure. WAFL creates a new data copy snapshot by replicating the root data structure. The root data structure is only 128 bits, and no additional data blocks need to be copied, and a new snapshot consumes almost no additional disk storage space until the user modifies or deletes data from the file system.

Figure 4 NetApp Snapshot principle

5. Veritas Snapshot

Veritas offers a variety of implementations for different levels of snapshots, with two types of snapshots in the volume level (VxVM, Veritas Volume Manager) and file system level (VxFS, the Veritas file systems), broken down into five types of snapshots, are volume-level mirrored detach snapshots, space-optimized snapshots, full instant snapshots, and file system-level file system snapshots, checkpoint snapshots . According to the implementation technology, they can be divided into image separation, copy-on-write, clone snapshot of three types. Here is a brief introduction to two types of snapshots at the file system level.

Figure 5 Veritas Snapshots: File system snapshots and checkpoint snapshots

Volume-level snapshots capture data block changes, while Veritas file system snapshots capture file content and metadata changes. File system snapshots typically occur at the file system operation boundary, and when a file changes, the snapshot automatically senses these changes, taking snapshot operations before or after the file system transaction, not during the transaction. The two file system-level snapshots work as shown in Figure 5.

The file system snapshot (left in Figure 5) is a non-persistent, read-only snapshot of the VxFS file system that uses write-time replication technology and supports only single data volumes. This snapshot does not contain all the file system image data and metadata and cannot be used offline for other application processing. VXFS maintains a memory map that represents the mapping of the block address on the file system volume to the file block address on the snapshot volume, and the snapshot volume holds the image before the file system data and metadata block changes.

The checkpoint snapshot (Figure 5 right) is the same as the snapshot file system, and also uses the write-time replication technique, except that the snapshot is persistent unless it is deleted, that the snapshot is writable, is more widely used than the file system snapshot, and that the snapshot volume is not required to use the file system's own free storage space, which can be applied to the Coordination can be maintained between multiple snapshots.

Vi. Conclusions and Prospects

Snapshot technology is a major innovation in traditional data backup and replication technology, which solves the problem of backup window, effectively brings the recovery time goal and recovery time point target into effect, and becomes the storage industry standard. In this paper, the snapshot in the implementation of the method and related techniques are described in more detail, and briefly introduce the current major storage manufacturers snapshot technology and storage products.

Since the invention of snapshot technology, people have made a lot of significant improvements. The snapshot window is shrinking, from seconds to moments, and you can create snapshots at any time, with finer granularity, increasing volume, improved snapshot performance, reduced impact on hosts and applications, and increased snapshot flexibility, scalability, and manageability. However, there is always no limit to the progress of technology. For current solutions, snapshot technology still has a lot of room for improvement in terms of overall performance, flexibility, and manageability. The most powerful proof is that storage vendors are constantly introducing new snapshot storage products or new versions.

In recent years, the rapid development of storage technology, object-oriented storage, solid state drives, storage virtualization, data deduplication, new storage architectures, technologies and solutions are constantly emerging. These may have a more profound impact on the transformation of snapshot technology, creating a broader space for the development of snapshot technology. We can predict that future snapshot technologies have made significant strides in space efficiency, snapshot object granularity (file, file set), virtualized storage support, and performance and user experience, and will have a broader application perspective in the area of data replication and backup.

Reference Documents:

[1] Snapshot. http://www.snia.org/education/dictionary/s/#snapshot

[2] point in time copy. Http://www.snia.org.cn/dic.php?word=p

[3] Alain Azagiiry, Michael E Factor, Julian Satran. Point-in-time Copy:yesterday, Today and Tomorrow[c]. College Park, usa:the 19th IEEE Symposium on Mass Storage Systems. 2002:259-270.

[4] Snapshot. Http://www.ibm.com/developerworks/tivoli/library/t-snaptsm1/index.html

[5] Geot, Lin an. Analysis and comparison of several mainstream snapshot technologies. Microprocessor, 1th issue 2008.

[6] Wang Shupeng, Yun, CJCP. A review of the development of continuous data Protection (CDP) technology. Information Technology Bulletin, 6th, Volume 6th, 2008.

[7] EMC TimeFinder. Http://china.emc.com/products/detail/software/timefinder.htm

[8] EMC TimeFinder. Http://china.emc.com/collateral/software/data-sheet/1700-timefinder.pdf

[9] HDS ShadowImage. Http://www.hds.com/cn/products/storage-software/shadowimage-in-system-replication.html

[Ten] NetApp Snapshot. Http://www.netapp.com/us/products/platform-os/snapshot.html

[One] Veritas Snapshot. Http://eval.symantec.com/mktginfo/enterprise/yellowbooks/using_local_copy_services_03_2006.en-us.pdf