SEEK_HOLE and SEEK_DATA: efficiently archive/copy large sparse files
Posted on In QAHow to efficiently archive a very large sparse file, say 1TB? The sparse file may contains a small amount of data, say 32MB.
Table of Contents
SEEK_HOLE and SEEK_DATA
The SEEK_HOLE/SEEK_DATA functionalities play the trick and makes `tar` and `cp` handle the large sparse file very efficiently.
`lseek` with `SEEK_HOLE` returns the offset of the start of the next hole where the offset is greater than or equal to the supplied offset. `lseek` with `SEEK_DATA` sets the file pointer to the start of the next non-hole file region where the new offset is greater than or equal to the supplied offset.
More on SEEK_HOLE:
https://blogs.oracle.com/bonwick/seekhole-and-seekdata-for-sparse-files
http://lwn.net/Articles/440778/
http://lwn.net/Articles/260699/
The SEEK_HOLE is supported from Linux 3.1. From man page for lseek:
Seeking file data and holes Since version 3.1, Linux supports the following
additional values for whence:
SEEK_DATA Adjust the file offset to the next location in the file greater
than or equal to offset containing data. If offset points to data, then the
file offset is set to offset.
SEEK_HOLE Adjust the file offset to the next hole in the file greater than
or equal to offset. If offset points into the middle of a hole, then the file
offset is set to offset. If there is no hole past offset, then the file
offset is adjusted to the end of the file (i.e., there is an implicit hole at
the end of any file).
If you are curious the changes insides of Linux kernel, check fs: handle SEEK_HOLE/SEEK_DATA properly in all fs’s that define their own llseek.
Tools/library support
I dig a little bit in the libarchive source tree and find it is added possibly from this commit:
commit d216d028a78e56a37bab9e42a2f17f28714a6535
Author: Michihiro NAKAJIMA <ggcueroad@gmail.com>
Date: Tue Feb 2 06:09:17 2010 -0500
Determine sparse files through API such as lseek(HOLE).
SVN-Revision: 1856
After that, there are bug fixes. For example:
$ git show b76da87985101f7acdcc0d84490bb4f6a736d210
commit b76da87985101f7acdcc0d84490bb4f6a736d210
Author: Michihiro NAKAJIMA <ggcueroad@gmail.com>
Date: Sat Feb 25 18:38:13 2012 +0900
Fix a wrong check on a result of lseek.
diff --git a/libarchive/archive_read_disk_entry_from_file.c b/libarchive/archive_read_disk_entry_from_file.c
index 8fcd0ab..0fef3c7 100644
--- a/libarchive/archive_read_disk_entry_from_file.c
+++ b/libarchive/archive_read_disk_entry_from_file.c
@@ -1033,7 +1033,7 @@ setup_sparse(struct archive_read_disk *a,
goto exit_setup_sparse;
}
off_e = lseek(*fd, off_s, SEEK_HOLE);
- if (off_s == (off_t)-1) {
+ if (off_e == (off_t)-1) {
if (errno == ENXIO) {
off_e = lseek(*fd, 0, SEEK_END);
if (off_e != (off_t)-1)
I guess that it works starting from v3.0.4 after that bugfix:
$ git show v3.0.4 | head -n4
tag v3.0.4
Tagger: Andres Mejia <amejia004@gmail.com>
Date: Wed Mar 28 09:53:16 2012 -0400
$ git show v3.0.3 | head -n4
commit e235511e964cf8b13bf49a1e343bfdc5c11014da
Author: Tim Kientzle <kientzle@gmail.com>
Date: Fri Jan 13 00:32:07 2012 -0500
Tests on 3.6.5 kernel and 2.6.32 kernel
On Fedora 17 with 3.6.5 kernel:
[zma@office t]$ uname -a
Linux office 3.6.5-1.fc17.x86_64 #1 SMP Wed Oct 31 19:37:18 UTC 201
[zma@office tmp]$ ls -lh pmem-1
-rw-rw-r-- 1 zma zma 1.0T Nov 7 20:14 pmem-1
[zma@office tmp]$ time tar cSf pmem-1.tar pmem-1
real 0m0.003s
user 0m0.003s
sys 0m0.000s
[zma@office tmp]$ time cp pmem-1 pmem-1-copy
real 0m0.020s
user 0m0.000s
sys 0m0.003s
[zma@office tmp]$ ls -lh pmem*
-rw-rw-r-- 1 zma zma 1.0T Nov 7 20:14 pmem-1
-rw-rw-r-- 1 zma zma 1.0T Nov 7 20:15 pmem-1-copy
-rw-rw-r-- 1 zma zma 10K Nov 7 20:15 pmem-1.tar
[zma@office tmp]$ mkdir t
[zma@office tmp]$ cd t
[zma@office t]$ time tar xSf ../pmem-1.tar
real 0m0.003s
user 0m0.000s
sys 0m0.002s
[zma@office t]$ ls -lha
total 8.0K
drwxrwxr-x 2 zma zma 4.0K Nov 7 20:16 .
drwxrwxrwt. 35 root root 4.0K Nov 7 20:16 ..
-rw-rw-r-- 1 zma zma 1.0T Nov 7 20:14 pmem-1
For comparison, on Fedora 12 with a 2.6.32 kernel:
$ du -hs sparse-1
0 sparse-1
$ ls -lha sparse-1
-rw-rw-r-- 1 user1 user1 1.0T 2012-11-03 11:17 sparse-1
$ time tar cSf sparse-1.tar sparse-1
real 96m19.847s
user 22m3.314s
sys 52m32.272s
$ time gzip sparse-1
real 200m18.714s
user 164m33.835s
sys 10m39.971s
$ ls -lha sparse-1*
-rw-rw-r-- 1 user1 user1 1018M 2012-11-03 11:17 sparse-1.gz
-rw-rw-r-- 1 user1 user1 10K 2012-11-06 23:13 sparse-1.tar
$ time rsync --sparse sparse-1 sparse-1-copy
real 124m46.321s
user 107m15.084s
sys 83m8.323s
$ du -hs sparse-1-copy
4.0K sparse-1-copy
Comments and discussions
Well, I can run some virtual machines.
Another crazy thing that I have noticed.
dd if=/dev/zero of=xen-guest.img bs=1 count=0 seek=10G dd if=/dev/zero of=xen-guest.img bs=1 count=1 seek=10G
Notice the difference in the “count” parameter. They will both produce an empty sparse file.
However, only the one with “count=1” will be fast with bsdtar. The other one will take ages.
Btw, I also notices that bsdtar 3.0.3 does not support this at all. You have to use at least 3.0.4 for it to work.
I find it very strange that it’s so hard to find information about this online.
With clouds popping up everywhere, you would think that people would care about copying 200GB sparse files in minutes instead of hours.
This observation is interesting. There are some differences:
$ dd if=/dev/zero of=xen-guest.img bs=1 count=0 seek=10G
0+0 records in
0+0 records out
0 bytes (0 B) copied, 1.2637e-05 s, 0.0 kB/s
$ dd if=/dev/zero of=xen-guest2.img bs=1 count=1 seek=10G
1+0 records in
1+0 records out
1 byte (1 B) copied, 4.2406e-05 s, 23.6 kB/s
If count is 0, 0 byte is copied. While 1 byte is copied if count is 1. And the size of the xen-guest2.img is 1 byte larger than the size of xen-guest.img. There is totally no data in xen-guest.img.
My suspicion is that bsdtar is not designed to handle the situation that the file is totally empty or there is a bug there.
