Security aspects gain an increasing importance in the transfer of data via the Internet. Security conscious sites often refuse to provide access via ftp or rcp daemons. So the only remaining mean of data transfer is scp which provides a secure session control and a secure (encrypted) data transfer.
In the Internet security aspects gain increasing importance. Faster computers and more sophisticated software give more and better opportunities to hackers to get illegal access to computers in the Net. So the classical access methods like telnet , rlogin , ftp are no longer adequate. An increasing number of (high performance computing) sites reduce the access to their machines to the Secure Shell (ssh) . By using the Secure Shell and its data transfer tool scp (Secure Copy) all data transfer goes via encryption. The encryption of the user data slows the data transfer significantly down. So the transfer of 10 MB from a Sun to a CRAY-J90 takes 3.3 sec via ftp and 140(!) sec via scp .
By using ssh as session control mean at least the metadata to open and close the session goes via an encrypted (secure) line. The performance of the data transfer can be significantly increased if the user data stream is detached from the control stream and goes via an unencrypted line. This way the user data can be read by software sniffing the Internet, but no information is compromised that increases the chance to break into the hosts transmitting the data.
In this paper we propose dmscp , a UNIX tool that (based on the ideas mentioned above) reaches a performance near to transfer rates of ftp/rcp additionally providing a better success check and a secure session control. dmscp is portable and easy to install by any user without requiring any special system features. A user front end yscp hat shows a behavior similar to scp/rcp is provided.
The solution is a ssh based tool that
The classical data transfer tools and scp are not very useful to pipe data from/to a process at the local host. This is one of the reasons for not using the design of scp/rcp to define the new tool.
The intention was to provide a more common mean to transfer and handle big amounts of data on hosts with data migration ( DMF - SGI/Cray ) that helps to solve several task classes.
Copying data to/from a remote file from/to a local file or a local pipe is one of these task classes.
The basic idea is the remote object . The tool to be defined copies data to (or from) a remote object .
A remote object can be:
With a remote command processing the data, tasks like copying a file tree become possible. A model like
(cd sourcedir/. && tar -cf - sourcedirname) |\ process_on_remote_host(cd destdir && tar -xf - sourcedirname)
By piping data into a data transfer tool it becomes possible to get data streams bigger than the biggest file at the source host.
If the remote (server) host does not have any mean of data migration the amount of transmitted data is limited by the size of the large file system at the server. Not so with data migration, provided the data is moved to an external media during the copy process. But most data migration systems allow to migrate whole files only!
If the data stream is cut into a set of files significantly smaller than the disk cache (system disk base of a file system with data migration) and each file is migrated immediately after closing really large amounts of data can be stored.
On a native UNIX file system a ` disk full ' error is fatal and leads to an immediate abort of the writing process. On a file system with data migration the same situation is not an error at all - it only means that the tape system is slower than the data transfer via the network. It can be expected that disk space will be available soon. So it makes sense for the writing process to wait for disk space to become available and resume the writing.
Classical data transfer tools don't do any checks, they assume the network to work properly. We had to learn (the hard way) that one should not set too much trust into the network. Due to an error in lower network layers data bytes had been exchanged by zeros. The destination files had the same size as the source files but contained somewhere inside a lot of zero bytes.
The most safe way would be not only to compare the sizes of source and destination but also compute and compare their checksums. This is not only an expensive approach, it works only if the data source and destination are files (or fragments). If source or destination are processes (or pipes) you have to trust the operating system (you have to do that anyway :-) ) and check the data at the front to the network - in the transmission buffers.
So the tool to be provided has to
Our dmscp is a tool providing all the above mentioned features. It is designed as back end to build front ends for specific tasks. It runs on most UNIX platforms and provides DMF (SGI/Cray Data Migration Facility) specific functions. It is I/O optimized for UNICOS and IRIX.
dmscp is symmetric, meaning the same process runs at the local and the remote host. The local process invokes the remote process in a way that the remote process knows it is the server.
dmscp is easy to install. Except the Secure Shell daemon (sshd) on the remote host does not need any special system functions or daemons. It can be installed in a system directory or in the data area of the user, any user can install it by himself using a quickinstall script. dmscp only has to be in the search path of the user, a parameter allows to extend the search path before starting the remote dmscp .
dmscp is public domain. You can obtain the source via http://www.zib.de/zibdoc/zib/mass-storage/dmscp/ .
dmscp runs in read or in write mode (to or from the remote host). The (documented) parameters are shown in the following table (for more details see man dmscp past installation):
Every parameter can be at any place in the commandline. If a parameter occurs more than once the last occurrence is significant.
dmscp -checksum -r -s avampire -u bzbstoll \ -prefix 'setenv PATH "$HOME/bin:$PATH"' \ -R 'cd /tmp/ && tar -cf - ./bzbstoll' \ | ( cd /tmp/ && tar -xf - )
( cd /tmp/ && tar -cf - ./bzbstoll ) | \ dmscp -checksum -w -s avampire -u bzbstoll \ -prefix 'setenv PATH "$HOME/bin:$PATH"' \ -R 'cd /tmp/ && mkdir ._tmpdir.123 && cd ._tmpdir.123 && tar -xf - 1>&2 && cd .. && mv ._tmpdir.123/bzbstoll bzbstoll2 && rmdir ._tmpdir.123'
find . -print|cpio -ocv |\ dmscp -w -s server -u user -p 1024 -d /archivedir \ -F 2g -S 0 -T 36000 -f myarchive -FMigrate
dmscp -r -s server -p 1024 -u user -d /archivedir \ -F 1 -f myarchive -frelease -T 360000 |\ cpio -icdv
Several processes are needed to do the job. The initial process has the task to analyze the command line. If it learns that it is the server it does the job itself (see below). If it is the client it forks and the child does the real work. The parent process gathers the reporting messages and checks the success.
ssh user@server "cd destdir && dmscp\ -s -server -f destfile' >/tmp/<named_pipe>
If the server ( remote dmscp ) runs into a fatal error it calls a panic routine . panic cleans up (as good as possible), reports to the client child, closes the data socket and dies. The (client) child process reports to its parent, cleans up and dies. The client parent process reports the error to the user. The panic function in the client child process works similarly.
The network bandwidth of a file transfer program is highly influenced by the tcp windowsize.
(see http://www.psc.edu/networking/perf_tune.html and http://www.dast.nlanr.net/Features/Autobuf/ ) Unfortunately the proposed method to optimize the tcp windowsize does not work on all Unix platforms. So dmscp doesn't contain an automatic network tuning but an option for hand tuning. During installation time the default windowsize can be modified. It is predefined with 64Kbyte. With the option -wsz [k|m]# (example -wsz 8m ) dmscp sets the tcp windowsize.
A thumb rule for a good value is: tcp_windowsize=bandwidth*delay of the network connection.
At Konrad-Zuse-Zentrum dmscp is used for various tasks. First it was used for backup and archival problems not solvable by NetBackup ( Veritas ). Here we need the fragmentation ! The user front end is a set of Korn Shell ( ksh ) scripts using dump/restore, cpio etc. to pack/unpack the data. A secure tool has to be provided because the only way to address the server is the Secure Shell .
Lately we have been forced to decide to cut the telnet etc. lines to our high performance computers (some nice hacker twisted our arm) and found out that scp is not an adequate tool to transfer that kind of data our costumers are dealing with. So we got another dmscp front end on the track.
yscp ( Yet another Secure Copy Program ) does basically the same as scp with the exception that the user data is transmitted without encryption. This way we cannot avoid the overhead of connecting by ssh (in fact yscp connects twice to each remote site) but we gain the performance of ftp/rcp in the data transfer itself, only slightly diminished by the (optional!) checksum computing of dmscp .
yscp is much faster than scp , much more secure than ftp/rcp and safer than all of them thanks to the properties of dmscp .
yscp is a perl program of 500 lines, dmscp is 2700 lines of C code.
Examples (for more details see man yscp):
yscp sourcefile user@server:destdir/destfile
yscp sourcefile user@server:destdir
yscp -r sourcedir user@server:destdir
The following table shows some (wall clock) times in seconds spent on copying 100 MB from a SGI/O2 to some hosts via a fast local network (without any network optimization).
The following table shows some (wall clock) the transmission bandwidth between a Sun,Ultra-60 in the ZIB (hlr.zib.de) and several remote hosts with optimized tcp windowsize
The data migration related features of dmscp are designed for DMF (CRAY/SGI - UNICOS, UNICOS/mk, IRIX) only. A more common approach is to provide a set of parameters to define a command to be executed
Depending on demands from the user community asynchronous I/O would be possible for most platforms by using shared memory, doing the network I/O and the disk (pipe) I/O by different processes synchronized by signals.
1. (ftp://rsync.samba.org/pub/rsync/ ) rsync is a comfortable data transfer program designed to update `mirrors' of file systems. It uses asynchronous I/O and has sophisticated success checks.