The dmscp data transmission tool

 

Manfred Stolle

Abstract:




Security aspects gain an increasing importance in the transfer of data via the Internet. Security conscious sites often refuse to provide access via ftp or rcp daemons. So the only remaining mean of data transfer is scp which provides a secure session control and a secure (encrypted) data transfer.
Due to its poor performance scp is not very useful to transfer big amounts of data. In the paper we propose dmscp , a tool that provides a secure session control and an unencrypted data transfer between UNIX machines.

Author:

Manfred Stolle , Tel.: (030) 841 85-310, e-mail: stolle@zib.de

Version:

February 20, 2001

URL:

http://mss.zib.de/mss

 

1. Introduction
2. The Solution
2.1 Remote Objects
2.2 Why Fragmentation?
2.3 `disk full' on a Filesystem with Data Migration?
2.4 Checking the Quality of Data Transfer
3. The Data Transfer Tool dmscp
3.1 dmscp Parameters
3.2 How does dmscp work?
3.3 Optimizing the Network Transfer
3.4 User Front End
4. Performance
5. Future Plans

 

1. Introduction

In the Internet security aspects gain increasing importance. Faster computers and more sophisticated software give more and better opportunities to hackers to get illegal access to computers in the Net. So the classical access methods like telnet , rlogin , ftp are no longer adequate. An increasing number of (high performance computing) sites reduce the access to their machines to the Secure Shell (ssh) . By using the Secure Shell and its data transfer tool scp (Secure Copy) all data transfer goes via encryption. The encryption of the user data slows the data transfer significantly down. So the transfer of 10 MB from a Sun to a CRAY-J90 takes 3.3 sec via ftp and 140(!) sec via scp .

By using ssh as session control mean at least the metadata to open and close the session goes via an encrypted (secure) line. The performance of the data transfer can be significantly increased if the user data stream is detached from the control stream and goes via an unencrypted line. This way the user data can be read by software sniffing the Internet, but no information is compromised that increases the chance to break into the hosts transmitting the data.

In this paper we propose dmscp , a UNIX tool that (based on the ideas mentioned above) reaches a performance near to transfer rates of ftp/rcp additionally providing a better success check and a secure session control. dmscp is portable and easy to install by any user without requiring any special system features. A user front end yscp hat shows a behavior similar to scp/rcp is provided.

2. The Solution

The solution is a ssh based tool that

The classical data transfer tools and scp are not very useful to pipe data from/to a process at the local host. This is one of the reasons for not using the design of scp/rcp to define the new tool.

The intention was to provide a more common mean to transfer and handle big amounts of data on hosts with data migration ( DMF - SGI/Cray ) that helps to solve several task classes.

Copying data to/from a remote file from/to a local file or a local pipe is one of these task classes.

2.1 Remote Objects

The basic idea is the remote object . The tool to be defined copies data to (or from) a remote object .

A remote object can be:

With a remote command processing the data, tasks like copying a file tree become possible. A model like

	(cd sourcedir/. && tar -cf - sourcedirname) |\
	process_on_remote_host(cd destdir && tar -xf - sourcedirname)

would work.

2.2 Why Fragmentation?

By piping data into a data transfer tool it becomes possible to get data streams bigger than the biggest file at the source host.

If the remote (server) host does not have any mean of data migration the amount of transmitted data is limited by the size of the large file system at the server. Not so with data migration, provided the data is moved to an external media during the copy process. But most data migration systems allow to migrate whole files only!

If the data stream is cut into a set of files significantly smaller than the disk cache (system disk base of a file system with data migration) and each file is migrated immediately after closing really large amounts of data can be stored.

2.3 `disk full' on a Filesystem with Data Migration?

On a native UNIX file system a ` disk full ' error is fatal and leads to an immediate abort of the writing process. On a file system with data migration the same situation is not an error at all - it only means that the tape system is slower than the data transfer via the network. It can be expected that disk space will be available soon. So it makes sense for the writing process to wait for disk space to become available and resume the writing.

2.4 Checking the Quality of Data Transfer

Classical data transfer tools don't do any checks, they assume the network to work properly. We had to learn (the hard way) that one should not set too much trust into the network. Due to an error in lower network layers data bytes had been exchanged by zeros. The destination files had the same size as the source files but contained somewhere inside a lot of zero bytes.

The most safe way would be not only to compare the sizes of source and destination but also compute and compare their checksums. This is not only an expensive approach, it works only if the data source and destination are files (or fragments). If source or destination are processes (or pipes) you have to trust the operating system (you have to do that anyway :-) ) and check the data at the front to the network - in the transmission buffers.

So the tool to be provided has to

3. The Data Transfer Tool dmscp

Our dmscp is a tool providing all the above mentioned features. It is designed as back end to build front ends for specific tasks. It runs on most UNIX platforms and provides DMF (SGI/Cray Data Migration Facility) specific functions. It is I/O optimized for UNICOS and IRIX.

dmscp is symmetric, meaning the same process runs at the local and the remote host. The local process invokes the remote process in a way that the remote process knows it is the server.

dmscp is easy to install. Except the Secure Shell daemon (sshd) on the remote host does not need any special system functions or daemons. It can be installed in a system directory or in the data area of the user, any user can install it by himself using a quickinstall script. dmscp only has to be in the search path of the user, a parameter allows to extend the search path before starting the remote dmscp .

dmscp is public domain. You can obtain the source via http://www.zib.de/zibdoc/zib/mass-storage/dmscp/ .

3.1 dmscp Parameters

dmscp runs in read or in write mode (to or from the remote host). The (documented) parameters are shown in the following table (for more details see man dmscp past installation):

-r|-w

read mode or write mode

-encrypt

If the remote object is not a command the encrypted data transfer is supported

-p port

the sshd port on the remote host

-s server

the remote server (DNS name)

-u user

the remote user

-checksum

optional checksums are computed and compared

-f remote_file

name of the remote object (file or set of fragments)

-f remote_file

tcp windowsize in byte (leadying k or m is possible)

-F fragmentsize

max. size of a fragment (turns fragmentation on)

-S stubsize

max. size of the first fragment (default 4KB)

-force

overwrite existing fragments

-T timeout

timeout for waiting for I/O etc. (default 7200 sec)

-d remotedir

directory to be set at the remote host before the remote dmscp is started

-prefix command
(not documented)



a shell command to be invoked before dmscp is started
examples:
-prefix 'PATH="$HOME/bin:$PATH"
-prefix 'setenv PATH "$HOME/bin:$PATH"'
Be careful with this parameter!

-R command

remote command reading the data from stdin (write mode) or writing it to stdout.

-[w][f]migrate

Migrate the remote object (dmput)

f: Migrate each fragment immediately after file close

w: Wait for migration to be completed

-[w][f]release

In addition to -[w][f]Migrate the disk cache is purged

Every parameter can be at any place in the commandline. If a parameter occurs more than once the last occurrence is significant.

Examples :

 
	dmscp -checksum -r -s avampire -u bzbstoll \
	-prefix 'setenv PATH "$HOME/bin:$PATH"' \
	-R 'cd /tmp/ && tar -cf  - ./bzbstoll' \
	| ( cd /tmp/ && tar -xf  - )
 
 
	( cd /tmp/ && tar -cf  - ./bzbstoll ) | \
	dmscp -checksum -w -s avampire -u bzbstoll \
	-prefix 'setenv PATH "$HOME/bin:$PATH"' \
	-R 'cd /tmp/ && mkdir ._tmpdir.123 && cd ._tmpdir.123 && tar -xf  - 1>&2 && cd .. &&  mv ._tmpdir.123/bzbstoll bzbstoll2 && rmdir ._tmpdir.123'
 
 
	find . -print|cpio -ocv |\
	dmscp -w -s server -u user -p 1024 -d /archivedir \
	-F 2g -S 0 -T 36000 -f myarchive -FMigrate
 
 
	dmscp -r -s server -p 1024 -u user -d /archivedir \
	-F 1 -f myarchive -frelease -T 360000 |\
	cpio -icdv
 

3.2 How does dmscp work?

Several processes are needed to do the job. The initial process has the task to analyze the command line. If it learns that it is the server it does the job itself (see below). If it is the client it forks and the child does the real work. The parent process gathers the reporting messages and checks the success.

 
	ssh user@server "cd destdir && dmscp\
	-s -server -f destfile' >/tmp/<named_pipe>
 

If the server ( remote dmscp ) runs into a fatal error it calls a panic routine . panic cleans up (as good as possible), reports to the client child, closes the data socket and dies. The (client) child process reports to its parent, cleans up and dies. The client parent process reports the error to the user. The panic function in the client child process works similarly.

3.3 Optimizing the Network Transfer

The network bandwidth of a file transfer program is highly influenced by the tcp windowsize.
(see http://www.psc.edu/networking/perf_tune.html and http://www.dast.nlanr.net/Features/Autobuf/ ) Unfortunately the proposed method to optimize the tcp windowsize does not work on all Unix platforms. So dmscp doesn't contain an automatic network tuning but an option for hand tuning. During installation time the default windowsize can be modified. It is predefined with 64Kbyte. With the option -wsz [k|m]# (example -wsz 8m ) dmscp sets the tcp windowsize.

A thumb rule for a good value is: tcp_windowsize=bandwidth*delay of the network connection.

3.4 User Front End

At Konrad-Zuse-Zentrum dmscp is used for various tasks. First it was used for backup and archival problems not solvable by NetBackup ( Veritas ). Here we need the fragmentation ! The user front end is a set of Korn Shell ( ksh ) scripts using dump/restore, cpio etc. to pack/unpack the data. A secure tool has to be provided because the only way to address the server is the Secure Shell .

Lately we have been forced to decide to cut the telnet etc. lines to our high performance computers (some nice hacker twisted our arm) and found out that scp is not an adequate tool to transfer that kind of data our costumers are dealing with. So we got another dmscp front end on the track.

yscp ( Yet another Secure Copy Program ) does basically the same as scp with the exception that the user data is transmitted without encryption. This way we cannot avoid the overhead of connecting by ssh (in fact yscp connects twice to each remote site) but we gain the performance of ftp/rcp in the data transfer itself, only slightly diminished by the (optional!) checksum computing of dmscp .

yscp is much faster than scp , much more secure than ftp/rcp and safer than all of them thanks to the properties of dmscp .

yscp is a perl program of 500 lines, dmscp is 2700 lines of C code.

 

Examples (for more details see man yscp):

	yscp sourcefile user@server:destdir/destfile
	yscp sourcefile user@server:destdir
	yscp -r sourcedir user@server:destdir

4. Performance

The following table shows some (wall clock) times in seconds spent on copying 100 MB from a SGI/O2 to some hosts via a fast local network (without any network optimization).

destination platform

yscp with checksum

yscp without checksum

scp

rcp

rsync 1
via ssh

Sun,Ultra-60

56

20

135

20

145

SGI/O200

49

45

133

27

148

CRAY/T3E

68

43

379

32

729

Cray/J90

108

45

1480

34

600

The following table shows some (wall clock) the transmission bandwidth between a Sun,Ultra-60 in the ZIB (hlr.zib.de) and several remote hosts with optimized tcp windowsize

destination platform

tcp windowsize of best bandwidth

[KByte]

bandwidth of dmscp without checksum

[MB/sec]

bandwidth of ftp

[MB/sec]

avampire.zib.de / SGI O2

65

6.1

5.6

berte.zib.de / Cray T3E

260

1.1

0.5

komma.zedat.fu-berlin.de /
SGI Origin 3000

65

2.0

1.4

r8000.lrz-muenchen.de /
Hitachi SR8000

260

1.7

1.0

crashi.rz.uni-kiel.de /
Cray SV1

65

1.1

0.4

t3e.rrzn.uni-hannover.de /
Cray T3E

260

1.3

0.5

5. Future Plans

The data migration related features of dmscp are designed for DMF (CRAY/SGI - UNICOS, UNICOS/mk, IRIX) only. A more common approach is to provide a set of parameters to define a command to be executed

Depending on demands from the user community asynchronous I/O would be possible for most platforms by using shared memory, doing the network I/O and the disk (pipe) I/O by different processes synchronized by signals.


1. (ftp://rsync.samba.org/pub/rsync/ ) rsync is a comfortable data transfer program designed to update `mirrors' of file systems. It uses asynchronous I/O and has sophisticated success checks.



Copyright © 2000, Konrad-Zuse-Zentrum fuer Informationstechnik Berlin (ZIB). All rights reserved.