orbxfer2 - transport arbitrary files using the orb protocol
orbxfer2 [-p pf] [-w dir] [-v][file ...] orb
orbxfer2 [-p pf] [-S state] [-v] orb
orbxfer2 uses an orbserver to transport files. It has two modes,
transmit and receive. In transmit mode, indicated by files on
the command line or a directory to watch (the -w option), it
reads files, converting them into packets on the output orbserver.
In receive mode, it reads xfer packets from an orbserver and writes
these out to new files with the original base name. It can optionally
preserve directory structure, but all files are written into some
destination directory chosen in the parameter file.
When transferring files with orbxfer2, the preferred configuration is
the following:
-
On the source machine, an orbserver of a size somewhat larger than the
largest file to be sent (although smaller orbs are workable), run under
rtexec.
orbxfer2 may be run at the command line as needed to
put files into the orb; alternatively, orbxfer2 may be run continuously
under rtexec, watching some directory. To send a file, it should be
copied to this watched directory.
-
On the destination machine, orbxfer2 should be run -- continuously --
under rtexec, directly
reading from the source orbserver, and using a state file.
If there's only one reader from the source, that reader should never see
any missing files, as the source orbxfer2 waits for the reader
to catch up. However, if there is more than one reader, it's possible
for packets to fall off the source orb before they're copied by the
destination orbxfer2.
The orbxfer demo is an example of this setup.
-
-w dir
orbxfer2 watches the directory dir; any files which appear in
the directory
are copied to the output orbserver, and then removed from the directory.
If a file does not match the file_match/file_reject parameters from the
parameter file, it remains in the watched directory. The operator
must remove it manually.
-
-S state
The state file contains the pktid and time of the last "final"
packet
for any file read.
Using a state file usually ensures that no
files are missed, and no files are duplicated.
If a state file is not
specified, orbxfer2 (by default) positions itself to the old pktid.
-
directory
This is the directory where output files are written; the default is the current
working directory.
-
starting_position
When orbxfer2 starts and no state file is specified, it positions
itself to this pktid. Valid options are oldest and newest,
as well as specific pktids.
-
preserve_directories
If set to yes (default), orbxfer2 preserves the directory structure
for incoming files, creating any necessary directories below the reception
directory (specified above). Leading slashes are removed before
creating output files.
-
match
-
reject
These regular expressions dictate what packets orbxfer2 receives from the
orbserver.
orbxfer2 creates packets with the srcname pattern
`hostname`/xfer/id/basename
The id is related to the second of the current epoch day, and should
ensure that multiple transmissions of the same file never get mixed
together somehow. The basename is the basename of the file. The
default match pattern matches only these srcnames.
-
wait_match
-
wait_reject
If these are not both blank, orbxfer2 watches the output orb, looking
for clients which match these regular expressions. It computes a
lag parameter; if this parameter is too large, it suspends
transmission of new packets until the readers catch up.
However, if there is no matching reader client on the output orb,
orbxfer2 waits until one appears, sending no packets while waiting.
Specify empty wait_match and wait_reject parameters to avoid this test
altogether. In this case, orbxfer2 does not wait for any readers; this
can lead to lost files because packets may drop
off the output orbserver before the remote reader reads them.
-
test_freq
orbxfer2 checks the lag parameter only every n transmitted packets;
n is test_freq * max_pktid on the orbserver.
-
wait_seconds
When the lag parameter is too large, orbxfer2 sleeps for wait_seconds
seconds, and then checks again.
-
lag_trigger .5
-
lag_reset .4
The lag parameter is an estimate of where the slowest reader is
in the orbserver; zero corresponds to reading the newest packet,
while one corresponds to the oldest packet.
Once the lag exceeds the lag_trigger, transmission is shut down
until the lag falls below the lag_reset parameter.
-
bytes_per_packet
This dictates the maximum size of packets. It should not be
necessary to change this, but if transmission seems to be failing
mysteriously at some packet, changing this to a smaller number
may help.
-
file_match
-
file_reject
These regular expressions may be used to restrict the files actually
transmitted. If file_match is empty or the path exactly matches the file_match
regular expression, and file_reject is empty or the path does not
match the file_reject regular expression, the file is transmitted.
If a directory fails the file_match/file_reject test --
it is not selected -- then orbxfer2 does not descend into that
directory. Directories are never removed by orbxfer2.
% orbxfer2 -v /etc/motd /etc/skel :
orbxfer2: $Revision: 1.7 $ $Date: 2005/06/13 17:43:04 $
orbxfer2: starting '/etc/motd'
orbxfer2: finished '/etc/motd'
orbxfer2: starting '/etc/skel/.profile'
orbxfer2: finished '/etc/skel/.profile'
orbxfer2: starting '/etc/skel/local.cshrc'
orbxfer2: finished '/etc/skel/local.cshrc'
orbxfer2: starting '/etc/skel/local.login'
orbxfer2: finished '/etc/skel/local.login'
orbxfer2: starting '/etc/skel/local.profile'
orbxfer2: finished '/etc/skel/local.profile'
% orbxfer2 -v -w xfer : &
% cp /etc/motd xfer
% orbxfer2: starting 'xfer/motd'
orbxfer2: finished 'xfer/motd'
% orbxfer2 -v :
orbxfer2: $Revision: 1.7 $ $Date: 2005/06/13 17:43:04 $
orbxfer2: Starting at pktid=0
orbxfer2: receiving '/etc/motd'
orbxfer2: received etc/motd (60 bytes, 2 packets)
orbxfer2: receiving '/etc/skel/.profile'
orbxfer2: received etc/skel/.profile (144 bytes, 2 packets)
orbxfer2: receiving '/etc/skel/local.cshrc'
orbxfer2: received etc/skel/local.cshrc (124 bytes, 2 packets)
orbxfer2: receiving '/etc/skel/local.login'
orbxfer2: received etc/skel/local.login (607 bytes, 2 packets)
orbxfer2: receiving '/etc/skel/local.profile'
orbxfer2: received etc/skel/local.profile (582 bytes, 2 packets)
orbxfer2: receiving 'xfer/motd'
orbxfer2: received xfer/motd (60 bytes, 2 packets)
-
Using orbstat to inspect packets
The file HOSTNAME is short, and has only three packets: a header,
the file data, and a trailer. Also shown is the initial emptye packet
which orbxfer2 writes before any file packets.
orbstat -i ./data/forb@f
> 0
#0 'crestone/ch/orbxfer_start': 5/10/2005 (130) 20:09:03.484 : 0 bytes
> hdr
> 6
#6 'crestone/xfer/72544/HOSTNAME': 5/10/2005 (130) 20:09:03.491 : 50 bytes
id = 72544
sequence = 0
filename = data/simple/HOSTNAME
mode = 100664
digest = 30616361356633373132373532323434 ndata = 21
> +
#7 'crestone/xfer/72544/HOSTNAME': 5/10/2005 (130) 20:09:03.491 : 45 bytes
id = 72544
sequence = 1
digest = 0aca5f37127522444ad7672e9e6ed341 ndata = 16
> +
#8 'crestone/xfer/72544/HOSTNAME': 5/10/2005 (130) 20:09:03.491 : 29 bytes
id = 72544
sequence = 2
digest = 30616361356633373132373532323434 *trailer packet*
>
-
waiting for orb readers
-
waited 30.000 seconds for orb readers
orbxfer2 avoids overfilling the orb by watching orb readers
which match (by default) (orb2orb|orbxfer2). If no readers
are present, it stalls until a reader starts. Thereafter,
it stalls if that reader appears to be falling well behind.
orbxfer(1) (the original)
$ANTELOPE/demo/orbxfer
-
orbxfer2 does not save files with filenames which have non-printable
characters.
-
The file basename is part of the orb packet srcname, along with the computer hostname.
Confusion and transmission failure could result if two files with the same name
were transmitted at the same time from the same computer to the same destination orb.
-
There is no attempt to correct for transmission errors, like missing packets or
checksum failures.
-
orbxfer2 overwrites files in the output directory without warning when presented
with an incoming file of the same name. However, new files are first written
to a file with an extra dash "-" suffix; when the file is completely received,
it is renamed to the correct filename.
-
The file_match/file_reject regular expressions are matched against
the complete path (from the starting directory); matching just the
basename might be better. In addition, to be a match, the regular
expression must match the entire path, not just some fragment.
-
Files must be smaller than 2 gigabytes in size.
Daniel Quinlan
Table of Contents
Antelope Release 4.7 Darwin 7.9.0 2005-07-18
Boulder Real Time Technologies, Inc
For more information, contact support@brtt.com