orb2db - create continuous database from orb ring buffer
orb2db [-a wffile] [-c chanmatch]
[-m srcmatch] [-r srcreject]
[-p pf-file] [-S state-file] [-s datatype]
[-t time-tags] [-T tolerance] [-w wfname] [-v]
orb db [start-time [window]]
orb2db copies data from the ring buffer orb to a continuous
database db. A starting time may be specified, otherwise
orb2db attempts to coordinate with an existing database, starting
after the earliest of the maximum endtimes for each station/channel
in the database.
orb2db reads packets from the ring buffer and attempts to fit them
into continuous waveform files, typically on day boundaries. It expects
packets to fit together; i.e., the packet times and number of
samples should fit into a continuous waveform file with a single
sample rate. This presumption is not always true: data loggers often
have inconstant clocks, packets may be lost, connections may go up
and down. For these and other reasons, there may be multiple wfdisc
records for a channel in a single day, though in the best case
there would be only one.
When starting against an existing database, orb2db attempts to begin
saving data immediately after the last data for that channel for that
day in the existing database. This avoids overlapping data, but
doesn't require a complicated state file, as it's ok to start a little
before the time where orb2db previously stopped.
In addition to copying data, orb2db
accumulates some statistics about packet errors into
the gap, changed, retransmit and ratechange tables.
orb2db expects the sample rate from the packet to be approximately
correct, in the sense that the time tags of adjacent packets are within
1/2 sample period of the time calculated from the previous time tag and
number of samples and the sample rate. It attempts to handle a clock
(or data logger like the Quanterra) where the actual sample rate is
slightly different from the rate specified in the packet, by
calculating a rate based on the time tags and requiring that the time
calculated with this rate and the number of samples match the actual
time tag to within 1/2 a sample period. New wfdisc rows are generated
when this criterion fails.
orb2db is continuously modifying a set of wfdisc records (usually
for the current day) as it runs.
As with any Datascope database, no other process should be
modifying the output database while orb2db is running. However,
in the real time system, it is useful to be able trim this output
database by deleting rows representing older data. This trimming
improves performance of other programs running against the database.
orb2db implements a simple mechanism for accomplishing this trimming
without shutting down and restarting orb2db. A special file is
created named after the output database: database.MSGFILE.
The first 8 bytes of this file contain a flag which is
set while a cleanup is performed, and an integer count of
cleanups performed.
orb2db tests the flag and the count at every packet;
when the flag is set, or the count has changed, orb2db closes its open database, and
waits for the flag to be reset.
When the flag is reset, it
reopens the output database, and
finds the new record numbers for the records it is modifying.
The MSGFILE may be monitored or modified using the program orb2db_msg(1).
Because there is no acknowledgement (from orb2db that it has paused),
it is important
to allow sufficient time for orb2db to stop itself, before beginning to
modify the database. orb2db_msg sleeps for some period of time before
returning, in an attempt to assure this time window.
In addition, the time required to
clean up the database must be considerably less than the time allowed
by the ring buffer size, so that no data is lost.
For certain data loggers and transmission schemes, data packets
may arrive in the orb out of time order. Duplicate packets may
be present, and some packets may be lost completely.
orb2db has an option to reorder these out of order packets and
eliminate duplicates before writing waveforms to disk. This results
in cleaner databases, with fewer wfdisc records and fewer overlaps.
When performing this reordering, orb2db keeps a stack of the data from
multiple packets for each channel. The size of these stacks is dictated
by the max_out_of_sequence parameter.
When max_out_of_sequence is zero,
no reordering is performed, and almost every packet received is written to
the disk. (The exception is at startup; at startup, orb2db
tries to match the start time for each wfdisc record with the previous
end time for that channel, eliminating any overlap).
When reordering packets, the max_out_of_sequence parameter dictates the
size of each stack, and therefore the maximum number of packets which
orb2db waits for a single late packet. (If there are multiple
late packets, then it may wait longer, until the stack is full).
Using the max_out_of_sequence parameter significantly increases the
overhead and memory usage of orb2db. Each channel of data typically
requires a 4096 byte buffer for writing the waveforms, plus other overhead.
Adding a stack adds
(4 bytes * #samples) + some overhead
For example, for max_out_of_sequence=50, packets with 100 samples each,
that's about 500 bytes * 50/stack,
or 50 kbytes for each channel. Compare this with the original 5 kbytes
per channel for standard processing.
There is also a startup cost. orb2db knows a packet is late
when it sees a packet with a time later than it expects based on the
previous packet. When there is no previous packet, it waits until the
entire stack is full before concluding that no previous packet is coming.
So, using the example above with a stack of 100 1 second packets, it
would wait for 100 1 second packets before starting to write data.
In addition to arriving late, some packets may be duplicated.
It's easy to detect a duplicate packet if its duplicate is still
in the stack; oftentimes, however, the previous copy is already
written to disk. orb2db does not keep old copies around. In addition,
sometimes the data loggers clocks misbehave, and the packet times may
then jump around: sometimes by a second or two, sometimes by days,
months or years.
orb2db differentiates the following possibilities:
-
packet time seems to fit within the time range of the previously written packet:
presume there's a timing error, or q3302orb has been restarted
and start a new wfdisc.
-
packet time is within the estimated time range for a full stack:
ignore packet as duplicate
-
packet time is older:
start new wfdisc.
During startup, orb2db uses the wfdisc table as a state file, and
tries to seamlessly start recording new waveforms at the last point
written previously. Because the previous shutdown may have been untidy,
and because of compression and buffering, the endpoints may be ragged.
orb2db attempts to detect various error conditions like waveform files
which don't match the wfdisc entry, and various kinds of errors in
the wfdisc like impossible nsamp or foff values, or unlikely time ranges.
If the time range is greater than the specified
preferred_waveform_file_range in the parameter file, or (if too_new is specified)
endtime is in the future by at least the parameter too_new,
then the wfdisc record is ignored,
but endtime is set to time - 1/samprate.
-
-a wffile
write a log file which announces waveform files opened and closed;
each line contains a leading open or close, followed
by the name of the waveform file.
When orb2db quits, a single line finished is written rather
than a close line for every open waveform file.
-
-c chanmatch
As each packet is unstuffed, the complete net_sta_chan_loc
name for each channel in the packet is composed and
compared to this regular expression chanmatch.
Only channels which match the regular expression
are accumulated and written to the output database.
This option is for separating out certain channels from
multi-channel packets, and is of limited utility; generally,
you would use the -m and -r
options to select networks and stations of interest.
-
-C
Print additional information when a packet causes a new wfdisc
to be generated.
-
-m srcmatch
Only packets which match the regular expression srcmatch
are requested from the orb server.
-
-r srcreject
packets which match the regular expression srcreject
are not collected from the orb server.
-
-p pf-file
Specify an alternative parameter file name pf-file
instead of orb2db.pf.
-
-S statefile
orb2db saves the pktid of the latest packet it processed
in this file when it quits abnormally
-
-s datatype
Specify the output datatype; this is normally done in the parameter
file, but may be overridden here.
-
-t file
As a debugging tool, record each packet into a time tag file
as data from the packet is saved.
-
-T tolerance
Specify alternate tolerance for deviation of packet
time tags from wfdisc straight line, in sampling periods.
The default is 0.5 periods, as specified in trdefaults.pf.
Making this larger can result in fewer wfdisc records, renders
the (calculated) timing information interior to a single wfdisc
waveform segment less accurate.
-
-v
Be more verbose; -vv shows every packet read.
-
-w wfname
see trwfname(3) -- this allows choosing the output
file naming convention.
Note that delayed packets (srcname has "/@" appended) are
saved into delayed waveform files, which similarly have "@" appended.
Similarly, they have independent wfdisc records.
-
start-time
-
window
A time range may be specified. The first parameter is the time
in any of the formats accepted by epoch(1). The second may be
either an ending time, or a time-window. A time window is most
conveniently specified as hours and minutes like: hh:mm or
hh:mm:ss. The parameter file specifies a maximum acceptable
time window (default is 24 hours); longer time windows are usually
command line errors, but a longer time window can be specified using an
end time rather than a window.
-
max_out_of_sequence
orb2db waits up to max_out_of_sequence packets for a retransmitted,
out of order packet. This parameter should be zero if your data stream
does not contain out of order packets.
When this is non-zero,
out of order packets which are later than
max_out_of_sequence are discarded.
-
preferred_waveform_file_range
Waveforms for a single net/sta/chan are stored in a single
file, covering this range of seconds. If the waveform
is discontinuous (has gaps), multiple wfdisc records may be created,
depending on the setting of max_gap_to_fill below.
(foff is set to the appropriate value). Note that when the
output data type is miniseed (sd), every waveform segment
occupies at least one miniseed data block. If there are lots of
small segments, miniseed data compression does not save space.
-
preferred_waveform_file_offset
waveform volumes start and end at a range boundary + offset
So, if you desire to have day volumes, but to make the day volumes
correspond to local time rather than UTC, you might specify offset
as the local offset in seconds from UTC.
-
max_gap_to_fill
This defines a small gap.
Small gaps are filled with
a special missing data value (see trdefaults.pf.5).
Gaps larger than max_gap_to_fill samples
are not marked, but cause a new wfdisc record to
be generated.
However, gaps which have fewer than
1/2 the number of samples in a packet are not filled. This is to avoid
hiding extremely small gaps, like a single sample gap. Such gaps
are likely to indicate a data logger problem.
-
min_gap_to_fill
However, some dataloggers at low sample rates can
create packets with small gaps. This parameter allows overriding
the default behavior, and fill even 1 sample gaps. Set min_gap_to_fill
to zero if you know the datalogger behaves this way.
-
max_open_files
This limit is only used if the system call getrlimit(2) fails. In
Solaris, this limit is usually 1024 file descriptors; orb2db needs
some of these for standard i/o, but should be able to handle
at least 1000 separate channels, if the cpu and disk can keep up.
-
max_pkt_period
When a start time is specified on the command line, the orbafter call
used to initially position the ring buffer specifies the command line start
time minus max_pkt_period, so that packets containing the desired
time are not skipped.
-
datatype
data format for waveform files; some typical choices are
sd for steim 2 compression, s4 for 32 bit integers,
and as for ascii.
For continuous data, sd is the usual choice.
For triggered and/or event data, it's probably desirable to
choose an uncompressed format (like s4) and set
the parameter waveform_buffer_size to 0. Otherwise, orb2db
may hold data internally until the next event causes it to
be flushed to disk.
There are a host of other datatypes, described in
either the css schema datatype field (use dbhelp(1)),
or in the trdefaults.pf parameter file, or in wftypes(5).
You may even add your own new data type; see addwf(3) and
examples_c_addwf(5).
-
waveform_buffer_size
This is the size of the data buffers used (per channel) when the data type
is other than sd (steim compressed data). Usually it is useful to buffer
many packets worth of data before writing the data to disk, to avoid the large
overhead of many system calls. This buffer size
should probably be comparable with the system page size.
-
max_time_waiting
When collecting data for a specified time period, it's possible for
stations to go off line. orb2db can then hang for a long period
waiting for the data, or for a packet later than the requested time
from that station. This sets a limit on how long to wait.
-
decent_interval
This is the interval at which orb2db checks for channels with old
(perhaps triggered) data. Any channel where the newest data point
is older than the most recent packet time - max_time_waiting is flushed
to disk.
In addition, the state file is refreshed at this interval.
Some kinds of problems can be quietly saved into the database tables
changed, retransmit, ratechange and gap. The parameters below allow
also printing error messages at regular intervals beyond a particular
threshold error rate.
-
chatter_limit
each type of error message is printed at most once per
chatter_limit seconds.
-
min_problem_count
-
min_problem_time
error messages are output only when there are at least min_problem_count
problems within min_problem_time
-
max_window
If a time range (rather than an end time) is specified on the command line,
then that range must be less than this parameter; the default is 24 hours.
Longer time windows are usually
command line errors.
The following parameters affect whether more detailed information about packet anomalies
is saved into database tables. The tables make more detailed analysis possible, but when
transmissions are poor, or clocks wander a lot, the tables may grow unreasonably large
unreasonably quickly. The default is not to write into the tables.
-
record_changed
record records to the changed table when the samprate, calib or tick registration
specified in the packet changes.
-
record_ratechange
add records to ratechange table when the observed sample rate changes. The sample
rate is calculated from the difference between packet time tags and the number of samples
in a packet.
-
record_gap
add records to the gap table when a waveform has a gap; the gaps can be
determined from the waveforms directly, but this is tedious because the
waveforms may have small gaps indicated internally by a missing data value.
When a packet arrives out of time order, it is presumed to represent a retransmission
of the packet. There are several categories of retransmission; recording each type
may be individually suppressed.
-
record_normal_retransmission
These are the normal out of time order packets, presumably retransmitted
because they were missing in the original data stream.
-
record_duplicate_retransmission
These are duplicates of a previously seen packet.
-
record_overlapping_packets
These are packets where the time frame overlaps (but is not the same as) another packet,
and probably indicate some problem with the data logger or clock.
-
record_rejected_packets
These are packets which arrive so far out of time order they are dropped;
they may indicate that the max_out_of_sequence parameter above is too small.
-
pause_timeout
maximum time (in seconds) to wait after pause for the matching
continue. This is a failsafe, in case the program issuing the
original pause fails to deliver a continue for some reason. The
period should probably be less than the ring buffer size, and
definitely greater than the worst case performance of any backup or
cleanup script. Having this number too short risks corrupting the
output database and causing an awful mess. Having this number too
long risks losing some data.
-
too_old
Some positive time difference between now and the packet time may
be specified here; if a packet is older than now() - too_old, the
packet is not saved to the database. However, it can optionally
be saved to the discards file.
The time can be straight seconds, or
anything understood by str2epoch (e.g., "72:00" is 72 hours or 3 days).
-
too_new
Some positive time difference between the packet time and now may
be specified here; if a packet is newer than now() + too_new, (i.e., it's
from the future) the packet is not saved to the database.
However, it can optionally
be saved to the discards file.
The time can be straight seconds, or
anything understood by str2epoch (e.g., "72:00" is 72 hours or 3 days).
-
discards
A file may be specified here, relative to the output database directory,
where packets that are too old or too new are saved in forb(5) format.
This might facilitate debugging the data logger problem implied by packets
from the distant past or far future. It might even allow recovery of the
improperly time tagged data.
The following parameters relate to a check for space
available on the output waveform filesystem.
-
path
This must be a path to the filesystem where waveforms are written. The
default is ".", but you must change this if your waveforms are kept
on a different filesystem.
-
min_freespace
This is the minimum space in megabytes which must be free; otherwise
orb2db pauses until more space is available.
-
frequency
Free space is checked every frequency packets. min_freespace and frequency
should be adjusted to ensure that the filesystem doesn't fill up between
checks; if it does, orb2db dies.
-
sec
Once orb2db has stopped for lack of space, it checks the filesystem every
sec seconds to see if more space has become available.
orb2db adds records to the wfdisc table, and optionally to the gaps, retransmit, ratechange,
and changed tables. It also creates waveform files, following the defaults dictated by trwfname(3)
and trdefaults.pf(5) or the command line argument. Waveform files corresponding to delayed
packets are saved into files with the same path, but with "@" appended.
see antelopeenv(5)
Continuously collect data from the BHZ channel of station RDM from a
ring buffer on XYZ to a database rdm.
% orb2db -m '.*RDM.*' -c '.*_RDM_BHZ' XYZ rdm
Collect data for the last 10 minutes
from the BHZ channels of stations from a ring buffer on XYZ to a
database cels.
% orb2db -c '.*_BHZ' XYZ cels '1996298 15:37' -0:10 0:10
Generally, orb2db soldiers on in the face of errors; however,
it gives up when writes or database updates start to fail.
-
"Can't close 'wfdisc->path'."
-
"Couldn't save pktid #pktid database packet for 'srcname'"
-
"Can't create new wfdisc record"
-
"Can't create new waveform data file"
-
"Can't open wfdisc->path to write trace data."
-
"Can't open output database database"
-
"Can't open output table 'database'"
Something is amiss in the output database.
-
"input data for pktid #pktid (net_sta_chan) was clipped to fit into output format"
-
"truncation to integers lost resolution during conversion to output format"
-
"unknown return code unstuff from unstuffPkt for pktid=pktid at s=strtime(pkttime)"
unstuffPkt(3) was unable to unstuff the specified packet.
-
"Can't read parameter file"
-
"Can't open ring buffer 'orbname'"
The orbserver(1) may not be running, or may be rejecting connections
from this machine.
-
"orbselect 'match' failed"
-
"orbreject 'reject' failed"
Something is wrong with the regular expression match or reject, and
the calls to orbselect(3) or orbreject(3) failed.
-
"orbget to get current server time fails"
-
"orbafter to strtime(params.from) failed"
-
"couldn't compile pattern 're'"
Something is wrong with the regular expression re.
orbmonrtd(1)
orbstat(1)
orb2db_msg(1)
Because orb2db keeps many files open, it's often necessary to raise the limit
of open files; try unlimit descriptors
If the start and stop times are more granular than the packet size,
the actual start and stop times vary from the specification; Data
gaps can also cause start and stop times which vary somewhat from the
spec.
orb2db does not detect any problem
when given a source match which doesn't select any sources; it waits
forever for a source matching the regular expression, even if
there is an explicit ending time.
orb2db should be run so that its database and its waveform files
are on local, not nfs-mounted partitions. Having the waveforms
on an nfs-mounted partition is particularly troublesome when the partition fills
up.
Daniel Quinlan
Boulder Real Time Technologies, Inc.
Table of Contents
Antelope Release 4.10 Mac OS X 10.4.11 2008-05-02
Boulder Real Time Technologies, Inc
For more information, contact support@brtt.com