NAME

cdorb2db - create continuous database from cd1 packets

SYNOPSIS


cdorb2db [-a wffile] [-c chanmatch]
       [-m srcmatch] [-r srcreject]
       [-p pf-file] [-S state-file]
       [-t time-tags] [-w wfname] [-v]
           orb db [start-time [window]]

DESCRIPTION

cdorb2db copies data from the ring buffer orb to a continuous database db. A starting time may be specified, otherwise cdorb2db attempts to coordinate with an existing database, starting after the earliest of the maximum endtimes for each station/channel in the database.

cdorb2db reads cd1 packets from the ring buffer and attempts to fit them into continuous uncompressed four byte integer waveform files, typically on day boundaries. It requires that packets fit together; ie, the packet times and number of samples should fit into a continuous waveform file with a single sample rate. This presumption may not be true: data loggers often have inconstant clocks, packets may be lost, connections may go up and down. However, cdorb2db ignores any problems and writes all data into a single volume.

The uncompressed waveform files may be compressed at a later date -- after all the data from the channel has been acquired -- using db2msd(1).

In addition to copying data, cdorb2db accumulates some statistics about packet errors into the gap, changed, retransmit and duplicate tables.

cdorb2db is continuously modifying a set of wfdisc records (usually for the current day) as it runs. As with any Datascope database, no other process should be modifying the output database while cdorb2db is running. However, in the real time system, it is useful to be able trim this output database by deleting rows representing older data. This trimming improves performance of other programs running against the database.

cdorb2db implements a simple mechanism for accomplishing this trimming without shutting down and restarting cdorb2db. A special file is created named after the output database: database.MSGFILE. The first 8 bytes of this file contain a flag which is set while a cleanup is performed, and an integer count of cleanups performed. cdorb2db tests the flag and the count at every packet; when the flag is set, or the count has changed, cdorb2db closes its open database, and waits for the flag to be reset. When the flag is reset, it reopens the output database, and finds the new record numbers for the records it is modifying.

The MSGFILE may be monitored or modified using the program orb2db_msg(1). Because there is no acknowledgement (from cdorb2db that it has paused), it is important to allow sufficient time for cdorb2db to stop itself, before beginning to modify the database. orb2db_msg sleeps for some period of time before returning, in an attempt to assure this time window. In addition, the time required to clean up the database must be considerably less than the time allowed by the ring buffer size, so that no data is lost.

cd1 data packets may arrive in the orb out of time order. Duplicate packets may be present, and some packets may be lost completely. cdorb2db simply writes packets into the waveform files as they are read from the orb, backing up in the orb as necessary.

OPTIONS

PARAMETER FILE

Some kinds of problems can be quietly saved into the database tables changed, retransmit, ratechange and gap. The parameters below allow also printing error messages at regular intervals beyond a particular threshold error rate.

The following parameters affect whether more detailed information about packet anomalies is saved into database tables. The tables make more detailed analysis possible, but when transmissions are poor, or clocks wander a lot, the tables may grow unreasonably large unreasonably quickly. The default is not to write into the tables.

FILES

cdorb2db adds records to the wfdisc table, and optionally to the gaps, retransmit, ratechange, and changed tables. It also creates waveform files, following the defaults dictated by trwfname(3) and trdefaults.pf(5) or the command line argument.

ENVIRONMENT

see antelopeenv(5)

EXAMPLE

Continuously collect data from the BHZ channel of station RDM from a ring buffer on XYZ to a database rdm.


% cdorb2db -m '.*RDM.*' -c '.*_RDM_BHZ' XYZ rdm

Collect data for the last 10 minutes from the BHZ channels of stations from a ring buffer on XYZ to a database cels.


% cdorb2db -c '.*_BHZ' XYZ cels '1996298 15:37' -0:10 0:10

DIAGNOSTICS

Fatal Errors

Generally, cdorb2db soldiers on in the face of errors; however, it gives up when writes or database updates start to fail.

Non-Fatal Errors

Command Line errors and Initialization errors

SEE ALSO

db2msd(1)
orbmonrtd(1)
orbstat(1)
orb2db_msg(1)
orb2db(1)

BUGS AND CAVEATS

cdorb2db has many fewer tests and checks than orb2db(1). It presumes that the datalogger clocks are correct and that every packet contains the right time, and the clock must be perfectly correct over the recording period. The latest packet contents overwrite any previous data.

Be cautious about using both cdorb2db and orb2db. If you do, you must be careful to have disjoint sets of packets going to corb2db and orb2db. If both orb2db and cdorb2db get the same packets, they are likely to try to save them into the same file in different formats, leaving behind an awful mess.

However, it should be possible to switch from orb2db to cdorb2db or vice versa, once per day, without messing up the database.

Because cdorb2db keeps many files open, it's often necessary to raise the limit of open files; try unlimit descriptors

If the start and stop times are more granular than the packet size, the actual start and stop times vary from the specification; Data gaps can also cause start and stop times which vary somewhat from the spec.

cdorb2db does not detect any problem when given a source match which doesn't select any sources; it waits forever for a source matching the regular expression, even if there is an explicit ending time.

cdorb2db should be run so that its database and its waveform files are on local, not nfs-mounted partitions. Having the waveforms on an nfs-mounted partition is particularly troublesome when the partition fills up.

cdorb2db writes big-endian integer (s4) datatype on big-endian machines (SPARC and power-pc), and little-endian integer (i4) datatype on small-endian machines (intel architectures).

AUTHOR

Daniel Quinlan
Boulder Real Time Technologies, Inc.

Table of Contents
Antelope Release 4.8 Darwin 8.6.0 2006-06-28
Boulder Real Time Technologies, Inc For more information, contact support@brtt.com