NAME

SYNOPSIS

orb2db [-a wffile] [-c chanmatch]
       [-m srcmatch] [-r srcreject]
       [-p pf-file] [-S state-file] [-s datatype]
       [-t time-tags] [-T tolerance] [-w wfname] [-v]
           orb db [start-time [window]]

DESCRIPTION

In addition to copying data, orb2db accumulates some statistics about packet errors into the gap, changed, retransmit and duplicate tables.

orb2db expects the sample rate from the packet to be approximately correct, in the sense that the time tags of adjacent packets are within 1/2 sample period of the time calculated from the previous time tag and number of samples and the sample rate. It attempts to handle a clock (or data logger like the Quanterra) where the actual sample rate is slightly different from the rate specified in the packet, by calculating a rate based on the time tags and requiring that the time calculated with this rate and the number of samples match the actual time tag to within 1/2 a sample period. New wfdisc rows are generated when this criterion fails.

orb2db is continuously modifying a set of wfdisc records (usually for the current day) as it runs. As with any Datascope database, no other process should be modifying the output database while orb2db is running. However, in the real time system, it is useful to be able trim this output database by deleting rows representing older data. This trimming improves performance of other programs running against the database.

orb2db implements a simple mechanism for accomplishing this trimming without shutting down and restarting orb2db. A special file is created named after the output database: database.MSGFILE. The first 8 bytes of this file contain a flag which is set while a cleanup is performed, and an integer count of cleanups performed. orb2db tests the flag and the count at every packet; when the flag is set, or the count has changed, orb2db closes its open database, and waits for the flag to be reset. When the flag is reset, it reopens the output database, and finds the new record numbers for the records it is modifying.

The MSGFILE may be monitored or modified using the program orb2db_msg(1). Because there is no acknowledgement (from orb2db that it has paused), it is important to allow sufficient time for orb2db to stop itself, before beginning to modify the database. orb2db_msg sleeps for some period of time before returning, in an attempt to assure this time window. In addition, the time required to clean up the database must be considerably less than the time allowed by the ring buffer size, so that no data is lost.

OPTIONS

• -a wffile
  write a log file which announces waveform files opened and closed; each line contains a leading open or close, followed by the name of the waveform file. When orb2db quits, a single line "finished" is written rather than a close line for every open waveform file.
• -c chanmatch
  Only net_sta_chan channels which match the regular expression chanmatch are accumulated and written to the output database.
• -C
  Print additional information when a packet causes a new wfdisc to be generated.
• -m srcmatch
  Only packets which match the regular expression srcmatch are requested from the orb server.
• -r srcreject
  packets which match the regular expression srcreject are not collected from the orb server.
• -p pf-file
  Specify an alternative parameter file name pf-file instead of orb2db.pf.
• -S statefile
  orb2db saves the pktid of the latest packet it processed in this file when it quits abnormally
• -s datatype
  Specify the output datatype; this is normally done in the parameter file, but may be overridden here.
• -t file
  As a debugging tool, record each packet into a time tag file as data from the packet is saved.
• -T tolerance
  Specify alternate tolerance for deviation of packet time tags from wfdisc straight line, in sampling periods. The default is 0.5 periods, as specified in trdefaults.pf. Making this larger can result in fewer wfdisc records, renders the (calculated) timing information interior to a single wfdisc waveform segment less accurate.
• -v
  Be more verbose; -vv shows every packet read.
• -w wfname
  see trwfname(3) -- this allows choosing the output file naming convention.
• start-time
  
• window
  A time range may be specified. The first parameter is the time in any of the formats accepted by epoch(1). The second may be either an ending time, or a time-window. A time window is most conveniently specified as hours and minutes like: hh:mm or hh:mm:ss. The parameter file specifies a maximum acceptable time window (default is 24 hours); longer time windows are usually command line errors, but a longer time window can be specified using an end time rather than a window.

PARAMETER FILE

• max_out_of_sequence
  orb2db waits up to max_out_of_sequence packets for a retransmitted, out of order packet. This parameter should be zero if your data stream does not contain out of order packets. When this is non-zero, out of order packets which are later than max_out_of_sequence are discarded.
• preferred_waveform_file_range
  Waveforms for a single net/sta/chan are stored in a single file, covering this range of seconds. If there are large enough gaps in the data, multiple wfdisc records are created, with foff set to the correct values. "Small" gaps are filled with a special missing data value.
• preferred_waveform_file_offset
  waveform volumes start and end at a range boundary + offset So, if you desire to have day volumes, but to make the day volumes correspond to local time rather than UTC, you might specify offset as the local offset in seconds from UTC.
• max_gap_to_fill
  This defines a small gap -- gaps larger than max_gap_to_fill samples are not marked with a missing data value, but cause a new wfdisc record to be generated.
• max_open_files
  This limit is only used if the system call getrlimit(2) fails. In Solaris, this limit is usually 1024 file descriptors; orb2db needs some of these for standard i/o, but should be able to handle at least 1000 separate channels, if the cpu and disk can keep up.
• max_pkt_period
  When a start time is specified on the command line, the orbafter call used to initially position the ring buffer specifies the command line start time minus max_pkt_period, so that packets containing the desired time are not skipped.
• datatype
  data format for waveform files; some typical choices are sd for steim 2 compression, s4 for 32 bit integers, and as for ascii.
  
  For continuous data, sd is the usual choice. For triggered and/or event data, it's probably desirable to choose an uncompressed format (like s4) and also set the parameter flush_wf_writes to yes. Otherwise, orb2db holds the data internally until the next event causes it to be flushed to disk.
  
  There are a host of other datatypes, described in either the css schema datatype field (use dbhelp(1)), or in the trdefaults.pf parameter file, or in wftypes(5). You may even add your own new data type; see addwf(3) and examples_c_addwf(5).
• waveform_buffer_size
  This is the size of the data buffers used (per channel) when the data type is other than sd (steim compressed data). Usually it is useful to buffer many packets worth of data before writing the data to disk, to avoid the large overhead of many system calls. This buffer size should probably be comparable with the system page size.
• max_time_waiting
  When collecting data for a specified time period, it's possible for stations to go off line. orb2db can then hang for a long period waiting for the data, or for a packet later than the requested time from that station. This sets a limit on how long to wait.
• decent_interval
  maximum number of seconds to wait for data from a channel during continuous recording, or between saving state files. This should be larger than max_time_waiting.
• flush_wf_writes
  If this is set to yes (or ok or 1 or true), writes to waveform files are flushed immediately. For seed format, writes are always flushed. For other formats, the default is to fill the buffer before writing; flushing each write could be expensive in execution time.

Some kinds of problems can be quietly saved into the database tables changed, retransmit, ratechange and gap. The parameters below allow also printing error messages at regular intervals beyond a particular threshold error rate.

• chatter_limit
  each type of error message is printed at most once per chatter_limit seconds.
• min_problem_count
  
• min_problem_time
  error messages are output only when there are at least min_problem_count problems within min_problem_time
• max_window
  If a time range (rather than an end time) is specified on the command line, then that range must be less than this parameter; the default is 24 hours. Longer time windows are usually command line errors.

The following parameters affect whether more detailed information about packet anomalies is saved into database tables. The tables make more detailed analysis possible, but when transmissions are poor, or clocks wander a lot, the tables may grow unreasonably large unreasonably quickly. The default is not to write into the tables.

• record_changed
  record records to the changed table when the samprate, calib or tick registration specified in the packet changes.
• record_ratechange
  add records to ratechange table when the observed sample rate changes. The sample rate is calculated from the difference between packet time tags and the number of samples in a packet.
• record_gap
  add records to the gap table when a waveform has a gap; the gaps can be determined from the waveforms directly, but this is tedious because the waveforms may have small gaps indicated internally by a missing data value.

When a packet arrives out of time order, it is presumed to represent a retransmission of the packet. There are several categories of retransmission; recording each type may be individually suppressed.

• record_normal_retransmission
  These are the normal out of time order packets, presumably retransmitted because they were missing in the original data stream.
• record_duplicate_retransmission
  These are duplicates of a previously seen packet.
• record_overlapping_packets
  These are packets where the time frame overlaps (but is not the same as) another packet, and probably indicate some problem with the data logger or clock.
• record_rejected_packets
  These are packets which arrive so far out of time order they are dropped; they may indicate that the max_out_of_sequence parameter above is too small.
• pause_timeout
  maximum time (in seconds) to wait after pause for the matching continue. This is a failsafe, in case the program issuing the original pause fails to deliver a continue for some reason. The period should probably be less than the ring buffer size, and definitely greater than the worst case performance of any backup or cleanup script. Having this number too short risks corrupting the output database and causing an awful mess. Having this number too long risks losing some data.

ENVIRONMENT

EXAMPLE

Continuously collect data from the BHZ channel of station RDM from a ring buffer on XYZ to a database rdm.

% orb2db -m '.*RDM.*' -c '.*_RDM_BHZ' XYZ rdm

Collect data for the last 10 minutes from the BHZ channels of stations from a ring buffer on XYZ to a database cels.

% orb2db -c '.*_BHZ' XYZ cels '1996298 15:37' -0:10 0:10

DIAGNOSTICS

Fatal Errors

• "Can't close 'wfdisc->path'."
• "Couldn't save pktid #pktid database packet for 'srcname'"
• "Can't create new wfdisc record"
• "Can't create new waveform data file"
• "Can't open wfdisc->path to write trace data."
• "Can't open output database database"
• "Can't open output table 'database'" Something is amiss in the output database.

Non-Fatal Errors

• "input data for pktid #pktid (net_sta_chan) was clipped to fit into output format"
• "truncation to integers lost resolution during conversion to output format"
• "unknown return code unstuff from unstuffPkt for pktid=pktid at s=strtime(pkttime)"
  unstuffPkt(3) was unable to unstuff the specified packet.

Command Line errors and Initialization errors

• "Can't read parameter file"
• "Can't open ring buffer 'orbname'"
  The orbserver(1) may not be running, or may be rejecting connections from this machine.
• "orbselect 'match' failed"
• "orbreject 'reject' failed" Something is wrong with the regular expression match or reject, and the calls to orbselect(3) or orbreject(3) failed.
• "orbget to get current server time fails"
• "orbafter to strtime(params.from) failed"
• "couldn't compile pattern 're'"
  Something is wrong with the regular expression re.

SEE ALSO

orbmonrtd(1)
orbstat(1)
orb2db_msg(1)

BUGS AND CAVEATS

If the start and stop times are more granular than the packet size, the actual start and stop times vary from the specification; Data gaps can also cause start and stop times which vary somewhat from the spec.

orb2db requires the rt1.0 schema, for the additional tables gaps, changed, ratechange, and retransmission.

orb2db does not detect any problem when given a source match which doesn't select any sources; it waits forever for a source matching the regular expression, even if there is an explicit ending time.

orb2db should be run so that its database and its waveform files are on local, not nfs-mounted partitions. Having the waveforms on an nfs-mounted partition is particularly troublesome when the partition fills up.

AUTHOR

• NAME
• SYNOPSIS
• DESCRIPTION
• OPTIONS
• PARAMETER FILE
• ENVIRONMENT
• EXAMPLE
• DIAGNOSTICS
• • Fatal Errors
  • Non-Fatal Errors
  • Command Line errors and Initialization errors
• SEE ALSO
• BUGS AND CAVEATS
• AUTHOR

• http://www.brtt.com
  
  2045 Broadway, Suite 400
  Boulder, CO 80302
  USA