NAME

rt2orb - acquire data from a Reftek RT130 datalogger to an ORB

SYNOPSIS 

rt2orb [-pf pfname] [-calib_db dbname] [-v]
       targetname [orbtag1 orbname1 [orbtag2 orbname2 [...]]]

DESCRIPTION

rt2orb is a program for interacting with one or more Reftek RT130 dataloggers. Data packets along with log messages, status messages, command responses and debug messages are written to a set of output ORBs. This program was designed to support UDP-based acquisition of digital data from Reftek RT130 dataloggers using the Reftek protocol documented in appendix B of RTPD Installation and Users Guide, Refraction Technology, RTPD-OP-005-D, August 26, 2004. Although in theory the same protocol can be used with older Reftek dataloggers, such as the RT72A dataloggers, we have not yet attempted to acquire data from these other dataloggers using this program.

OPTIONS

ORB OUTPUT AND INPUT PACKETS

rt2orb writes out waveform data strictly in SEED packet format. (Note that this is the only output format supported at this time.) For each datalogger, individual SEED channels are preserved as individual ORB packets. The original SEED compression and packaging from the datalogger is preserved in the output ORB packets. Note that only the c0 Reftek data format is supported (you cannot used rt2orb with the uncompressed integer format).

rt2orb writes out a set of log messages as <target_name>/log packets, for log messages relating to the rt2orb instance and as <datalogger_name>/log packets, for log messages relating to RT130 dataloggers. These messages are simple ASCII strings. These log messages can be displayed by the program dlmon(1) and can be archived with orb2logs(1).

rt2orb keeps a set of status information for each attached datalogger. All of this status information is written as a parameter file packet (with name <target_name>/pf/st) to the output orb at regular intervals. These status packets can be displayed by the program dlmon(1).

Commands to rt2orb are all read by rt2orb from the command ORB. The ORB packet naming conventions for commands and command responses are discussed in the section COMMANDS.

PROGRAM PARAMETER FILE

The rt2orb parameter file contains all of the information necessary for connecting to and acquiring data from the RT130 dataloggers. Following is an example rt2orb parameter file. 


#
#  This the rt2orb parameter file used to specify all of the settings needed to
#  communicate with a set of RT130 dataloggers.
#

#  Following are global parameters:

pfstatusreport_interval  20         # This is the time interval in seconds for output
                                    #    status report packets
cmd_orbtag               dataorb    # command line orb tag for command packets
log_orbtag               dataorb    # command line orb tag for log packets
status_orbtag            dataorb    # command line orb tag for status packets

packet_defs &Arr{                   # ORB data packet definitions
    sd &Arr{                        # Standard mini-SEED definition
        data_orbtag      dataorb    # command line orb tag for data packets
        twin             0          # preserve SEED packing
        multiplexed      no         # Is this packet channel-wise multiplexed?
        suffix           SEED       # data type
        subcode                     # subcode fields in ORB srcname
    }
}

#  Following are global default datalogger parameters:

statusrequest_interval   20         # This is the time interval in seconds for requesting
                                    #    datalogger status
statusreport_interval    20         # This is the time interval in seconds for reporting
                                    #    client status information as waveforms
datarate_interval        10.0       # This is the time interval in seconds for computing
                                    #    data rate averages
timeout_data             20         # read timeout in secs for data
timeout_control          20         # read timeout in secs for control responses
local_port_rtp           2543       # "well known" UDP port used for RTP communications
control_port_rtc         5000       # RT130 TCP port used for command and control

debug_data               0          # Put out data debugging log messages?
debug_cmd                2          # Put out control debugging log messages?
debug_udp                0          # Put out udp read/write log messages?
dump_rtp                 0          # Put out rtp packet hex dumps

pchan_map &Arr{ # These map rt130 physical channels to SEED net-sta-chan-loc codes
#   pchan net_sta_chan[_loc] calib calper  segtype
    0     $DLNET_$DLSTA_HHZ  1.266 -1.0    V
    1     $DLNET_$DLSTA_HHN  1.266 -1.0    V
    2     $DLNET_$DLSTA_HHE  1.266 -1.0    V
}

rate_mask &Arr{ # These are mask values that modify the pchan_map SEED
#   stream srate  chanm	(locm)
    0      1.0    L..           # Stream 1
    1      40.0   B..           # Stream 2
    2      100.0  H..           # Stream 3
    3      200.0  E..           # Stream 4
}

acq_matrix &Tbl{ # acquisition matrix
#   str0 str1 str2 etc.
    sd   sd   sd   sd   # pchan 0
    sd   sd   sd   sd   # pchan 1
    sd   sd   sd   sd   # pchan 2
}

#  Following are datalogger parameter templates:

datalogger_templates &Arr{
    az &Arr{
        rate_mask &Arr{ # These are mask values that modify the pchan_map SEED
#           stream srate  chanm (locm)
            0      200.0  E..
            1      40.0   B..
        }
    }
}

#  Following is a list of dataloggers:

dataloggers &Tbl{
#   dlname   dlnet dlsta  rt_unitid_number templ   disposition
    PI_R936F PI    R936F  936F             default startacq
    PI_R948C PI    R948C  948C             default startacq
    AZ_R930B AZ    R930B  930B             az      startacq
}

#   Following are individual datalogger overrides:

The first part of the parameter file consists of a set of default values that will be used by all dataloggers and also a set of datalogger packet format and template definitions that are used to define which channels are acquired and how the individual SEED channels are written to the output ORB packets. The second part of the parameter file consists of a single dataloggers table that defines each of the individual RT130s for acquisition. The third part of the parameter file lists individual datalogger parameters that override the parameters specified in the previous two sections.

Parameters from the default section of the parameter file are as follows:

The following parameters relate to debug log messages.

The following parameters relate to mapping SEED names to the physical RT130 channels and sample rate streams.

The following parameters relate to specifying exactly which physical channels and streams are to be acquired and how these channels will be output into ORB packets.

These default parameters define an acquisition "template" which we assign the token default. The default template can be used for any and all of the dataloggers if desired. Other acquisition templates can be defined through the datalogger_templates associative array (see below). This provides a convenient mechanism for defining a set of "default" configurations.

Once all of the default parameters and the datalogger templates have been defined, it is relatively easy to set up acquisition from a particular set of RT130 dataloggers. This is done in the dataloggers table which is defined below:

Finally, any of the parameters can be overridden for particular dataloggers. This is done by specifying associative arrays whose names are set to the dlname fields in the dataloggers array. The contents of these associative arrays can contain any of the global default parameters which causes the values for those parameters to be overridden only for the specific datalogger.

COMMANDS

rt2orb takes on the role of a command and control proxy both for itself and for all of the RT130 dataloggers that it is communicating with (there are no currently defined commands that are sent to the dataloggers themselves). rt2orb continuously looks for special command ORB packets on the ORB specified by the cmd_orbtag parameter. ORB packets that are read by rt2orb as commands are in a special parameter file format and must have the ORB srcname <targetname>/pf/dlcm. Packets of this type are produced by the program dlcmd(1). Packets with this ORB srcname are read by rt2orb and their commands are executed. A single command can be directed to the rt2orb instance itself or to any of the RT130 dataloggers that it is servicing. A typical command packet looks like this. 


#2153703 'TOLLrt/pf/dlcm':  3/18/2006 (077) 21:14:48.058 : 259 bytes
    parameter file data packet
cmdtime       1142716488.1
command       stop
delayhangup   1.00
dlnames       BR_R936F
dsig          2536876889e9f758ec382591c9e6929d
hostname      ruper.brtt.com
ip            207.174.76.133
reply         dlcmd/pf/dlcmr
sequence      BR_R936F_ruper.brtt.com_7273_0
targets       TOLLrt
types         rt
username      danny

The response to the command, if any, is specified by the reply parameter as an ORB srcname. In this case the response would be placed back on the command ORB with srcname dlcmd/pf/dlcmr. In this case the command, specified by the command parameter, is stop, which causes current connections with one or more dataloggers to be stopped and for the associated data threads to go into a hibernation, or standby state until they are started again (with a start command or by restarting rt2orb). The target dataloggers are specified in the dlnames parameter which is matched as a UNIX regular expression against all of the datalogger names managed by the instance of rt2orb. In this case only one datalogger will be stopped. This particular command generated the following response. 



#68212 'dlcmd/pf/dlcmr':  2/20/2006 (051) 22:18:42.730 : 288 bytes
    parameter file data packet
command         stop
disposition     done
dlcom           207.174.76.155:2543
dlname          BR_R936F
dlserial        936F
sequence        BR_R936F_ruper.brtt.com_7273_0
sequence_target toll_5125_1
target          TOLLrt
thostname       toll.brtt.com
tip             207.174.76.153
tpid            5125
tusername       rt
type            rt

The disposition value of done indicates that the stop command was received by the target rt2orb, it was acted upon and the action resulted in no errors. dlcmd(1) waits on this response and prints it out for you.

The command parameter is exactly what you type into the command line arguments cmd [cmdarg1 [...]] in the program dlcmd(1) and can contain a typical full command line syntax. Following is the list of commands. Note that currently all commands are really directed to the rt2orb programs themselves whereas no commands are directed to the RT130 dataloggers.

BUGS AND CAVEATS

There are undoubtedly bugs in this new release of rt2orb. Because of the complexity related to acquiring data from modern dataloggers, we have found that it takes years for a new data acquisition software module to become thoroughly exercised and for all of its bugs to be uncovered. This is our first release of this program so we ask our users to report any bugs or unusual behavior and to be understanding when encountering problems.

We think it is advisable for our users to become familiar with the operating characteristics of the RT130 dataloggers and how these characteristics affect the operations of the entire remote datalogger, communications, local software acquisition system. The RT130 datalogger is derived heavily from its standalone, no communications, predecessors and many aspects of its operation will be familiar to those who have used the older Reftek dataloggers in portable no-comm acquisition environments. However, in some cases these traditional approaches do not fit well into continuous telemetry based systems. In this section we will endeavor to explain in detail how RT130 dataloggers operate in continuous telemetry mode and how its operating characteristics impinge upon, and in some cases limit, various design considerations in rt2orb. We urge all new users to go through this section thoroughly so that they will be able to understand certain operating restrictions and how they should configure the RT130 dataloggers for telemetry using Antelope's rt2orb program.

RT130 DATA FORMATS - Understanding how a RT130 datalogger telemeters its data starts with an understanding of the data formats used by the RT130. Going back 10 or more years, Reftek dataloggers had been used for portable stand-alone deployments with no real-time communications, such as those supported by IRIS' PASSCAL program. In the "old" days real-time communications were usually not available for the types of remote deployments that employed Refteks. Instead the Reftek dataloggers digitized data at multiple sample rates and stored the data on local mass storage devices, as the RT130 continues to do today on its removable flash disk devices. Because of size and power derived limitations of the then available mass storage, the early Reftek dataloggers were most often operated in event recording mode where only events, determined by STA/LTA triggers, were stored on the local mass storage. Accordingly, Reftek devised a data recording format in which the data is organized by "event" files, each file containing all of the waveform data for a single event (see Reftek publication 130 Recording Format Specification, 130-RF-001-E, August, 2004). As mass storage size limitations eased over the years, the Reftek dataloggers became increasingly used for continuous recordings. Reftek implements continuous recordings as contiguous fixed time length "continuous" triggers resulting in fixed time duration event files with the same format and data content as the original event triggered event files. Real-time telemetry in the modern RT130 dataloggers is implemented as verbatim real-time transmissions of the event files as they are being (optionally) assimilated within the RT130 local mass store. The point here is that the RT130 real-time telemetry data format is basically identical to the format used to store the data on the RT130's local mass store, which is the original Reftek event file format.

Reftek event files consist of 1024 byte fixed size data blocks. Most of the data blocks are single channel-stream waveform samples stored in either straight integer or Steim II compressed formats. Each 1024 byte data block has a common header that identifies the unitid (serial number), event number, packet sequence number and time stamp. Each of the waveform data blocks contain another fixed header that identifies the channel number, stream number, data format and number of samples. Note that individual waveform data blocks do not contain any SEED codes, or, more importantly, the sample rate for that particular block, only the numerical channel and stream numbers. Also note that although it is possible to configure the RT130 to produce Steim II compressed data samples that can go unmodified into miniSEED blockettes, the literal miniSEED blockette header is not present in the Reftek event file data blocks, i.e. the miniSEED header, which contains information like SEED codes and sample rate, must be assimilated by the downstream acquisition software from the available event file blocks.

In addition to the waveform data blocks, there are a number of other blocks that appear infrequently and relate to configuration, status and state-of-health information, including the ad hoc datalogger log messages. One of the more important of these config/status blocks, the "event header" blocks (EH), always appear near the beginning of each event file before any of the waveform data blocks. There is an EH block at the beginning of each event file for each channel-stream of waveform data. The EH blocks contain the "missing" information from the waveform data blocks; Seed codes, sample rates and other useful stuff like gains and GPS coordinates. Note that there is only one EH block per channel-stream-event file that comes at the beginning of the file and is not repeated elsewhere in the file, except for the "event trailer" (ET) block at the very end of the event file. The EH blocks can be used to determine the sample rates that correspond to each stream number. In fact this is the usual way that the sample rates are determined from event files. When processing event files in batch mode from a stand-alone deployment without communication, this is a reliable and straightforward approach to formatting the data since the EH blocks will always preceed any waveform blocks in the event file processing. Note that in Reftek dataloggers, the correspondence between sample rates and stream numbers is not fixed, but instead is configured by the user. Therefore, when reading the data, knowing the sample rates for each stream number requires processing of the EH blocks.

DETERMINING SAMPLE RATE - However, in real-time telemetry mode, the Reftek event files are not transmitted as actual files, but instead each of the 1024 byte event file data blocks is transmitted in a time ordered sequence of data blocks. When acquisition software like rt2orb connects to a RT130 datalogger, the datalogger determines where the data block reading starts within its own internal buffers. To our knowledge, it is not possible to use the UDP data link to request the datalogger to start at any particular point in the stream of event file data blocks. Therefore, at initial connection the data block sequence can start anywhere in relation to the event file boundaries, not necessarily at the beginning of an event file; this behavior is normal for telemetry connections to a RT130. Since continuous event file durations are arbitrary and configurable at the datalogger by the user, it is possible for EH blocks to occur at very infrequent intervals, up to many hours. This means that when rt2orb connects to a RT130 datalogger it could take hours before the first EH block is read. The one critical piece of missing information that rt2orb needs before it can assimilate a full miniSEED blockette for dispatch to its output data ORB is sample rate, which normally can only be found in the EH blocks.

There are a number of approaches that we could take to deal with the missing sample rate problem. 1) - Just wait for the EH block before outputting any miniSEED data to the ORB. 2) - Try to "guess" at the sample rate by comparing time stamps and numbers of samples from adjacent waveform blocks. 3) - Query the sample rates using the RT130 TCP-based command and control server. 4) - Get the sample rates from rt2orb's parameter file. We eliminated option 1 out of hand; this either involves buffering up large amounts of data while waiting for the EH blocks or throwing the data away until an EH block arrives, neither approach we considered to be acceptable. At first glance option 2 may seem feasible, but we considered this approach to be unreliable in the presence of time tears or real gaps caused by buffer overflows, etc. We wanted an approach that would be as fool-proof as possible and at least consistent in its errors, so we eliminated option 2 as well. Basically we decided on a combination of options 3 and 4 to solve the sample rate problem.

RT130 COMMAND AND CONTROL SERVER - The primary method that rt2orb uses for determining channel-stream sample rates after a telemetry connection has been established is to make a query to the RT130's TCP-based command and control server (see Reftek publication 130 Command Format Specification, 130-CR-001-C, March, 2004). RT130 dataloggers are configured and controlled through a TCP server running on each datalogger that listens for incoming connection requests and services command and control requests through TCP connections to remote (to the datalogger) clients. In addition to servicing command requests these same links can be used by acquisition programs like rt2orb to request certain configuration information, like the mapping between stream numbers and sample rates. The application-level protocols and formats for the TCP command links are entirely different from those of the UDP data links.

So far, so good. In theory rt2orb has a means to determine stream number to sample rate mappings on demand at UDP data link startup by sending a query to the datalogger's C&C server through a TCP connection. However, in practice there are some problems with this approach. The first problem relates to security restrictions that are encountered in many institutional settings. The port number used by the datalogger TCP C&C server is hardwired to 5000. As far as we know there is no way to change this port number at the dataloggers. TCP port 5000 is a well known target for various hacker attacks (just google "port 5000 hack" and see what you get) and is therefore routinely blocked on most firewalls. Users must take care to insure that an unrestricted path exists between the host where rt2orb is running and the RT130 dataloggers for connection to TCP port 5000. Note that there is a parameter in rt2orb's parameter file, control_port_rtc, that can be used to set this port number on the rt2orb side. This is there in case Reftek makes the TCP C&C TCP port number configurable in the future and to support NAT-based schemes for getting around the port 5000 problem. The second problem with using the datalogger's C&C server for getting the stream number to sample rate mappings is that the RT130 C&C server is apparently singly threaded in its ability to service C&C requests. What this means is that only one TCP connection to one client at a time is allowed by the RT130 C&C server. (Note that it is possible to route many different TCP sockets from different ip addresses through the same TCP port number. This is how orbserver(1) works.) If you see a "connection refused" message in rt2orb's error log, then probably this means that somebody else is connected to the RT130 C&C server. When rt2orb uses the TCP C&C link to a datalogger, it opens the connection, gets the information it needs and immediately closes the connection so that other clients have an opportunity to access the datalogger's C&C server. rt2orb only does this once for each running instance of the program and for each datalogger-channel-stream at the first connection instance and caches and reuses these mappings for subsequent UDP data link cycles.

Unfortunately not all other software that is in use to access the RT130 command and control servers behaves in this way. Specifically, certain Reftek software can be configured to open connections and "camp out" on the datalogger C&C TCP ports 5000. When this happens it will not be possible for rt2orb to get the stream to sample rate mappings, because of the singly threaded nature of the servers running on the dataloggers. We have found that between the port 5000 security problems and the single threaded server bottleneck at the dataloggers, it is fairly common for the attempt by rt2orb to get the sample rate mappings from the RT130 command and control server to fail. There is a timeout, timeout_control, that is used by rt2orb so that it will eventually give up trying to get sample rate information in this way. When rt2orb has been unsuccessful in getting the stream to sample rate mappings through the RT130 command and control server, it falls back to its "last resort" approach, which is to take the mappings from the rate_mask specifications in the rt2orb parameter file. Users should exercise care to make sure that the stream to sample rate mappings are correct in their rt2orb parameter files. Should these values be wrong and there be no access to the datalogger command and control servers, then the resulting output data miniSEED ORB packets will be pretty well hosed.

RT130 COMMUNICATIONS AND OTHER CONSIDERATIONS - Users should be aware of some other aspects of the RT130 communications protocols and formats and certain other operational restrictions when used with rt2orb. First, in addition to the hardwired TCP port 5000, the dataloggers also expect the acquisition software to be listening for connections at a hardwired UDP port number, 2543. Once again, the user can specify this port with the parameter local_port_rtp, to accommodate possible future changes in the Reftek firmware and NAT-based port translation schemes. However, aside from NAT tricks, this hardwired port number means that it is not generally possible to run more than one instance of rt2orb on a single host or to run rt2orb if any other software is camping on UDP port 2543, such as Reftek's RTPD software.

There are some RT130 related communications considerations that users should take into account, specifically relating to long haul and high RTT UDP-based data telemetry communications. Probably less critical is the fixed UDP packet size of 1024 bytes, which, adding the standard ETH/IP/UDP headers, translates into a raw IP packet size of 1068 bytes. Unlike TCP which is byte stream oriented, UDP packets must be maintained without fragmentation through communications links. Some communication links have limitations on the maximum size of a packet, or Maximum Transmission Unit (MTU) (try google 'udp mtu limitation'). The Reftek RTP UDP packets have a fixed and unchangeable MTU of 1068. Users should ascertain that a particular candidate communication infrastructure will support the largish RT130 UDP MTU.

Another problem with the RT130 UDP packets is that there are no robust application-level crc or checksum checks for verifying data integrity; this is done entirely based upon the boilerplate checksums in the IP and UDP headers. This has probably been done because the UDP data packets are exactly the same as the event file blocks and these application-level crcs were probably not consiered to be necessary for on-site local storage. However, these checks are very necessary for reliable UDP-based telemetry over the broad range of imperfect long haul communications infrastructures. We have definitely seen cases involving UDP data packets over long haul links (although with another manufacturer's product) in which corrupted packets were delivered to the receiving application program. A UDP packet communication pathogen that comes quickly to mind is that the standard IP-level UDP checksum is not very robust when compared to more sophisticated crc methods. This means that there are a fairly large number of distinctly different packet byte patterns that can produce the same UDP checksum, which increases the likelyhood that a random UDP packet corruption will be undetected by the UDP checksum. Another more insidious and unfortunately more common pathogen is attributable to the "imperfect" nature of some communications infrastructures we have encountered. Some communication providers consider UDP to be a "private" protocol that is used by themselves to monitor and control their own networks with UDP-based services like ping and SNMP. They allow user-level UDP packets, but they still sometime expect that these packets conform to a few well-known standard uses. We have encountered situations where UDP packet payloads are purposefully transformed and the UDP checksums are recomputed somewhere in the communication infrastructure cloud. We believe that the transformations are a misguided attempt to automatically translate ip-addresses that happen to appear, randomly and coincidentally, in the UDP packet payloads. Although uncommon, we have seen this happen. Without an application-level data integrity check, there is no way to screen out packets that have been munged in this way.

Another more common pathogen that can be attributed to "imperfect" communication services is the systematic filtering of UDP packets with certain payload signatures. This is a problem that we have encountered fairly often (so far strictly with certain VSAT providers) and we believe it is, once again, a misguided attempt by the communications provider to route or filter certain UDP packets based on their contents instead of on the actual IP headers. When this happens, if the same exact UDP packet content is retransmitted because it never appeared at the receiving end, it is exactly filtered out again in the same way as before which, eventually, results in holes in the telemetry data. A way around this problem is to scramble each UDP packet at the application level before transmission in such a way that retransmitted packets have different byte patterns from the original packets, but this is not a feature of the RT130 dataloggers. We have found that this problem seems to be only present on low-end consumer-grade communications providers and not with higher-end commercial-grade providers. However, users should be aware of the problem and its high latency and missing data side effects.

SEE ALSO

dlcmd(1) dlmon(1)

AUTHOR

Danny Harvey
Boulder Real Time Technologies, Inc.

Table of Contents
Antelope Release 4.8 Darwin 8.6.0 2006-04-11
Boulder Real Time Technologies, Inc For more information, contact support@brtt.com