CTudp captures data from UDP packets into time-stamped CloudTurbine files. This is useful for high speed data packets, such as for capturing streaming video.  CTudp can also optionally capture a CSV string and save the data in constituent channels.  Other options allow CTudp to communicate with a UDP server or even start a local UDP server, useful for testing.

UDP2CT is a related application which focuses on custom UDP data parsers and supports CTrollaball-style output.  Since there is some feature overlap between CTudp and UDP2CT, we are considering consolidating these applications.

Usage:

The “-bps” option is used to specify the number of blocks per segment.  Without this option (or if you specify “-bps 0”) the segment layer is not used in the CloudTurbine folder hierarchy.

Use either the “-c” or “-csplit” option to specify the name(s) of the output CloudTurbine channel(s).

  • Using “-c” will save the UDP packet data as a byte array
  • Use “-csplit” when the UDP packet data is a CSV string; this comma-delimited string is broken up and saved to constituent channels (see below for further details).

If the incoming data is a CSV string, then the “-csplit” flag can be used to specify individual output channel names corresponding to the components of the incoming comma-separated string.  The number of components in the CSV string from the UDP packet must match the number of channel names specified with the “-csplit” argument.  If this doesn’t match, data from the incoming CSV string is not parsed/saved.  The data type of the saved components from the CSV string depends on the channel name suffixes: “.txt” channels save data as strings, “.csv” channels or channels with no suffix saves data as numeric, “.f32” channels save data as 32-bit floating point, “.f64” channels save data as 64-bit floating point.  As an example, consider launching CTudp as follows:

Say an incoming UDP packet contains the following CSV string:

Data will be saved as follows:

See “Local UDP server for CTudp testing” below for how to use the “-csplit”, “-testserver” and “-udpserver” options together to create a simple, local stand-alone CSV test system.

The “-d” option is used to specify the interval when receiving regularly-spaced channel data.  Data will be time-stamped using increments of this period instead of wall clock arrival time.  Wall clock arrival time is used if no “-d” option is specified or if “-d 0” is used.

The “-e <exception val>” flag is used when the “-csplit” flag is enabled.  It specifies a default floating point value to use if a numeric channel receives a non-numeric entry in the CSV string.  For example, consider a “-csplit” channel named “voltage.f32”.  The corresponding field in each incoming CSV string should be a number, which will be saved as a 32-bit floating point value in channel “voltage.f32”.  Say there is an error with this CSV field and instead of a number, the entry in the CSV string is “blah”.  Since this string (“blah”) can’t be interpreted as a number, the “voltage.f32” entry will be set to the “<exception val>” specified with the “-e” flag.  If no “-e” flag is specified, 0.0 is used as the default exception value.

The “-f” option specifies how frequently data will be flushed to the output CloudTurbine source.  Data is saved in ZIP files, and thus this flag specifies how much data is stored in each ZIP file and how frequently the ZIP files are flushed to disk.

To receive multicast UDP messages, use the “-m” option to specify the multicast address.

The “-p” option specifies the port number to listen for packets on.  Not to be used with the “-udpserver” option.

Specifying the “-pack” flag will pack blocks of CloudTurbine data.  Each block (i.e., each ZIP file) will contain a single file per channel; all of the data for a channel over the time range of the block will be stored in this single file.  Timestamps are interpolated over the data block (i.e., individual point times are not stored), thus packed data can save considerable storage space.  In addition to storage space savings, CloudTurbine access to packed channel data is very efficient.  Since individual point times are not saved with the data, packing is most optimal for periodically sampled data with regular “delta-T” time steps between samples.

Use the “-s” option to specify the name of the output CloudTurbine source.

Use the “-t” flag to only maintain a certain amount of data (number of seconds) in the CloudTurbine source; older data is deleted automatically.

If the “-testserver” option has been specified, a UDP server will be started at the specified port on the local machine to serve test UDP packets at the specified rate.  This test server is only capable of serving one connected client.  The output UDP packet stream will commence after the client sends a UDP message to the server (the content of this message does not matter); the server extracts the client’s IP address and port from the client packet and begins streaming UDP packets to the client at the rate specified by the period given in the “-testserver” command line argument.  If the “-csplit” option is not specified on the CTudp command line, the UDP server will output a simple text message.  If the user has specified a “-csplit” option, the UDP server will output CSV strings whose format (number of channels and data type) are consistent with the given “-csplit” channel list.  Note the following special features for this CSV data:

  • If the “-csplit” option contains a channel with the name “time.txt”, the corresponding output CSV string entry will contain the current time in ISO 8601 format “yyyy-MM-dd’T’HH:mm:ss.SSS”
  • “.txt” channels (other than “time.txt”) will contain a pseudo-random text string (a universally unique identifier, UUID)
  • The first numeric channel in the channel list (i.e., the first “.f32”, “.f64” or “.csv” channel) will contain a value that ramps from 0 – 60, corresponding to the current wall-clock second (with some added random noise in the range 0 <= x < 1).  In addition, to “stress test” downstream processing of this channel’s data, during the 10 second and 40 second mark of every minute, the numeric value that normally resides in this channel will be replaced with a string (“bogus”).
  • Other numeric (“.f32”, “.f64” or “.csv”) channels will contain a value equal to their integer position in the “-csplit” channel list with some added random noise (0 <= x < 0.5).  For example, if “chan3.f64” is the third channel in the “-csplit” list, its corresponding CSV value will be a random number in the range 3.0 <= x < 3.5.

Although not required, generally the “-udpserver” option will be used along with the “-testserver” option to direct the stream of test packets to this instance of CTudp (i.e., the same CTudp that launches the test server will also receive UDP packets from the test server).  See “Local UDP server for CTudp testing” below for how to use the “-csplit”, “-testserver” and “-udpserver” options together to create a simple, local stand-alone CSV test system.

Use the “-udpserver” flag to periodically send a short keep-alive/heartbeat message via UDP to a UDP server (see figure below).

The comma-delimited list of arguments to the “-udpserver” flag specify the IP address (or DNS name) and port of the UDP server and the period between keep-alive messages (in milliseconds).  The message sent to the server is “hello from CTudp”.  As an example, if you wanted to send a keep-alive message to the server at 192.168.1.1:5678 every second, the “-udpserver” command line option would be configured as follows:

Note that the “-p” and “-udpserver” options should not be used together.

See “Local UDP server for CTudp testing” below for how to use the “-csplit”, “-testserver” and “-udpserver” options together to create a simple, local stand-alone CSV test system.

The “-w” option is used along with the “-csplit” option to split on whitespace rather than commas.

Use “-x” to turn on CloudTurbine debugging.

Listening on multiple ports

CTudp can listen for data on multiple UDP ports by specifying a comma-delimited list of channel names (“-c” option), port numbers (“-p” option), and (if desired) delta-time periods (“-d” option).  The number of channels specified with the “-c” flag must match the number of ports specified with the “-p” flag.  The “-csplit” option should not be used when listening to multiple UDP ports.

Local UDP server for CTudp testing

Using the “-csplit”, “-testserver” and “-udpserver” options together creates a simple, local stand-alone CSV test system (see figure below).  The “-testserver” option specifies to launch a UDP server; this server is executed in a separate thread within the CTudp program.  The “-udpserver” option will send periodic “keep-alive” messages to the UDP server; when the server starts receiving these messages, it will begin the stream of UDP packets back to the client’s IP and port.

This setup can be useful for local, closed-loop CTudp testing prior to operation with a “live” system.  The server will output UDP packets containing CSV strings in a format consistent with the given “-csplit” channels (correct number and format of the channels).  For example, say the CTudp command is as follows:

In this example, the local UDP server will be started at port 8765.  Based on the “-udpserver” flag, CTudp will send a keep-alive message to this local server (at port 8765) every 2 seconds.  After the UDP server receives the first keep-alive message, it will start to output CSV packets every 100 msec (i.e. 10Hz) back to the CTudp client.  These CSV strings will be compatible with the given “-csplit” channels.  For the example shown above, the UDP server will output a CSV string containing 3 channels: a text channel followed by 2 numeric channels.