CTlogger is a file parser that converts lines of comma-separated-value (CSV) log-file data to CloudTurbine format. It can automatically extract new lines of CSV data as they are added to the end of a log file in real-time. This provides a simple interface compatible with a wide range of dataloggers, and is easy to adapt to new data sources.
java -jar CTlogger.jar -h
-a,--noappend Turn off append mode (i.e., do not append to
end of existing CT data).
-b,--nobackwards No backwards-going time allowed.
-B,--blockMode Turn on CloudTurbine writer block mode
(multiple points per output data file,
-f Flush interval (sec); default = "3600".
-g,--gzipmode Turn on gzip for extra compression.
-h,--help Print this message.
-H Optional CSV list of channel names; if not
supplied, this is read from the first line
in the logger file.
-i Leading ID string (IWG1 compliant).
-k In logger file, the num lines to skip after
the header line to get to the first line of
data; default = "2".
-l Name of the logger data file; required argument.
-N,--newfilemode Re-parse entire logger file every time it is checked.
-n Replace NAN with this; default = "0".
-o,--outputfolder Location of output files (source is created
under this folder); default = "CTdata".
-p If repeatFetch option has been specified,
recheck the logger data file at this polling
interval (sec); default = "60".
-r,--repeatFetch Turn on repeat fetch (auto-fetch data loop).
-s CloudTurbine source name; default = "CTlogger".
-t,--storeTime Store time string as a channel; time is the
first data entry in each line; if this
option is not specified, then the time
channel is skipped/not saved to CloudTurbine.
-T Trim (ring-buffer loop) time (sec)
(trimTime=0 for indefinite); default = "0.0".
-x,--debug Turn on debug output.
-z,--nozip Turn off zip mode (it is on by default).
Consider the following simple example. Say we have a data file called “log.txt” which contains time and 3 one-dimensional positions (“posA”, “posB”, “posC”). The top of the data file looks like the following:
We can parse the data in this file and send it to CloudTurbine using CTlogger as follows:
java -jar CTlogger.jar -f 30 -k 0 -l log.txt -t -r
CloudTurbine channel names are taken from the first line of the file (“time”,”posA”,”posB”,”posC”). Alternatively, channel names can be specified using the “-H <HeaderLine>” command line argument.
The following command line arguments have been used:
“-f 30” specifies to flush data to CloudTurbine every 30 seconds
“-k 0” specifies to not skip any lines between the header line (which is the first line in the file) and the first data line
“-l log.txt” specifies the name of the logger file to parse
“-t” specifies that time (the first entry in each CSV data string) will be stored in its own CloudTurbine channel
“-r” specifies to re-check the data file (“log.txt” in this case) for new data periodically; in this case the default period (every 60 seconds) is used, but this period can be specified using the “-p <pollInterval>” command line argument if desired
CTlogger to CloudTurbine interface for Northern Temperate Lakes (NTL) demonstration.
Note the simplicity of the data source interface: simply write updated data to a CSV file in a shared folder. Authorized collaborators can view data via any connected viewer (e.g. WebScan, DataTurbine RDV, Matlab)
CTlogger to CloudTurbine Interface is noted in red. The system illustrated above also utilizes the CTplugin utility, a CloudTurbine to DataTurbine connector.