CTpack is a CloudTurbine utility to convert and compact file block formats.
A basic attribute of CloudTurbine files is the block-interval used for storage. Many short, small data blocks enable low-latency, fast update data streaming. Fewer, large data blocks are more efficient in terms of disk space usage and file-access (playback) performance. For some long-duration, large scale recordings, small data blocks may be required for real-time viewers, but larger more efficient data blocks may be desired for efficient archiving and playback.
CTpack helps provide a solution for this situation. CTpack reads CloudTurbine files and rewrites them to a new folder hierarchy. In so doing, it enables you to change the number, size, and type of segments, blocks, and zip options for previously recorded archives.
The Figure below illustrates an original file structure for three CT sources, along with CTpack generated equivalent compacted structures. The original data was acquired at ~1 second intervals, and CTpack repacked them at 10 second intervals for a 10:1 reduction in the number of files. Note also the conversion from absolute to relative timestamps.
CTpack can convert between the various file formats discussed in the related CTblocktest application. In the limit, CTpack can convert a CT source into a single zip file. This trades all stream-ability for the maximum convenience and compaction of a more traditional “media” file.
CTpack operates from a simple command-line interface:
java -jar CTpack.jar -h
usage: CTpack [options] sourceFolder (no default), where [options] can be:
-1,--singleFile single zip file mode, default: false
-f,--binarymode <arg> convert CSV to float binary, -f32, -f64
-h,--help Print this message
-o <packSource> name of output source, default: sourceFolder.pack
-p,--packmode pack mode, default: true
-r <rootFolder> rootFolder, default: CTdata
-s <segBlocks> blocks per segment, default: 10
-t <timePerBlock> time per output block (sec), default: 10.0
-x,--debug debug mode, default: false
-z,--zipmode zip mode, default: true
A graphical user interface (GUI) would be a desirable new feature.
Time-precision can be lost when converting data from unpacked, single timestamp per sample data into packed, interpolated time per block. (Or actually improve accuracy if original data was evenly sampled yet had “jittery” timestamps). If significant time-gaps occur in the data, this interpolation can result in correspondingly significant time-stamp errors. An automatic detection and splitting data blocks across time-gaps is a to-be-implemented feature.
Still to be implemented is a pacing-capability so that CTpack can also “unpack” an archive into smaller, incremental files at a paced rate, enabling you to effectively re-stream an archive for playback.