There are four basic ways to get data into CloudTurbine:
- use an existing source application
- manually create the CloudTurbine folder/file structure and content
- use a shell script to create the CloudTurbine folder/file structure and content
- create your own CloudTurbine source app using one of the CloudTurbine APIs
Each method is discussed in a separate tab below.
The CloudTurbine distribution includes a variety of applications that can be used to output source data, including:
- CTstream: create a movie (series of images) captured from a region of your computer desktop; includes support for optional audio capture
- CTaudio: record audio from your computer
- CTserial: capture comma-separated value (CSV) data from a serial port to CloudTurbine files; includes a mode for generating simulated data
- CTtext: writes text as-you-type to CloudTurbine
- CTarchive: saves data from one or more DataTurbine channels to CloudTurbine
- CTandroidAV and CTandroidACL: Android apps which stream audio/video and accelerometer data (respectively) to CloudTurbine files via FTP
- CTlogger: a file parser that converts lines of comma-separated-value (CSV) log-file data to CloudTurbine format
- CTudp: captures data from UDP packets and saves it to CloudTurbine files
- CTsync: monitors a file folder in real-time and converts new files to CloudTurbine’s time-stamped folder structure
- CTblocktest: CloudTurbine test utility which generates a variety of CloudTurbine data formats
- CTbigtest: CloudTurbine test utility which generates a very large number of CloudTurbine sources and data
- CTsource: simple example of a CloudTurbine source utility
Details for all of these source applications can be found on the Apps page (see the Source, Utility and Example tabs).
CloudTurbine data is written out to a series of folders and files with a defined structure. The files are easy to navigate – go ahead and open a file browser on your computer and dive down into a CloudTurbine source folder. In the simplest structure, there is a top level folder whose name is the source name; below this are several layers of time/sub-time folders; the lowest time folder contains one or more data files where each file corresponds to one channel (the name of the file being the name of the channel). While data files can be of any format, they typically contain simple text or binary formatted data. Details of the CloudTurbine data file structure are provided in the Structure document.
Following this structure, you can manually construct a “home made” CloudTurbine source. For example, create the simple source depicted in the tree below. Set the top-level folder, CTdata, to be viewed using the CTweb server. The source folder, SimpleSource, is under CTdata. The name of the folder directly under SimpleSource is the root time for the source (officially called a “session” time), specified by an epoch time (with millisecond resolution). In this case, the root time (1488313139000) corresponds to 2017-02-28 20:18:59 GMT. Under this root time folder is a “block” level folder, “0”, which contains 2 “point” time folders “0” and “1000”. The point time folders contain data for one channel, “output.txt”. The names of the point time folders, “0” and “1000”, specify that the data in the “output.txt” channel files are at times “root time + 0 msec” and “root time + 1000 msec”; that is, the samples are 1 second apart. Put the word “Hello” in the first “output.txt” file and and “World” in the second “output.txt” file.
With this folder/file structure, make the following URL request to the locally running CTweb server:
The “d=1” flag tells the server we want to see the newest 1 second worth of data. This retrieves the following “timestamp,data” pairs:
which are the data point from our 2 files!
Due to CloudTurbine’s simple format (see the Structure document), it is easy to develop a shell script which creates a CloudTurbine source. Consider the example below, which periodically adds images from a Dartmouth College web camera to a CloudTurbine source.
if [ "$#" -eq 1 ]; then
mkdir -p $CTsession
mkdir -p $CTpoint
curl -s -S -o temp.jpg "$webcamURL"
mv temp.jpg "$CTpoint/webcam.jpg"
A portion of the source structure from a sample run of this script is shown in the figure below. The top-level folder under which all sources reside is typically called “CTdata”; under this is the source-level folder named “Dartmouth”; under this is the “Session” time folder (in seconds since epoch); the two levels under the Session folder are the “Block” and “Point” folders. A channel data file (“webcam.jpg”) resides in each Point folder.
Creating a CloudTurbine source using this shell script provides a good working example to explore how to view data in a browser with WebScan and CTweb. See the WebScan documentation page (specifically the section titled “Setup your own CloudTurbine data service”) for details.
Manually formatting CloudTurbine data (as described under the “Manual” tab) can be useful to help understand the technology, but it becomes tedious for all but the simplest sources. Using a shell script (as described under the “Shell script” tab) affords a step up in capability but still has limited utility (for example, it is not integrated into a larger program and may lack features such as ZIP compression). When you need to create a custom CloudTurbine source, the preferred method is to develop your own source app using one of the CloudTurbine APIs. For an introduction to the using the Java API, see Writing CT source apps. For using the .NET/C# API, see the CloudTurbine C# library post.