The Arduino must first collect data before streaming it! The Arduino's built-in A/D channels or various add-on shields and boards can be used to collect data. The best solution is one which meets a "real-world" need.
Capture Arduino serial data
Develop a CloudTurbine "CTserial" application to capture serial CSV data from an Arduino, parse it, and store it in CloudTurbine.
Save CT data on SD card
In data-logger fashion, save data in CloudTurbine format to an SD card directly on the Arduino using an SD-enabled shield such as Adafruit’s https://www.adafruit.com/products/1141. Part of this task can be to develop a simple Arduino library to mimic basic CTwriter functionality. Demonstrate saving the CloudTurbine-formatted data from the SD card to a desktop machine and viewing it with WebScan.
Wireless serial demonstration
This is a "wireless" version of the previous "Capture Arduino serial data” task. Use low-energy or standard Bluetooth to stream serial data from the Arduino. Use boards such as Adafruit’s Bluetooth LE Shield (https://www.adafruit.com/products/2746) or Bluetooth EZ-Link (https://www.adafruit.com/products/1588).
Wireless Ethernet demonstration
Stream CSV sensor data from the Arduino as UDP and capture it with the CTudp application. Use a WiFi shield such as Adafruit’s WiFi Shield 101 (https://www.adafruit.com/products/2891; note that this will work with UNO but it is reported that it takes up much of the available memory, see https://www.arduino.cc/en/Main/ArduinoWiFiShield101) or SparkFun’s WiFi Shield (https://www.sparkfun.com/products/13287).
Send data to single-board/embeddable computer
Demonstrate a complete end-to-end data solution sending data from the Arduino to a small embeddable computer such as a Raspberry Pi.
Using Adafruit’s ADS1115-based data acquisition card (https://www.adafruit.com/products/1085) attached to an Arduino UNO, we collected 4 channels of data, formatted them as a CSV string and wrote them out from the Arduino. A new serial capture utility, CTserial, was developed which captures the serial CSV strings, parses them and writes them out as CloudTurbine-formatted files. A description of this demonstration was included in the News post at https://www.cloudturbine.com/arduino-to-ct-demo/. CTserial is available in the CloudTurbine repository on GitHub (https://github.com/cycronix/cloudturbine/tree/master/JavaCode/CTserial).
A WiFi card has been purchased for the Arduino UNO (Adafruit’s ATWINC1500 WiFi Breakout, https://www.adafruit.com/product/2999); we will test sending UDP data out from the Arduino to CT soon!
This WiFi breakout board seemed like the best option for our Arduino UNO. There is a WiFi shield from Sparkfun for the UNO that we could also try (https://www.sparkfun.com/products/13287); there is also a WiFi shield which works for the Arduino Zero but not for the UNO (https://www.adafruit.com/products/2891).
Adafruit’s ATWINC1500 WiFi Breakout (https://www.adafruit.com/product/2999) has been used to send UDP packets containing CSV data strings to CloudTurbine.
Pictures of the hardware setup are shown below. The Adafruit tutorials for both the A/D card and WiFi card were straightforward and resulted in a working system.
A custom Arduino program was developed which reads the 4 A/D channels, formats the data into a CSV string and sends it out via serial and UDP. Screenshots below show UDP data displayed in Wireshark and serial data in the Arduino IDE Serial Monitor. To demonstrate the correlation between these two modes of output, the same line of data is highlighted in each picture.
(As a side note, to get the “Data” column in Wireshark, go to Edit > “Preferences…” > Appearance > Columns; add a new column and set the following fields: Title = Data, Type = Custom, Fields = data.text. Also select Edit > “Preferences…” > Protocols > Data; click on “Show data as text”.)
To get data into CloudTurbine, the UDP packets were captured using CTudp. A screenshot displaying some of the captured data in a web browser window is shown below.
The same WiFi/ADC hardware from the previous update has been “cleaned-up” by mounting the boards on an Adafruit Proto Shield for Arduino (https://www.adafruit.com/products/2077). The proto shield has a 0.1″ pitch prototyping area, breaks out all the Arduino pins, includes several sets of GND/5V rails and comes with stacking or plain headers (we used the plain headers because there would be no way to stack another shield on top of our hardware!). As can be seen in the pictures below, the WiFi and ADC boards were mounted vertically using right-angle headers (the WiFi board is the larger of the two). Since we don’t have sensors hooked up to our prototype hardware, a potentiometer (see the small blue pot with brass screw wiper adjustment) is used with GND/5V to provide a pseudo-input to the first channel on the ADC board.
I’ve noticed that the UDP stream stops after about 2-3 days; this has happened twice. Is this something with our WiFi card, with our Arduino program or something else? TBD
We purchased and setup a Raspberry Pi 3 Model B (includes WiFi!) and compiled the CloudTurbine library and applications on it. (For setup details, see the Raspberry Pi demonstration App-Kit, https://www.cloudturbine.com/app_kit/raspberry-pi-demonstration/).
UDP packets containing CSV strings of sensor data were sent from the Arduino to the Raspberry Pi, where they were captured and saved in CloudTurbine format using CTudp. The first figure below shows the Arduino hardware with wireless card (background) and the new Raspberry Pi (foreground). Second figure below shows UDP data capture to CloudTurbine on the Raspberry Pi.
CTweb was run on the Raspberry Pi…
…and CSV data was accessed from a browser on a Windows 10 machine on the same LAN.
Other interesting things that could be done for this demo:
- run the Raspberry Pi in headless mode; launch CTudp and CTweb automatically at startup
- setup Raspberry Pi WiFi as a wireless access point for Arduino connection; Raspberry Pi is then connected to the office/lab LAN using wired Ethernet (https://learn.adafruit.com/setting-up-a-raspberry-pi-as-a-wifi-access-point/overview)
We had a fun time trying out “tshark” (terminal version of Wireshark) on the Raspberry Pi to view the UDP packets and their CSV payloads. To see the basic incoming UDP packet information (source, destination, protocol, ports) we ran:
- tshark -Y “udp and not icmp and ip.src == 192.168.2.76 and ip.dst == 192.168.2.75”
An example of the output from this command is shown below.
To view the packet data in hex and ascii format, we ran the following:
- tshark -Y “udp and not icmp and ip.src == 192.168.2.76 and ip.dst == 192.168.2.75” –x
An example of the output from this command is shown below. In the ascii output on the right hand side, you can see one of the CSV strings, namely “20391534,1.897,0.541,0.541,0.542”.
But how can we view just the ascii CSV packet data? We arrived at two ways of doing this:
The first way “almost” works but the stderr packet counter that tshark displays in the terminal gets prepended to some of the output lines:
- tshark -Y “udp and not icmp and ip.src == 192.168.2.76 and ip.dst == 192.168.2.75” -T fields -e data -l | xxd -r –p
The following 2-step approach is better: First, we capture stdout from tshark to file; this captures the hex data only, not including tshark’s packet counter:
- tshark -Y “udp and not icmp and ip.src == 192.168.2.76 and ip.dst == 192.168.2.75” -T fields -e data -l > foo.txt
Second, tail the output file from the first step and pipe it to xxd to convert from hex to ascii:
- tail -f foo.txt | xxd -r -p
An example of the output using this second method is shown below.
See the new Arduino/RPi/CT system diagram below (kudos to NASA Armstrong for the suggestion!). This diagram provides an overview of the data flow and processing for the Arduino/RPi/CT App-Kit. Starting in the upper left corner, an Arduino UNO collects data using a 16-bit ADS1115 DAQ card (https://www.adafruit.com/product/1085), formats the data as a CSV string and sends it out via UDP using Adafruit’s ATWINC1500 WiFi Breakout (https://www.adafruit.com/product/2999). Data is captured by CTudp running on a Raspberry Pi 3 Model B (which includes WiFi). CTudp parses the data into its constituent channels and saves it to a CT source on the RPi. CTweb is also run on the RPi, supporting local and remote (wireless or wired) access to the CT data.