FilePump and FileWatch are test utilities which work together to analyze the performance of third-party file sharing services such as Dropbox, Google Drive and FTP.  Characterizing a file sharing service’s performance gives users an estimate of how well it will operate in a CloudTurbine network.  FilePump produces a stream of small test files at a user-specified rate and saves them to a user-specified directory – either a local directory (which could be shared using a file sharing service) or a remote directory via FTP/SFTP.  FileWatch observes this incoming stream of test files, noting the time and order in which they arrive.  At the end of the test, FileWatch calculates throughput and latency metrics based on the statistics of the creation time, arrival time and order of arrival of the test files.

Prior to using FilePump and FileWatch to conduct a file sharing test, we recommend reading the documentation for both apps to get an idea of their operation – both their individual roles and how they work together to perform a test.

FilePump produces a stream of small test files at a user-specified rate and saves them to a user-specified directory – either a local directory (which could be shared using a file sharing service) or a remote directory via FTP/SFTP.

FilePump can be run in batch mode or via a graphical user interface.

Usage

FilePump can output files in one of three modes: write files to a local folder or send them to a remote FTP or SFTP server.  The output mode is specified either by the “-mode” command line flag or in the Settings dialog box.

To run in batch mode (no graphical user interface), specify all options using the command line flags and include “-x” on the command line to automatically run at startup.

Without the “-x” command line flag, FilePump will launch the graphical user interface shown below.

File menu items
Menu item Description
Settings… Pops up a dialog to set all FilePump options. If FilePump is writing out files when this menu item is selected, the current run will be stopped.
Exit Exit the application

 

Selecting the Settings… item from the File menu pops up the following dialog. All FilePump options can be set from this dialog.

After specifying the desired settings, click OK in the FilePump Settings dialog. You can then click the Start pump button on the main window to start writing out files. When FilePump changes to write mode, the green Start pump button changes to a red Stop pump button and the File count label periodically updates to show the number of files that have been written out.

Clicking the Finish test button signals the end of the file test. FilePump will stop producing files (if it is currently running) and write a special “signal” file (“end.txt”) to the output directory. When FileWatch receives this file it will stop watching for new files and will process and write out the test data. Stopping FilePump (by clicking the Stop pump button) and signalling the end of the test (by clicking the Finish test button) are 2 separate steps, which is especially helpful when there is significant latency in transferring files to FileWatch. That is, after you have stopped FilePump (i.e., clicked Stop pump) you may want to wait some time before signalling the end of the test (i.e., clicking Finish test) to allow all files to transfer and be recorded by FileWatch.

Name format and content of the output files

FilePump writes out files with names having following format:

where <epoch_time_in_millis> is the time the file was created and <sequence_num> is the sequence number of this file in the test. The sequence number sequentially increments over consecutive files. For example, if FilePump wrote out its third file in a test on March 24, 2017 at exactly 16:00:00 GMT, the name of this file would be:

This filename informs FileWatch exactly when the file was written and in what order it was written in a series of files.

To make the files unique and avoid deduplication by third-party file sharing services, each file contains a 6-digit random number.

FileWatch observes an incoming stream of test files from FilePump, noting the time and order in which they arrive.  At the end of the test, FileWatch calculates throughput and latency metrics based on the statistics of the creation time, arrival time and order of arrival of the test files.

FileWatch can be run in two different modes

Normal/one-way mode: FilePump writes small test files to a shared folder; for instance, a shared Dropbox or Google Drive folder or a remote folder accessed via FTP/SFTP. The name of each test file includes the timestamp when it was created and the order in which it was created. FileWatch uses this creation time and order along with the arrival time at the destination folder to calculate throughput and latency. Accurate latency depends on the two system clocks (the clock on the computer running FilePump and the clock on the computer running FileWatch) being synchronized. The figure below gives an illustration where these two clocks are not synchronized, which will lead to inaccurate latency metrics. Throughput is unaffected by clock synchronization.

Recaster/round-trip mode: This mode works similar to the normal/one-way mode presented above except FileWatch runs in “recaster” mode on the remote side to redirect the stream of incoming test files back to the system where FilePump is running. A separate FileWatch runs on the same computer as FilePump; it observes the incoming files and calculates throughput and latency in the same manner as the normal/one-way mode above. Since FilePump and this second FileWatch run on the same computer, clock synchronization (important for latency calculations) is not an issue. Since test files travel twice the distance as in a normal/one-way test, the calculated latency metrics will be 2x their expected value; use the “-a” command line flag for this second FileWatch to adjust the output latency results to an equivalent “one-way” value.

 

Usage

Command line arguments, additional information
Option Description
-a,–adjust_latency The normal mode of sharing files is from source to destination. Running a test in “recaster” mode determines throughput and latency using files which have traveled round trip, i.e. from source (FilePump) to destination (where files are “recasted”) back to a FileWatch running on the source end. This avoids clock synchronization issues which affect latency calculations since the same system clock is used when producing and interpreting the test data files. Since the test files travel round-trip when using a recaster, latency is doubled. Use this command line flag to divide the latency metrics in half when presenting them to the user, thereby normalizing them to the equivalent “one-way”/source-to-destination results.
-d,–display_results_in_recaster_mode We are not typically interested in the metrics calculated by the “recaster” FileWatch. In fact, the point of recaster mode is that we don’t trust the clock synchronization between the source machine (where FilePump is running) and the destination machine. However, there may be occasions where you would like to see the throughput and latency metrics from the recaster FileWatch. Use this command line flag to output results for a FileWatch running in recaster mode. If FileWatch is not running in recaster mode, this flag has no affect.
-h,–help Display the above listed usage information.
-i <watchdir> Directory to watch for incoming test data files. This is the destination folder where the third-party file sharing service puts the test files produced by FilePump. Must be an existing directory. Defaults to the current directory (“.”).
-o <outfilename> Name of the output data file. Metric results (such as average latency and file throughput) are included in this file as well as all of the raw data.
-p <pollinterval_msec> FileWatch uses one of two methods to determine that new files have shown up in the incoming “watchdir”: (1) The default method is an event-based mechanism using Java “WatchService”. (2) A polling/file scan mechanism (activated using this “-p” command line flag) where FileWatch continually refreshes the list of files in the incoming “watchdir” and determines what files are new; to avoid pegging the system CPU, FileWatch will sleep for pollinterval_msec (milliseconds) between each folder scan (we recommend using 10msec as a starting point for this value).
-r <recasterdir> Turns on “recaster” mode and specifies the name of the folder where FileWatch should put the redirected test files. The directory must already exist. The redirected test files should be shared using the same file sharing service as was used for the original test files from FilePump; for example, if Dropbox was used to share the original test files from FilePump, then Dropbox should also be used to share the redirected test files to the final FileWatch.
-t <plottitle> Adds a custom title to the throughput and latency plots which are displayed at the conclusion of the test.

 

Test metrics

The goal of running a FilePump/FileWatch test is to characterize a file sharing service in order to anticipate its performance when used in a CloudTurbine network. Here’s a description of the metrics calculated by FileWatch; these values are printed to the screen at the end of a test as well as saved in the output data file.

  • actual source rate (files/sec): The actual FilePump output test file rate, or “source rate”.
  • Rku (files/sec): “Keep up rate”. This is the achieved file rate at the sink end; i.e., how many files/sec FileWatch observes coming in. To determine this value, FileWatch performs a linear regression of the cumulative number of files which have arrived versus time, the slope of which we define as Rku.
  • Lav (sec): “Average latency”. This is the average time it takes a file to go from source (FilePump) to destination (FileWatch). FileWatch calculates latency for each file it receives over time; a linear regression is applied to this latency versus time data, the y-intercept of which we define as Lav.
  • Lmax observed (sec): FileWatch calculates latency for each file it receives over time. “Lmax observed” is the maximum of these latency values.
  • Lmax calculated (sec): This is the “3 sigma” latency, a statistically derived maximum latency value defined as the average latency (Lav) plus 3 times the standard deviation of the latency values.
  • Lgr (sec/file): “Latency growth”. This is an estimate of the growth in latency with each file transferred by the file sharing service. In general, file sharing services have an overhead associated with an accumulation of files so that the service performs slower (higher latency) when dealing with a larger number of files. To determine Lgr, it is ideal to run a slow file test (e.g., 1 file/sec) for many hours. Lgr is defined as the slope of the linear regression of the latency vs. file index values (where “file index” is the order of creation of the file from FilePump).

Test types

Think of a file sharing service as a “pipe” into which files are fed at one end (the source end) and from which files stream out at the other end (the sink end). With this simplified model in mind, consider that there are two different types of tests used to characterize throughput and latency of a third-party file sharing service.

  • To successfully measure throughput, we need to fill the “pipe” quickly to see how many files per second are able to be transferred to the sink end. This is a fast “keep up rate” type of test. We are specifically interested in the sustainable throughput the file sharing service is capable of – that is, the file rate that the service could sustain indefinitely. This rate will of course be less than the service’s maximum possible throughput rate. You may need to run a series of FilePump/FileWatch tests in order to “zero in” on the sustainable throughput rate. If the throughput calculated by FileWatch nearly matches the output rate from FilePump, bump up FilePump‘s output rate and test again. You are very close to the sustainable rate when FilePump‘s rate is somewhat higher than the rate calculated by FileWatch. For each test, we recommend streaming files from FilePump anywhere from 10 minutes to 1 hour. Note that you don’t need to use “recaster” mode for throughput tests because wall-clock time isn’t referenced.
  • On the other end of the spectrum, to successfully measure latency, we need to slowly feed files into the “pipe” and see how long it takes them to arrive at the sink end. By “slowly feed” we mean at a rate which is well below the file sharing service’s sustainable throughput rate. We typically run FilePump at 1 file/sec for normal/one-way tests and slower (0.5 files/sec) for recaster/round-trip tests. Run FilePump for one to several hours for a throughput test; the goal is to get an average latency observed over a length of time. If clock synchronization is questionable between the two computer systems involved in the test, we recommend running a recaster/round-trip test as described above.

Test procedure

Steps to setup and conduct a test using FilePump and FileWatch.

1. If you are conducting a latency test and are not running in recaster/round-trip mode, make sure the system clocks (FilePump computer and FileWatch computer) are in sync.

2. How you launch FileWatch depends on what type of test is being conducted:

  • For a normal/one-way test, FileWatch is launched on the remote/destination computer as follows (folder name per the normal/one-way diagram above):

  • For a recaster/round-trip test (typically used when measuring latency), two instances of FileWatch are launched as follows (folder names per the recaster/round-trip diagram above):

3. Launch FilePump. Set the file rate based on the type of test being conducted: faster rate for throughput tests, slower rate (0.5 – 1 file/sec) for latency tests. Start the flow of test files by clicking the Start pump button in FilePump. Shortly after starting FilePump, you should see debug output from FileWatch: one “.” for each received file.

4. Allow FilePump to stream files for an “appropriate” length of time, where “appropriate” is based on the type of test being executed: one to several hours for latency tests; 10 minutes to an hour for throughput tests.

5. After you have stopped FilePump, wait for files to finish transferring to FileWatch (i.e., the stream of “.” in the FileWatch console window stop). When file transfers have stopped, click the Finish test button in FilePump, which writes a special signal file (“end.txt”) to the FilePump output directory. When this file has been transferred to the FileWatch input directory, FileWatch calculates and writes out the metrics data file and displays latency and throughput plots.

6. If you are running throughput tests, examine the output results and run additional tests if needed to “zero in” on the file sharing service’s sustainable throughput rate. As mentioned above, if the throughput calculated by FileWatch nearly matches the output rate from FilePump, bump up FilePump‘s output rate and test again. You are at the sustainable level when the rate FileWatch has observed is somewhat less than FilePump‘s output rate.