CloudTurbine uses Gradle for its build engine.  This post introduces the CloudTurbine build system, including some of its features and benefits, its structure, how to import CloudTurbine into an IDE, and how to extend the core build system to include additional projects.

Selecting Gradle as the CloudTurbine build system

A software project’s build system is an important component affecting the system’s structure and flexibility.  We wanted CloudTurbine’s build system to meet the following criteria:

  • easy setup: any build system will have a learning curve, but our desire was to select a modern build system which will allow us and other CloudTurbine community participant developers to quickly come up to speed on how to use and maintain the code
  • project hierarchy: CloudTurbine is made up of a number of individual projects. including core functionality, utilities and example programs; the build system should include multi-project and hierarchical support
  • Android support: CloudTurbine includes a number of Android projects, so it is critical that the build system support this operating system
  • Eclipse support: CloudTurbine developers often use Eclipse, so it is important that the build system be supported in this integrated development environment
  • easily extendable: it should be easy to add new projects to the build system

After considering the three main build system contenders (Ant, Maven and Gradle), Gradle was selected as the CloudTurbine build system.  Gradle (https://gradle.org/) is a modern build system which has garnered a lot of attention – it is being used by companies such as LinkedIn and Netflix and it is also the build tool for Android Studio (the official Android IDE).  Gradle can be used either as an imperative build system (like Ant) or a declarative build system (like Maven).  It uses Groovy or Kotlin for scripting and supports a multi-project build hierarchy.

Multi-project setup and Gradle benefits

As the figure below shows, CloudTurbine projects are organized into two top-level folders containing Android projects (CTandroidACL, CTandroidAV, etc.) and Java projects (CTlib, CTserver, etc.).  Folders with green check-mark icons are included in the GitHub repository; those without a check contain build outputs and only reside locally.

Gradle_layout_figure

Besides the individual Android and Java projects (CTandroidACL, CTlib, etc.), the file hierarchy contains the following additional folders:

  • In AndroidCode:
    • Common: contains needed build dependency JAR files; includes two JARs that are automatically copied from Java projects (CTlib.jar and CTserver.jar) and a third-party JAR (commons-net)
    • gradle: contains files for the Gradle “wrapper”, which allows a user to run Gradle commands without explicitly installing Gradle first; behind the scenes, the wrapper downloads and runs the appropriate version of Gradle
    • JavaDoc: includes a series of documentation sub-folders, one for each Android project
  • In JavaCode:
    • Distribute: contains a copy of the output JAR file from each Java project; these are “fat JAR” files, i.e. they include all needed dependencies embedded in the JAR
    • gradle: contains files for the Gradle “wrapper”, which allows a user to run Gradle commands without explicitly installing Gradle first; behind the scenes, the wrapper downloads and runs the appropriate version of Gradle
    • JavaDoc: includes a series of documentation sub-folders, one for each Java project
    • Third_Party: contains needed third party JAR files (RBNB and commons-net)

The updated build system is flexible.  Gradle tasks can be run from the command line or from an integrated development environment (we have tested Eclipse with the Buildship plugin for Java applications and Android Studio for Android applications).  You can execute a “top down” build from either AndroidCode or JavaCode to have all of the corresponding Android or Java projects built, or selectively go into a project folder and run the build for that individual project.

Another benefit of the Gradle-based build system is that it is easy to add new Android or Java projects by copying the structure of an existing project.  See the section “Add a new project” for details.

Gradle tasks

The following Gradle tasks can be run either for an individual project or at the AndroidCode/JavaCode level to execute the task for all child Android or Java projects:

  • “gradle tasks”: lists all available tasks with a brief description
  • “gradle build”: build executable outputs; this will create “.apk” files for Android projects and “.jar” files for Java projects; outputs are put in individual “build” directories within each project; content of the “build” directories are intentionally not included in the CloudTurbine GitHub repository
  • “gradle clean”: remove the project “build” directory
  • “gradle javadoc”: produce Javadoc documentation for the project

 

 

The core of the Gradle build system is made up of the following files:

  • AndroidCode/settings.gradle – lists the Android subprojects included in the multi-project build
  • AndroidCode/build.gradle – Android root project build file
  • AndroidCode/<subproject>/build.gradle – build file for each Android subproject
  • JavaCode/settings.gradle – lists the Java subprojects included in the multi-project build
  • JavaCode/build.gradle – Java root project build file
  • JavaCode/<subproject>/build.gradle – build file for each Java subproject

Each of these files will be examined in turn.  Selected portions of the files are presented below.  Please reference the full version of each file for additional comments and features not discussed here.

AndroidCode/settings.gradle

This file lists all of the Android subprojects to be included in the multi-project build.  The file is shown below.  If a new Android project is added to AndroidCode, its name should be added to this list.

The ‘:’ prefix to each project name represents the root project (AndroidCode, in this case); it can be thought of as the ‘/’ file path separator.

AndroidCode/build.gradle

This is the build file for the root Android project.  Although a stub “build” directory is created under AndroidCode, the main purpose of this root project is to configure the various Android subprojects (CTandroidACL, CTandroidAV, etc.).  There are two main sections to this file, each considered in turn below.

This section, from the top of the build file, specifies which version of the Android Plugin for Gradle will be used.

The code and settings specified in this subprojects closure will be applied to each of the Android subprojects.  Gradle refers to this as “configuration injection”; it is a way to define common settings so they don’t need to be repeated in each subproject’s build.gradle file (https://docs.gradle.org/current/userguide/organizing_build_logic.html#sec:injected_configuration).  Several items are specified for the subprojects:

  • apply the Android plugin; this defines a series of Gradle tasks useful for building and supporting Android applications
  • android closure specifies the versions of the Android SDK and build tools
  • repositories closure specifies places to search for library dependencies
  • define a javadoc task to build Javadoc documentation
  • extend the clean task to specify that the project’s Javadoc output folder should be deleted; this is performed at the start of the clean task (“doFirst”)
  • extend the build task to print out a message that the project has been built; this is performed at the end of the build task (“<<“; could also be specified as “doLast” similar to how “doFirst” was used with the clean task)
  • Specify task ordering.  The clean and javadoc tasks will be run prior to running build; additionally, the clean task will be run before the preBuild task.  See additional comments in build.gradle for the reasons behind this ordering.

AndroidCode/<subproject>/build.gradle

Each subproject has its own build.gradle file to add to the settings from the subprojects section of the root build file examined above.  For example, consider AndroidCode/CTandroidAV/build.gradle:

The only item being added to what was already defined in the subprojects closure in AndroidCode/build.gradle is to specify this subproject’s library dependencies: CTlib.jar and the Apache Commons Net library.  This dependencies closure specifies what libraries the subproject requires, whereas the repositories closure that we saw in AndroidCode/build.gradle specifies where to look for these libraries.  CTlib.jar will be found in AndroidCode/Common and the Apache Commons Net library will be downloaded from the Maven Central repository.  Further information about library dependencies is included under the “Library dependencies” tab.

JavaCode/settings.gradle

This file lists all of the Java subprojects to be included in the multi-project build.  The file is shown below.  If a new Java project is added to JavaCode, its name should be added to this list.

JavaCode/build.gradle

This is the build file for the root Java project.  The main purpose of this root project is to configure the various Java subprojects (CTlib, CTserver, CTudp, etc.).

The code and settings specified in the subprojects closure will be applied to each of the Java subprojects.  Gradle refers to this as “configuration injection”; it is a way to define common settings so they don’t need to be repeated in each subproject’s build.gradle file (https://docs.gradle.org/current/userguide/organizing_build_logic.html#sec:injected_configuration).  Several items are specified for the subprojects:

  • apply the Java plugin; this defines a series of Gradle tasks useful for building and supporting Java applications
  • repositories closure specifies places to search for library dependencies
  • Specify task ordering.  The clean and javadoc tasks will be run prior to running build; additionally, the clean task will be run before the compileJava task.
  • extend the build task to copy the JAR file to JavaCode/Distribute; this is performed at the end of the build task (“<<“; could also be specified as “doLast”)
  • extend the javadoc task to define settings for building Javadoc documentation
  • Extend the clean task to remove the subproject’s JAR file from JavaCode/Distribute and its Javadoc folder from JavaCode/JavaDoc.  These deletions are performed at the start of the clean task (“doFirst”).

JavaCode/<subproject>/build.gradle

Each subproject has its own build.gradle file to add to the settings from the subprojects section of the root build file examined above.  For example, consider JavaCode/CTudp/build.gradle:

Several items are defined for the CTudp project in this file:

  • Specify JAR file settings.  The “doFirst” specification is applied to force this task to run during Gradle’s “execution phase”.  Without this specification, this task would be run during the “configuration phase”, which would fail if CTlib.jar didn’t yet exist.  During “execution phase”, CTlib.jar will exist by the time this jar task is run.
  • Within the jar task enclosure, two things are specified:
    • The from closure specifies what to include in the JAR file.  Since we want this to be a self-contained “fat” JAR file, all classes on the “runtime” configuration path are included in the JAR file.
    • The “Main-Class” and “Class-Path” settings are specified for the Manifest file.  “Main-Class” should be the fully qualified name (i.e. including package name) of the application’s entry point.
  • repositories closure specifies places to search for needed libraries; this will be added to the repositories setting specified in the root Gradle build file, JavaCode/build.gradle; repositories tells Gradle where to look for needed libraries
  • dependencies closure specifies what libraries to include, in this case CTlib.jar and the Apache Commons CLI library (used for command line argument processing).  CTlib.jar will be found in JavaCode/CTlib/build/libs and the Apache Commons Net library will be downloaded from the Maven Central repository.  Further information about library dependencies is included under the “Library dependencies” tab.

 

Each application has its own library requirements.  These are specified in build.gradle files as “repositories” (defines where to find libraries) and “dependencies” (defines which libraries are required).

As an example of how libraries are specified, consider the CTudp Java application which requires the main CloudTurbine library (CTlib.jar) and the Apache Commons CLI library (for command line argument processing).

Let’s first examine the root Gradle build file at JavaCode/build.gradle.  As specified in the snippet shown below, Gradle will look in two places to locate libraries for the subprojects (CTlib, CTudp, etc.): the local JavaCode/Third_Party directory and the Maven central repository.  Other commonly used library locations can be added to this “repositories” section.

Next, let’s look at the “repositories” and “dependencies” sections from the CTudp Gradle build file, JavaCode/CTudp/build.gradle:

In this snippet, we see that a new repository location is being added to those specified in the root Gradle build file: “$rootProject.projectDir/CTlib/build/libs”; this allows CTudp to locate the CTlib JAR file.

The “dependencies” section in this snippet specifies the following:

  • Make sure CTlib is up-to-date before building CTudp
  • Specify CTlib as a library dependency (this will be found at “$rootProject.projectDir/CTlib/build/libs”)
  • Specify Apache Commons CLI as a library dependency (this will be downloaded from the Maven central repository)

Additional information on the synatax used in these Gradle build file snippets is specified below.

In JavaCode/build.gradle, what are “subprojects”?  We have setup JavaCode as a multi-project Gradle build.  JavaCode is the name of the root project; it contains several subprojects (CTlib, CT2CSV, CTapps, etc.).  The Gradle instructions specified in the “subprojects” section will be applied to all of the subprojects.  Gradle refers to this as “configuration injection”; it is a way to define common settings so they don’t need to be repeated in each subproject’s build.gradle file (https://docs.gradle.org/current/userguide/organizing_build_logic.html#sec:injected_configuration).

Curly brackets define a Groovy “closure”, in which code or settings can be defined.

Specifying the Apache Commons CLI library.  The “group”, “name” and “version” tags provide information about the specific library we want to use.  These tags can be used to locate the library in a central repository (such as the Maven central repository).  For CTudp, we want to use “commons-cli-1.3.1.jar”.  To determine values for the “group”, “name” and “version” tags, it can be useful to lookup the library at https://mvnrepository.com/.  The information for “commons-cli-1.3.1.jar” is at https://mvnrepository.com/artifact/commons-cli/commons-cli/1.3.1.  On this page, click on the “Gradle” tab to see the syntax shown above.

 

 

A feature of the CloudTurbine Gradle-based build system is that the software can be built either from the command line or from within an Integrated Development Environment (IDE).  Building from the command line can be simple and convenient, but IDEs offer many built-in features to assist with editing and code development and therefore it is also useful to integrate Gradle build tasks in the IDE.  Since Google’s termination of Android project support in Eclipse as of the end of 2015 (https://www.infoq.com/news/2015/06/google-android-eclipse), our CloudTurbine development is split between two IDEs: Java applications (projects under the JavaCode folder) are developed in Eclipse and Android applications (projects under the AndroidCode folder) are developed in Android Studio.

Importing Java projects into Eclipse

We use the Eclipse Buildship plug-ins (https://projects.eclipse.org/projects/tools.buildship) to provide Gradle support in Eclipse.  Following the instructions at http://www.vogella.com/tutorials/EclipseGradle/article.html, it is easy to setup Buildship and import all of the projects under the JavaCode folder.

Eclipse will automatically create “.classpth” and “.project” files in each JavaCode sub-project folder (CTlib, CTserver, CTudp, etc.)  There should be no need to edit these automatically created Eclipse files; projects should only be configured via the Buildship plugin and build.gradle files.

Note that Buildship v2.0 and later do not include the ability to specify a Java JDK version (i.e., what JDK would you like to use to build your Gradle project).  This is now done by specifying the path to a JDK with an org.gradle.java.home setting in a gradle.properties file (https://docs.gradle.org/current/userguide/build_environment.html).  This file can be located, for instance, in a .gradle folder in your computer account home directory.

Although not a universal issue, we have seen the following occur on a Windows machine after importing the projects into Eclipse: the CTsync project had a number of “Access restriction” errors. In our case, the issue was easily resolved by following the suggestion given at http://stackoverflow.com/questions/860187/access-restriction-on-class-due-to-restriction-on-required-library-rt-jar; namely, removing and then adding back the JRE System library as follows:

  1. Bring up the Properties window for the CTsync project
  2. Go to the “Java Build Path”
  3. Click on the “Libraries” tab
  4. Remove the currently listed “JRE System Library”
  5. Click on “Add Library…” and add back the “JRE System Library”

Importing Android projects into Android Studio

The collection of Android applications in AndroidCode are easily imported into Android Studio.  In the “Welcome to Android Studio” window that pops up when the application launches, click on “Import project (Eclipse ADT, Gradle, etc.)” and then select the “settings.gradle” file located in AndroidCode. All of the projects should automatically load.

After loading CloudTurbine projects into Android Studio, you may see the following popup:

AndroidCode_unregistered_root_warning

This is letting you know that Android Studio’s version control integration features have not been enabled for this project. You can select one of the given options:

  • Add root: If Git is configured in Android Studio, selecting this option will register the project to use Android Studio’s Git integration.  If Git has not been fully configured, a Git settings dialog will pop up (you may, for instance, need to enter the path to the Git executable).
  • Configure: Android Studio’s Settings dialog box pops up showing a list of “Unregistered roots” (i.e., projects which don’t have version control enabled).  Select the CloudTurbine project from the list and click the small “+” button to register the project.
  • Ignore: Click this option if you don’t want to use the Git version control tools in Android Studio.  Clicking “Ignore” will avoid this warning popup from being displayed on subsequent launches of Android Studio.  Many of the version control tools mentioned below will not be available if you click this option.

Version control options and features are available in several places in the Android Studio user interface:

  • VCS menu in the main menu bar
  • Settings are available by selecting File > Settings… > Version Control
  • Display a Version Control tool window by clicking Alt+9 on Windows and Linux platforms or Command+9 on Mac platforms
  • If you have enabled Git integration for the CloudTurbine projects in Android Studio, each file’s right-click menu will have a new “Git” menu
  • There is a “Git” popup menu available on the status bar at the bottom of the user interface

 

 

Use the following procedure to add a new CloudTurbine project to the Gradle-based build system.

Before adding the new project: If AndroidStudio or Eclipse are open, save your work and exit.

Adding a new Android project

  1. Add a new, empty folder under the AndroidCode directory; the name of this new folder will be the name of the new Gradle project.
  2. Copy build.gradle from one of the existing Android projects into this new folder (for example, CTandroidAV/build.gradle).  Adjust the “dependencies” section of this build file to include those libraries required for the new project.  This file (“build.gradle”) is the only file which should be copied from the existing project directory.
  3. Add new source code files under src/main/java in the new project folder.  Additional sub-folders will be added depending on the package name of the new project.  For instance, let’s say the package name is com.foo.androidproj.  In this case, the source code files would go in src/main/java/com/foo/androidproj.
  4. Resources should be placed under src/main/res.
  5. A manifest file may be placed at src/main/AndroidManifest.xml.
  6. Add the name of the new project in AndroidCode/settings.gradle; this file specifies all projects included in the “multi-project” build.

Adding a new Java project

  1. Add a new, empty folder under the JavaCode directory; the name of this new folder will be the name of the new Gradle project.
  2. Copy build.gradle from one of the existing Java projects into this new folder (for example, CTudp/build.gradle).  Note that this one file is the ONLY file which should be copied from the existing project directory into your new project directory.  There are some variations in build.gradle amongst the projects.  CTlib’s build.gradle is possibly the “most different” variation of build.gradle as it specifies Java compatibility version, it doesn’t build a “fat jar” file and it copies the compiled JAR over to the AndroidCode area.  CTserver is also a bit different as it also specifies Java compatibility version and copies the compiled JAR over to the AndroidCode area.  The remaining build.gradle files are very similar.  One change that will be necessary is to adjust the “Main-Class” setting for the manifest file.  In addition, adjust the “repositories” and “dependencies” sections to include needed library files.
  3. Add new source code files under src/main/java in the new project folder.  Additional sub-folders will be added depending on the package name of the new project.  For instance, let’s say the package name is com.foo.javaproj.  In this case, the source code files would go in src/main/java/com/foo/javaproj.
  4. Add the name of the new project in JavaCode/settings.gradle; this file specifies all projects included in the “multi-project” build.

Add the new Java project to your Eclipse workspace

If you have added a new Java project and are using the Eclipse IDE, the following procedure can be followed to add this new project to your Eclipse workspace alongside your other existing Java projects.  It is assumed you have already imported the other CloudTurbine Java projects into Eclipse using the Buildship plugin (see the “Import into Eclipse and Android Studio” tab for additional details).

  1. Start Eclipse and open your workspace which contains the JavaCode projects.
  2. In Eclipse’s Package Explorer panel, open up the JavaCode branch.  You should see items such as a Distribute folder, a gradle folder, etc.  One of the items you will see is the build.gradle file for the root JavaCode project.  Right-click on build.gradle and select “Refresh”.  Then, right-click on JavaCode again and select “Refresh Gradle Project” from the Gradle sub-menu.  Your new project should automatically be inserted into your workspace alongside the existing JavaCode projects.

Note that Eclipse will automatically create “.classpth” and “.project” files in your new sub-project folder.  There should be no need to edit these automatically created Eclipse files; the new project should only be configured via the Buildship plugin and build.gradle file.

 

 

We occasionally update the Gradle build files included in the CloudTurbine distribution or the version of Gradle itself being used to build CloudTurbine.  In addition, updates are occasionally released to other tools used for CloudTurbine development (Eclipse and Android Studio, for instance).

Follow these instructions to update your local development and build system:

  1. Exit Eclipse and Android Studio before proceeding.
  2. Developers can use the Gradle wrapper files which come with CloudTurbine.  You use the Gradle wrapper files if you build CloudTurbine projects from within Eclipse or if you execute “gradlew” commands from a terminal window.  If this is your situation, proceed to the next step.  However, if you use Gradle directly (for instance, you execute “gradle” commands from a terminal window) do the following:
    1. Consider installing the most recent Gradle version (https://gradle.org/install).  When you install an update, make sure to also update your GRADLE_HOME and path environment variables.  Double-check your new installation by executing “gradle –version” from a command window to make sure the new version is running.
    2. If you are a core CloudTurbine developer and have updated your Gradle version in the previous step and you want to update the Gradle wrappers to use this new version of Gradle: bring up a terminal window and execute “gradle wrapper” in both the AndroidCode and JavaCode directories.
  3. Update your local CloudTurbine files using git fetch or git pull or GitHub Desktop.
  4. Install updates to Android Studio (note: this may automatically adjust the version of the Android Plugin for Gradle referenced in AndroidCode/build.gradle).
  5. Open your Eclipse workspace that contains the CloudTurbine projects.  Install updates to Eclipse and to the Buildship plugin (select “Check for Updates” from the Help menu).  Note that Buildship v2.0 and later do not include the ability to specify a Java JDK version (i.e., what JDK would you like to use to build your Gradle project).  This is now done by specifying the path to a JDK with an org.gradle.java.home setting in a gradle.properties file (https://docs.gradle.org/current/userguide/build_environment.html).  This file can be located, for instance, in a .gradle folder in your computer account home directory.
  6. You shouldn’t need to re-import the CloudTurbine Gradle projects in Eclipse.  Instead, right-click on the “JavaCode” project under Package Explorer and chose “Refresh Gradle Project” from the Gradle menu.
  7. Execute a full build of all Java projects in Eclipse and Android project in Android Studio.  You can also execute full builds from a terminal window by executing “gradlew build” in the JavaCode and AndroidCode directories.