Setting up a Jenkins CI engine for embedded projects

Welcome to the sixth part of our “embedded testing” series. From our former blogposts, you are now familiar with PlatformIO and its powerful features like unit testing, remote controlling and continuous integration. On the fifth embedded testing blogpost, we’ve been diving into Travis CI in order to automate build and testing processes, deployment of software artifacts to servers. CI tools like TravisCI and Jenkins services can completely automate build and testing processes, as well as deployment of software artifacts to servers (and embedded, of course). It would be nice to have something comparable for the world of IoT as well. TravisCI is a cloud solution, meaning it will always build on fresh servers (or better: containers), but remote. Maybe you'd want to have your builds running locally. Today we have chosen Jenkins, a well-known CI service that’s used often in software development processes, and show how to set it up on a RaspberryPi, in conjunction with PlatformIO

Jenkins CI

Jenkins CI is an open source automation server, provides a magnitude of plugins to support building, deploying and automating your project. Easy installation, easy configuration, plugins, extensions and multiple platforms’ compatibility makes Jenkins a great choice. And we're going to use it for our embedded projects as well! The physical setup we’ve been using consists on a RaspberryPi and a NodeMCU that is connected to RPi via USB. The above PlatformIO project is for NodeMCUs firmware, which will be built, uploaded and tested locally from Jenkins CI.

Let’s plug in our RPi, login and start with the login to PlatformIO:

$ pio account login
<your login data>

It will be necessary to get & save our PIO Token - PLATFORMIO_AUTH_TOKEN flag - in order to authenticate the CI server later:

$ pio account token
<Output>
PlatformIO Plus (https://pioplus.com) v0.10.11
Password:
Personal Authentication Token: xxxxxx11122233444aaabbbcccddee

Copy the token and store it somewhere else for later usage.

For using the Jenkins on RPi board, we need Java JDK-8. Please type this and select JDK-8 by giving the selection number:

$ sudo update-alternatives --config java
  Selection      Path                                                   Priority
------------------------------------------------------------
   0            /usr/lib/jvm/java-7-openjdk-armhf/jre/bin/java          1063      Automatic
   1            /usr/lib/jvm/java-7-openjdk-armhf/jre/bin/java          1063      Manual
   2            /usr/lib/jvm/java-8-openjdk-armhf/jre/bin/java          1063      Manual
 * 3            /usr/lib/jvm/jdk-8-oracle-arm32-vfp-hflt/jre/bin/java   318       Manual

Now it is time to install Jenkins to our RPi, starting with making a new directory, then install Jenkins:

$ mkdir jenkins && cd jenkins
$ wget -q -O - https://jenkins-ci.org/debian/jenkins-ci.org.key | sudo apt-key add -
$ sudo sh -c 'echo deb http://pkg.jenkins-ci.org/debian binary/ > /etc/apt/sources.list.d/jenkins.list'
$ sudo apt-get update
$ sudo apt-get install Jenkins

Now, Jenkins is installed and should automatically be running. For our demo setup we also place the Git Repository on the RaspberryPi, but this can of course be on GitHub, a hosted GitLab or some other repository server in your network. We put ours in /opt/git:

$ mkdir /opt/git 
$ cd /opt/git
$ mkdir ci-with-platformio && chmod g+rwx ci-with-platformio
$ ( cd ci-with-platformio; git init --bare --shared )

Now we have an empty, bare repository at /opt/git/ci-with-platformio. Since both repository and jenkins instance run on the same server, we're only cloning locally and need to take care of file system access rights to this directory. We'll put both our pi and jenkins users into the same group and give group rights to the directory. Additionally, we'd like to have jenkins build our project automatically whenever a git push occurs. Let's put a post-receive hook into the repo by:

$ cd /opt/git/ci-with-platformio/
echo '#/bin/bash
/usr/bin/curl \
--user <YOUR_USERNAME>:<YOUR_PASS> \ 
-s \
http://127.0.0.1:8080/git/notifyCommit?url=/opt/git/ci-with-platformio
' >hooks/post-receive
$ chmod g+rwx hooks/post-receive

Good, now we need some code to work with :-) Let's take the code from the last blog posts about unit testing. It's at github.com/emirez/ci-with-platformio, and we can simply push this into our local ci-with-platformio repo:

$ cd ~
$ git clone https://github.com/emirez/ci-with-platformio.git
$ cd ci-with-platformio
$ git remote add local /opt/git/ci-with-platformio
$ git push local
$ cd ..
$ rm -fr ./ci-with-platformio
$ git clone /opt/git/ci-with-platformio
$ cd ci-with-platformio
$ # we can make changes here and push to the local repo (later)

For restarting Jenkins:

$ sudo /etc/init.d/jenkins restart

Here is how your project looks like:

$ tree
├── README.md
├── lib
│   └── mod1
│       └── src
│                ├── mod1.cpp
│                └── mod1.h
├── src
│        └── main.cpp
├── platformio.ini
├── .travis.yml
├──test
│        └── native
│        ├── test_first.cpp
│        ├── nodemcuv2
└        └── test_main.cpp

By opening our browser and typing the following address, we will see the Jenkins CI interface at http://<your_device_name_or_ip>:8080

For our setup, it's http://tfpi.local:8080

Hint 1: Jenkins CI interface uses the language of your browser. If you want to change the language of the Jenkins CI, please change your browser language.

Image 1: Jenkins CI UI

If you already have an account, please login. Otherwise, please navigate to “Create an Account”. After creating your account, you can login and see the next page.

Image 2: Jenkins CI UI on 8080 Port

To start with a new project, please click on New Item.

Image 3: Jenkins CI New Item Initiation

It is the new item page that allows us to select the type of the project that we will build and test automatically. After giving a name, select Freestyle Project and click on OK.

The fine tuning parameters, environment variables, sketches and etc. should be specified now:

  • Type some information about the project under description.
  • Git project and it’s path will be written under source code management.
  • Build triggers and build environment can be selected arbitrarily, you can basically set them as you.
  • Build sketches are the commands that you want to automatize. Please see the global environment flag PLATFORMIO_AUTH_TOKEN, set it to the token value that you obtained before by $ pio account token command.
  • Save and apply.

Image 4: Jenkins CI Project Settings

After every push to our local git repository, a new build will be started automatically due to the post-receive hook we set up above:

$ cd ci-with-platformio-jenkins
$ # change something ..
$ git add .
$ git commit -m "first Jenkins commit"
$ git push

In order to see the build console output, please click on the arrow next to the project name:

Image 6: Project build output on Jenkins

Here is the console output:

Image 7: Console Test Output

You can always build the project by clicking on Build now in the context menu of a project:

Image 8: Build now on context menu

The weather icon shows the condition about your last builds. If it is rainy, that means there are some fails at your last 5 builds. If your last 5 builds are successful, then you will see the sun. Hint 2: Probably you see the blue ball whenever your builds are successful. You can paint it green by installing an extension to the Jenkins. To do so, please navigate to Manage Jenkins > Manage Plugins and install the Green Balls extension. Restart your Jenkins and see them in green color.

Finalising today: So what did we actually achieve? Using our PlatformIO’s project template, we’re able to automatise all the build & testing process, including the installation of Jenkins CI. However, devices can still be connected locally, i.e. a notebook or as in our setup, a RPi simulation our IoT integration farm. Remote function of PIO would be also used in order to reach the integration farm remotely. The goal of this topic is to getting deeper into Agile Embedded process. Every code push triggers a build and testing process, and ideally flashes firmware to some test devices, and runs on-device test processes as well. So every team member is able to inspect the current build and test status on a dashboard and see where the team stands in terms of delivery. This concludes the sixth part of these series, and we’re evolving from testing into running a complete IoT Toolchain. Stay tuned for the upcoming parts, where we’ll be looking under the hood of PlatformIO’s build system and see how to integrate special actions.

Eren

Andreas