• Spring Web MVC + JDBC database driver
• Spring Web MVC + R2DBC database driver
• Spring WebFlux + JDBC database driver
• Spring WebFlux + R2DBC database driver

• Process CPU usage
  User and kernel time (based on /proc/PID/stat)
• Memory usage
  Private and shared process memory (based on /proc/PID/maps)
• Response time
  As reported by wrk
• Throughput (number of requests)
  As reported by wrk

Concurrency on the x-axis and CPU usage on the y-axis

Memory usage [Mb] on the y-axis and concurrency on the x-axis

Response time [ms] on the y-axis and concurrency on the x-axis

Number of requests which could be processed in 60 seconds on the y-axis and concurrency on the x-axis

• At high concurrency, the benefit of using R2DBC instead of JDBC is obvious. This allows your cores to be utilized more efficiently since less CPU is required to process a single request. Response times at high concurrency are also better when using R2DBC instead of JDBC.
• WebFlux provides also several benefits when used instead of Web MVC such as stable and higher throughput at high concurrency.
• You're not required to have a completely non-blocking stack to reap the benefits of using R2DBC or WebFlux.

• Be careful with your memory usage when using R2DBC, especially in combination with WebFlux.
• JPA cannot deal with reactive repositories such as provided by Spring Data R2DBC. This means you will have to do more things manually when using R2DBC.
• There are other reactive drivers around such as for example Quarkus Reactive Postgres client (which uses Vert.x). This does not use R2DBC and has different performance characteristics (see here).
• Limited availability
  Not all relational databases have reactive drivers available. For example, Oracle does not (yet?) have an R2DBC implementation. 
• Application servers still depend on JDBC.
  Do people still use those for non-legacy in this Kubernetes age?
• When Java Fibers will be introduced (Project Loom, could be Java 15), the driver landscape might change again and R2DBC might not become JDBCs successor after all.

• Every node uses its own local registry. The Makefile used does provide ways to load the same image to the different registries but what I actually want is all the nodes to use the same registry.
• There is no shared PersistentVolume available within the Kubernetes environment which can be used. Shared storage. Preparations for that have been made though since /shared in every VM is mounted to the same local folder.
• We have not installed anything in the environment yet but a small dashboard. I want to have Jenkins running inside and be able to deploy applications.

Kubernetes has become the de facto container orchestration platform to run applications on. Java applications are no exception to this. When using a PaaS provider to give you a hosted Kubernetes, sometimes that provider also provides a CI/CD solution. However this is not always the case. When hosting Kubernetes yourself, you also need to implement a CI/CD solution.

Jenkins is a popular tool to use when implementing CI/CD solutions. Jenkins can also run quite easily in a Kubernetes environment. When you have Jenkins installed, you need to have a Git repository to deploy your code from, a Jenkins pipeline definition, tools to wrap your Java application in a container, a container registry to deploy your container to and some files to describe how the container should be deployed and run on Kubernetes. In this blog post I'll describe a simple end-to-end solution to deploy a Java service to Kubernetes. This is a minimal example so there is much room for improvement. It is meant to get you started quickly.

The Java application

Installing Jenkins

For my Kubernetes environment I have used the setup described here. You also need a persistent storage solution as prerequisite for Jenkins. In a SaaS environment, this is usually provided, but if it is not or you are running your own installation, you can consider using OpenEBS. How to install OpenEBS is described here. kubectl (+ Kubernetes configuration .kube/config) and helm need to be installed on the machine from which you are going to perform the Jenkins deployment.

After you have and a storage class, you can continue with the installation of Jenkins.

First create a PersistentVolumeClaim to store the Jenkins master persistent data. Again, this is based on the storage class solution described above.

 kubectl create ns jenkins  

 kubectl create -n jenkins -f - <<END  
 apiVersion: v1  
 kind: PersistentVolumeClaim  
 metadata:  
  name: jenkins-pv-claim  
 spec:  
  storageClassName: openebs-sc-statefulset  
  accessModes:  
   - ReadWriteOnce  
  resources:  
   requests:  
    storage: 8Gi  
 END  

Next install Jenkins. Mind that recently the Jenkins repository for the most up to date Helm charts has moved.

 cat << EOF > jenkins-config.yaml  
 persistence:  
   enabled: true  
   size: 5Gi  
   accessMode: ReadWriteOnce  
   existingClaim: jenkins-pv-claim  
   storageClass: "openebs-sc-statefulset"
 EOF  

 helm repo add jenkinsci https://charts.jenkins.io  
 helm install my-jenkins-release -f jenkins-config.yaml jenkinsci/jenkins --namespace jenkins  

Now you will get a message like:

Get your 'admin' user password by running:

 printf $(kubectl get secret --namespace jenkins my-jenkins-release -o jsonpath="{.data.jenkins-admin-password}" | base64 --decode);echo

Get the Jenkins URL to visit by running these commands in the same shell and login!

 export POD_NAME=$(kubectl get pods --namespace jenkins -l "app.kubernetes.io/component=jenkins-master" -l "app.kubernetes.io/instance=my-jenkins-release" -o jsonpath="{.items[0].metadata.name}")

 kubectl --namespace jenkins port-forward $POD_NAME 8080:8080

Now visit http://localhost:8080 in your browser

Configuring Jenkins

The pipeline

Tool configuration

  tools {  
   jdk 'jdk-11'
   maven 'mvn-3.6.3'
  }  

  stages {  
   stage('Build') {  
    steps {  
     withMaven(maven : 'mvn-3.6.3') {  
      sh "mvn package"
     }  
    }  
   }  

 sh 'curl -LO "https://storage.googleapis.com/kubernetes-release/release/$(curl -s https://storage.googleapis.com/kubernetes-release/release/stable.txt)/bin/linux/amd64/kubectl"'
 sh 'chmod u+x ./kubectl'
 sh './kubectl apply -f k8s.yaml'

 kubectl apply -f https://raw.githubusercontent.com/metallb/metallb/v0.9.3/manifests/namespace.yaml
 kubectl apply -f https://raw.githubusercontent.com/metallb/metallb/v0.9.3/manifests/metallb.yaml
 # On first install only
 kubectl create secret generic -n metallb-system memberlist --from-literal=secretkey="$(openssl rand -base64 128)"

 kubectl apply -f - <<END 
 apiVersion: v1
 kind: ConfigMap
 metadata:
   namespace: metallb-system
   name: config
 data:
   config: |
     address-pools:
     - name: default
       protocol: layer2
       addresses:
       - 192.168.122.150-192.168.122.255
 END

 kubectl create deployment spring-boot-demo --image=docker.io/maartensmeets/spring-boot-demo --dry-run=client -o=yaml > k8s.yaml
 echo --- >> k8s.yaml
 kubectl create service loadbalancer spring-boot-demo --tcp=8080:8080 --dry-run=client -o=yaml >> deployment.yaml

 withKubeConfig([credentialsId: 'kubernetes-config']) {  
      sh 'curl -LO "https://storage.googleapis.com/kubernetes-release/release/$(curl -s https://storage.googleapis.com/kubernetes-release/release/stable.txt)/bin/linux/amd64/kubectl"'
      sh 'chmod u+x ./kubectl'
      sh './kubectl apply -f k8s.yaml'
     }  

 withCredentials([usernamePassword(credentialsId: 'docker-credentials', usernameVariable: 'DOCKER_USERNAME', passwordVariable: 'DOCKER_PASSWORD')]) {  
      sh "mvn jib:build"
     }  

Jenkins job configuration

The configuration of the Jenkins job is actually almost the least exciting. The pipeline is defined outside of Jenkins. The only thing Jenkins needs to know is where to find the sources and the pipeline.

Create a Multibranch Pipeline job. Multibranch is quite powerful since it allows you to build multiple branches with the same job configuration.

Webhook configuration

  smee -u https://smee.io/z8AyLYwJaUBBDA5V -t http://localhost:8080/github-webhook/

Finally

Anchore Engine is a popular open source tool for container image inspection and vulnerability scanning. It is easily integrated in a Kubernetes environment as an admission controller or in a Jenkins build pipeline using a plugin. A while ago I took a look at Anchore Engine and created a small introductory presentation and Katacoda scenario for it. The Katacoda scenario allows you to try out Anchore Engine without having to setup your own container environment. In this blog I'll go a step further and illustrate how you can incorporate an Anchore Engine container scan inside your Java build pipeline which I illustrated here. Anchore Engine is deployed to Kubernetes, configured in Jenkins (which also runs on Kubernetes) and incorporated in a Jenkins Pipeline during a build process. Only if the container has been deemed secure by the configured Anchore Engine policy, is it allowed to be deployed to Kubernetes. I will also show how to update policies using the CLI.

Prerequisites

In order for the following steps to work, you need to have a Kubernetes cluster running and kubectl + helm configured to allow deployments to it. The Kubernetes cluster is expected to have internet access in order to download the container image from DockerHub and to update the Anchore Engine vulnerability lists. I've used the following Kubernetes environment using the following storage configuration and the following Jenkins installation for this.

Deploying Anchore Engine to Kubernetes

The following can be used to deploy Anchore Engine to Kubernetes. This is based on the following documentation. The Helm chart installs a PostgreSQL database, but you can also use one supplied externally. In production you should definitely use your own PostgreSQL installation which has been deployed in a high-available setup. Of course I would also change Welcome01 to a more secure one if you need it.

 cat << EOF > anchore_values.yaml  
 postgresql:  
  postgresPassword: Welcome01  
  persistence:  
   size: 10Gi  

 anchoreGlobal:  
  defaultAdminPassword: Welcome01  
  defaultAdminEmail: maarten.smeets@amis.nl  
 EOF  

 helm repo add anchore https://charts.anchore.io  
 helm repo update  

 kubectl create ns anchore  
 helm install anchore-release -n anchore -f anchore_values.yaml anchore/anchore-engine

In order to check the deployment, you are provided with several handy commands after the installation is finished such as a command to start a container which has the anchore-cli installed. You also get information on the URL at which you can access Anchore Engine from within the Kubernetes cluster. You need this information for the Jenkins configuration.

 kubectl run -i --tty anchore-cli --restart=Always --image anchore/engine-cli --env ANCHORE_CLI_USER=admin --env ANCHORE_CLI_PASS=Welcome01 --env ANCHORE_CLI_URL=http://anchore-release-anchore-engine-api.anchore.svc.cluster.local:8228/v1/    

 [anchore@anchore-cli anchore-cli]$ anchore-cli system status  
 Service simplequeue (anchore-release-anchore-engine-simplequeue-75b49c55c5-bktbv, http://anchore-release-anchore-engine-simplequeue:8083): up  
 Service policy_engine (anchore-release-anchore-engine-policy-5576ff74b4-8rv5r, http://anchore-release-anchore-engine-policy:8087): up  
 Service apiext (anchore-release-anchore-engine-api-5b5ddb8cd6-k66hm, http://anchore-release-anchore-engine-api:8228): up  
 Service catalog (anchore-release-anchore-engine-catalog-5648d4df64-5rtq9, http://anchore-release-anchore-engine-catalog:8082): up  
 Service analyzer (anchore-release-anchore-engine-analyzer-5588cc6964-489h7, http://anchore-release-anchore-engine-analyzer:8084): up  

 Engine DB Version: 0.0.13  
 Engine Code Version: 0.8.1  

Configuring Jenkins

The Jenkins configuration is relatively straightforward. You need to install the Anchore Container Image Scanner plugin:

Next you need to configure the Anchore Engine location and credentials. The location is the URL as previously obtained from the output of the Helm chart installation and the credentials as supplied in the anchore_values.yaml file.

Configure and run your pipeline

I created a minimal Jenkins Java build pipeline a while back which I wanted to expand with Anchore Engine scanning. I added the following to my pipeline:

   stage('Anchore analyse') {  
    steps {  
     writeFile file: 'anchore_images', text: 'docker.io/maartensmeets/spring-boot-demo'
     anchore name: 'anchore_images'
    }  
   }  

Next I ran it:

kubectl run -i --tty anchore-cli --restart=Always --image anchore/engine-cli --env ANCHORE_CLI_USER=admin --env ANCHORE_CLI_PASS=Welcome01 --env ANCHORE_CLI_URL=http://anchore-release-anchore-engine-api.anchore.svc.cluster.local:8228/v1/    

Next determine and download the used policy

 [anchore@anchore-cli anchore-cli]$ anchore-cli policy list  
 Policy ID                  Active    Created           Updated             
 2c53a13c-1765-11e8-82ef-23527761d060    True     2020-10-04T11:30:19Z    2020-10-04T11:30:19Z      
 [anchore@anchore-cli anchore-cli]$ anchore-cli policy get 2c53a13c-1765-11e8-82ef-23527761d060 --detail > /tmp/policybundle.json  

When you edit the file with vi, you can find the dockerfile warning. You can also see at the bottom of the file a whitelists definition. In this whitelist we can add the following in the items tag to exclude this specific warning.

The gate and trigger_id can be determined from the report given by the Anchore plugin in Jenkins or by looking up the vulnerability in the json file you just downloaded.

Next update the policy and activate it:

 [anchore@anchore-cli anchore-cli]$ anchore-cli policy add /tmp/policybundle.json   
 Policy ID: 2c53a13c-1765-11e8-82ef-23527761d060  
 Active: False  
 Source: local  
 Created: 2020-10-04T11:30:19Z  
 Updated: 2020-10-04T14:03:25Z  

 [anchore@anchore-cli anchore-cli]$ anchore-cli policy activate 2c53a13c-1765-11e8-82ef-23527761d060  
 Success: 2c53a13c-1765-11e8-82ef-23527761d060 activated

When you now execute the pipeline again, you will notice the warning is gone (listed as Go) and it is indicated as whitelisted.

Finally

Deploying Anchore Engine to your Kubernetes cluster is relatively easy. Using it in a Jenkins pipeline is also. By adding it to your build process, you can confirm images used do not have vulnerabilities you might want to avoid. Also using the Anchore CLI you can edit the policy which determines if the build succeeds or not. 

Anchore Engine also provides a notification system. If you scan a container and new vulnerabilities are discovered because vulnerability feeds get updated, you can be informed by webhook calls. This allows you to take action when needed or feed dashboards (like an ELK stack).

What this does not provide is in-debt scanning of images, just the libraries which are present in the container. It depends on a publicly available vulnerability database. Anchore has an Enterprise version available which adds additional vulnerability feeds. It also provides an option to create a local proxy for the vulnerability feeds which you are not dependent on an internet connection. Also you get reports on entire Docker registries and a GUI for configuring policies. The Enterprise version has advanced SSO/RBAC options and an easy way to integrate notifications with Slack, Jira, GitHub, etc. If you want to use Anchore Engine at scale, I would take a look at the Enterprise version.

If you want to scan for example Java library dependencies, you can use the OWASP dependency check (of which (amongst others) a Maven plugin and Gradle plugin are available). By using several measures to scan your application/container/image for vulnerabilities, you can make sure your application is safe to run! In this blog post I tried to show that implementing Anchore Engine (in a basic setup) is a piece of cake and there is no good reason not to use it.

The OWASP top 10 has listed the following vulnerability for several years (at least in 2013 and 2017): using components with known vulnerabilities. But software nowadays can be quite complex consisting of many dependencies. How do you know the components and versions of those components do not contain known vulnerabilities? Luckily the OWASP foundation has also provided a dependency-check tool with plugins for various languages to make detecting this more easy. In this blog post I'll show how you can incorporate this in a Jenkins pipeline running on Kubernetes and using Jenkens and SonarQube to display the results of the scan.

Prerequisites

I used the environment described here. This includes a ready configured kubectl and helm installation. For the Jenkins installation and basic pipeline configuration I used the following here. In order to execute the below steps you should have at least a Kubernetes and Jenkins installation ready. If you want to use the literal code samples, you also require the Jenkins configuration as described including this.

OWASP dependency-check

The OWASP foundation provided Dependency-Check plugins for various build tools such as Ant, Gradle and Maven and a Jenkins plugin. They also have a standalone CLI tool available. Mind that the more specific a plugin you use, the more relevant the findings will be. You can for example use the Dependency-Check Jenkins plugin to perform a scan, but it will not understand how dependencies inside a pom.xml work so will not give sufficiently useful results. You will get something like below:

When you implement the Maven Dependency-Check plugin to produce results and the Jenkins Dependency-Check plugin to get those results visible in Jenkins, you get results which are specific to a Maven build of your Java application. This is quite useful! When using this you will get more accurate results like below.

The Jenkins Dependency-Check plugin (which can be used within a pipeline) also produces trend graphs and html reports inside Jenkins.

Thus use the Maven Dependency-Check plugin to scan your project and use the Jenkins plugin to publish the results generated from the scan to Jenkins. After you have installed and configured SonarQube, you can use the same results to publish them to SonarQube.

Maven plugin configuration

You can find the pom.xml file I used here. You can execute the scan by running mvn dependency-check:check.

<plugin>  
<groupId>org.owasp</groupId>  
<artifactId>dependency-check-maven</artifactId>  
<version>6.0.2</version>  
<executions>  
<execution>  
<goals>  
<goal>check</goal>  
</goals>  
</execution>  
</executions>  
<configuration>  
<failBuildOnCVSS>7</failBuildOnCVSS>  
<!-- Generate all report formats -->  
<format>ALL</format>  
<!-- Don't use Nexus Analyzer -->  
<centralAnalyzerEnabled>false</centralAnalyzerEnabled>  
<!-- Am I the latest version? -->  
<versionCheckEnabled>true</versionCheckEnabled>  
</configuration>  
</plugin>  

Notice the format ALL is specified. This generates an HTML, JSON, XML and CSV report in the target folder. SonarQube will use the JSON report and Jenkins the XML report. I have not looked at the central analyzer (this appears to be a feature which is part of the commercial Nexus distribution). This might help reduce build time since the vulnerability files can be shared across scans and do not need to be downloaded every time.

You can browse the HTML report yourself if you like. Using this you can confirm that the dependencies were identified and actually scanned. My scan found 0 vulnerabilities so I was worried the scan was not executed correctly but the results showed the different dependencies and how they were evaluated to they were correct. A new Spring Boot version does not contain vulnerable dependencies as it should!

Installing SonarQube

There are various ways you can install SonarQube on Kubernetes. There are Docker images available which require you to create your own Kubernetes resources, but also a Helm chart. The Helm chart is really easy to use. For example, it creates a PostgreSQL database for the SonarQube installation for you without additional effort. The PostgreSQL database which is created is of course not highly available, clustered, etc. Using Helm value overrides, you can specify your own PostgreSQL DB to use which you can setup to fit your availability needs. The default username and password is admin. Of course change this in a production environment. A small drawback is that the chart is not 100% up to date. Currently it installs version 8.3 while 8.5 is already available. I'm not sure if this is an indication not much maintenance is being done on the chart. I hope this is not the case of course, else I would not recommend it. If you do use it, keep this in mind. For a lab scenario like this one I do not care whether I get version 8.3 or 8.5.

Installing SonarQube on Kubernetes can be done with the following commands using the helm chart;

 helm repo add oteemocharts https://oteemo.github.io/charts  
 helm repo update  
 kubectl create ns sonar  
 helm install -n sonar sonar-release oteemocharts/sonarqube  

After the installation is complete, you can access it with:

 export POD_NAME=$(kubectl get pods --namespace sonar -l "app=sonarqube,release=sonar-release" -o jsonpath="{.items[0].metadata.name}")  
 kubectl -n sonar port-forward $POD_NAME 8080:9000  

And going to http://localhost:8080. Username and password are as indicated admin

Configuring SonarQube in Jenkins

In the previously described setup, you can access Jenkins with:

 export POD_NAME=$(kubectl get pods --namespace jenkins -l "app.kubernetes.io/component=jenkins-master" -l "app.kubernetes.io/instance=my-jenkins-release" -o jsonpath="{.items[0].metadata.name}")  
 printf $(kubectl get secret --namespace jenkins my-jenkins-release -o jsonpath="{.data.jenkins-admin-password}" | base64 --decode);echo  
 kubectl --namespace jenkins port-forward $POD_NAME 8081:8080   

The second line gives the password of the admin user you can use to login. The last command makes a Jenkins pod available on localhost port 8081. 

In SonarQube you can obtain a secret. The SonarQube pipeline plugin in Jenkins can be configured to use the secret to store results from the build/dependency-check in SonarQube.

   stage ('OWASP Dependency-Check Vulnerabilities') {  
    steps {  
     withMaven(maven : 'mvn-3.6.3') {  
      sh 'mvn dependency-check:check'
     }  

     dependencyCheckPublisher pattern: 'target/dependency-check-report.xml'
    }  
   }  

   stage('SonarQube analysis') {  
    steps {  
     withSonarQubeEnv(credentialsId: 'sonarqube-secret', installationName: 'sonarqube-server') {  
      withMaven(maven : 'mvn-3.6.3') {  
       sh 'mvn sonar:sonar -Dsonar.dependencyCheck.jsonReportPath=target/dependency-check-report.json -Dsonar.dependencyCheck.xmlReportPath=target/dependency-check-report.xml -Dsonar.dependencyCheck.htmlReportPath=target/dependency-check-report.html'
      }  
     }  
    }  
   }  

Executing the pipeline

When you execute the pipeline, SonarQube gets fed with various results and can also produce a measure of technical debt in your project. In this case test coverage (produced by the Maven Jacoco plugin) and data produced by the OWASP Dependency-Check. 

SonarQube makes a verdict on whether the build passes or not and this is displayed in Jenkins by the SonarQube Scanner plugin.

You can of course further expand this to include more thorough code quality checks by including Maven plugins such as Checkstyle, PMD and JDepend.

There are a lot of use-cases in which you might want to automate a web-browser. For example to automate tedious repetitive tasks or to perform automated tests of front-end applications. There are also several tools available to do this such as Selenium, Cypress and Puppeteer. Several blog posts and presentations by Lucas Jellema picked my interest in Playwright so I decided to give it a try. I'm not a great fan of JavaScript so I decided to go with Python for this one. I also did some tests with wrk, a simple yet powerful HTTP bench-marking tool, to get an indication about how Playwright would handle concurrency and was not disappointed.

Introduction

Playwright

The following here gives a nice comparison of Selenium, Cypress, Puppeteer and Playwright. Microsoft Playwright has been created by the same people who created Google Puppeteer and is relatively new. Playwright communicates bidirectionally with the browser. This allows events in the browser to trigger events in your scripts (see here). This is something which can be done but is more difficult using something like Selenium (see here, I suspect polling is usually involved). With Selenium you also often need to build delays inside your scripts. Playwright provides extensive options for waiting for things to happen and since the interaction is bidirectional, I suspect polling will not be used which increases performance and makes scripts more robust and fast.

I used PlayWright on Python. The Node.js implementation is more mature. On Python, before the first official non-alpha release, there might still be some breaking changes in the API so the sample provided here might not work in the future due to those API changes. Since the JavaScript and Python API are quite similar, I do not expect major changes though.

Python

Python 3.4 introduced the asyncio module and since Python 3.5 you can use keywords like async and await. Because I was using async libraries and I wanted to wrap my script in a webservice, I decided to take a look at what webservice frameworks are available for Python. I stumbled on this comparison. Based on the async capabilities, simple syntax, good performance and (claimed) popularity, I decided to go with Sanic.

Google Translate API

Of course Google provides a commercial translate API. If you are thinking about seriously implementing a translate API, definitely go with that one since is is made to do translations and provide SLAs. I decided to create a little Python REST service to use the Google Translate website (a website scraper). For me this was a tryout of Playwright so in this example, I did not care about support, performance or SLAs. If you overuse the site, you will get Capcha's and I did not automate those away.

Getting things ready

I started out with a clean Ubuntu 20.04 environment. I tried JupyterLabs first but Playwright and JupyterLabs did not seem to play well together since probably JupyterLabs also extensively uses a browser itself. I decided to go with PyCharm. PyCharm has some code completion features which I like among other things and of course the interface is similar to IntelliJ and DataGrip which I also use for other things.

 #Python 3 was already installed. pip wasn't yet  
 sudo apt-get install python3-pip  
 sudo pip3 install playwright  
 sudo pip3 install lxml  
 #Webservice framework  
 sudo pip3 install sanic  
 sudo apt-get install libenchant1c2a  
 #This installs the browsers which are used by Playwright  
 python3 -m playwright install  
 #PyCharm  
 sudo snap install pycharm-community --classic  

 from playwright import async_playwright  
 from sanic import Sanic  
 from sanic import response  

 app = Sanic(name='Translate application')  

 @app.route("/translate")  
 async def doTranslate(request):  
   async with async_playwright() as p:  
     sl = request.args.get('sl')  
     tl = request.args.get('tl')  
     translate = request.args.get('translate')  
     browser = await p.chromium.launch() # headless=False  
     context = await browser.newContext()  
     page = await context.newPage()  
     await page.goto('https://translate.google.com/?sl='+sl+'&tl='+tl+'&op=translate')  
     textarea = await page.waitForSelector('//textarea')  
     await textarea.fill(translate)  
     waitforthis = await page.waitForSelector('div.Dwvecf',state='attached')  
     result = await page.querySelector('span.VIiyi >> ../span/span/span')  
     textresult = await result.textContent()  
     await browser.close()  
     return response.json({'translation':textresult})  

 if __name__ == '__main__':  
   app.run(host="0.0.0.0", port=5000)  

You can also find the code here.

How does it work?

The app.run command at the end starts an HTTP server on port 5000. @app.route indicates the function doTranslate will be available at /translate. The async function receives a request. This request is used to obtain the GET arguments which are used to indicate 'target language' (tl), 'source language' (sl) and the text to translate (translate). Next a Chrome browser is started in headless mode. Headless is the default for Playwright but during development, it helps to disable this headless mode so you can see what happens in the browser. The Google translate site is opened with the source and target language as GET parameters. Next Playwright waits until a textarea appears (the p[age is fully loaded). It fills the text area with the text to be translated. Next Playwright waits for the translation to appear (the box 'Translations of  auto' in the screenshot below). The result is selected and saved. First I select span.VIiyi (a CSS selector) and within that span I select ../span/span/span (an XPATH selector). After the result is obtained, I close the browser and return it

When you start the service, you will get something like

 [2021-01-24 09:16:38 +0100] [3278] [INFO] Goin' Fast @ http://0.0.0.0:5000  
 [2021-01-24 09:16:38 +0100] [3278] [INFO] Starting worker [3278]  

Next you can test the service. You can do this with 

 curl 'http://localhost:5000/translate?sl=nl&tl=en&translate=auto'

It will translate the Dutch (sl=nl) 'auto' (translate=auto) to English (tl=en). The result will be car.

 {"translation":"car"}  

Performance

Of course performance tests come with a big disclaimer. I tried this on specific hardware, on a specific OS, etc. You will not be able to exactly reproduce these results. Also the test I did was relatively simple just to get an impression. I additionally logged the responses since the script I used did not include proper exception handling.

I used wrk, an HTTP benchmark tool on the service.

 wrk -c2 -t2 -d30s --timeout=30s 'http://localhost:5000/translate?sl=nl&tl=en&translate=auto'

WRK used 2 threads and kept 2 connections open at the same time (2 concurrent requests). Per request I set the timeout to 30s. This timeout was never reached.

This gave me average results of 2.5s per request. At 4 concurrent requests, this became 3s on average. At 8 concurrent requests this became 4.5s on average and started to give some errors. A 'normal' API can of course do much better. I was a bit surprised though Playwright could handle 8 headless browsers simultaneously on an 8Gb Ubuntu VM which also had PyCharm open at the same time (knowing what a memory-hog Chrome can be). I was also surprised Google didn't start to bother me with Capcha's yet.

Tips for development

Automatically generate scripts

You can use Playwright to automatically generate scripts for you from manual browser interactions. This can be a good start of a script. See for example here. The generated scripts however do not contain smart logic such as waiting for certain elements in the page to appear before selecting text from other elements.

Use browser developer tools

In order to automate a browser using Playwright, you need to select elements in the web-page. An easy way to do this is by looking at the developer tools which are present in most web-browsers nowadays. Here you can easily browse the DOM (document object model, the model on which the browser bases what it shows you). This allows you to find specific elements of interest to manipulate using Playwright.

Approximate human behavior

One of the dangers of creating screen scrapers is that if the site changes, your code might not work anymore. In my sample script I used specific identifiers like Dwvecf and VIiyi and queried for elements based on the sites DOM. When you look at a website yourself, you select elements in a more visual way. The better you can approximate the way a human would interact with a website, the more stable your script will be. For example, selecting the first textarea on the site is more stable then expecting the result to be in span.VIiyi and in that element under span/span/span. 

The right tool for the job

If an API is available and you can use it directly, that is of course preferable to using a website scraper since an API is made for automated interaction and a web-site is made for human interaction. You usually get much better performance and stability when using an official API instead. Playwright includes an API to monitor and modify HTTP and HTTPS requests done by the browser. This might help you in determining back-end APIs so you can try if you can use them directly.

When using a tool like Playwright to automate browser interaction in for example tests for custom developed applications, you can get better stability since you know what will change when in the site and can make sure the automation scripts keep working. When you're using Playwright against an external website, you will have less control. Google will not inform me when they change https://translate.google.com and this script will most likely break because of it.

There are situations where you want to change Java code not at the source code level but at runtime. For example when you want to instrument code for logging purposes but do not want to change the source code (because you might not have access to it). This can be done by using a Java Agent. For example, Dynatrace uses a Java agent to collect data from inside the JVM. Another example is the GraalVM tracing agent (here) which helps you create configuration for the generation of native images. Logging is one use-case but you can also more dramatically alter runtime code to obtain a completely different runtime behavior.

This blog post is not a step by step introduction for creating Java Agents. For that please take a look at the following. In this blog post I have created a Java agent which rewrites synchronized methods to use a ReentrantLock instead (see here). The use-case for this is to allow applications to use Project Loom's Virtual Threads more efficiently.

You can find the code of the agent here.

The Java Agent

When I created the agent, I encountered several challenges, which I listed below and how I dealt with them. This might help you to overcome some initial challenges when writing your own Java Agent.

Keep the dependencies to a minimum

First, the JAR for the agent must include its dependencies. You often do not know in which context the agent is loaded and if required dependencies are supplied by another application. The maven-dependency-plugin with the descriptor reference jar-with-dependencies helps you generate a JAR file including dependencies.

When you include dependencies however in the JAR containing also your agent, they might cause issues with other classes which are loaded by the JVM (because you usually run an application or even application server). The specific challenge which I encountered was that I was using slf4j-simple and my application was using Logback for logging. The solution was simple; remove slf4-simple. More generally; use as few external dependencies as viable for the agent. I used only javassist. I needed this dependency to easily do bytecode manipulation.

Trigger class transformation for every class

There are several solutions available to obtain the correct bytecode. Be careful though that your bytecode transformer is triggered for every class that is loaded when it is loaded. Solutions which only look at currently loaded classes (instrumentation.getAllLoadedClasses()) or a scan based on reflection utils like javassist (example here) won't do that. You can register a transformer class that will be triggered when a specific class is loaded, but if you want your agent to be triggered for every class, it is better to register a transformer without having a specific class specified; instrumentation.addTransformer(new RemsyncTransformer());. The agent will be the first to be loaded when using the premain method / the javaagent JVM switch when starting your application.

Of course, if you want to attach the agent to an already running JVM (using the agentmain method), you still need to process the already loaded classes.

Obtain the bytecode

I've seen several examples on how to obtain the bytecode of the class to edit in the transformer class. The signature of the transform method of transformer class is: 

 public byte[] transform(ClassLoader loader, String className, Class<?> classBeingRedefined,  
               ProtectionDomain protectionDomain, byte[] classfileBuffer) throws IllegalClassFormatException  

It is tempting to use the loader, className and classBeingRedefined to obtain the relevant CtClass instance. The CtClass instance can be edited using javassist. For example by doing something like: 

 String targetClassName = className.replaceAll("\\.", "/");  
 ClassPool cp = ClassPool.getDefault();  
 CtClass cc = cp.get(targetClassName);  

I'm however not a great fan of string parsing / editing to obtain the correct name of a class and then fetch it. The classfileBuffer contains the bytecode of the class being loaded and this can be edited directly. I implemented this like below (based on this which contains a more elaborate explanation).

  private final ScopedClassPoolFactoryImpl scopedClassPoolFactory = new ScopedClassPoolFactoryImpl();  

  ClassPool classPool = scopedClassPoolFactory.create(loader, ClassPool.getDefault(),ScopedClassPoolRepositoryImpl.getInstance());  
  CtClass ctClass = classPool.makeClass(new ByteArrayInputStream(classfileBuffer));  

Editing the bytecode

I used the following code to add a private instance variable, remove the synchronized modifier and wrap the contents of the method in a try/finally block:

 try {  
   ctField = ctClass.getDeclaredField("lockCustomAgent");  
 } catch (NotFoundException e) {  
   ctClass.addField(CtField.make("final java.util.concurrent.locks.Lock lockCustomAgent = new java.util.concurrent.locks.ReentrantLock();", ctClass));  
 }  

 method.instrument(new ExprEditor() {  
   public void edit(MethodCall m) throws CannotCompileException {  
     m.replace("{ lockCustomAgent.lock(); try { $_ = $proceed($$); } finally { lockCustomAgent.unlock(); } }");  
   }  
 });  

 modifier = Modifier.clear(modifier, Modifier.SYNCHRONIZED);  
 method.setModifiers(modifier);  

What this did was change the following Java code

   synchronized void hi() {  
     System.out.println("Hi");  
   }  

to the following

   final java.util.concurrent.locks.Lock lockCustomAgent = new java.util.concurrent.locks.ReentrantLock();  

   void hi() {  
    lockCustomAgentStatic.lock();  
    try {   
      System.out.println("Hi");  
    } finally {   
      lockCustomAgentStatic.unlock();   
    }  
   }  

This is exactly as described here at 'Mitigating limitations'.

Generate your JAR manifest

A manifest file helps the JVM to know which agent classes to use inside the JAR file. You can supply this file and package it together with the compiled sources in the JAR or let it be generated upon build. I choose the second option. The following will generate a manifest indicating which agent class needs to be started by the JVM. 

<plugin>  
<artifactId>maven-assembly-plugin</artifactId>  
<version>3.3.0</version>  
<executions>  
<execution>  
<phase>package</phase>  
<goals>  
<goal>single</goal>  
</goals>  
</execution>  
</executions>  
<configuration>  
<descriptorRefs>  
<descriptorRef>jar-with-dependencies</descriptorRef>  
</descriptorRefs>  
<archive>  
<manifestEntries>  
<Premain-Class>nl.amis.smeetsm.agent.RemsyncInstrumentationAgent</Premain-Class>  
<Agent-Class>nl.amis.smeetsm.agent.RemsyncInstrumentationAgent</Agent-Class>  
<Can-Redefine-Classes>true</Can-Redefine-Classes>  
<Can-Retransform-Classes>true</Can-Retransform-Classes>  
</manifestEntries>  
</archive>  
</configuration>  
</plugin>  

Finally

The agent described here illustrated some of the options you have for editing bytecode. I noticed though when creating this agent, that there are many situations where the agent cannot correctly rewrite code. This is probably because my rewrite code is not elaborate enough to deal with all code constructs it may encounter. An agent is probably only suitable for simple changes and not for complex rewrites. One of the strengths of using a method like this is though that you do not need to change any source code to check out if a certain change does what you expect. I was surprised at how easy it was to create a working agent that changes Java code at runtime. If you have such a use-case, do check out what you can do with this. PS for Project Loom, just using virtual threads and rewriting synchronized modifiers did not give me better performance.

You can find the code of the agent here.

It is often expected of a DevOps team to also take security into consideration when delivering software. Often however, this does not get the attention it deserves. In this blog post I'll describe some easy to use, CI/CD pipeline friendly, open source tools you can use to perform several checks during your Java software delivery process which will help you identify and fix issues with minimal effort.

You can view my sample project here which implements all these tools. There is also a docker-compose.yml file supplied. SonarQube en Jenkins however do not come preconfigured in this setup. You can look at the Jenkinsfile (pipeline definition) to see what Jenkins configuration is required (installing plugins and creating credentials). 

This is provided as an overview and small example only. It is not meant as a walkthrough to setup a CI/CD environment or a manual how to use the different tools. Also the mentioned tools are not all the tools which are available but a set of tools which are freely available, popular and which I managed to get working without too much effort. Using them together allows you to improve the security of your Java applications during your software delivery process and provide quick feedback to developers. This can also help increase security awareness. When you have some experience with these tools you can implement more strict policies (which can let a build fail) and quality gates.

Static code analysis

What does it help you solve?

Static application security testing (SAST) helps you identify several issues in your source code by looking at the code itself before compilation. Some issues which static analysis can help prevent are;

• Exposed resources
  Static analysis tools can help detect exposed resources. These might be vulnerable to attacks.
• Hardcoded keys or credentials
  If keys or passwords are hardcoded inside your  application, attackers might be able to extract them by for example decompiling the Java code
• Stability issues
  This can be related to memory leaks such as not closing resources which will no longer be used. This can also be related to performance. Bad performance can lead to stability issues. An attacker might purposefully try to bring your application down by abusing performance issues. 

Which tools can you use?

When talking about Java applications, the following tools are free to use to perform static analysis.

• PMD (here)
  PMD can be used as a Maven or Gradle plugin (and probably also in other ways). It has a Jenkins plug-in available and the SonarQube Maven plugin can be supplied with parameters to send PMD reports to SonarQube. PMD can find a small number of actual security vulnerabilities. It can provide quite a lot of performance related suggestions.
• SpotBugs (here)
  SpotBugs is the spiritual successor of FindBugs. It can (like PMD) easily be integrated in builds and CI/CD pipelines. The SonarQube Maven plugin also supports sending PMD reports to SonarQube without additional plugin requirements. SpotBugs can find bugs in the area of security and performance among others (here).
• FindSecBugs (here)
  This is an extension of SpotBugs and can be used similarly. It covers the OWASP TOP 10 and detects /classifies several security related programming bugs.

Dynamic Application Security Testing (DAST)

Some things cannot easily be automatically identified from within the code. For example, how the application looks on the outside. For this it is not unusual to perform a penetration test. Penetration tests can luckily be automated!

What does it help you solve?

• Scan web applications and analyse used headers
• Check exposed endpoints and security features of those endpoints
• Check OpenAPI / SOAP / Websocket communication/services
• Check traffic between browser and service

Which tools can you use?

• OWASP ZAP (Zed Attack Proxy)
  Can run in a standalone container, started from a Maven build using a plugin, configured as a tool within Jenkins. You can install a plugin (here) in SonarQube to visualize results and use the HTML publisher plugin in Jenkins to show the results there. OWASP ZAP can also be used together with Selenium to passively scan traffic between the browser and the service or actively modify requests.

I did not take a look at other tools to do similar things yet. 

Dependency analyses

What does it help you solve?

• Java programs use different libraries to build and run. You can think of frameworks, drivers, utility code and many other re-usable assets. Those dependencies can contain vulnerabilities. Usually for older libraries, more vulnerabilities are known. Knowing about vulnerabilities in specific versions of libraries and their severity, can help you in prioritizing updating them.

Container image vulnerability scanning

What does it help you solve?

• A Java program can run in a container on a container platform. A container is usually created by using a base image and putting an application inside. The base image contains operating system libraries or other software (such as a JVM) which can contain vulnerabilities.

Which tools can you use?

• Anchore Engine (here).
  Anchore Engine is available as a container image. Once started, it will download information from various vulnerability databases. You can request a certain image to be analyzed. It has good integration with SonarQube and Jenkins. Read my blog posts about using Anchore Engine here and here.

OWASP ZAP or Zed Attack Proxy is an open source dynamic application security testing (DAST) tool. It is available here and has a website with documentation here. I recently encountered it when looking for open source security test tools to embed in a CI/CD pipeline (here). I was surprised by how versatile this tool is. In this blog post I'll summarize several ways how you can use it. 

Demo application

I've used a small Spring Boot service to illustrate the different ways you can use OWASP ZAP. You can find it in DockerHub here and in GitHub here. 

To create a Docker network and start the application, do the following;

 docker network create demo-network

 docker run --network demo-network --name spring-boot-demo -p 8080:8080 maartensmeets/spring-boot-demo  

ZAP CLI

OWASP ZAP CLI can be run from a container (read here). You can for example execute a scan like listed in the command below. Mind that by default only High severity issues are displayed. That's why there is '-l Low' at the end.

 docker run --network demo-network -i owasp/zap2docker-stable zap-cli quick-scan --spider --self-contained --recursive --start-options '-config api.disablekey=true' http://spring-boot-demo:8080/rest/demo/ -l Low  

This can give you output like;

 [INFO]      Starting ZAP daemon  
 [INFO]      Running a quick scan for http://spring-boot-demo:8080/rest/demo/  
 [INFO]      Issues found: 1  
 +---------------------------------------+--------+----------+-----------------------------------------+  
 | Alert                 | Risk  |  CWE ID | URL                   |  
 +=======================================+========+==========+=========================================+  
 | X-Content-Type-Options Header Missing | Low  |    16 | http://spring-boot-demo:8080/rest/demo/ |  
 +---------------------------------------+--------+----------+-----------------------------------------+  
 [INFO]      Shutting down ZAP daemon  

Using a GUI

Webswing allows you to run OWASP ZAP from a container with a GUI which you can open in a browser. Read here. 

  docker run -u zap -p 8081:8080 -p 8090:8090 --network demo-network -i owasp/zap2docker-stable zap-webswing.sh  

Now you can go to localhost:8081/zap to open the GUI. 

Quickscan

You can perform an automated scan from the GUI to quickly scan a specific endpoint. This will yield results similar to the zap-cli example above.

Proxy requests

You can also use port 8090 of the container as proxy. URLs which pass the proxy will be listed in the GUI and you can easily attack all detected endpoints. 

It will ask you to accept the self-signed ZAP certificate. You can import the ZAP root CA certificate to permanently to avoid the warning in the future. Read the following here on how to do that.

You can also see the request and response in the ZAP GUI.

As you can see, this gives more information than only a scan of a specific endpoint since OWASP ZAP can now look at the traffic between browser and service. Here it detects an additional header missing. It is also possible to set breakpoints and alter requests. In addition, there is scripting support available. See for example here.

Embedded in a CI/CD process

There are various ways in which you can embed an OWASP ZAP scan in your CI/CD process. Since it is a DAST scan, it requires a running application. Also ZAP itself needs to be running in order to execute scans. The specific method is highly dependent on your environment. 

Install

Depending on your environment you can spin up ZAP as a container or install ZAP yourself by for example using the Linux installer which you can find here. 

Start and stop

Ideally, you want to start ZAP, execute a scan and clean-up. This in order not to have it claim resources while not in use (and saving you money, especially in a cloud environment).

You can use a plugin such as this one to start and stop a ZAP container or a local installation. If the plugin does not suffice, it is easy to script it yourself as part of your pipeline.

Execute

Once your application and ZAP are running, there are various ways to initiate a scan. You can access its API (directly, using the CLI or by using for example a plugin) and ask it to do scans of specific endpoints. Using a plugin is relatively easy.

You can embed a ZAP scan in a Selenium test so you can first proxy requests and than analyse the proxied traffic (passive) and attack those endpoints (active). See for example the following presentation here on how to set that up. This requires a bit more work.

Process ZAP results

If you want to integrate the results of the scan in for example SonarQube, you can use the following plugin here. You can also integrate it in Azure DevOps by transforming the result XML file to a format Azure DevOps understands (see here).

Example

I created a docker-compose.yml file which starts ZAP and my test application here. Next from the pom.xml file (Maven, it is a Java application) I use a plugin during the build process to access ZAP using the API and initiate a scan. The result is published to Jenkins and imported in SonarQube. A setup such as this one is of course not viable for real-life use since the ZAP spin-up and spin-down are not part of it (I start everything at the same time when I do docker-compose up -d).

Finally

I was happily surprised by how many integration options OWASP ZAP has and the different ways you can run ZAP. It has a GUI, a CLI and a powerful API. This makes it easy to embed it into any CI/CD pipeline to dynamically scan endpoints. The proxy option in combination with an automated GUI test is very powerful. It allows ZAP to first do a passive scan of traffic (identifying endpoints, analyzing requests/responses) and with the information obtained, to do an active attack. Such elaborate scans can take a while though, but they can provide a lot of information on potential vulnerabilities. In my opinion, there is no good reason not to embed OWASP ZAP in your software delivery process to make the world a little bit safer by performing automated security tests.

I'm a regular user of GitHub. Recently I discovered GitHub also has a build-in CI/CD workflow solution called GitHub Actions. Curious about how this would work I decided to try it out. I had previously build a Jenkins Pipeline to perform several static and dynamic application security tests on a Java project and decided to try and rebuild this pipeline using GitHub Actions. This blog described my first experiences and impressions.

First impressions

No external environment required

One of the immediate benefits I noticed was that the Jenkins Pipeline requires a Jenkins installation to be executed. GitHub Actions work on their own in your GitHub repository so no external environment is required. Do mind that GitHub Actions have some limitations. If you want to use them extensively, you will eventually probably need to pay.

Integrated with GitHub

GitHub Actions are of course specific to GitHub and integrate well with repository functionality. Because of this, you would probably only use them if your source code is in GitHub. There are quite a lot of options to have a workflow triggered. For example on a push or pull request, on a schedule or manually. Because GitHub Actions are integrated with GitHub and linked to your account, you will get automatic mails when a workflow succeeds or fails.

If you want to integrate Jenkins with GitHub, the communication is usually done via webhooks. This requires you to expose your Jenkins environment in a way GitHub can reach it. This can be challenging to do securely (might require you to punch holes in your firewall or use a polling mechanism). Also if you want to be informed of the results of a build by e-mail, you need to define mail server settings/credentials in Jenkins. For GitHub Actions this is not required.

Reporting capabilities

My Jenkins / SonarQube environment provide reporting capabilities. At the end of a build you can find various reports back in Jenkins and issues are created in SonarQube. In SonarQube you can define quality gates to let the build fail or not.

Using GitHub Actions, this works differently. Letting a build fail is something to explicitly script instead of configured using a GUI. At the end of a build you can export artifacts. These can contain reports but you need to open them yourself. Another option is to publish them to GitHub Pages (see here). Thus no 'out of the box' capability to do reporting for GitHub Actions.

GitHub has so-called "Issues" but these appear not to be meant to register individual findings of the different static code analysis tools. They work better to just register a single issue for a failed build. Of course, there are GitHub Actions available for that.

Extensibility

To be honest, I did not dive into how to create custom pipeline steps in my Jenkins environment. I noticed you can code some Groovy and integrate that with a Pipeline but it would probably require quite a time investment to obtain the required knowledge to get something working. For GitHub Actions, creating your own Actions is pretty straightforward and described here. You create a repository for your Action, a Dockerfile and a yaml describing the inputs and outputs of the Action. Next you can directly use it in your GitHub workflows. As the Dutch would say; 'een kind kan de was doen' (literally: 'a child can do laundry'. It is a saying which means: 'it's child's play')

Challenges rebuilding a Jenkins Pipeline

I had created a simple Jenkins Pipeline to compile Java code, perform static code analysis, create a container, deploy it to Docker Hub, scan the container using Anchore Engine, perform an automated penetration test using OWASP ZAP, feed the results to SonarQube and check a Quality Gate.

The main challenges I had with rebuilding the pipeline were working with containers during the build process and how to deal with caching.

Starting and stopping containers

The main challenges I had when building the Jenkins Pipeline was that Jenkins was running inside a container and I wanted to spawn new containers for Anchore, ZAP and my application. Eventually I decided to use docker-compose to start everything upfront and do a manual restart of my application after the container image in Docker Hub was updated. This challenge is similar when running Jenkins on K8s.

Using GitHub Actions, the environment is different. You choose a so-called runner for the build. This can be a self-hosted environment but can also be GitHub hosted. GitHub offers various flavors of runners. In essence, this gives you for example a Linux VM on which you can run stuff, start and stop containers and more or less do whatever you want to perform your build. The Jenkins challenge of starting and stopping containers from within a container is just not there! 

For GitHub Actions, you should be aware there are various ways to deal with containers. You can define service containers which will start when your workflow starts. You can also execute certain actions in a container you specify with the 'uses' keyword. When these do not suffice, you can call docker directly yourself from a run command or by calling an external script. I chose the last option in my workflow. See my script here for starting the OWASP ZAP scan in a container.

Caching artifacts

To reduce build time, you can cache artifacts such as Maven dependencies but also of container images. For the container images you can think of images cached by Google Jib or images cached by the Docker daemon. I tried caching images from the Docker daemon but that increased my build time. Most likely because of how the cache worked. Every image which was saved in the cache and later restored, included all underlying layers. Probably certain layers are part of multiple images so they are saved multiple times. It appeared downloading only the required layers was quicker than caching them and restoring them  on a next build. If you are however more worried about network bandwidth than storage, you can of course still implement this. For my project, caching the Maven and Jib artifacts caused my build time to go from 7m 29s to 4m 45s so this appeared quite effective. Probably because artifacts are only downloaded once (no doubles) and there are many small artifacts which all require creating a connection to a Maven repository. Applying a saved repository was quicker than downloading in this case.

Setting environment variables

Setting environment variables in GitHub Actions was not so much of a challenge but in my opinion, the method I used felt a bit peculiar. For pushing an image from my Maven build to Docker Hub, I needed Docker Hub credentials. Probably I should have used a Docker Hub access token instead for added security but I didn't.

I needed to register my Docker Hub credentials in GitHub as secrets. You can do this by registering repository secrets and refer to them in your GitHub Actions workflow using something like: ${{ secrets.DOCKER_USERNAME }} .

   steps:  
    - name: Set environment variables  
     run: |  
      echo "DOCKER_USERNAME=${{ secrets.DOCKER_USERNAME }}">> $GITHUB_ENV  
      echo "DOCKER_PASSWORD=${{ secrets.DOCKER_PASSWORD }}">> $GITHUB_ENV  

I could also have specified the environment at the step level or at the workflow level using the env keyword like;

  env: # Set environment variables
    DOCKER_USERNAME: ${{ secrets.DOCKER_USERNAME }}
    DOCKER_PASSWORD: ${{ secrets.DOCKER_PASSWORD }} 

The first option is very flexible since you can use any bash statement to fill variables. The second option however is more readable.

Getting started with GitHub Actions

Creating GitHub Actions is relatively simple. You create a file called .github/workflows/main.yml in your GitHub repository. You can use a provided starters from GitHub specific to a language or create one from scratch for yourself.

When you have created such a file, you can go to the Actions tab in GitHub after you have logged in;

You can in my case open the CI workflow and run it manually (I choose to do it manually but as mentioned, you can use other triggers). When performing a build manually, you can define inputs but I did not use that functionality.

When you open a build, you can view logging and generated artifacts (under summary). In my example I published the test reports as artifacts and pushed my created container to Docker Hub. I was not so much interested in the generated JAR file.

Mind that I didn't need to setup any environment on my own but just used the freely available GitHub Actions functionality.

Pipeline code

You can browse the pipeline here.

 name: CI  

 # Controls when the action will run.   
 on:  
  # Allows you to run this workflow manually from the Actions tab  
  workflow_dispatch:  

 # A workflow run is made up of one or more jobs that can run sequentially or in parallel  
 jobs:  
  # This workflow contains a single job called "build"
  build:  
   # The type of runner that the job will run on  
   runs-on: ubuntu-20.04  

   # Steps represent a sequence of tasks that will be executed as part of the job  
   steps:  
   - name: Set environment variables  
     run: |  
      echo "DOCKER_USERNAME=${{ secrets.DOCKER_USERNAME }}">> $GITHUB_ENV  
      echo "DOCKER_PASSWORD=${{ secrets.DOCKER_PASSWORD }}">> $GITHUB_ENV  

   # Checks-out your repository under $GITHUB_WORKSPACE, so your job can access it  
   - name: Checkout sources  
     uses: actions/checkout@v2  

   # Runs a single command using the runners shell  
   - name: Setup Java 11  
     uses: actions/setup-java@v1  
     with:  
      java-version: 11  

    #- name: Cache Docker images  
    # uses: satackey/action-docker-layer-caching@v0.0.11  
    # # Ignore the failure of a step and avoid terminating the job.  
    # continue-on-error: true  

   - name: Cache Maven packages and Google Jib cache  
     uses: actions/cache@v2  
     with:  
      path: |   
       ~/.m2  
       ~/.cache/google-cloud-tools-java/jib  
      key: ${{ runner.os }}-m2-${{ hashFiles('**/pom.xml') }}  
      restore-keys: ${{ runner.os }}-m2       

   - name: Static checks  
     run: mvn --batch-mode --update-snapshots dependency-check:check pmd:pmd pmd:cpd spotbugs:spotbugs  

   - name: Publish image to Docker Hub  
     run: mvn --batch-mode --update-snapshots compile jib:build  

   - name: Anchore scan  
     uses: anchore/scan-action@v1  
     with:  
      image-reference: "docker.io/maartensmeets/spring-boot-demo"
      fail-build: true  

   - name: Start service  
     run: |  
      docker network create zap  
      docker run --pull always --name spring-boot-demo --network zap -d -p 8080:8080 docker.io/maartensmeets/spring-boot-demo

   - name: OWASP ZAP scan  
     run: |  
       # make file runnable, might not be necessary  
       chmod +x "${GITHUB_WORKSPACE}/.github/zap_cli_scan.sh"
       # run script  
"${GITHUB_WORKSPACE}/.github/zap_cli_scan.sh"
       mv owaspreport.html target  

   - name: 'Publish Test Report'
     if: always()  
     uses: actions/upload-artifact@v2  
     with:  
      name: 'test-reports'
      path: |  
        target/*.html  
        target/site/  
        ./anchore-reports/  

As you can see, the workflow is pretty readable, except maybe the OWASP ZAP part. The different steps have already been described generally before. It was a shame I did need to fall back a couple of times on running direct shell commands and scripts instead of completely relying on re-usable GitHub Actions. Improving that might be something for the future.

It is nice you can generate a badge to use in your GitHub README file indicating the status of the build.

Finally

The good

GitHub Actions are out of the box CI/CD workflow functionality which you can use (with some limitations) when you have a free GitHub account. Because it is out of the box functionality, you are not required to setup your own CI/CD environment to achieve the same. It runs within GitHub and is quite nicely integrated with for example repository events and secrets functionality. There are a lot of GitHub Actions already available and creating/using your own is a piece of cake. The workflows can be triggered in several ways and you can fall back on manual scripting if required. The provided runner images contain a Docker installation so working with containers also works nicely (which can be challenging when your own CI/CD environment runs within a container). You can even use your own runner in for example a different cloud to get improved performance and more resources for the runner if you need it. To summarize, GitHub Actions are powerful, flexible and well integrated with other GitHub provided functionality.

The bad

GitHub Actions are great when you have your project in GitHub and don't want to setup your own CI/CD environment. When you are for example using a private repository outside of GitHub, using GitHub Actions is not the obvious choice.

I was disappointed by the out of the box reporting capabilities. I could generate reports, however I could not publish them anywhere easily without introducing additional dependencies. Scripting the generation of GitHub Issues also did not seem like it was the way to go to solve this. I could use GitHub Pages, however, a link would not be quickly accessible from the build itself (like I was used to in Jenkins) and the logic on how to deal with reports from different builds/branches and collecting the reports in a single accessible page, is something to create for yourself (or use a specific provider). Much of the functionality of a tool like SonarQube was also something I missed, such as fine grained issue management and the ability to define quality gates.

GitHub Actions allow you to do most CI/CD tasks for free, directly from your GitHub repository. One of the challenges however is that there is no build-in facility like for example SonarQube to manage code quality. Luckily, SonarSource provides SonarCloud; a SonarQube SaaS offering which is free for public projects! It is also easy to feed SonarCloud from GitHub Actions. In this blog post I'll describe how you can do this.

Limitations

There are of course some limitations on usage for the free GitHub and SonarCloud accounts. Next to that however, SonarCloud does not allow 3rd party plugins. It is a SaaS offering and allowing 3rd party plugins would cause an additional burden on managing the environment and in addition possible licensing issues. For some code quality aspects however, using 3rd party plugins is currently the only option. Examples of these are the OWASP Dependency-Check and OWASP ZAP. Processing output of those tests is currently not supported in SonarCloud. You can however feed it with SpotBugs (the spiritual successor of FindBugs), PMD and code coverage data. To work around the 3rd party plugin limitation, you could possibly convert the Dependency-Check and ZAP data and merge it with the SpotBugs/PMD output and feed that to SonarQube. I haven't tried that yet however.

GitHub repository

I used the following GitHub repository with some Java code to generate code quality information. In order to do that I had the following entries in my pom.xml file;

SonarCloud configuration

In order to feed data to SonarCloud, some preparations needed to be done on that side. First login to SonarCloud using your GitHub account.

SonarCloud is a hosted SonarQube SaaS solution which does not allow 3rd party plugins to be installed. This puts some limitations on the kind of data you can put in SonarCloud. For Java this is limited to Checkstyle, PMD and SpotBugs results. OWASP provides a Dependency-Check plugin to identify vulnerable dependencies in for example your pom.xml file. In this blog post I'll show how to get OWASP Depedency-Check data in SonarCloud without using a 3rd party plugin! Disclaimer: this solution has been created in very little time (~2 hours) and has not been seriously tested, optimized or used in production environments. Use at your own risk!

Method used

SonarCloud can import CheckStyle, PMD, SpotBugs result data. The output XML files which are generated by those plugins, conform to a specific format specified in XSDs.

• SpotBugs
• PMD
• CheckStyle doesn't have one (read here)

Jenkins is a solid CI/CD platform which has proven itself over the years. Many organizations use it to build, test and deploy their applications. In Jenkins it is possible to define credentials or to use an external credential store. You can then use these inside your pipelines and jobs. Direct access to credentials can be limited. Even with limited access, there are still various ways in which you can extract credentials. 

This blog post illustrates how you can display credentials in log files as base64 encoded strings so they are not masked and you can easily copy / paste / base64 decode them to obtain and (ab)use them. The method described is not specific to Jenkins but can also be used in various other CI/CD platforms on-premises and in the cloud (such as GitHub Actions and Bamboo). 

Note: this is not meant as an encouragement to break rules or laws. Often legislation does not allow you to try and access systems you are not officially authorized to. It is meant to create awareness and to allow you to think about if and how you might want to prevent this in your own CI/CD platform.

Pipeline as code

You often see pipeline definitions which are shipped together with the code which needs to be build, tested, deployed (think Jenkinsfile or GitHub Actions). These pipelines contain commands which are allowed to use credentials. These credentials are often available in environment variables so they can easily be used for example to execute deployments. Since the pipeline is bundled with the application, you can perform version control on it and easily put the responsibility for maintaining the pipeline at the team which builds and maintains the code. Credentials are often defined as secrets within the platform. See for example below for the definition of a Jenkins secret.

Creating a new pipeline

When I'm able to create a new Jenkins pipeline or update an existing pipeline I can obtain credentials defined in the CI/CD platform.

I can define a pipeline outside of version control as part of the job definition. I do require access to an agent to execute the pipeline though.

When I'm required to change the pipeline code inside a version control system, my actions will be far more easy to trace and I might not be so eager to try this. Creating and removing a job after having used it to obtain credentials, often leaves few traces. 

Access to Jenkins credentials

Limit UI and API options

Within Jenkins, you can use matrix based security to limit access to credentials

This will limit your options in the Jenkins UI and via the Jenkins API

A limited user cannot create or edit credentials.

Mask credentials in logging

Printing credentials in the Jenkins logging by using a plain echo command, will give you a warning and mask the credential. Even when wrapping the displaying of the credential in a more complicated command, Jenkins still masks it (although without warning). When however base64 encoding the credentials, Jenkins doesn't recognize them anymore and displays the base64 encoded credentials which you can easily decode using unsafe websites such as this one. Thank you very much!