Code Quality Testing

Learn how code quality testing of pipelines works and how it can improve the quality of your deployments.

Introduction

Code quality testing evaluates your application code based on a set of quality rules. It is the primary purpose of a code-quality only pipeline and is executed immediately following the build step in all production and non-production pipelines.

See Configuring Your CI-CD Pipeline to learn more about different types of pipelines.

Code Quality Rules

Code quality testing scans the source code to ensure that it meets certain quality criteria. This is implemented by a combination of SonarQube and content package-level examination using OakPAL. There are over 100 rules, combining generic Java rules and AEM-specific rules. Some of the AEM-specific rules are created based on best practices from AEM Engineering and are referred to as custom code quality rules.

NOTE

You can download the complete list of rules with this link.

Three-Tiered Ratings

Issues identified by code quality testing are assigned to one of three categories.

  • Critical - These are issues which cause an immediate failure of the pipeline.

  • Important - These are issues which cause the pipeline to enter a paused state. A deployment manager, project manager, or business owner can either override the issues, in which case the pipeline proceeds, or they can accept the issues, in which case the pipeline stops with a failure.

  • Info - These are issues which are provided purely for informational purposes and have no impact on the pipeline execution

NOTE

In a code quality only pipeline, important failures in the Code Quality gate cannot be overridden since the code quality testing step is the final step in the pipeline.

Ratings

The results of this step is delivered as Ratings.

The following table summarizes the ratings and failure thresholds for each of the critical, important and information categories.

Name Definition Category Failure Threshold
Security Rating A = No vulnerabilities
B = At least 1 minor vulnerability
C = At least 1 major vulnerability
D = At least 1 critical vulnerability
E = At least 1 blocker vulnerability		 Critical < B
Reliability Rating A = No bugs
B = At least 1 minor bug
C = At least 1 major bug
D = At least 1 critical bug
E = At least 1 blocker bug		 Critical < D
Maintainability Rating Defined by the outstanding remediation cost for code smells as a percentage of the time that has already gone into the application
  • A = <=5%
  • B = 6-10%
  • C = 11-20%
  • D = 21-50%
  • E = >50%
 Important < A
Coverage Defined by a mix of unit test line coverage and condition coverage using the formula:
Coverage = (CT + CF + LC)/(2*B + EL)
  • CT = Conditions that have been evaluated as true at least once while running unit tests
  • CF = Conditions that have been evaluated as false at least once while running unit tests
  • LC = Covered lines = lines_to_cover - uncovered_lines
  • B = total number of conditions
  • EL = total number of executable lines (lines_to_cover)
 Important < 50%
Skipped Unit Tests Number of skipped unit tests Info > 1
Open Issues Overall issue types - Vulnerabilities, Bugs, and Code Smells Info > 0
Duplicated Lines Defined as the number of lines involved in duplicated blocks. A block of code is considered duplicated under the following conditions.
Non-Java Projects:
  • There should be at least 100 successive and duplicated tokens.
  • Those tokens should be spread over at least:
  • 30 lines of code for COBOL
  • 20 lines of code for ABAP
  • 10 lines of code for other languages
Java Projects:
  • There should be at least 10 successive and duplicated statements regardless of the number of tokens and lines.
Differences in indentation and in string literals are ignored when detecting duplicates.		 Info > 1%
Cloud Service Compatibility Number of identified cloud service compatibility issues Info > 0
NOTE

See SonarQube’s Metric Definitions for more detailed definitions.

NOTE

To learn more about the custom code quality rules run by Cloud Manager, see Custom Code Quality Rules.

Dealing with False Positives

The quality scanning process is not perfect and will sometimes incorrectly identify issues which are not actually problematic. This is referred to as a false positive.

In these cases, the source code can be annotated with the standard Java @SuppressWarnings annotation specifying the rule ID as the annotation attribute. For example, one common false positive is that the SonarQube rule to detect hardcoded passwords can be aggressive about how a hardcoded password is identified.

The following code is fairly common in an AEM project, which has code to connect to some external service.

@Property(label = "Service Password")
private static final String PROP_SERVICE_PASSWORD = "password";

SonarQube will then raise a blocker vulnerability. But after reviewing the code, you recognize that this is not a vulnerability and can annotate the code with the appropriate rule ID.

@SuppressWarnings("squid:S2068")
@Property(label = "Service Password")
private static final String PROP_SERVICE_PASSWORD = "password";

However, if the code was actually this:

@Property(label = "Service Password", value = "mysecretpassword")
private static final String PROP_SERVICE_PASSWORD = "password";

Then the correct solution is to remove the hardcoded password.

NOTE

While it is a best practice to make the @SuppressWarnings annotation as specific as possible, that is, annotate only the specific statement or block causing the issue, it is possible to annotate at a class level.

NOTE

While there is no explicit security testing step, there are security-related code quality rules evaluated during the code quality step. See Security Overview for AEM as a Cloud Service to learn more about security in Cloud Service.

Content Package Scanning Optimization

As part of the quality analysis process, Cloud Manager performs analysis of the content packages produced by the Maven build. Cloud Manager offers optimizations to accelerate this process, which are effective when certain packaging constraints are observed. Most significant is the optimization performed for projects that output a single content package, generally referred to as an “all” package, which contains a number of other content packages produced by the build, which are marked as skipped. When Cloud Manager detects this scenario, rather than unpack the “all” package, the individual content packages are scanned directly and sorted based on dependencies. For example, consider the following build output.

  • all/myco-all-1.0.0-SNAPSHOT.zip (content-package)
  • ui.apps/myco-ui.apps-1.0.0-SNAPSHOT.zip (skipped-content-package)
  • ui.content/myco-ui.content-1.0.0-SNAPSHOT.zip (skipped-content-package)

If the only items inside myco-all-1.0.0-SNAPSHOT.zip are the two skipped content packages, then the two embedded packages are scanned in lieu of the “all” content package.

For projects that produce dozens of embedded packages, this optimization has been shown to save upwards of 10 minutes per pipeline execution.

A special case can occur when the “all” content package contains a combination of skipped content packages and OSGi bundles. For example, if myco-all-1.0.0-SNAPSHOT.zip contained the two embedded packages previously mentioned and one or more OSGi bundles, then a new, minimal content package is constructed with only the OSGi bundles. This package is always named cloudmanager-synthetic-jar-package and the contained bundles are placed in /apps/cloudmanager-synthetic-installer/install.

NOTE
  • This optimization does not impact the packages which are deployed to AEM.
  • Because the matching between the embedded content packages and the skipped content packages is based on file names, this optimization cannot be performed if multiple skipped content packages have exactly the same file name or if the file name is changed while embedding.

On this page