By John Burns and Emily Yuan
Introduction
At Netflix, we function utilizing a polyrepo technique with tens of hundreds of Java repositories. Which means that we have to have methods of sharing frequent construct logic throughout these repositories. On the JVM Ecosystem group inside Java Platform, we construct tooling such because the Nebula suite of Gradle plugins to supply customary methods to construct initiatives, maintain dependencies up-to-date, and publish artifacts reliably throughout the Java ecosystem. Our mission additionally entails offering build-time suggestions to the developer once they deviate from the paved highway, or when their code base incorporates technical debt.
Case Examine
After a Netflix incident regarding a library releasing a backwards-incompatible change, our group was requested to supply some tooling and practices to enhance the Java library lifecycle administration. This was not a easy case of a library making a reckless breaking change. The code eliminated had been deprecated for years. Library authors typically battle to know when it’s protected to take away deprecated code, or refactor code that’s not meant for use by downstream functions. Fleet-wide migrations, resembling upgrading main Spring Boot variations, additionally contain deprecated code elimination. To assist with this, we established a collection of API lifecycle annotations:
- @Deprecated from the Java customary library
- @Public A customized annotation to make use of on APIs meant for use downstream
- @Experimental A customized annotation for brand new APIs which can not but be secure
- All different APIs are assumed to be “inside”
Library authors can annotate their APIs with these annotations. Nonetheless, how will they know which downstream initiatives are utilizing their API incorrectly, based mostly on these?
As we sought to enhance the paved highway for JVM-based libraries at Netflix, we would have liked a great way of figuring out this type of technical debt, not just for the advantage of the Java Platform-provided libraries, however any group delivering shared libraries to the group. For this, we checked out ArchUnit.
ArchUnit is a well-liked OSS library (3.5k stars, 84 contributors) used to implement “architectural” code guidelines as a part of a JUnit suite. It’s used internally by Gradle, Spring, and is supplied as a part of the Spring Modulith platform. The foundations engine, which is constructed immediately on high of ASM, can be utilized for all kinds of use instances. It’s highly effective sufficient to be a normal function static evaluation software with the next distinctive options:
1. Works cross-language (JVM), as a result of it makes use of ASM/bytecode, not AST parsing.
2. Exposes a builder API sample that makes it simple to put in writing guidelines
3. Additionally has a decrease stage API very best for writing extra advanced customized guidelines.
The limitation of ArchUnit is that it’s designed for use as a part of a JUnit suite in a single repository. The Nebula ArchRules plugins give organizations the power to share and apply guidelines throughout any variety of repositories. Guidelines may be sourced from OSS libraries or non-public inside libraries. This makes the plugin typically helpful for any JVM+Gradle engineering group.
Why ArchUnit?
Earlier than we go into how ArchRules works, it’s good to grasp why we might wish to use ArchUnit on this manner as a substitute of different static evaluation instruments.
AST vs Bytecode
Some instruments, resembling PMD, course of guidelines towards an AST (summary syntax tree). An AST is a structured illustration of supply code. This sort of software could have guidelines which can be syntax dependent. Guidelines that have to assist a number of JVM languages, resembling Kotlin or Scala, typically have to be rewritten for every language. It additionally permits code which ought to be discovered to be hidden below syntactic sugar not anticipated by the rule writer. ArchUnit makes use of ASM to investigate precise compiled bytecode, which suggests it doesn’t matter how that code was produced. What’s analyzed is the precise code that shall be run.
Rule Authorship
Instruments like PMD and Spotbugs are usually not optimized for customized rule authorships. Most utilization of those instruments run built-in supplied guidelines, or add in pre-made third occasion plugins. Check out what a customized rule for PMD would possibly look like:
<![CDATA[
//AllocationExpression/ClassOrInterfaceType[
@Image='DateTime' and (
(count(..//Name[@Image='DateTimeZone.UTC'])<=0)
and
(depend(..//Title[@Image='DateTimeZone.forID'])<=0)
) or (
(
(depend(..//Title[@Image='DateTimeZone.UTC'])>0)
or
(depend(..//Title[@Image='DateTimeZone.forID'])>0)
) and (../Arguments/ArgumentList and depend(../Arguments/ArgumentList/Expression) = 1)
)
]
]]>
This rule ensures that DateTimes are usually not instantiated with out an specific zone. This can be a uncooked string meant for use inside PMD’s xpath parser. There isn’t any IDE steering on crafting it. To check it, an entire separate PMD course of must be wired as much as interpret the rule and consider it towards a supply file. Let’s see how an analogous rule would look with ArchUnit:
ArchRuleDefinition.precedence(Precedence.MEDIUM)
.noClasses()
.ought to()
.callConstructorWhere(
// constructor doesn't have a zone arguement
goal(doesNot(have(rawParameterTypes(DateTimeZone.class))))
// constructor is for DateTime
.and(targetOwner(assignableTo(DateTime.class)))
)
That is type-safe Java code with a fluent API. It’s also easy to unit take a look at, as ArchUnit has a way to go a rule object and sophistication references to guage the rule towards these lessons.
Class Relations
As a result of ArchUnit processes all the classpath with ASM, it retains a graph of the category knowledge, permitting guidelines to simply traverse class relationships and name websites. This enables guidelines to have way more context concerning the code it’s evaluating.
Guidelines Libraries
Step one was to construct the power to put in writing ArchUnit guidelines which may be shared and printed. With a purpose to do that, we’ve the ArchRules Library Plugin. This plugin provides a further supply set to your Gradle challenge known as archRules. On this supply set, you possibly can create a category which implements the ArchRulesService interface. This interface has a single summary technique which returns a Map<String, ArchRule>. The keys of this map are the names of your guidelines, and the ArchRule is the rule you want to outline utilizing the usual ArchUnit API. Right here is an instance:
public class GuavaRules implements ArchRulesService {
static remaining ArchRule OPTIONAL = ArchRuleDefinition.precedence(Precedence.MEDIUM)
.noClasses()
.ought to()
.dependOnClassesThat()
.haveFullyQualifiedName("com.google.frequent.base.Non-compulsory")
.as a result of("Java Non-compulsory is most well-liked over Guava Non-compulsory");
@Override
public Map<String, ArchRule> getRules() {
Map<String, ArchRule> guidelines = new HashMap<>();
guidelines.put("guava non-compulsory", OPTIONAL);
return guidelines;
}
}This code and its dependencies is not going to be bundled along with your predominant code. It’s bundled right into a separate Jar with the arch-rules classifier. When publishing, your library will publish this jar as a separate variant with the utilization attribute set to arch-rules. Which means that to ensure that downstream initiatives to make use of these guidelines, they have to use Gradle Module Metadata for dependency decision. There are 2 flavors of guidelines Libraries: Standalone guidelines libraries, bundled rule libraries.
Standalone Rule Libraries
A Standalone Rule library incorporates no predominant code: solely archRules. These are helpful for outlining guidelines for code you don’t personal, resembling Core Java APIs or OSS libraries. They’re additionally helpful for generic guidelines that may apply to any code, resembling “don’t use code marked as @Deprecated”. We preserve a set of OSS Standalone rule libraries which anybody is free to make use of, and function examples of the sorts of guidelines it’s possible you’ll wish to write your self. Nonetheless, the actual energy of ArchRules is in “bundled rule libraries”.
Bundled Rule Libraries
A bundled rule library is a library with each predominant and archRules sources. The primary supply set will include helpful library code, no matter it might be. The archRules will include guidelines particular to the utilization of that library. For instance, guidelines scoped to that library’s package deal, or referencing that library’s particular API. Each time doable, we advocate writing guidelines on this bundled manner. That’s as a result of the ArchRules Runner Plugin will be capable to robotically detect these guidelines and run them in solely the supply units that use this library as a dependency. An instance of this may be seen in our Nebula Check library.
In any case, the library plugin will robotically generate a service loader registration entry in your ArchRulesService in order that the runner can uncover your guidelines.
Operating Guidelines
The ArchRules Runner Plugin permits guidelines to be evaluated towards your code. Standalone rule libraries may be evaluated towards all supply units by including them to the archRules configuration in your construct. For instance:
dependencies {
archRules("your:guidelines:1.0.0")
}As talked about earlier than, bundled guidelines shall be evaluated robotically. To do that, the runner plugin creates a separate configuration for every of your supply units. In every of those configurations, the archRules classpath is mixed with the runtimeClasspath with the arch-rules variant chosen. This configuration is the classpath used when the ServiceLoader discovers implementations of ArchRulesService. Within the following instance, we’ve a Mission which makes use of a take a look at helper library as a testImplementation dependency, and in addition provides a standalone guidelines library to the archRules configuration. The take a look at runtime classpath will solely include the implementation jar for the helper library, however the arch guidelines runtime will include the archrules jar for the bundled guidelines and standalone guidelines. This all occurs robotically.
As soon as the principles classpath is decided, the runner plugin will create a Gradle work motion to guage guidelines towards that particular supply set. This motion runs with classpath isolation utilizing the *archRuleRuntime configuration. Inside this motion, a ServiceLoader is used to find rule definitions. The motion ends by writing a binary serialization of rule violations to a file for reporting.
In a challenge working guidelines, you even have the power to customise rule configurations utilizing the archRules extension. For instance, you possibly can override a rule’s precedence stage:
archRules {
ruleClass("com.netflix.nebula.archrules.deprecation") {
precedence("HIGH")
}
}Different customizations embrace disabling working guidelines on sure supply units and configuring the failure threshold (i.e., excessive precedence failures will trigger the construct to fail).
Reporting
The ArchRules runner plugin has two built-in studies: JSON and console. The json report will acquire the output from all supply units inside a challenge and create a single json file with the entire knowledge. The console report additionally collects the output from all supply units inside a challenge, nevertheless it prints to the console a simple to learn report, for instance:
Observe that failure particulars characteristic an in depth plain English description, together with a pointer to the precise line of code in violation.
For customized reporting, you possibly can both use the JSON file, or create your personal job that reads the binary information. Check out the supply code for the ArchRules runner plugin’s report duties for an instance of how one can do this.
Case Examine Answer
Going again to our unique drawback, utilizing ArchRules, we had been in a position to ship a platform for library authors to trace the utilization of their APIs. They write ArchRules to detect utilization of the annotations, scoped to their library’s package deal, such as:
ArchRuleDefinition.precedence(Precedence.MEDIUM)
.noClasses().that(resideOutsideOfPackage(packageName + ".."))
.ought to()
.dependOnClassesThat(resideInAPackage(packageName + "..").and(are(deprecated())))
.orShould().accessTargetWhere(targetOwner(resideInAPackage(packageName + ".."))
.and(goal(is(deprecated())).or(targetOwner(is(deprecated())))))
.allowEmptyShould(true)
.as a result of("Deprecated APIs are topic to elimination");
NB: the deprecated() predicate comes from nebula-archrules.
Our inside Nebula customary Gradle wrapper and plugin suite robotically allow the ArchRules runner on each challenge, and supplies a customized reporter which sends the report knowledge to our Inner Developer Portal on each main-branch CI construct. This manner, library authors can simply see a report of all downstream shoppers utilizing their experimental, deprecated, or personal APIs, giving them confidence to make “breaking” adjustments, figuring out that it’s going to not really break downstream shoppers. If their adjustments are presently blocked by downstream utilization, they will simply see precisely which initiatives are reporting these usages.
OSS Rule Libraries
Whereas probably the most highly effective manner to make use of ArchRules is so that you can write your personal guidelines, we’ve constructed some OSS rule libraries that anybody is free to make use of, or reference as examples.
Nullability
These guidelines implement correct nullability annotation in Java, for instance, that each public class is marked with JSpecify’s @NullMarked. It’s sensible sufficient to exclude Kotlin code, as Kotlin has built-in nullability.
Gradle Plugin Finest Practices
Writing Gradle plugins may be arduous, particularly since there are various APIs and patterns that shouldn’t be used anymore. These guidelines assist implement present finest practices when writing Gradle plugins.
Joda / Guava Guidelines
These rule libraries discourage using Joda Time and Guava lessons (respectively) as these have been outdated by java.time and customary library enhancements.
Safety Guidelines
These guidelines assist mitigate CVEs by detecting utilization of identified susceptible APIs. Ideally, we maintain dependencies updated to mitigate CVEs. However typically that’s not instantly possible, and in these instances, a compile time examine to make sure the particular susceptible API isn’t used is commonly good sufficient.
Conclusion
We at the moment are working 358 (and counting) guidelines throughout over 5,000 repositories detecting over almost 1 million points. About 1,000 of those points are for “Excessive” precedence guidelines. Having the ability to run these guidelines on this scale permits us to rapidly acquire perception into our massive fleet of microservices, and determine the areas carrying probably the most essential technical debt. This makes it simpler to focus and prioritize our efforts.
Going ahead, we shall be exploring how one can tie auto-remediation options into the ArchRules findings. ArchUnit presently supplies very particular and detailed details about failures in studies, which makes a really robust enter sign to an auto remediation software. We’ll discover deterministic options resembling OpenRewrite and non-deterministic options resembling LLMs. Pairing the simple rule authorship and deterministic outcomes of ArchUnit with an auto-remediation software that may accurately interpret the outcomes to resolve the problem at hand shall be a really highly effective mixture.
We additionally will examine how one can get ArchRule failure data surfaced within the IDE as inspections.
If in case you have questions or suggestions about Nebula ArchRules, attain out to us by posting within the #nebula channel on the Gradle Group Slack.
Scaling ArchUnit with Nebula ArchRules was initially printed in Netflix TechBlog on Medium, the place individuals are persevering with the dialog by highlighting and responding to this story.
