Gradle build and release FAQ
This FAQ has been written with building and releasing the Spring Framework in mind, but is also general enough to serve as a guide for any Spring project that publishes to http://repo.spring.io. The audience is primarily geared for project leads and committers, but should be valuable for anyone building from source or wanting to understand better how Spring is published, etc. The focus is on Gradle-based builds, but keep in mind that Maven-based Spring projects can take advantage of the same general steps, e.g. using the Artifactory Maven 3 Task instead of the Artifactory Gradle Task, etc. Note that if you're unfamiliar with the benefits of Artifactory and repo.spring.io in general, you may want to check out the Spring repository FAQ.
build-related
- How do I check out and build the Framework?
- How long should a build take?
- Why are there so many Javadoc warnings?
- Why are compile-time warnings suppressed?
- What are the most important tips for Gradle newbies?
- Why do I get 401/403 errors when downloading dependencies?
release-related
- How and where are snapshots published?
- How do I perform a Milestone, RC, or GA release?
- What about publishing artifacts to Maven Central?
- How are docs and schemas and distribution zips published?
# How do I check out and build the Framework? See [building from source](https://github.com/spring-projects/spring-framework#building-from-source) in the README. This also includes information on importing projects into Eclipse/STS/IDEA.
# How long should a build take? When running `./gradlew build` for the first time, it is likely that the Gradle wrapper script will need to download Gradle for you. Gradle distributions run around 30MB, so this step will be connection-speed dependent.
You will also need to download all of Spring's compile- and test-time dependencies. This currently runs around 120MB. Keep in mind here that almost all of these dependencies are optional at runtime for applications that use Spring. It's only when building from source that you actually need them all!
Once you've bootstrapped a Gradle distribution and downloaded dependencies, you won't need to do it again; they are cached in your $HOME/.gradle directory. A complete ./gradle clean build at this point will take between 5 and 15 minutes depending on your clock and disk speed. A solid state drive makes a huge difference here!
As is also mentioned below in the 'tips' section, you'll want to break yourself of any habits executing the clean task during normal development iterations. Gradle has excellent incremental build support, meaning that once you've built Javadoc, compiled, run test, etc, Gradle won't execute these tasks again unless the inputs for those tasks (e.g. .java files) have changed. As a general rule, just run ./gradle build or ./gradle test (without clean) to keep things snappy.
Also, consider running with the -a flag to avoid evaluating other subprojects you depend on. For example, if you're iterating on changes in spring-webmvc, cd into the spring-webmvc directory and run ../gradlew -a build to tell gradle to evaluate and build only that subproject.
Finally, the Gradle daemon helps greatly in eliminating startup overhead. This feature will soon be 'on by default', but in the meantime you need to run gradle with the --daemon flag, or alternatively, export your GRADLE_OPTS environment variable to include -Dorg.gradle.daemon=true
# Why are compile-time warnings suppressed? You'll notice that `build.gradle` includes the following line:
[compileJava, compileTestJava]*.options*.compilerArgs = ['-Xlint:none']
This tells Gradle to suppress all warnings during compilation. The reason for this is that the framework currently has many warnings, most of which are related to generics usage -- particularly raw type warnings -- e.g. using Class instead of Class<?>. This is an artifact switching to Java 5 in Spring 3. As with the Javadoc warnings mentioned above, committers are encouraged to fix these warnings whenever possible. Once the bulk of them are eliminated, we can switch to -Xlint:all. In the meantime, it's just creates unnecessary noise in the build output.
# What are the most important tips for Gradle newbies?
As mentioned above, Gradle has excellent incremental build capabilities, meaning that it will only compile, run tests, generate javadoc, etc. if content has actually changed. However, telling gradle to clean everything eliminates this benefit. It is a common practice -- particularly for longtime Maven users -- to type 'clean' with every build as a matter of course. Break yourself of this habit and you'll find builds running much faster.
Following onto the item above, there are certainly cases in which you want to clean, but you may not want to clean everything. Instead of typing ./gradlew clean build, consider using selective cleaning, in which you clean only the outputs for a given task. For example
$ ./gradlew :spring-oxm:cleanTest build
will eliminate the only the spring-oxm/build/classes/test directory and then run the entire build. This means that you still benefit from incremental build, but for sure will re-execute spring-oxm's test suite. The rule here is "clean", where is the capitalized name of the task you want to clean. Other examples would be: cleanJar, cleanJavadoc, cleanDistZip, etc.
As mentioned earlier in this document, the -a switch tells Gradle to ignore evaluting and building any other subprojects. An example would be when iterating on changes to spring-webmvc. You may be running ./gradlew test over and over again in the spring-webmvc directory, and each time you do this, Gradle will check all the other subprojects that spring-webmvc depends on to see if they've changed. This is generally useful functionality, but if you know better, you may want to save yourself the extra few seconds that these checks can take. Simply run ../gradlew -a test within the spring-webmvc directory, and you'll see the difference.
Rather than running ./gradlew build every time you want to simply compile and test, run ./gradlew test. If you want only to generate the jar for a particular subproject, run ../gradlew jar in that subproject's directory.
Run ./gradlew tasks to get an overview of available tasks and their descriptions. Run ./gradlew tasks --all to get an even more exhaustive list.
When you run ./gradlew build from the root directory, Gradle recurses through all projects, finding those that actually have a 'build' task, and executing it if it exists. This kind of heuristic behavior makes for convenient usage, but you can also be more specific. For example, from the root directory you can run ./gradlew :spring-core:test and Gradle will run the 'test' task only within the spring-core subproject.
A particularly relevant example of this is when running the integration tests for Spring Framework. As of the switch to Gradle, there is no longer a dedicated integration-tests subproject. Rather, these tests now live in the root project's src/test directory. If you run ./gradlew test from the root directory, Gradle will run not only the tests in the root src/test directory, but also the tests for every subproject! To narrow it down and run only the root project tests, type ./gradlew :test. The ':' here says "the 'test' task in the root project". Contrast this again with ./gradlew :spring-core:test.
Perhaps you'd like to run a complete clean build, but don't want to take the time to run the test suites? Do this with ./gradlew clean build -x test. -x tells Gradle to exclude the task from the build lifecycle. Specify multiple tasks to exclude with multiple '-x' switches, e.g. ./gradlew clean build -x javadoc -x test
Perhaps you're iterating on a particular unit test in spring-context and don't want to run the thousands of tests in its test suite just to see if your latest changes work. Run ./gradlew test -Dtest.single=MyTests. There's no need to qualify the packaging here. Gradle will simply find any class named 'MyTests' and run just those tests.
By default, Gradle's command line output is blissfully minimalistic. It will only tell you at a high level what it's doing right now, e.g.:
:spring-asm:compileJava UP-TO-DATE
:spring-asm:processResources UP-TO-DATE
:spring-asm:classes UP-TO-DATE
:spring-asm:repackageAsm UP-TO-DATE
:spring-asm:jar UP-TO-DATE
:spring-core:compileJava
If there are warnings, e.g. compiler or Javadoc warnings, those will show up, because Gradle wants you to see that there's something wrong. But if everything is going well, Gradle just keeps quiet. In certain cases, however, you might want to see exactly which tests are running when, or exactly what requests are being made to download dependencies, etc. Run ./gradlew -i (or --info) to get a bit more verbosity. Run with -d / --debug to switch on the firehose.
Prior to the switch to Gradle, we manually maintained Eclipse and IDEA project metadata. This a hassle and error-prone to say the least. Gradle provides robust support for generating this metadata, and for this reason we no longer check these files into source control. You'll notice that all these files now have entries in .gitignore. Please don't check them in!
See the import-into-eclipse.sh and import-into-idea.md files in the root of the source tree for detailed instructions.
Note that both IDEA and STS are hard at work on even more advanced Gradle tooling from the IDE side. At the time of this writing, neither one is quite capable of meeting all the needs of a build as complex as Spring's, but they'll get there soon. In the meantime, Gradle's built-in support works quite well.
This commit provides all the detail you'll ever want on how we manage optional and provided dependencies with regard to our generated Maven poms.
See the FAQ item on authentication below.
If you need to compile against a locally built version of a dependency, you'll need to add your local $HOME/.m2 repository to the set of repositories that Gradle searches during the build.
This can be done directly in your build script by adding mavenLocal() as follows:
repositories {
mavenLocal()
maven { url "http://repo.spring.io/libs-release" }
}
However, it's recommended that you do this via an init script, to avoid accidentally checking in the mavenLocal() entry. If you're already using the authentication init script described above, you can append the following to it, or if not, just create a new file in $HOME/.gradle named init.gradle with the following:
allprojects {
repositories {
mavenLocal()
}
}
If you're building Spring artifacts from source in order to compile against them from another project (perhaps another Spring project, even, like Spring Batch or Spring Integration or the Spring Data-* family), you'll need to publish jars into your local $HOME/.m2 cache in order to pick them up from the dependent project. Do this with ./gradlew install.
The Gradle wrapper is a big help, but it can be cumbersome to have to use relative pathing, e.g. ./gradlew, ../gradlew when working with the build. Follow the instructions here to install a simple shell script that allows you to type gradle anywhere and finds the nearest gradlew or falls back to your system-wide Gradle installation (if you have one): https://github.com/cbeams/shell-scripts/blob/master/find-gradle
# Why do I get 401/403 errors when downloading dependencies? Our Artifactory instance at http://repo.spring.io is configured such that requesting and caching external dependencies, e.g. junit or cglib from Maven Central) can only be performed by authenticated users, i.e. Spring team members. Once such a dependency has been successfully cached in the repository, then even anonymous/unauthenticated users may download them. This configuration helps ensure that repo.spring.io contains only Spring project artifacts and their transitive dependencies, and that we don't act as a kind of 'open proxy'.
This means you can receive a 401/403 while downloading a dependency, possibly because it's being downloaded for the first time and you're not authenticated. A typical example might be when upgrading to a newly-released version of JUnit or Hibernate. Occasionally this can happen also in error. Here is what to do about it.
If you get a 401/403 and you are not a Spring team member, you can add the Maven central repository to your build as a temporary workaround. Meanwhile do report the issue by sending an email to buildmaster@springframework.org.
If you are a Spring team member, the solution to this problem is to provide your Artifactory authentication credentials to Gradle, such that it can successfully respond to 401 challenges when Artifactory issues them. The best way to do this is with a Gradle init script, which keeps your username and password separate from your build script.
See https://github.com/spring-projects/gradle-init-scripts#readme for simple instructions on how to dowload and install an init script that does just this. This is recommended for all committers.
# How and where are snapshots published? The [primary CI build plan](https://build.springsource.org/browse/SPR-B40X) is configured to use the [Artifactory Bamboo plugin](http://wiki.jfrog.org/confluence/display/RTF/Bamboo+Artifactory+Plug-in). Each time commits are pushed to [spring-projects/master](https://github.com/spring-projects/spring-framework/tree/master), this build plan
- executes
./gradlew build, producing jars and distribution zip files - publishes these artifacts to http://repo.spring.io/libs-snapshot-local (browse in Artifactory tree view)
Along with individual artifacts, build metadata is also published to Artifactory, including information about the environment and principal involved in the build, as well as allowing for bi-directional links between Bamboo and Artifactory. Click on the 'Artifactory' tab for any successful CI build to navigate to the associated Artifactory build.
Likewise, click on "Show in CI Server" from any individual Artifactory build to navigate back to the CI build that published it.
Users wishing to consume snapshot builds may add http://repo.spring.io/snapshot to the list of repositories in their build script. See Downloading Spring artifacts for more details.
# How do I perform a Milestone, RC, or GA release?
The steps are simple, and almost everything is done via the Bamboo and Artifactory UIs.
Configure your CI build plan to use the Artifactory Maven 3 or Artifactory Gradle tasks as appropriate. For "Deployer Username", use "buildmaster" (password on request).
- Stage the release into the libs-staging-local repository
- Verify and test the staged artifacts
- Promote the release to libs-milestone-local (or libs-release-local as appropriate).
- Merge release branch
- Announce the release
The Artifactory Bamboo plugin mentioned above also includes sophisticated Release Management capabilities. This feature allows for publishing releases directly from CI, including creating a release branch and/or tag; incrementing the project version; and publishing to the libs-staging-local, libs-milestone-local or libs-release-local repositories as appropriate.
To access this feature, click on the "Default Job" for the Spring 3.2.x build plan, where you'll see a link to "Artifactory Release Management". Fill out the form fields there and click "Build and Release to Artifactory". Typical values -- in this case for a milestone release -- look something like the following:
In the example above, the version property key refers to the property of the same name declared in gradle.properties in the root of the source tree. This value will actually be modified and updated in source control during the release process.
Using a 'release branch' is optional, but recommended. This means that updates to the gradle.properties file will occur on a branch named 3.2.0.M1, helping to isolate the release from changes on the master branch, and also allowing for simplified rollback in case a last minute change needs to be made after staging the release.
'Create VCS Tag' is also checked, indicating that a git tag named 'v3.2.0.M1' should be created, pointing to the commit where the version property is incremented.
Notice that 'Next development version comment' is blank. Because this is a milestone release, the 'next integration value' of the version property is configured to return to it's previous value of '3.2.0.BUILD-SNAPSHOT'. The tooling is smart enough here to avoid creating an additional commit, so no comment is necessary.
Importantly, notice that we're publishing to the 'libs-staging-local' repository - this is just what it sounds like: a staging area that allows us to test out the release internally before finally promoting it to the actual 'libs-milestone-local' repository and announcing it to the world.
With these values supplied, click 'Build and Release to Artifactory'.
When the staging build and release process is complete, you can navigate to the associated build record in Artifactory to verify that all modules were published as expected, e.g.:
Note that in the Artifactory tree view, you can easily drill into jars and zips to inspect their contents, e.g. manifest files, javadoc, reference docs, etc, e.g.:
In the example above, clicking the 'Download' button will open the API javadocs in the browser -- a nice convenience.
You may also wish to have internal team members 'smoke test' the release, e.g. change their sample projects and dependent framework builds to point to http://repo.spring.io/libs-staging-local and compile/test/run against the staged artifacts.
When verification is complete, return to the build in Bamboo from which you staged the release and click 'Default Job' and 'Artifactory' at the top, below the Job status bar. Make sure you have the side-bar shown in order to see this. You'll now see 'Promotion' options as follows:
The 'Target promotion repository' is set to 'libs-milestone-local'. Click 'Update' to move all artifacts from 'libs-staging-local' to 'libs-milestone-local'.
At this point, the release is complete and successful, so the release branch should be merged back into master, e.g.
$ cd spring-framework # your local spring-framework working copy
$ git checkout master
$ git fetch --all # to fetch the branch created during the release
$ git merge springsource/master # make sure you're up to date
$ git merge springsource/3.2.0.M1 # assuming your remote is named 'springsource'
$ vi gradle.properties # bump the version to 3.1.0.BUILD-SNAPSHOT
$ git ci -am"Increment version to 3.2.0.BUILD-SNAPSHOT"
$ git push springsource master:master
At this point, announcements may be made and users may consume the released artifacts by adding http://repo.spring.io/libs-milestone-local to their build scripts.
# What about publishing artifacts to Maven Central?
GA releases of Spring Framework are published not only to http://repo.spring.io/libs-release-local, but also to Maven Central at http://repo1.maven.org. This allows for maximum convenience for the majority of Spring users, given that most users have Maven-based builds and Maven resolves artifacts by default from Maven Central.
The preferred way of releasing artifacts to Maven Central is via Sonatype's Nexus server at oss.sonatype.org (OSO). This is explained in detail in Sonatype's OSS usage guide.
The Spring Artifactory repository has been customized with a "nexus-push" plugin that allows for automatic propagation of builds from Artifactory to the Nexus server at OSO for publication into Maven Central.
All Spring projects -- that is, all projects having groupid org.springframework -- can publish to OSO under the shared 'springsource' account. This has already been set up in the nexus-push plugin, so there's no additional setup necessary at OSO, even for new projects.
The Artifactory Bamboo plugin supports use of the nexus-push plugin through it's UI. Step 3 of the the FAQ entry above on publishing releases described the process for promoting a build out of staging. If the build is a GA release, simply choose the 'Push to Nexus' option, and select 'libs-release-local' as the target repository:
Choosing this option means that the build will first be published into a staging repository at OSO and 'closed' in Nexus terminology. The 'closing' process will check your build artifacts to ensure they meet the requirements for publication into Maven Central, e.g. that POMs are properly formed, that all artifacts have checksums, PGP signatures, etc. If there are any errors in the repository closing process, they will be displayed in the Bamboo UI and the promotion process will fail. At this point you'll need to correct the issues and walk through the release staging process described above once again.
Note that with regard to requirements for OSO onboarding, Artifactory automatically generates sha1 and md5 checksums for all artifacts, so you don't need to worry about this. Furthermore, a custom plugin has been developed for repo.spring.io that adds PGP signatures (.asc files) on the fly during upload using the buildmaster@springframework.org PGP key. You simply need to make sure that your build produces jars (including -sources and -javadoc jars) and well-formed poms. Any zip files such as distribution or doc zips are excluded from the promotion process to OSO.
When the promotion process is complete, i.e. closing the staging repository at OSO succeeds, there is one additional step - you must log into http://oss.sonatype.org with the 'springsource' account and manually 'release' to Maven Central.
- Go to http://oss.sonatype.org and click 'log in'
- username: springsource; password: on request
- Click on "Staging Repositories".
- You will see your closed staging repository there; click to select it.
- Click the 'Release' button. It looks something like this:
When you're prompted for a description, you can leave it blank.
Pressing the release button means that your artifacts will be published into Maven Central, but beware -- there's no going back after this point!
Synchronization to Maven Central should be complete within three hours of pressing the 'release' button. In practice, it is usually two hours or less, but with just the right timing (or wrong timing as the case may be), you may need to wait the full three. If your artifacts do not show up at http://search.maven.org within four hours, email buildmaster@springframework.org and ask about escalation to Sonatype.
# How are docs and schemas and distribution zips published? First, you'll find three important tasks in the root build.gradle script: `docsZip`, `schemaZip`, and `distZip`. As you might guess, these create zip files containing docs and schemas in the case of _docsZip_ and _schemaZip_ respectively; _distZip_ aggregates the contents of the first two and adds in all classes jars, -sources jars and -javadoc jars.
As described in the release process documentation above, all artifacts produced by the Spring Framework build are published into Artifactory, including these zip archives. This provides a consistent storage mechanism, but ultimately the docs and schema zips need to be published and unpacked at http://static.springframework.org/spring-framework/docs and http://www.springframework.org/schema, respectively. For example:
- http://static.springsource.org/spring-framework/docs/3.1.0.RELEASE (changelog, api javadocs, reference docs)
- http://www.springframework.org/schema/context (xsd files for spring-context)
Performing these uploads directly from the build script is problematic. It requires the build to use SSH libraries in order to SCP and unpack the zip files, which is already complex, but worse it requires that the operator of the build script has the correct SSH key authentication configured on the remote servers, and that they are within the VMware VPN.
To avoid this complexity, a separate process called 'autorepo' runs periodically (every 20 minutes), querying Artifactory for these for docs and schema zips. When new ones are found, the autorepo process does the heavy SSH lifting to upload and unpack them at the sites mentioned above. This script is currently under development, but you can see the results of the prototype effort at http://static.springsource.org/autorepo/.
In order for autorepo to work properly, these artifacts must be annotated in Artifactory with custom metadata. This metadata is attached to the artifacts on upload by the Artifactory Bamboo plugin. Here's the configuration in the Spring Framework 3.2.x build plan:
And here are those properties in copy-pastable form:
archives *:*:*:*@zip zip.name:spring-framework, zip.displayname:Spring Framework, zip.deployed:false
archives *:*:*:docs@zip zip.type:docs
archives *:*:*:dist@zip zip.type:dist
archives *:*:*:schema@zip zip.type:schema
Note: Full documentation for this feature can be found in the Artifactory Gradle plugin documentation.
The zip.type property tells autorepo that the artifact is a 'docs', 'schema', or 'dist'. The zip.deployed property tells autorepo whether it has already uploaded and unpacked this artifact. When autorepo detects a new docs or schema zip (zip.deployed == false), it performs the uploading and unpacking, and then sets the zip.deployed property to true.
The dist zip, on the other hand, remains within Artifactory.
The zip.displayname value is used by autorepo to determine where in the community download page the artifact should show up. So this value should match whatever name your project already has on the community download page.
Once autorepo is working, you may want to see autorepo version updating