Synchronize GitHub Source Code Mappings to Java Build Tool Model

[gilday]

Problem Statement

I would like Sentry tooling to synchronize the source code mappings in my Sentry GitHub integration with my Java project structure.

Consider a multi-project Gradle project with subprojects foo, bar, and baz. In this case, I would like an automated way to create source code mappings for directories foo/src/main/java, bar/src/main/java, and baz/src/main/java.

Solution Brainstorm

Java build tool (e.g. Maven and Gradle) are aware of projects' directory structures and the difference between production and test code. A Sentry Maven / Gradle plugin could query the project's structure and synchronize the GitHub source code mappings.

Product Area

Issues

[getsantry]

Assigning to @getsentry/support for routing ⏲️

[getsantry]

Routing to @getsentry/product-owners-settings-integrations for triage ⏲️

[hubertdeng123]

Hmm, will transfer to sentry-java to see if they have anything to add here.

[adinauer]

Hello @gilday we already have https://github.com/getsentry/sentry-android-gradle-plugin which allows you to upload source code. You can find docs here.

Does that do what you want?

[gilday]

I tried the plugin to see if it has the same behavior as that described by the source code mappings (i.e. link to source in GitHub), but the plugin does not work in my Gradle build. It fails with the error:

    Reason: Task ':codemodder-orchestrator:workflow-executor:sentryCollectSourcesJava' uses this output of task ':codemodder-orchestrator:workflow-executor:quarkusGenerateCode' without declaring an explicit or implicit dependency. This can lead to incorrect results being produced, depending on what order the tasks are executed.

[adinauer]

@romtsn do you know how to fix this?

[romtsn]

@gilday what Gradle version are you on? Also, do you know where does the quarkusGenerateCode task put the generated code? Is it src/main/java?

[gilday]

------------------------------------------------------------
Gradle 8.2.1
------------------------------------------------------------

Build time:   2023-07-10 12:12:35 UTC
Revision:     a38ec64d3c4612da9083cc506a1ccb212afeecaa

Kotlin:       1.8.20
Groovy:       3.0.17
Ant:          Apache Ant(TM) version 1.10.13 compiled on January 4 2023
JVM:          17.0.7 (Eclipse Adoptium 17.0.7+7)
OS:           Mac OS X 13.4.1 aarch64

I believe quarkusGenerateCode writes to build/classes/java/quarkus-generated-sources.

[romtsn]

I've moved this to the SAGP repo where it should belong I believe. We'll have to investigate quarkus and see what it does (I've got a feeling that something they are doing is off, but maybe it's also us).

[lbloder]

Hi @gilday,

Could you try to add the following to your build.gradle or build.gradle.kts file:

tasks.named("sentryCollectSourcesJava") {
    mustRunAfter("quarkusGenerateCode", "quarkusGenerateCodeDev", "quarkusGenerateCodeTests")
}

This should fix the dependency issue you are facing during the build.

Please let us know if that helped.

[adinauer]

Hello @gilday have you had a chance to test the fix @lbloder suggested?

[Glyphack]

Hi @adinauer I had the same issue and this comment fixed it:
#546 (comment)

My issue was not exactly this but similarly a tool was generating code at build time and I had to add this dependency. I think we can add this to troubleshooting or setup gradle plugin docs. I came here from this page:
https://docs.sentry.io/platforms/java/guides/logback/gradle/

[adinauer]

Thanks for the feedback @Glyphack we'll improve troubleshooting docs.

[gilday]

Can confirm that workaround is effective.

[gilday]

It's my understanding that the Gradle plugin uploads sources to provide source context. With this working, I can see my sources alongside stack traces, but there are still no links to GitHub.

The feature I described in the description of this issue pertains to GitHub code mappings which feels different than only seeing sources.

Is it accurate to say that the Gradle plugin provides source context but does not help with configuring GitHub code mappings?

[romtsn]

@gilday so, the source context is a prerequisite for displaying github links afaik, otherwise even if you set up correct code mappings, they just won't be shown in the UI without the source context. So you need both in this case, and the gradle plugin helps with the former.

Have you tried setting up code mappings in your project settings as describe in the guide you linked?

[gilday]

I have tried setting up those code mappings. They're not working, but I now see that code mapping requires a Business payment tier but I have a Team payment tier at the moment.

I'm not likely to upgrade for this feature alone, because manually configuring code mappings for my multi-module Gradle project is too tedious. I have dozens of projects in my multi-module build, and each one needs its own code mapping. The mappings I set-up in August are already woefully out of date with the current state of the repository.

This brings me to the original new capability request in the description of this issue: I'd like for the Sentry Gradle plugin to manage my code mappings for this project. Without that, the code mapping feature gives me more toil than value.

[adinauer]

Not sure how this works exactly in the Sentry backend but we could try to make it smarter by recognizing popular build tools (Gradle, Maven) and auto mapping modules after scanning the repo.

[gilday]

Not sure how this works exactly in the Sentry backend but we could try to make it smarter by recognizing popular build tools (Gradle, Maven) and auto mapping modules after scanning the repo.

I too build SaaS developer tools, and having been down this road before, you may also find that Gradle project directory structures don't always conform to conventions that you can depend on from the outside looking in . The most reliable way to find the sources is to be in the Gradle build i.e. a plugin. JetBrains project model syncing is an example: the IDE needs to know is where the sources are, and it finds them by ways of a Gradle plugin.

[adinauer]

Ah thanks for the info. Good thing we already have a gradle and maven plugin, maybe they just need to be told the repo URL and offer some task that can set up mappings.

[wzieba]

hi there 👋 I believe this issue should not be closed: it started with

tooling to synchronize the source code mappings in my Sentry GitHub integration with my Java project structure

but was closed with a commit mentioning documentation about setting dependency with code generation tasks.

To circle back to the original request from @gilday, I'd like to second to

I have dozens of projects in my multi-module build, and each one needs its own code mapping. The mappings I set-up in August are already woefully out of date with the current state of the repository.

this is very on point 🎯 . I guess it's unlike that a team of multimodule Android project would invest into maintaining dozens of Code Mappings manually; therefore they lose plenty of great features you prepared (Stack Trace Links, Source Contexts etc.).

Would you consider:

• introducing a feature to Sentry Gradle Plugin which would provide Code Mappings
• or at lest API for this, so we could prepare a script ourselves
• or any other solution to this?

[adinauer]

Sorry this was closed via the PR that added docs for a fix mentioned in this issue. Reopening.

[adinauer]

@wzieba afaict there's currently no API to configure code mappings. So achieving this seems a little more comlex than assumed.

[wzieba]

Thank you @adinauer for opening this again!

there's currently no API to configure code mappings

It's unfortunate :(. It'd be really great if there was one - ideally, mapping would be associated with the specific build via some kind of id (the same as Source Context or Proguard mappings are) as mapping can change from build to build.

I hope this feature will be prioritized some day.

[adinauer]

Yeah we'll likely have to make this more generic and rework integrations as well. It's now back in our backlog but I can't say when we'll get to it.

[adinauer]

@wzieba you could take a look at the Sentry UI in browser dev tools and inspect API requests performed when adding code mappings, then replicate this in a script.

[wzieba]

Good idea @adinauer ! I've come up with something like this:

custom task

tasks.register("createSentryMappings") {
    doLast {
        fun makeRequest(dryRun: Boolean, sourceRoot: String, stackRoot: String) {
            println("About to send mapping\nsourceRoot: $sourceRoot\nstackRoot: $stackRoot\n---")
            if (dryRun) return

            val body = HttpRequest.BodyPublishers.ofString(
                """
                    {
                      "defaultBranch": "main",
                      "integrationId": "<id>",
                      "projectId": "<id>",
                      "repositoryId": "<id>",
                      "sourceRoot": "$sourceRoot",
                      "stackRoot": "$stackRoot"
                    }
                """.trimIndent()
            )
            val request = HttpRequest
                .newBuilder(URI("https://us.sentry.io/api/0/organizations/<org>/code-mappings/"))
                .setHeader(
                    "Authorization",
                    "Bearer <token>"
                )
                .setHeader("Content-Type", "application/json")
                .POST(body)
                .build()

            val response =
                HttpClient.newHttpClient().send(request, HttpResponse.BodyHandlers.ofString())

            logger.info(response.body())
        }
        rootProject.allprojects
            .filter {
                it.name !in listOf(
                    rootProject.name,
                    <projects_to_exclude>
                )
            }
            .forEach {
                makeRequest(
                    false,
                    sourceRoot = "modules/${it.project.parent!!.name}/${it.project.name}/src/main/java/au/com/shiftyjelly/pocketcasts/${it.project.name}",
                    stackRoot = "au/com/shiftyjelly/pocketcasts/${it.project.name}",
                )
            }
    }
}

It's tailored for my use case with hard-coded source and stack roots, but I guess it can be a starting point for someone else looking for a way to automate sending mappings.

[adinauer]

Thanks @wzieba that's great 🚀