Dependency management for Kafka 2.5 on nonexistent org.apache.kafka:kafka_2.11:2.5.0

[otaconix]

I tried upgrading one of my projects' dependency on spring-boot-dependencies to 2.3.0, only to be met by errors mentioning a dependency not found.
It turns out that spring-boot-dependencies does dependency management for a bunch of Kafka artifacts, and that includes kafka_2.11. Unfortunately, support for Scala 2.11 has been dropped in Kafka 2.5.0, meaning this dependency no longer exists.

I'm not sure what the best course of action would be, but it may be wise to simply remove the dependency on kafka_2.11 (assuming no Spring Boot project actually depends on it, which I don't know).

[snicoll]

Thanks for the report. I can see we're providing dependency management for a non existent artifact but that shouldn't lead to any problem per se on your end.

What concrete problems are you facing? Sharing a sample project if you have one would be ideal.

[otaconix]

kafka_2.11 is a transitive dependency of ours. By importing spring-boot-dependencies in our pom.xml, we're forcing (managing) versions of many Kafka artifacts, which is fine when the forced versions exist, but causes problems when they don't.

Reproduction case (however contrived):

<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
         xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
    <modelVersion>4.0.0</modelVersion>

    <groupId>org.example</groupId>
    <artifactId>spring-boot-kafka_2.11-issue-repro</artifactId>
    <version>1.0-SNAPSHOT</version>

    <dependencyManagement>
        <dependencies>
            <dependency>
                <groupId>org.springframework.boot</groupId>
                <artifactId>spring-boot-dependencies</artifactId>
                <version>2.3.0.RELEASE</version>
                <type>pom</type>
                <scope>import</scope>
            </dependency>
        </dependencies>
    </dependencyManagement>

    <dependencies>
        <dependency>
            <groupId>net.mguenther.kafka</groupId>
            <artifactId>kafka-junit</artifactId>
            <version>2.3.0</version>
        </dependency>
    </dependencies>
</project>

I can fix the problem by adding another managed dependency, but it'd be nice not to have to.

[snicoll]

Thanks for the short sample and I can see how that's a problem. However, if you're relying on kafka_2.11 you're not really compatible with Kafka 2.5 so you need to fix that before upgrading to Spring Boot 2.3, isn't it?

[otaconix]

What you say is absolutely true. If we required compatibility with Kafka 2.5, we'd have another problem. But we don't require Kafka 2.5 (and we don't use any Spring Boot-provided Kafka integration).
Maybe a case could be made that we shouldn't use spring-boot-dependencies, but it's very convenient, so I'd like to keep on using it if at all possible.

[snicoll]

Ah that makes total sense now. Sorry I've been a bit slow here. And sorry for breaking you.

[wilkinsona]

We should have removed the dependency management for kafka_2.11 (and I suspect kafka-streams-scala_2.11) when we upgraded to Kafka 2.5. I think we should remove the dependency management now. We should also double-check that Bomr detects that those modules are missing.

[snicoll]

It does.

Kafka 2.4.1:
  1: 2.5.0 (some modules are missing: org.apache.kafka:kafka-streams-scala_2.11, org.apache.kafka:kafka_2.11)

I didn't noticed as I upgraded to Kafka 2.5 manually as part of upgrading to Spring Kafka SNAPSHOT 🤦‍♂️

[puppylpg]

It does.

Kafka 2.4.1:
  1: 2.5.0 (some modules are missing: org.apache.kafka:kafka-streams-scala_2.11, org.apache.kafka:kafka_2.11)

I didn't noticed as I upgraded to Kafka 2.5 manually as part of upgrading to Spring Kafka SNAPSHOT

I encountered the same problem today and notiecd you have deleted kafka_2.11 in spring-boot-dependencies. May I know the version number of the fixed release and the date?
Thanks

[snicoll]

@puppylpg perhaps you're looking to this page on a phone? There is a milestone on the right hand side that provides the version (2.3.1). It was released an hour ago, or something.

[puppylpg]

@puppylpg perhaps you're looking to this page on a phone? There is a milestone on the right hand side that provides the version (2.3.1). It was released an hour ago, or something.

Oh sorry, I didn't notice the 2.3.1 version and just searched in maven repo but found nothing. Maybe later it'll be ok.
Thanks sincerely~