Consider bumping scala-xml to 2.x

[ckipp01]

Problem outline

Not really a bug, just something a bunch of people are going to hit on. I've reported this to scala/bug, but these updates may have to go hand in hand if we do them. I'll copy over what I posted there since the reasoning is the same:

I know this was a conversation a long time ago in scala/scala#9743 but recently this conversation has surfaced again. Recently in sbt-scoverage I had some reports of conflicts since I was still on 1.x, but some other libraries like scalatest bumped to 2.x making the problem hit people a bit harder. Due to that, I went ahead and updated to 2.x. Due to me updating, we got even more reports of conflicts with sbt-native-packager, which caused them to update to 2.x as well. Now we're in a position where people are getting hit with stuff like this:

[error]         * org.scala-lang.modules:scala-xml_2.12:2.1.0 (early-semver) is selected over {1.3.0, 1.0.6, 1.2.0}
[error]             +- org.scoverage:scalac-scoverage-reporter_2.12:2.0.2 (depends on 2.1.0)
[error]             +- com.github.sbt:sbt-native-packager:1.9.10 (sbtVersion=1.0, scalaVersion=2.12) (depends on 2.1.0)
[error]             +- org.scala-sbt:testing_2.12:1.7.1                   (depends on 1.3.0)
[error]             +- org.scala-sbt:sbinary_2.12:0.5.1                   (depends on 1.0.6)
[error]             +- org.scala-sbt:main_2.12:1.7.1                      (depends on 1.3.0)
[error]             +- org.scala-sbt:librarymanagement-core_2.12:1.7.0    (depends on 1.2.0)
[error]             +- org.scala-lang:scala-compiler:2.12.16              (depends on 1.0.6)
[error]             +- io.get-coursier:lm-coursier-shaded_2.12:2.0.10     (depends on 1.3.0)

Where common sbt plugins are conflicting with sbt and Scala 2.12 itself. Looking back at past conversations I understand the breaking changes between 1.x and 2.x of scala-xml are extremely small so doing the following is pretty safe:

in your plugins.sbt

ThisBuild / libraryDependencySchemes ++= Seq(
  "org.scala-lang.modules" %% "scala-xml" % VersionScheme.Always
)

But this isn't ideal to have this all over the place, and will cause confusions for users. Is it time to update scala-xml in 2.12? Is there something holding this back? We're in a weird position now where have the ecosystem is update for 2.12 and half isn't, causing issues.

[guizmaii]

Related to this issue, is there any plan to update to Scala 2.13? 🤔

[ckipp01]

Related to this issue, is there any plan to update to Scala 2.13? 🤔

From my understanding of past discussions, no. Updating to 2.13 would have huge ramifications to the sbt plugin ecosystem causing all plugins to have to update to 2.13. So I believe the plan is for sbt 2.x to be on Scala 3, so the plugin ecosystem would only need to go through that migration once. @eed3si9n can correct me if I'm wrong on that.

[guizmaii]

Make sense. Thanks for the answer @ckipp01 :)

[eed3si9n]

Commented on scala/bug#12632 (comment)

Yea. Unless something prevents us from going to Scala 3, sbt 2.x should be on 3 (see https://eed3si9n.com/sudori-part3/ etc).

[martijnhoekstra]

would declaring the dependency as "[1.3,3.0[" maybe work as intended?

[mkurz]

Got this fixed with #7021?

[guizmaii]

Well, this issue is blocking Twirl (see playframework/twirl#548 (comment)), which is blocking Caliban (see ghostdogpr/caliban#1472)

As suggested by @mkurz in his comment in the Twirl repo, can we maybe get a 1.8.0-M1 release using Scala 2.12.17 to be able to unblock everyone, maybe?

[mkurz]

To be honest that whole scala-xml upgrade thinig is a mess. The fact that sbt fails with an error on startup just by upgrading a librarie's patch version because it upgraded scala-xml is very very annoying.

Next time this has should be coordinated somehow (next candidate is scala-parser-combinators...

[guizmaii]

Next time this has should be coordinated somehow

I'm not against but, IMO, it's quite hard to coordinate all these projects maintained by many different people unless one person or a small group of persons takes the responsibility to coordinate everything.

The few of us who are involved in this scala-xml migration in Scala 2.12 and sbt are making PRs here and there to update things. It takes time, it's open-source work made in people's free time. Without this person (or small group) coordinating and making the required work, I don't see how it can be done more quickly.

[mkurz]

True, but bumping in patch releases is also not the best idea. That crashes sbt just because upgrading sbt-native-packager from 1.9.9 to 1.9.10... That's what's really bothers me.

[mkurz]

True, but bumping in patch releases is also not the best idea. That crashes sbt just because upgrading sbt-native-packager from 1.9.9 to 1.9.10... That's what's really bothers me.

And suddenly I have to talk to 5 other maintainers to convice them to upgrade scala-xml because I can't, in a clean way, upgrade a patch version...

[ckipp01]

For what it's worth, I understand your frustration, but this thread on sbt probably isn't the correct place to air it as it's not the fault of sbt at all.

There is more context in here but this wasn't the first time scala-xml updates have caused issues. Projects have bumped back and forth to handle this, published multiple versions, etc. I totally agree that it's a bit of a nightmare. I'll even take part of the blame as I'm the one who bumped scoverage. However, I bumped because someone else bumped and I got complaints about it. That project probably bumped because someone else bumped. Thus is the nature of developing with Scala, the JVM, semver, etc.

As @guizmaii mentioned these things are really hard to coordinate without a single person at the helm guiding it, but even with that, you'll have projects that won't bump or respond, etc. There are various open prs and issues around the ecosystem to bump. I encourage people to continue to push for this, comment on issues, and lend a hand where they can. This will hopefully get everything over the tipping point and we can forget about scala-xml 1.x and this issue for good.

[mkurz]

This will hopefully get everything over the tipping point and we can forget about scala-xml 1.x and this issue for good.

scala-parser-combinators like: hold my beer 😉

[ckipp01]

@eed3si9n do you have any timeline on when you'd like to get 1.8.x out? I noticed that with the latest 1.7.3 even just using addDependencyTreePlugin globally (which I normally do) causes issues:

[error] java.lang.RuntimeException: found version conflict(s) in library dependencies; some are suspected to be binary incompatible:
[error]
[error]         * org.scala-lang.modules:scala-xml_2.12:2.1.0 (early-semver) is selected over {1.3.0, 1.2.0, 1.0.6}
[error]             +- org.scala-lang:scala-compiler:2.12.17              (depends on 2.1.0)
[error]             +- org.scala-sbt:testing_2.12:1.7.3                   (depends on 1.3.0)
[error]             +- org.scala-sbt:sbinary_2.12:0.5.1                   (depends on 1.0.6)
[error]             +- org.scala-sbt:main_2.12:1.7.3                      (depends on 1.3.0)
[error]             +- org.scala-sbt:librarymanagement-core_2.12:1.7.1    (depends on 1.2.0)
[error]             +- io.get-coursier:lm-coursier-shaded_2.12:2.0.12     (depends on 1.3.0)

I've seen others report this as well. However it's not clear where 2.12.17 is coming in from in this scenario. This also made me realize it wasn't bumped in lm-coursier-shaded, so I sent in coursier/sbt-coursier#427.

Either way the ball is sort of rolling, so how do we think it's best to proceed, and how can I help?

[ajozwik]

It's weird that setting VersionScheme.Always for scala-xml doesn't work for my local machine (Apple M1 Max, macOS Monterey v12.6) if I use sbt 1.7.3

Sounds like a separate bug report, if you can manage to come up with reproduction steps that work for others

For reproduction clone: https://github.com/ajozwik/quill-generic
And change build.properties to 1.7.3

[ckipp01]

@SethTisue I just tested it with another Apple M1 (Pro) and it's fine there. 😵‍💫

This sounds like it could be something in your global ~/.sbt/1.0/plugins.sbt or something. For example if you're using addDependencyTreePlugin locally it will fail for any project on your machine using 1.7.3. So fon't forget to check there.

[ckipp01]

. For reproduction clone: https://github.com/ajozwik/quill-generic
And change build.properties to 1.7.3

Locally with this repo bumping to 1.7.3 gives me no issues. Same as above, it could be something in your global config actually causing an issue here.

[ajozwik]

. For reproduction clone: https://github.com/ajozwik/quill-generic
And change build.properties to 1.7.3

Locally with this repo bumping to 1.7.3 gives me no issues. Same as above, it could be something in your global config actually causing an issue here.

Yes,
I have:
addDependencyTreePlugin
in ~/.sbt/1.0/plugins/plugins.sbt

[eed3si9n]

I am guessing that in some situations sbt-coursier's Scala version (2.12.17) is bubbling up. I've opened coursier/coursier#2573 to request 2.12.16 version of sbt-coursier for 1.7.x branch.

[kevin-lee]

@SethTisue I just tested it with another Apple M1 (Pro) and it's fine there. 😵‍💫

This sounds like it could be something in your global ~/.sbt/1.0/plugins.sbt or something. For example if you're using addDependencyTreePlugin locally it will fail for any project on your machine using 1.7.3. So fon't forget to check there.

@ckipp01 Yeah! I should have checked your comment early! 😂
I just found that disabling all plugins I have at ~/.sbt/1.0/plugins/sbt-plugins.sbt solves the issue. I'll check one by one and share what caused it later.

[kevin-lee]

I have 7 plugins in ~/.sbt/1.0/plugins/sbt-plugins.sbt and only addDependencyTreePlugin caused the problem.
So it's the same as Andrzej's one.

[ckipp01]

In case anyone here missed it and is following along, v1.8.0-RC1 was realeased which does bring in the 2.x version of scala-xml. Please do test it out and report back if you're hitting on any scala-xml problems. I'm assuming there may be some, and then we can chase them down in the various repos.

[guizmaii]

FYI, migrating to v1.8.0-RC1 seems to work fine for us

[fourth44]

I still had problems with version conflicts when trying to use addDependencyTreePlugin in plugins.sbt (and sbt.version=1.8.0-RC1), but removing metals.sbt and .bloop/ folders did help (which weren't touched upon for a while).

[eed3si9n]

ok. I am closing this as done. If you encounter related issues (esp years from now) please open a fresh issue to avoid pinging everyone on this thread.

[sjfloat]

In case anyone here missed it and is following along, v1.8.0-RC1 was realeased which does bring in the 2.x version of scala-xml. Please do test it out and report back if you're hitting on any scala-xml problems. I'm assuming there may be some, and then we can chase them down in the various repos.

1.8.0-RC1 did not clear this up for me.

[eed3si9n]

Some builds will continue to fail if the plugin depends on scala-xml 1.x. Either case the workaround would be to put the following in project/plugins.sbt:

ThisBuild / libraryDependencySchemes += "org.scala-lang.modules" %% "scala-xml" % VersionScheme.Always

[guizmaii]

FYI, some members of my team, when they moved to 1.8.0-RC1, were experiencing this issue:

To fix it, they had to delete the project and clone it again from Github.

Weirdly enough, it seems that it was impacting people having an M1 mac 🤷‍♂️

[sjfloat]

Some builds will continue to fail if the plugin depends on scala-xml 1.x. Either case the workaround would be to put the following in project/plugins.sbt:

ThisBuild / libraryDependencySchemes += "org.scala-lang.modules" %% "scala-xml" % VersionScheme.Always

@ckipp01 suggested this to me on discord and it works for us. Thanks!

[eed3si9n]

@guizmaii Thanks for that. I'll try to remove those transitive dependencies on scala-xml 1.x.

[ckipp01]

Just a heads up (last post here I promise), I created #7075 as a way to track various plugins or issues that people may be hitting on. If you're still hitting on issues related to this, feel free to head there first before creating an issue or commenting here.

[zzvara]

I also needed this for settings:

ThisBuild / evictionErrorLevel := Level.Info

[ckipp01]

I also needed this for settings:

ThisBuild / evictionErrorLevel := Level.Info

I'd actually caution you not to use this setting. By doing this you're telling your whole build that instead of erroring on an eviction that it should error on, just ignore it. While for some cases like scala-xml this may be ok, for most libraries this will cause things to blow up. Using

ThisBuild / libraryDependencySchemes += "org.scala-lang.modules" %% "scala-xml" % VersionScheme.Always

to only target scala-xml in the case that you need to should be enough. If you're having to use what you listed, then you probably have some other conflicts as well that you don't want to be ignoring.

With that being said, the core of this issue in sbt has been solved as sbt 1.8.0 is on 2.x of scala-xml, so it's recommended you update. If you still have issues, then there is probably an issue with something else in your build that I recommend you hunt down or open a new issue about.

[zzvara]

@ckipp01, thanks for reminding me about that one!

However, this did not work for me in plugins.sbt:

ThisBuild / libraryDependencySchemes += "org.scala-lang.modules" %% "scala-xml" % VersionScheme.Always

This is the issue:

01-Jan-2023 19:27:04 | [error] (core / update) found version conflict(s) in library dependencies; some are suspected to be binary incompatible:
-- | --
01-Jan-2023 19:27:04 | [error]
01-Jan-2023 19:27:04 | [error]         * org.scala-lang.modules:scala-xml_2.13:2.1.0 (early-semver) is selected over {1.3.0}
01-Jan-2023 19:27:04 | [error]             +- org.scoverage:scalac-scoverage-reporter_2.13:2.0.7 (depends on 2.1.0)
01-Jan-2023 19:27:04 | [error]             +- systems.enliven.invoicing.hungarian:core_2.13:1.0.109 (depends on 1.3.0)
01-Jan-2023 19:27:04 | [error]             +- org.scalatra.scalate:scalate-util_2.13:1.9.7       (depends on 1.3.0)

I'm on sbt 1.8.0.

Plugins are as follows:

logLevel := Level.Warn

addDependencyTreePlugin
addSbtPlugin("com.github.cb372" % "sbt-explicit-dependencies" % "0.2.16")
addSbtPlugin("org.scoverage" % "sbt-scoverage" % "2.0.6")
addSbtPlugin("com.github.sbt" % "sbt-release" % "1.1.0")
addSbtPlugin("ch.epfl.scala" % "sbt-scalafix" % "0.10.4")
addSbtPlugin("de.heikoseeberger" % "sbt-header" % "5.9.0")
addSbtPlugin("io.gatling" % "gatling-sbt" % "4.2.4")
addSbtPlugin("com.eed3si9n" % "sbt-assembly" % "2.0.0")
addSbtPlugin("pl.project13.scala" % "sbt-jmh" % "0.4.3")
addSbtPlugin("com.typesafe.sbt" % "sbt-license-report" % "1.2.0")

ThisBuild / libraryDependencySchemes += "org.scala-lang.modules" %% "scala-xml" % VersionScheme.Always

[ckipp01]

However, this did not work for me in plugins.sbt:

Looking at the dependency issue, it's actually an issue with 2.13 artifacts, meaning it's not an issue with your plugins. If you'd move that same libraryDependencySchemes to your build.sbt it'd probably fix it.