Cannot resolve an SBT plugin published to a maven repo - sbt

I created a reproducible example about a problem I'm experiencing with SBT plugin resolution:
https://github.com/NicolasRouquette/sbt.problem.example
This example has two SBT projects:
test.plugin a simple SBT plugin
test.app a simple SBT project that uses the test.plugin
There is also a local Maven repository to which the test.plugin is published with a POM file that includes properties like this:
<properties>
<git.branch>master</git.branch>
<git.commit>fe2dc11d6fbb85c5ce0e83b031bbd425997bbd59</git.commit>
<git.tags></git.tags>
</properties>
<properties>
<scalaVersion>2.10</scalaVersion>
<sbtVersion>0.13</sbtVersion>
<extraDependencyAttributes xml:space="preserve">+e:sbtVersion:###:+0.13:###:+module:###:+sbt-license plugin:###:+e:scalaVersion:###:+2.10:###:+organisation:###:+com.banno:###:+branch:###:+##:NULL:##:###:+revision:###:+0.1.5:###:
+e:sbtVersion:###:+0.13:###:+module:###:+sbt-license-report:###:+e:scalaVersion:###:+2.10:###:+organisation:###:+com.typesafe.sbt:###:+branch:###:+##:NULL:##:###:+revision:###:+1.0.0:###:
+e:sbtVersion:###:+0.13:###:+module:###:+sbt-git:###:+e:scalaVersion:###:+2.10:###:+organisation:###:+com.typesafe.sbt:###:+branch:###:+##:NULL:##:###:+revision:###:+0.8.5:###:
+e:sbtVersion:###:+0.13:###:+module:###:+aether-deploy:###:+e:scalaVersion:###:+2.10:###:+organisation:###:+no.arktekk.sbt:###:+branch:###:+##:NULL:##:###:+revision:###:+0.16:###:
</extraDependencyAttributes>
</properties>
I can't run SBT on test.app because SBT fails to resolve the test.plugin:
addSbtPlugin("org.test" % "test-plugin" % "1.0")
[warn] ::::::::::::::::::::::::::::::::::::::::::::::
[warn] :: UNRESOLVED DEPENDENCIES ::
[warn] ::::::::::::::::::::::::::::::::::::::::::::::
[warn] :: org.test#test-plugin;1.0: not found
[warn] ::::::::::::::::::::::::::::::::::::::::::::::
[warn]
[warn] Note: Some unresolved dependencies have extra attributes. Check that these dependencies exist with the requested attributes.
[warn] org.test:test-plugin:1.0 (sbtVersion=0.13, scalaVersion=2.10)
[warn]
[warn] Note: Unresolved dependencies path:
[warn] org.test:test-plugin:1.0 (sbtVersion=0.13, scalaVersion=2.10) (/opt/local/imce/users/nfr/github.imce/example/test.app/project/plugins.sbt#L6-7)
[warn] +- default:test-app-build:0.1-SNAPSHOT (sbtVersion=0.13, scalaVersion=2.10)
sbt.ResolveException: unresolved dependency: org.test#test-plugin;1.0: not found
at sbt.IvyActions$.sbt$IvyActions$$resolve(IvyActions.scala:294)
at sbt.IvyActions$$anonfun$updateEither$1.apply(IvyActions.scala:191)
at sbt.IvyActions$$anonfun$updateEither$1.apply(IvyActions.scala:168)
at sbt.IvySbt$Module$$anonfun$withModule$1.apply(Ivy.scala:155)
at sbt.IvySbt$Module$$anonfun$withModule$1.apply(Ivy.scala:155)
at sbt.IvySbt$$anonfun$withIvy$1.apply(Ivy.scala:132)
at sbt.IvySbt.sbt$IvySbt$$action$1(Ivy.scala:57)
...
Based on trying to understand what's going on with the debugger, I get the impression that:
resolving plugins happens very early when running SBT; which means that it involves a lot of SBT/Ivy machinery
I don't quite understand how the SBT/Ivy machinery resolves SBT plugins published to a Maven repository (though I've seen some kind of Maven/Ivy conversion somewhere)
it seems that the Maven POM properties are not analyzed to retrieve the sbtVersion (e.g. 0.13) and scalaBinaryVersion (e.g. 2.10) of an artifact
Can someone confirm/correct my analysis?
Is there a way to make this maven-published plugin dependency lookup work?

Thanks for the repro project.
First issue is that you have two <properties> element. sbt will use both the URL layout and the content of the POM to resolve the plugin. Here's how to merge <properties> element. Not sure if there's more elegant way:
makePom := {
val old = makePom.value
val pom = xml.XML.loadFile(old)
val additionalProperties =
(<git.branch>{git.gitCurrentBranch.value}</git.branch>
<git.commit>{git.gitHeadCommit.value.getOrElse("N/A")+(if (git.gitUncommittedChanges.value) "-SNAPSHOT" else "")}</git.commit>
<git.tags>{git.gitCurrentTags.value}</git.tags>)
val newPom = pom.copy(child = pom.child.toSeq map {
case elem: xml.Elem if elem.label == "properties" =>
elem.copy(child = elem.child ++ additionalProperties)
case x => x
})
xml.XML.save(old.toString, newPom, enc = "UTF-8", xmlDecl = true)
old
}
Next, you should've seen something like this when you tried to resolve the plugin:
[warn] ==== Local Test: tried
[warn] file:/xxx/sbt.problem.example/test.app/project/../local.repo/org/test/test-plugin_2.10_0.13/1.0/test-plugin-1.0.pom
See that test.app/project/../local.repo/ would become test.app/local.repo. Instead of ".." here's what you can do in test.app/project/plugins.sbt:
resolvers += new MavenCache("Local Test", baseDirectory.value.getParentFile.getParentFile / "local.repo")
addSbtPlugin("org.test" % "test-plugin" % "1.0")

Related

sbt behind local artifactory proxy

When setting up a sbt project with a local artifactory / maven proxy I see the following message:
In order to specify that all resolvers added in the sbt project should
be ignored in favor of those configured in the repositories
configuration, add the following configuration option to the sbt
launcher script:
-Dsbt.override.build.repos=true
Add the following to your build.sbt file:
resolvers +=
"Artifactory" at "http://url/artifactory/virtualRepository/"
But what I would like to achieve is a behaviour similar to maven i.e. not manually overriding resolvers in the SBT file, but rather via the configuration.
Is this possible as well? If yes how?
Desired behaviour
the project should compile fine without local artifactory proxy
when available / configured in repositories the local one should be used as the source / cache for quicker access
currently, I only get unresolved dependencies for sbt plugins:
::::::::::::::::::::::::::::::::::::::::::::::
[warn] :: UNRESOLVED DEPENDENCIES ::
[warn] ::::::::::::::::::::::::::::::::::::::::::::::
[warn] :: org.scalariform#sbt-scalariform;1.6.0: not found
[warn] :: org.scoverage#sbt-scoverage;1.5.0: not found
[warn] :: org.scalastyle#scalastyle-sbt-plugin;0.8.0: not found
[warn] :: net.virtual-void#sbt-dependency-graph;0.8.2: not found
[warn] ::::::::::::::::::::::::::::::::::::::::::::::
With warnings of
[warn] module not found: org.scalariform#sbt-scalariform;1.6.0
[warn] ==== typesafe-ivy-releases: tried
[warn] https://repo.typesafe.com/typesafe/ivy-releases/org.scalariform/sbt-scalariform/scala_2.10/sbt_0.13/1.6.0/ivys/ivy.xml
[warn] ==== sbt-plugin-releases: tried
[warn] https://repo.scala-sbt.org/scalasbt/sbt-plugin-releases/org.scalariform/sbt-scalariform/scala_2.10/sbt_0.13/1.6.0/ivys/ivy.xml
[warn] ==== local: tried
[warn] d:\users\heilerg\.ivy2\local\org.scalariform\sbt-scalariform\scala_2.10\sbt_0.13\1.6.0\ivys\ivy.xml
[warn] ==== my-ivy-proxy-releases: tried
[warn] http://url/artifactory/virtualRepositoryScala/org.scalariform/sbt-scalariform/scala_2.10/sbt_0.13/1.6.0/ivys/ivy.xml
[warn] ==== my-maven-proxy-releases: tried
[warn] http://url/artifactory/virtualRepositoryScala/org/scalariform/sbt-scalariform_2.10_0.13/1.6.0/sbt-scalariform-1.6.0.pom
[warn] ==== Artima Maven Repository: tried
[warn] http://repo.artima.com/releases/org/scalariform/sbt-scalariform_2.10_0.13/1.6.0/sbt-scalariform-1.6.0.pom
[info] Resolving org.scoverage#sbt-scoverage;1.5.0 ...
and SBT logs will show
[ERROR] (o.a.r.RemoteRepoBase:766) - IO error while trying to download resource 'repo1:org/scalariform/sbt-scalariform_2.10_0.13/1.6.0/sbt-scalariform-1.6.0.pom': org.artifactory.api.repo.exception.maven.BadPomException: The target deployment path 'org/scalariform/sbt-scalariform_2.10_0.13/1.6.0/sbt-scalariform-1.6.0.pom' does not match the POM's expected path prefix 'org/scalariform/sbt-scalariform/1.6.0'. Please verify your POM content for correctness and make sure the source path is a valid Maven repository root path.
Somewhere people mention to use the following option in Artifactory to "suppress POM consistency checks", but in the current version of artifactory I can not finde such an option.
edit
I can see only these options
As answered in the comment, a common workaround is "Suppress POM Consistency Checks" - Advanced Settings.
If that is not desirable, another method might be to re-publish the plugin with a valid POM. I've written POM consistency for sbt plugins to show that can be done.
// set some unique postfix
ThisBuild / version := "0.15.0-Pets1"
lazy val root = (project in file("."))
.enablePlugins(SbtPlugin)
.settings(
name := "sbt-assembly",
....
publishMavenStyle := true,
// add this
pomConsistency2021DraftSettings,
)
// Add the following
lazy val pomConsistency2021Draft = settingKey[Boolean]("experimental")
/**
* this is an unofficial experiment to re-publish plugins with better Maven compatibility
*/
def pomConsistency2021DraftSettings: Seq[Setting[_]] = Seq(
pomConsistency2021Draft := Set("true", "1")(sys.env.get("POM_CONSISTENCY").getOrElse("false")),
moduleName := {
if (pomConsistency2021Draft.value)
sbtPluginModuleName2021Draft(moduleName.value,
(pluginCrossBuild / sbtBinaryVersion).value)
else moduleName.value
},
projectID := {
if (pomConsistency2021Draft.value) sbtPluginExtra2021Draft(projectID.value)
else projectID.value
},
)
def sbtPluginModuleName2021Draft(n: String, sbtV: String): String =
s"""${n}_sbt${if (sbtV == "1.0") "1" else if (sbtV == "2.0") "2" else sbtV}"""
def sbtPluginExtra2021Draft(m: ModuleID): ModuleID =
m.withExtraAttributes(Map.empty)
.withCrossVersion(CrossVersion.binary)

SBT SFTP resolver not trying hard enough

I am converting our mixed build environment into an sbt based ecosystem. And I am at a bit of a road block with this issue.
I have deployed a library to a remote repository via sftp but when I try to load it as a dependency I have an issue.
In build.sbt I have:
resolvers += {
val privateKeyFile = new java.io.File(sys.env("HOME") + "/.ssh/id_rsa")
Resolver.sftp("MY REPO", "aHost", "/repoBase/") as ("aUser", privateKeyFile)
}
update in interactive shows:
[info] Resolving com.test.it.out#myLib_2.11;1.0 ...
[warn] module not found: com.test.it.out#myLib_2.11;1.0
[warn] ==== local: tried
[warn] /home/bday/.ivy2/local/com.test.it.out#myLib_2.11/1.0/ivys/ivy.xml
[warn] ==== public: tried
[warn] https://repo1.maven.org/maven2/com.test.it.out#myLib_2.11/1.0/jsshfs_2.11-1.0.pom
[warn] ==== MY REPO: tried
[info] Resolving jline#jline;2.12 ...
[warn] ::::::::::::::::::::::::::::::::::::::::::::::
[warn] :: UNRESOLVED DEPENDENCIES ::
[warn] ::::::::::::::::::::::::::::::::::::::::::::::
It never lists any path information for MY REPO.
I have tried multiple ways to configure the resolver, even users/hosts that should fail and the results don't change.
show resolvers
[info] List(SftpRepository(MY REPO,SshConnection(Some(KeyFileAuthentication(aUser,/home/aUser/.ssh/id_rsa,None)),Some(aHost),None),Patterns(ivyPatterns=List(), artifactPatterns=List(/repoBase/[organisation]/module(_[sbtVersion])/[revision]/[artifact]-revision.[ext]), isMavenCompatible=true, descriptorOptional=false, skipConsistencyCheck=false)))
I did see someone had the same issue and said switching from SSH to SFTP fixed it, in 12.?. I am using 13.7
I have come to the end of the internet on this one, please extends it a bit.
Thanks!
I had the similar issue, it was not listing any urls that it tried. I added the same Pattern that I was using while publishing, and it fixed the problem. Also, I have put my resolver as the first one in the order.
resolvers in ThisBuild := (Resolver.sftp("MY Repo", "example.com", 22, "sbt-repos/releases")(Patterns("[organisation]/[module](_[scalaVersion])(_[sbtVersion])/[revision]/[type]s/[artifact](-[classifier]).[ext]"))) +: resolvers.value
I am using 0.13.9. I searched a lot for this one and finally figured it out after so many trials. Hope this will help somebody.
val sftpRes = {
val privateKeyFile = new java.io.File(sys.env("HOME") + "/.ssh/id_rsa")
Resolver.sftp("REPO", "gitHost", "/path/to/data/")(Resolver.ivyStylePatterns) as ("user", privateKeyFile)
}
publishMavenStyle := false
publishTo := Some(sftpRes)
Using SBT 1.2.4
In the end this is what I got to work. I tried Resolver.mavenStylePatterns, but in that case as with others SBT never really tries to find files on the resolver.
Using Ivy patterns on the publish and resolve worked for me.

SBT doesn't use ssh-based resolver to resolve dependency

I have the following definition in my build.sbt:
libraryDependencies += "com.bubblefoundry" %% "something" % "0.1-SNAPSHOT"
resolvers += {
val privateKeyFile = new java.io.File(sys.env("HOME") + "/.ssh/id_rsa")
Resolver.ssh("Bubble Foundry", "bubblefoundry.com", "/usr/local/repository/") as ("peter", privateKeyFile) withPermissions("0644")
}
When sbt tries to resolve the dependency, it fails:
[info] Resolving com.bubblefoundry#something_2.10;0.1-SNAPSHOT ...
[warn] module not found: com.bubblefoundry#something_2.10;0.1-SNAPSHOT
[warn] ==== local: tried
[warn] /Users/peter/.ivy2/local/com.bubblefoundry/something_2.10/0.1-SNAPSHOT/ivys/ivy.xml
[warn] ==== Bubble Foundry: tried
[warn] ==== public: tried
[warn] http://repo1.maven.org/maven2/com.bubblefoundry/something_2.10/0.1-SNAPSHOT/something_2.10-0.1-SNAPSHOT.pom
[warn] ::::::::::::::::::::::::::::::::::::::::::::::
[warn] :: UNRESOLVED DEPENDENCIES ::
[warn] ::::::::::::::::::::::::::::::::::::::::::::::
[warn] :: com.bubblefoundry#something_2.10;0.1-SNAPSHOT: not found
[warn] ::::::::::::::::::::::::::::::::::::::::::::::
It appears like it didn't even connect to the server to look for the dependency. Why is that? Am I doing something wrong?
The dependency has been published (using the same resolver definition) to /usr/local/repository/com/bubblefoundry/...
After updating to sbt 0.13, I can use my private repo by adding the following line to build.sbt
resolvers += Resolver.ssh("Company Maven Repo", "git#github.com:company/company-repo.git", "/raw/master")
A dialog pops up asking for my github username and password
It's a bug!
For example I am using github to host a private/internal maven repo accessed via SSH. I can pull artifacts without any difficultly when using Maven/POM files as you'd expect.
But can't get SBT (0.12.2) to work. Most frustratingly it just says
== REPO_NAME: tried
Even if I specify invalid authentication or give a bad ssh url it does the same i.e. there is no error reporting.
Spent a few hours researching and trying combinations to no avail. Therefore I suggest SSH repos don't work properly.
I found a solution: switch from ssh to sftp:
resolvers += {
val privateKeyFile = new java.io.File(sys.env("HOME") + "/.ssh/id_rsa")
Resolver.sftp("Bubble Foundry", "bubblefoundry.com", "/usr/local/repository/") as ("peter", privateKeyFile)
}

What's the proper setup of sbt-idea with sbt 0.11?

I am creating a Scala project with sbt 0.11.2 and sbt-idea and I am getting UNRESOLVED DEPENDENCIES on the gen-idea task.
I've just installed sbt (downloaded jar and made script as instructed in the wiki), followed the sbt-idea setup here, made an empty directory for my project, and run sbt and then run the gen-idea task.
It can't find the dependency because it only uses the built-in repos. How do I tell sbt to check another repo?
When I place the build.sbt file in the plugins dir and run sbt it starts resolving things, one of which is Resolving com.github.mpeltonen#sbt-idea;0.11.0 ...
Later in the process it downloads it successfully:
[info] downloading http://mpeltonen.github.com/maven/com/github/mpeltonen/sbt-idea_2.9.1_0.11.2/0.11.0/sbt-idea-0.11.0.jar ...
[info] [SUCCESSFUL ] com.github.mpeltonen#sbt-idea;0.11.0!sbt-idea.jar (592ms)
When I run the gen-idea task, things look good at first...
> gen-idea
[info] Trying to create an Idea module default-b91f2c
It moves on to creating .idea directories and such, which seem to be created just fine. It then starts resolving things again (scala tools, sbt, commens-*, etc)
Eventually it tries to resolve sbt-idea:
[warn] module not found: com.github.mpeltonen#sbt-idea;0.11.0
[warn] ==== local: tried
[warn] /home/scaladev/.ivy2/local/com.github.mpeltonen/sbt-idea/scala_2.9.1/sbt_0.11.2/0.11.0/ivys/ivy.xml
[warn] ==== typesafe-ivy-releases: tried
[warn] http://repo.typesafe.com/typesafe/ivy-releases/com.github.mpeltonen/sbt-idea/0.11.0/ivys/ivy.xml
[warn] ==== public: tried
[warn] http://repo1.maven.org/maven2/com/github/mpeltonen/sbt-idea_2.9.1_0.11.2/0.11.0/sbt-idea-0.11.0.pom
[warn] ==== Scala-Tools Maven2 Repository: tried
[warn] http://scala-tools.org/repo-releases/com/github/mpeltonen/sbt-idea_2.9.1_0.11.2/0.11.0/sbt-idea-0.11.0.pom
[warn] ==== Scala-Tools Maven2 Snapshots Repository: tried
[warn] http://scala-tools.org/repo-snapshots/com/github/mpeltonen/sbt-idea_2.9.1_0.11.2/0.11.0/sbt-idea-0.11.0.pom
[info] Resolving commons-io#commons-io;2.0.1 ...
[warn] ::::::::::::::::::::::::::::::::::::::::::::::
[warn] :: UNRESOLVED DEPENDENCIES ::
[warn] ::::::::::::::::::::::::::::::::::::::::::::::
[warn] :: com.github.mpeltonen#sbt-idea;0.11.0: not found
[warn] ::::::::::::::::::::::::::::::::::::::::::::::
[warn]
[warn] Note: Some unresolved dependencies have extra attributes. Check that these dependencies exist with the requested attributes.
[warn] com.github.mpeltonen:sbt-idea:0.11.0 (sbtVersion=0.11.2, scalaVersion=2.9.1)
[warn]
I understand that it wouldn't find it at those locations, but I don't understand why it didn't try the github repo, as it did when configuring the plugin. I was expecting to see a line looking something like this:
[warn] ==== sbt-idea-repo: tried
gen-idea plugin for sbt 0.11.2 has not yet been published but 0.11.1-SNAPSHOT version should work as expected :
resolvers += "sbt-idea-repo" at "http://mpeltonen.github.com/maven/"
addSbtPlugin("com.github.mpeltonen" % "sbt-idea" % "0.11.1-SNAPSHOT")
This is documented in the sbt-idea README file here. Specifically:
Add the following lines to ~/.sbt/plugins/build.sbt or PROJECT_DIR/project/plugins.sbt
resolvers += "sbt-idea-repo" at "http://mpeltonen.github.com/maven/"
addSbtPlugin("com.github.mpeltonen" % "sbt-idea" % "0.11.0")
NOTE: If you experience problems with sbt 0.11 installation, see this.

How to define new resolvers?

I have defined several additional resolvers, which are displayed by resolvers:
[clearspan]> show resolvers
[info] List(bt: http://maven.[redacted].com/artifactory/repo/, Local Maven Repository: file:///home/dan/.m2/repository, Oracle: http://download.java.net/maven/2, localMaven: file:///home/dan/.m2/repository)
However, when I run update, it only tries Maven Central. Any idea why?
> update
[info] Updating {file:/home/dan/l/clearspan/}cs-trading-processor...
[warn] [NOT FOUND ] javax.resource#connector;1.0!connector.jar (44ms)
[warn] ==== public: tried
[warn] http://repo1.maven.org/maven2/javax/resource/connector/1.0/connector-1.0.jar
[warn] ::::::::::::::::::::::::::::::::::::::::::::::
[warn] :: FAILED DOWNLOADS ::
[warn] :: ^ see resolution messages for details ^ ::
[warn] ::::::::::::::::::::::::::::::::::::::::::::::
[warn] :: javax.resource#connector;1.0!connector.jar
[warn] ::::::::::::::::::::::::::::::::::::::::::::::
[info]
[warn] :: problems summary ::
[warn] :::: WARNINGS
[warn] [NOT FOUND ] javax.resource#connector;1.0!connector.jar (44ms)
[warn] ==== public: tried
[warn] http://repo1.maven.org/maven2/javax/resource/connector/1.0/connector-1.0.jar
[warn] ::::::::::::::::::::::::::::::::::::::::::::::
[warn] :: FAILED DOWNLOADS ::
[warn] :: ^ see resolution messages for details ^ ::
[warn] ::::::::::::::::::::::::::::::::::::::::::::::
[warn] :: javax.resource#connector;1.0!connector.jar
[warn] ::::::::::::::::::::::::::::::::::::::::::::::
[info]
[info] :: USE VERBOSE OR DEBUG MESSAGE LEVEL FOR MORE DETAILS
[error] {file:/home/dan/l/[redacted]/}[redacted]/*:update: sbt.ResolveException: download failed: javax.resource#connector;1.0!connector.jar
[error] Total time: 2 s, completed Sep 7, 2011 4:51:11 PM
According to the documentation Library dependencies section Resolvers:
Not all packages live on the same server; sbt uses the standard Maven2
repository by default. If your dependency isn’t on one of the default
repositories, you’ll have to add a resolver to help Ivy find it.
You can do it using resolvers setting:
resolvers += name at location
You did it and it didn't work.
In Overriding default resolvers you can read:
resolvers does not contain the default resolvers; only additional ones
added by your build definition.
sbt combines resolvers with some default repositories to form
externalResolvers.
Therefore, to change or remove the default resolvers, you would need
to override externalResolvers instead of resolvers.
In your answer you said using externalResolvers helped, but I doubt that (sorry).
I think the issue was where you defined the new resolvers. It appears that the resolvers come from the top-level project while you may have been using multi-project build and the resolvers setting didn't get picked up.
I think a solution is to define the resolvers setting for the build in general using in ThisBuild or in [projectName] for a given project with projectName replaced with the proper project name the value of the setting should be set to.
I should have been setting externalResolvers instead.

Resources