HTTP 400 - Unable to parse remote repository npm metadata - artifactory

We have 2 remote NPM registries inside of a virtual repository. One of them is the NPM Registry, the other one is from a software provider. When I add the second repository to the virtual repository, I am getting HTTP 400 messages at random.
For example: if I want to install a package from the npm-registry, I see through the logs that Artifactory is trying to get the package from the other repository (which does not have the package) and tries to parse the response as json. The response from the other repository gives back a html file though which results in the following error message:
2017-02-23 09:39:05,424 [http-nio-8080-exec-7112] [ERROR]
(o.a.a.n.r.NpmRemoteRepoHandler:362) - Error while parsing the response of a remote npm
JSON query on 'https://repository.domain.com/api/npm/public/file-loader':
Unexpected character ('<' (code 60)): expected a valid value (number, String, array, object,
'true', 'false' or 'null')
at [Source:org.artifactory.storage.db.binstore.service.UsageTrackingBinaryProvider$ReaderTrackingStream#7360bc6c; line: 1, column: 2]
As you can see, Artifactory is trying to get the package from the other repository. The JSON response of our artifactory, when I try to get the package manually is :
{
"errors" : [ {
"status" : 400,
"message" : "Unable to parse remote repository npm metadata."
} ]
}
Any help would be greatly appreciated, since this makes the NPM Registry completely useless as some requests are returning this HTTP 400 error.
fyi: We are using Artifactory Pro 4.5.1

There are 2 things you should do to avoid this behavior
Configure the virtual repository resolution order so the NPM registry is approached before the software provider registry. The resolution order is controlled by the order they are presented in the Selected Repositories list.
Use include/exclude patterns to control which packages are resolved from the software provider registry. Assuming there is a way to identify the packages which should be resolved from software provider you can define patterns which will limit this registry only for the resolution of certain packages.
Another thing to check is whether the software provider remote repository configured properly. Normally it should not return an HTML response for an API call.

Related

How do I use JARs stored in Artifactory in spark submits?

I am trying to configure the spark-submits to use JARs that are stored in artifactory.
I've tried a few ways to do this
Attempt 1: Changing the --jars parameter to point to the https end point
Result 1: 401 Error. Credentials are being passed like so: https://username:password#jfrog-endpoint. The link was tested using wget and it authenticates and downloads the JAR fine.
Error
Attempt 2: Using a combination of --packages --repositories
Result 2: URL doesn't resolve to the right location of the jar
Attempt 3:Using combination of --packages and modified ivysettings.xml (containing repo and artifact pattern)
ivy settings
Result 3: URL resolves correctly but still results in "Not Found"
After some research it looks like the error might say "Not Found" and the it looks like it has "tried" the repo, it could still very well be a 401 error.
Error
Any ideas would be helpful! Links i've explored:
Can i do spark-submit application jar directly from maven/jfrog artifactory
spark resolve external packages behind corporate artifactory
How to pass jar file (from Artifactory) in dcos spark run?
https://godatadriven.com/blog/spark-packages-from-a-password-protected-repository/
https://spark.apache.org/docs/latest/submitting-applications.html#advanced-dependency-management
You can use https://username:password#jfrog.com/rep, however you need to specify port.
https://username:password#jfrog.com:443/repo
I discovered this using Artifactory's "set me up" tool for package type Ivy. If you look in the resolver url it specifies the port.

Internal error: Unexpected GitHub remote configuration: 'theirs'

I am currently working on a collaborative R-project, and after I forked the repository, I made a couple of changes and wanted to push them. However, when I tried to push it in the Rstudio, it raised the following error.
Internal error: Unexpected GitHub remote configuration: 'theirs'
To push the commits, I've used the following code :
pr_push()
Here are some context:
git remote -v
origin https://github.com/PPBDS/primer.git (fetch)
origin https://github.com/PPBDS/primer.git (push)
The code I used to fork the original repository.
library(usethis)
create_from_github("PPBDS/primer",
fork = TRUE,
destdir = "/mydest/",
protocol = "https")
The branch I've created to work on:
> pr_init(branch ='Python.v')
√ Pulling changes from 'origin/master'
√ Creating and switching to local branch 'Python.v'
I don't know how to fix this. Can anyone help me?
Your git remotes shows that the repo was cloned over https (no push allowed over https) and no fork was configured.
If I'm interpreting you correctly, the output should be:
origin git#github.com:AtillaColak/primer.git (fetch)
origin git#github.com:AtillaColak/primer.git (push)
upstream https://github.com/PPBDS/primer.git (fetch)
upstream https://github.com/PPBDS/primer.git (push)
Perhaps this will work? (Untested) (in a shell, not in R)
git remote rename origin upstream
git remote add origin git#github.com:AtillaColak/primer.git
If those commands do not work, the .git/config file should look like this:
[remote "origin"]
url = git#github.com:AtillaColak/primer.git
fetch = +refs/heads/*:refs/remotes/origin/*
[remote "upstream"]
url = https://github.com/PPBDS/primer.git
fetch = +refs/heads/*:refs/remotes/upstream/*
I don't encourage editing the .git/config file manually if you can avoid it, but this (if nothing else) may provide verification of other attempts.

Installing a package from private GitLab server on Windows

I am struggling with installing a package from a GitLab repository on a Windows computer.
I found different hints but still have problems to install my package from GitLab. First of all, I generated a public and private key with puttygen.exe. The files need to be changed afterwards, I had to remove comments and stuff so they look like my the file on my Unix system. So now, both public and private key files have just a single line.
I tried to install my package via devtools::install_git which takes very long and I get the error message
Error: Failed to install 'unknown package' from Git:
Error in 'git2r_remote_ls': Failed to authenticate SSH session: Unable to send userauth-publickey request
And with devtools::install_gitlab I get a different error message and I somehow have the feeling, the link which gets generated doesn't fit to my GitLab server.
Error: Failed to install 'unknown package' from GitLab:
cannot open URL 'https://gitlab.rlp.net/api/v4/projects/madejung%2FMQqueue.git/repository/files/DESCRIPTION/raw?ref=master'
My complete code to test at the moment is
creds <- git2r::cred_ssh_key(publickey="~/.ssh/id_rsa_gitlab.pub",
privatekey="~/.ssh/id_rsa_gitlab")
devtools::install_git(
url='git#gitlab.rlp.net:madejung/MQqueue.git',
quiet=FALSE,
credentials=creds)
devtools::install_gitlab(
repo='madejung/MQqueue.git',
host='gitlab.rlp.net',
quiet=FALSE,
credentials=creds
)
My id_rsa_gitlab.pub file looks like this and is just a single line:
ssh-rsa AAAA....fiwbw== rsa-key-20200121
The id_rsa_gitlab file has just the code:
AAABA.....3WNSIAGE=
Update
On my Mac system it works as expected after installing the libssh2 library via homebrew and and recompiling git2r with install.packages("git2r", type = "source").
So the working code on my machine is:
creds <- git2r::cred_ssh_key(publickey="~/.ssh/id_rsa_gitlab.rlp.net.pub",
privatekey="~/.ssh/id_rsa_gitlab.rlp.net")
devtools::install_git(
url='git#gitlab.rlp.net:madejung/MQqueue.git',
quiet=FALSE,
credentials=creds
)
For some strange reason, the devtools::install_git call needs about a minute to fail in the end. I have no idea where the problem here is.
After struggling for almost a day, I found a solution I can live with...
I first created a PAT (Personal Access Token) in my gitlab account and granted full API access. For some reason the read_only access didn't worked and I am now tired to figure out what the problem is.
After this I had still problems to install my package and for some reason, the wininet setting for downloading doesn't work.
I used the command capabilities("libcurl") to check if libcurl is available on my windows, which was and tried to overwrite wininet to libcurl by using method='libcurl' in the install function. Somehow, this was not enough so I overwrote the options variable download.file.method directly.
options("download.file.method"='libcurl')
devtools::install_gitlab(
repo='madejung/MQqueue',
auth_token='Ho...SOMETHING...xugzb',
host='gitlab.rlp.net',
quiet=FALSE, force=TRUE
)

Push replication on pypi repos

I've been trying to setup push replication in pypi repos, however that does not seem to work. If I setup the remote to be a generic repo, then the files get pushed. Otherwise, if it's a pypi repo, then I get the following error:
Error while deploying item 'test:gcovr/3.2/gcovr-3.2-py2.7.egg on Url:http://artifactory/artifactory/api/pypi/testpypi ': Method Not Allowed [405]
When trying to replicate the test repo to testpypi. Am I doing something wrong here? I've tried suffixing with simple or even just a /. Nothing seems to work.
You should use the following URL format when configuring the replication:
http://artifactory/artifactory/testpypi

After upgrade attempting to get artifact results in "Could not process download request: Binary provider has no content for"

I recently upgraded our artifactory repository from 2.6.5 to the current version 5.4.6.
However, something seems to have gone wrong in the process. There are some artifacts that throw a HTTP 500 error when attempting to access them. Here is an example using wget:
wget http://xyz.server.com:8081/artifactory/gradle/org/jfrog/buildinfo/build-info-extractor-gradle/2.0.12/build-info-extractor-gradle-2.0.12.pom
--2017-09-12 12:17:13--
http://xyz.server.com:8081/artifactory/gradle/org/jfrog/buildinfo/build-info-extractor-gradle/2.0.12/build-info-extractor-gradle-2.0.12.pom
Resolving xyz.server.com (xyz.server.com)... 10.125.1.28
Connecting to xyz.server.com (xyz.server.com)|10.125.1.28|:8081... connected.
HTTP request sent, awaiting response... 500 Internal Server Error
2017-09-12 12:17:13 ERROR 500: Internal Server Error.
I verified this by going to the artifactory site, browsing to the object in question, and trying to download it. The result was the following:
{
"errors" : [ {
"status" : 500,
"message" : "Could not process download request: Binary provider has no content for 'e52a9a9a58d6829b7b13dd841af4b027c88bb028'"
} ]
}
The problem seems to be in the final step of the upgrade process, upgrading from 3.9.5 to 5.4.6. The wget command above works on 3.9.5, but not on the 5.4.6 instance.
I found a reference to a "Zap Cache" function in older documentation and thought it might fix things, but I don't seem to be able to find that function in the current site.
Is anyone able to point me to: a way to fix this issue, or what I need to do/look for in the upgrade process in order to prevent it from occurring?
As a further data point, we're using an Oracle database for the full file store, if that matters in any way (using the tag: <chain template="full-db"> in binarystore.xml)
Thanks in advance....

Resources