I'm trying to locally train pytorch_lightning model and log metrics using pytorch_lightning.loggers.MLFlowLogger.
It was working fine until last weekend. Now training crashes with error:
mlflow.exceptions.RestException: BAD_REQUEST: Response: {'Error': {'Code': 'UserError', 'Severity': None, 'Message': 'Metric once published using sync API should always use sync API to publish following metrics', 'MessageFormat': None, 'MessageParameters': None, 'ReferenceCode': None, 'DetailsUri': None, 'Target': None, 'Details': [], 'InnerError': None, 'DebugInfo': None, 'AdditionalInfo': None}, 'Correlation': {'operation': '', 'request': ''}, 'Environment': 'northeurope', 'Location': 'northeurope', 'Time': '2021-07-27T14:06:23.7035319+00:00', 'ComponentName': 'run-history', 'error_code': 'BAD_REQUEST'}
How to fix this issue?
It was a bug in azure machine learning studio. Now it should be fixed.
Solution is to change versions of azure-mlflow and azure-core packages on training machine.
Currently (28.07.2021) logging metrics works with versions:
azureml-core==1.30.0
azureml-mlflow==1.30.0
Currently (02.08.2021) logging metrics works with versions:
azureml-core==1.18.0.post4
azureml-mlflow==1.18.0
Related
I'm attempting to clone a repository as part of my single branch pipeline in Jenkins. I'm using Sonatype Nexus as a 3rd party LFS server, with a repository that has submodules. Has anyone been able to clone successfully without using native Git LFS?
My cloning attempt looks like this at the moment:
stage('Clone Repo') {
steps {
script {
cleanWs()
dir('repo_clone') {
withCredentials([usernameColonPassword(credentialsId: 'git-token', variable: 'GIT_CREDENTIALS')]) {
checkout([
$class: 'GitSCM',
branches: [[name: 'refs/heads/master']],
extensions: [
[$class: 'GitLFSPull'],
[$class: 'SubmoduleOption', disableSubmodules: false, parentCredentials: true, recursiveSubmodules: true, reference: '', trackingSubmodules: false],
[$class: 'CheckoutOption', timeout: 20],
[$class: 'CloneOption', depth: 0, noTags: false, shallow: true, timeout: 120]
],
userRemoteConfigs: [[credentialsId: 'git-token', url: "https://github.com/username/repo.git"]]
])
}
}
}
}
}
However, this doesn't provide a way to inject Nexus credentials for the HTTPS request to my Nexus server. Typically while cloning on my machine, I use git/nexus windows managed credentials to assist with authentication.
I've investigated this for some time, but most of the results I see mention using maven, which I am not using. I haven't been able to find any resources detailing using external LFS servers with Jenkins. Does anyone have suggestions for how to accomplish this?
I've thought about trying to create the windows managed credentials within the pipeline, but haven't had a lot of success with that either. Any help would be greatly appreciated!
I am attempting to deploy new resources and update to existing using my original ARM Templates that I deployed with a few months ago. Unfortunately, the deployment does not get to Azure, as I do not see a deployment entry in my resource group. The error that is presented is local before things start calling into Azure.
I am deploying using the Visual Studio 2017 Arm Template Deployment Context menu.
Here is the error that is output. I get this and a nearly identical one when running the validation command as well.
08:58:22 - VERBOSE: Performing the operation "Creating Deployment" on target "MigrationPlaybook_Prod".
08:58:23 - New-AzureRmResourceGroupDeployment : Multiple error occurred: Forbidden,Forbidden. Please see details.
08:58:23 - At C:\workspaces\Migration Playbook\MigrationPlaybookRegion\ProductionResourceGroup\bin\Debug\staging\ProductionResourc
08:58:23 - eGroup\Deploy-AzureResourceGroup.ps1:108 char:5
08:58:23 - + New-AzureRmResourceGroupDeployment -Name ((Get-ChildItem $Templat ...
08:58:23 - + ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
08:58:23 - + CategoryInfo : CloseError: (:) [New-AzureRmResourceGroupDeployment], CloudException
08:58:23 - + FullyQualifiedErrorId : Microsoft.Azure.Commands.ResourceManager.Cmdlets.Implementation.NewAzureResourceGroupDep
08:58:23 - loymentCmdlet
08:58:23 -
08:58:24 -
08:58:24 - Template deployment returned the following errors:
08:58:24 - Multiple error occurred: Forbidden,Forbidden. Please see details.
Mitigations:
The template involves a KeyVault - I made sure that the ARM Template permission was Enabled
The project is several months old - I generated a new project to make sure that the PowerShell script, that is generated into the project at creation time, did not have any significant change
Account permissions - I have verified that my account permissions on the subscription has not changed in a way to prevent me from adding/modifying resources
While the mitigations address issues that arise when the ARM Template is being deployed the error, and its resulting records, suggest that there is an issue before it gets to Azure.
What could be the issue here and what can I do to remedy this?
I have seen the same error message when using a Key Vault to store username and password secrets but forgot to 'Enable for template deployment' on the Key Vault resource, and then referencing those secrets to pass credentials in the deployment of SQL server. Have you ensured that the Key Vault section in the ARM template has the option enabled where "enabledForTemplateDeployment": true?
Mine looks something like:
{
"type": "Microsoft.KeyVault/vaults",
"apiVersion": "2016-10-01",
"name": "[variables('keyVaultName')]",
"location": "[resourceGroup().location]",
"tags": "[parameters('baseParameters').tagValues]",
"scale": null,
"dependsOn": [],
"properties": {
"sku": {
"family": "A",
"name": "standard"
},
"tenantId": "[subscription().tenantId]",
"accessPolicies": [],
"enabledForDeployment": true,
"enabledForDiskEncryption": false,
"enabledForTemplateDeployment": true
}
},
When enabled, it will look like the following in the portal:
Just to check, I intentionally removed (disabled) the setting and the result looks similar to your error. Using the arguments -Verbose -Debug helped me to see the details.
I know that this isn't a real answer, but I don't have enough reputation yet to just comment. I've seen similar error messages in a subscription that had a limited number of cores and when I was trying to deploy more VMs. However I also am not convinced that this is your problem ...
what I wanted to ask was what happens when you try and deploy the ARM template from the Azure portal using the "Deploy a Custom Template"? This might give you some better hints as to what exactly is going wrong.
I am trying to simple meteor application using mupx. In my scenario, I want to bypass docker setup and all docker options because everything is already installed on my destination server.
how can I do that ? I am using mupx 1.5.2 version.
here is my mup.json file.
{
"servers": [
{
"host": "MY_HOSR",
"username": "USER",
"password": "PASSWORD",
"env": {}
}
],
"setupMongo": false,
"setupNode" : false,
"appName": "blank",
"app": "/home/rut2/meteor_workspace/blank",
"env": {
"PORT": 3000,
"ROOT_URL": "http://MY_HOST:3000"
},
"deployCheckWaitTime": 15,
"enableUploadProgressBar": true
}
The reason I want to disable this is, I never could deploy successfully with this options. It always gives me error while starting meteor/ verifying deployment stage. (I will discuss error if anyone wants to help on that one also.)
I had the lower version of mup (1.0.1, which is without docker options), and it was working perfectly.
Please help me guys. I have been stuck here since 3 days. tried every issues, every forum I found related.
Thank you in advance
You can't use mupx without docker. I had these kind of problems too, some solutions that works for me: upgrade server node version, increase deploy wait time, custom image.
I'm facing strange issue - I deployed my Meteor app with mup ( followed this instruction: https://www.vultr.com/docs/deploy-a-meteor-application-on-ubuntu ). Everything went successfully (see screenshot).
However the content doesn't appear when I try to access it via browser (I type IP-address).
As a server I use apache2 on Ubuntu 15.04, and mup.json as following:
{
// Server authentication info
"servers": [
{
"host": "IP_ADDRESS",
"username": "developer"
//"password": "password"
// or pem file (ssh based authentication)
//"pem": "~/.ssh/id_rsa"
}
],
// Install MongoDB in the server, does not destroy local MongoDB on future setup
"setupMongo": false,
// WARNING: Node.js is required! Only skip if you already have Node.js installed on server.
"setupNode": true,
// WARNING: If nodeVersion omitted will setup 0.10.36 by default. Do not use v, only version number.
"nodeVersion": "0.10.36",
// Install PhantomJS in the server
"setupPhantom": true,
// Show a progress bar during the upload of the bundle to the server.
// Might cause an error in some rare cases if set to true, for instance in Shippable CI
"enableUploadProgressBar": true,
// Application name (No spaces)
"appName": "meteor",
// Location of app (local directory)
"app": "~/app",
// Configure environment
"env": {
"PORT": 80,
"UPSTART_UID": "meteoruser",
"ROOT_URL": "http://IP_ADDRESS",//and I also tried http://localhost
//"MONGO_URL": "",
"METEOR_ENV": "production"
},
// Meteor Up checks if the app comes online just after the deployment
// before mup checks that, it will wait for no. of seconds configured below
"deployCheckWaitTime": 15
}
I would appreciate any help! Thanks in advance!
UPD: I made a better research and found a great tutorial - https://www.phusionpassenger.com/library/walkthroughs/deploy/meteor/ownserver/apache/oss/vivid/deploy_app.html . So, if anyone else curious about how to deploy Meteor app, please have a look :)
I'm trying to build Chrome under windows, I got the chromium trunk using tortoiseSVN and I believe I got everything correctly, but when I run "gclient runhooks" I get the error: "Error: client not configured; see 'gclient config'".
Now, I know that it happens because I don't have a ".gclient" file on the same directory, but I couldn't find .gclient file anywhere in the project. I tried to create .gclient file myself but it says there's a solution missing.
I'm probably missing something, can anyone help me with that? I'm pretty stuck!
Thanks!
gclient config http://src.chromium.org/svn/trunk/src
gclient runhooks
Or make a .gclient file with the following content, which skips the huge amount of webkit layout tests
solutions = [
{ "name" : "src",
"url" : "http://src.chromium.org/svn/trunk/src",
"deps_file" : "DEPS",
"managed" : True,
"custom_deps" : {
"src/third_party/WebKit/LayoutTests": None,
"src/chrome_frame/tools/test/reference_build/chrome": None,
"src/chrome/tools/test/reference_build/chrome_mac": None,
"src/chrome/tools/test/reference_build/chrome_win": None,
"src/chrome/tools/test/reference_build/chrome_linux": None,
},
"safesync_url": "",
},
]
The above solution is out-dated. Running with the SVN repository results in:
Error:
The chromium code repository has migrated completely to git.
Your SVN-based checkout is now obsolete; you need to create a brand-new
git checkout by following these instructions:
http://www.chromium.org/developers/how-tos/get-the-code
Now you need to create a .gclient file like this
solutions = [
{
"managed": False,
"name": "src",
"url": "https://chromium.googlesource.com/chromium/src.git",
"custom_deps": {},
"deps_file": ".DEPS.git",
"safesync_url": "",
},
]
and do:
gclient sync
Chromium does not include a preconfigured .gclient file for the Chromium build and does not automatically handle Visual Studio versioning changes and default Deploy toolkit hints. After you have successfully downloaded the deploy tools and the chromium source code as provided at chromium.org perform the following in the root directory where your deploy_tools and src code is located.
NOTE : If you receive errors try to start a new command prompt session and try again.
set DEPOT_TOOLS_WIN_TOOLCHAIN=0
set GYP_MSVS_VERSION = 2015
gclient config https://chromium.googlesource.com/chromium/src.git
gclient sync
gclient runhooks
cd src
ninja -C out\Debug chrome
The build will take some time gclient runhooks should generate the build folder.