I have set up my mesos cluser correctly with one master and two slaves. What I am trying to do is use the mesos-execute framework to run jar files on the cluster. I can use it to run simple commands like:
mesos-execute --master=mesosr:5050 --name="simple-test" --command=echo "hello"
Which will run as expected. However if I try to replace that echo "hello" command with something like "java -jar helloWorld.jar" it won't work.
I managed to identify the problem, but I don't know how to fix it. The issue is that the command doesn't run from the home directory, it runs from something similar to this
/var/lib/mesos/slaves/3f5439b1-7fab-45d6-876e-7e75b7c15fc9-S0/frameworks/3f5439b1-7fab-45d6-876e-7e75b7c15fc9-0043/executors/java-test/runs/7c20baff-080f-48ee-95fc-3662c388744b
I got that path by running "pwd" as a command on mesos-execute.
Now, my question is how do I get out from there? cd doesn't work.
Is there any way for me to get to the home folder or to a special folder where I can put my jars to make them accessible to mesos-execute?
The use case for this application is that there will be a lot of small jar files that will have to be run on the cluster. They don't have to stay alive, so I am not using anything like Marathon for these jars.
Thank you.
From mesos-execute -h
--task_group=VALUE The value could be a JSON-formatted string of TaskGroupInfo or a file path containing the JSON-formatted TaskGroupInfo. Path must be of the form file:///path/to/file or /path/to/file. See the TaskGroupInfo message in mesos.proto for the expected format. NOTE: agent_id need not to be set.
Example:
{
"tasks":
[
{
"name": "Name of the task",
"task_id": {"value" : "Id of the task"},
"agent_id": {"value" : ""},
"resources": [{
"name": "cpus",
"type": "SCALAR",
"scalar": {
"value": 0.1
}
},
{
"name": "mem",
"type": "SCALAR",
"scalar": {
"value": 32
}
}],
"command": {
"value": "sleep 1000"
}
}
]
}
What interest you most is command part. There you can define your task with all files it need to download to run correctly. All possible configuration options for command are specified in CommandInfo.
Ok, so I figured out how to do it.
I was going about it all wrong, perhaps not the best idea to tackle a new thing at the end of a workday.
What I was trying to do was change directory to the home directory as part of the mesos-execute command. This is not allowed. The way to run a jar that is located in the home directory is to specify the path of the jar in the java -jar command. So the final command, that works, looks like this:
mesos-execute --master=mesosr:5050 --name="simple-test" --command="java -jar /home/user/jarFile.jar"
This works, and the jar is executed on the cluster.
Related
I'm working on a large angular / .NET Core project and have to type e.g. dotnet run /path/to/subproject in the terminal often.
Can I use VSCode to store/manage these common commands? I've been through the vscode docs on launch.json and tasks.json but cannot find a good answer.
Thanks!
Yes, you have a couple of options.
(1) Set up a command to just rerun the last command - see Make a keybinding to run previous or last shell commands
{
"key": "alt+x", // choose your keybinding
"command": "workbench.action.terminal.sendSequence",
"args": { "text": "\u001b[A\u000d" }
},
or (2) just put your frequently-used command into a keybinding ala:
{
"key": "alt+x", // choose your keybinding
"command": "workbench.action.terminal.sendSequence",
"args": { "text": "dotnet run /path/to/subproject\u000d" },
// "when": "terminalFocus"
},
The \u000d is a return so the command runs immediately. I find it easiest to not have the when clause so I can run it from anywhere - editorFocus or terminalFocus, etc.
These go into your keybindings.json.
You can use variables where you have /path/to/subproject. See task - variable substitution and available variables which may help with your path.
I cannot get VS Code to build an empty class library while dotnet core can quite happily.
In PowerShell I create a folder called CoreTesting, navigate into it and launch VS Code with code.
I hit CTRL + ` to enter the terminal and navigate into the solution's folder.
I then enter dotnet new classlib --name Common, see the new folder Common and enter dotnet build .\Common\ to build the class library. All is well.
I add the Common folder to VS Code and hit CTRL + SHIFT + B and see No build task to run found. Configure Build Task..., so I hit return and see Create tasks.json file from template, so I hit return again and see:
MSBuild
maven
.NET Core
Others
So I select .NET Core and see that a .vscode folder is created containing tasks.json. This file contains:
{
// See https://go.microsoft.com/fwlink/?LinkId=733558
// for the documentation about the tasks.json format
"version": "2.0.0",
"tasks": [
{
"label": "build",
"command": "dotnet build",
"type": "shell",
"group": "build",
"presentation": {
"reveal": "silent"
},
"problemMatcher": "$msCompile"
}
]
}
I hit CTRL + SHIFT + B again and see the option build Common, so I hit return and see this:
> Executing task in folder Common: dotnet build <
Microsoft (R) Build Engine version 16.0.225-preview+g5ebeba52a1 for .NET Core
Copyright (C) Microsoft Corporation. All rights reserved.
MSBUILD : error MSB1003: Specify a project or solution file. The current working directory does not contain a project or solution file.
The terminal process terminated with exit code: 1
Terminal will be reused by tasks, press any key to close it.
The structure I can see is this:
Common\
.vscode\
tasks.json
bin\
obj\
Class1.cs
Common.csproj
What have I done wrong?
I was able to reproduce your problem on v1.41.1 for Windows. In doing so it created this tasks.json which is similar to yours...
{
// See https://go.microsoft.com/fwlink/?LinkId=733558
// for the documentation about the tasks.json format
"version": "2.0.0",
"tasks": [
{
"label": "build",
"command": "dotnet",
"type": "shell",
"args": [
"build",
// Ask dotnet build to generate full paths for file names.
"/property:GenerateFullPaths=true",
// Do not generate summary otherwise it leads to duplicate errors in Problems panel
"/consoleloggerparameters:NoSummary"
],
"group": "build",
"presentation": {
"reveal": "silent"
},
"problemMatcher": "$msCompile"
}
]
}
When you invoke a task, it defaults to using the workspace folder (CoreTesting) path as the working directory. However, the project file is a directory beneath in Common, hence the error The current working directory does not contain a project or solution file.
A quick fix for this is to simply open the directory with the project file as the workspace folder (i.e. File → Open Folder... → Select the Common directory).
Alternatively, if that solution is undesirable then with CoreTesting opened as the workspace folder you can configure the task to execute with a different working directory. To do this, set the task's options.cwd property...
{
/* snip */
"tasks": [
{
/* snip */
"problemMatcher": "$msCompile",
"options": {
"cwd": "${workspaceFolder}/Common"
}
}
]
}
I found this property in the Schema for tasks.json, and it's also mentioned in the Custom tasks section of the Tasks documentation. After making either change above the library builds successfully for me.
I am checking if I can use kairosdb for my project. I was checking out the REST api's and I have a use case where I need to save both my device state and status (state tells if device is on or off and status tells if my device is occupied or empty)
kairosdb version: 1.1.1
I came across this link https://kairosdb.github.io/docs/build/html/restapi/AddDataPoints.html
but when I try to post data from REST client I am getting the error 400 BAD Request error. The error is
{"errors":["Unregistered data point type 'complex-number'"]}
My request I am posting is ,
{
"name": "device_data",
"type": "complex-number",
"datapoints": [
[
1470897496,
{
"state": 0,
"status": "empty"
}
]
],
"tags": {
"device_id": "abc123"
}
}
In tried doing the same in Java as specified in https://kairosdb.github.io/docs/build/html/kairosdevelopment/CustomData.html
I get the same error i
Please let me know how to use complex-numbers or custom data types from REST
Recently, I figured out how to use this.
Using the example from the official document of KairosDB.
create 2 files called ComplexDataPoint.java and ComplexDataPointFactory.java and then paste the code provided by the tutorial on the doc: https://kairosdb.github.io/docs/build/html/kairosdevelopment/CustomData.html#example-for-creating-custom-types
download the KairosDB source, then extract the .zip file.
paste the 2 files in /KAIROSDB_DOWNLOADED_SOURCE/src/main/java/org/kairosdb/core/datapoints/
configure the CoreModule.java at /KAIROSDB_DOWNLOADED_SOURCE/src/main/java/org/kairosdb/core/, add the following line in the function protected void configure():
bind(ComplexDataPointFactory.class).in(Singleton.class);
open terminal, cd to KAIROSDB_DOWNLOADED_SOURCE/, then follow the instruction in the file how_to_build.txt
when complete, it will create a folder called build, the compiled kairosdb jar file is located in KAIROSDB_DOWNLOADED_SOURCE/build/jar
in your kairosdb installation folder, backup the kairosdb-X.X.X.jar file in YOUR_KAIROSDB_INSTALLATION/lib
mv kairosdb-X.X.X.jar kairosdb-X.X.X.jar.backup
mv the newly compiled jar file to YOUR_KAIROSDB_INSTALLATION/lib
modify the configuration file by adding the following line:
kairosdb.datapoints.factory.complex=org.kairosdb.core.datapoints.ComplexDataPointFactory
restart your kairosdb
For your query, since the registered name is kairosdb.datapoints.factory.complex, replace complex-number with complex in your query string.
Hope this will help you! I am now having a problem to plot the complex data. I am still figuring out...
I keep getting "Warning: Spawn ENOENT" problem when I am running grunt-open.
My setup
I have all my project files on Google Drive.
I am doing my development on the files directly on them (using Google Drive desktop).
I just want to open the index.html file when I run grunt. Just that simple.
Part of my Gruntfile.js
open: {
all: {
path : 'index.html'
}
},
But I couldn't get it working? What am I missing?
I have figured out.
You have to change your system environment variable to include the following if on PC.
%SystemRoot%\system32;
I had similar problem for grunt-run task:
run: {
jasmine: {
cmd: 'jasmine-node.cmd',
args: ['--autotest', '--test-dir', 'test']
}
},
You have to specify jasmine-node.cmd as cmd - not just jasmine-node which is enough when running from command line.
I'm trying to build Chrome under windows, I got the chromium trunk using tortoiseSVN and I believe I got everything correctly, but when I run "gclient runhooks" I get the error: "Error: client not configured; see 'gclient config'".
Now, I know that it happens because I don't have a ".gclient" file on the same directory, but I couldn't find .gclient file anywhere in the project. I tried to create .gclient file myself but it says there's a solution missing.
I'm probably missing something, can anyone help me with that? I'm pretty stuck!
Thanks!
gclient config http://src.chromium.org/svn/trunk/src
gclient runhooks
Or make a .gclient file with the following content, which skips the huge amount of webkit layout tests
solutions = [
{ "name" : "src",
"url" : "http://src.chromium.org/svn/trunk/src",
"deps_file" : "DEPS",
"managed" : True,
"custom_deps" : {
"src/third_party/WebKit/LayoutTests": None,
"src/chrome_frame/tools/test/reference_build/chrome": None,
"src/chrome/tools/test/reference_build/chrome_mac": None,
"src/chrome/tools/test/reference_build/chrome_win": None,
"src/chrome/tools/test/reference_build/chrome_linux": None,
},
"safesync_url": "",
},
]
The above solution is out-dated. Running with the SVN repository results in:
Error:
The chromium code repository has migrated completely to git.
Your SVN-based checkout is now obsolete; you need to create a brand-new
git checkout by following these instructions:
http://www.chromium.org/developers/how-tos/get-the-code
Now you need to create a .gclient file like this
solutions = [
{
"managed": False,
"name": "src",
"url": "https://chromium.googlesource.com/chromium/src.git",
"custom_deps": {},
"deps_file": ".DEPS.git",
"safesync_url": "",
},
]
and do:
gclient sync
Chromium does not include a preconfigured .gclient file for the Chromium build and does not automatically handle Visual Studio versioning changes and default Deploy toolkit hints. After you have successfully downloaded the deploy tools and the chromium source code as provided at chromium.org perform the following in the root directory where your deploy_tools and src code is located.
NOTE : If you receive errors try to start a new command prompt session and try again.
set DEPOT_TOOLS_WIN_TOOLCHAIN=0
set GYP_MSVS_VERSION = 2015
gclient config https://chromium.googlesource.com/chromium/src.git
gclient sync
gclient runhooks
cd src
ninja -C out\Debug chrome
The build will take some time gclient runhooks should generate the build folder.