I want to deploy and automatically run a script on a remote machine.
However, I don't know how I can issue a remote command.
I've tried several things and plugins but none of that stuff seems to work.
Is there a way to do it?
val deployAndRunTask = TaskKey[Unit]("deploy-run", "Deploy and run application.")
deployAndRunTask := {
// Deploy .jar file
val _ = deployTask.value
println("Running the script ..")
}
Here is a working example, which you can save in test.sbt in the top level directory of your SBT project. The shellRun method runs an arbitrary command. I use shellRun to run ssh, connect to the local machine, and list the files in my home directory.
import scala.sys.process.Process
val deployAndRunTask = TaskKey[Unit]("deploy-and-run-task", "Short example")
deployAndRunTask := {
def shellRun(command: String*) = Process(command.toSeq).!!.trim
val result = shellRun("/usr/bin/ssh", "localhost", "ls")
println(result)
}
Run it by typing:
sbt deployAndRunTask
Related
I'm trying to implement custom XCOM backend.
Those are the steps I did:
Created "include" directory at the main Airflow dir (AIRFLOW_HOME).
Created these "custom_xcom_backend.py" file inside:
from typing import Any
from airflow.models.xcom import BaseXCom
import pandas as pd
class CustomXComBackend(BaseXCom):
#staticmethod
def serialize_value(value: Any):
if isinstance(value, pd.DataFrame):
value = value.to_json(orient='records')
return BaseXCom.serialize_value(value)
#staticmethod
def deserialize_value(result) -> Any:
result = BaseXCom.deserialize_value(result)
result = df = pd.read_json(result)
return result
Set at config file:
xcom_backend = include.custom_xcom_backend.CustomXComBackend
When I restarted webserver I got:
airflow.exceptions.AirflowConfigException: The object could not be loaded. Please check "xcom_backend" key in "core" section. Current value: "include.cust...
My guess is that it not recognizing the "include" folder
But how can I fix it?
*Note: There is no docker. It is installed on a Ubuntu machine.
Thanks!
So I solved it:
Put custom_xcom_backend.py into the plugins directory
set at config file:
xcom_backend = custom_xcom_backend.CustomXComBackend
Restart all airflow related services
*Note: Do not store DataFrames that way (bad practice).
Sources I used:
https://www.youtube.com/watch?v=iI0ymwOij88
I'm using the sbt run command to run my project. My project uses the Logback logging mechanism and if I would like to enable logging, then I have to use the following command:
sbt -Dlogback.configurationFile=/path/to/log/file/app-logger.xml run
Is there a way that I could set this programmatically? I mean I would like to just say
sbt run
and it picks up automagically the app-logger.xml by itself via the application.
This is how I do it!
def loadLogger() = Option(System.getProperty("logback.configurationFile")) match {
case Some(logXml) =>
logger.info(s"using logger $logXml")
case None =>
val path = s"${System.getProperty("user.dir")}/conf/app-logger.xml"
System.setProperty("logback.configurationFile", path)
logger.info(s"using logger $path")
}
I want to write a grunt task, which will start a server, and run some tests based on that server:
grunt.initConfig({
shell: {
sbtRun: {
options: {
stdout: true
},
command: './sbt run'
}
}
});
grunt.loadNpmTasks('grunt-shell');
grunt.registerTask('run-test-only', ...);
grunt.registerTask('start-server-and-test', ['shell:sbtRun', 'run-test-only']);
But the problem is, the task ./sbt run is not running in daemon. When I run:
grunt start-server-and-test
It will blocking in the shell:sbtRun task forever.
Is there any way to start it in a child process? So the second task run-test-only will be called, and the server will be destroyed automatically after testing?
I was facing a similar issue. I wanted to connect to a database and store the result of a query in a .csv. I created a separate file to do all that work and I was successful in running that file through the command line using:
node filename.js
I used grunt-shell and grunt-exec to run this command but everytime I saw the same error:
const child = require('child_process').exec;
Then, I found another grunt plugin i.e grunt-execute which was able to create a child process and the task was completed successfully. You could find more about grunt-execute here, https://www.npmjs.com/package/grunt-execute.
Hope this helps.
$ wget --quiet http://download.qt-project.org/official_releases/qt/5.2/5.2.1/qt-opensource-windows-x86-msvc2012_64_opengl-5.2.1.exe
$
As seen above, I first downloaded the Qt package for Visual Studio in a Cygwin Bash shell.
A sidenote: The Qt library packaged within Cygwin is not useful for me because I need to use the Visual Studio C++ compiler.
First I set the correct permissions on the file
$ chmod 755 qt-opensource-windows-x86-msvc2012_64_opengl-5.2.1.exe
If I start it like this
$ ./qt-opensource-windows-x86-msvc2012_64_opengl-5.2.1.exe
a graphical window (GUI) is shown but that is not what I want as I would later like to have the installation procedure written into a Bash script that I could run in Cygwin.
If I add the option --help, like this
$ ./qt-opensource-windows-x86-msvc2012_64_opengl-5.2.1.exe --help
a new terminal window is opened with the following text
Usage: SDKMaintenanceTool [OPTIONS]
User:
--help Show commandline usage
--version Show current version
--checkupdates Check for updates and return an XML file describing
the available updates
--updater Start in updater mode.
--manage-packages Start in packagemanager mode.
--proxy Set system proxy on Win and Mac.
This option has no effect on Linux.
--verbose Show debug output on the console
--create-offline-repository Offline installer only: Create a local repository inside the
installation directory based on the offline
installer's content.
Developer:
--runoperation [OPERATION] [arguments...] Perform an operation with a list of arguments
--undooperation [OPERATION] [arguments...] Undo an operation with a list of arguments
--script [scriptName] Execute a script
--no-force-installations Enable deselection of forced components
--addRepository [URI] Add a local or remote repo to the list of user defined repos.
--addTempRepository [URI] Add a local or remote repo to the list of temporary available
repos.
--setTempRepository [URI] Set a local or remote repo as tmp repo, it is the only one
used during fetch.
Note: URI must be prefixed with the protocol, i.e. file:///
http:// or ftp://. It can consist of multiple
addresses separated by comma only.
--show-virtual-components Show virtual components in package manager and updater
--binarydatafile [binary_data_file] Use the binary data of another installer or maintenance tool.
--update-installerbase [new_installerbase] Patch a full installer with a new installer base
--dump-binary-data -i [PATH] -o [PATH] Dumps the binary content into specified output path (offline
installer only).
Input path pointing to binary data file, if omitted
the current application is used as input.
I don't know how to continue from here. Do you know how I could install the Qt 5.2.1 library (for Visual Studio) from the Bash shell in Cygwin?
Update: The advantage of writing the build script for a Cygwin environment is that commands like git, wget, and scp are available. This Stackoverflow answer describes how to invoke the MSVC compiler from a Cygwin bash script. Note, that the Qt application I'm building is not going to have any dependency on Cygwin.
I didn't test with Cygwin but I successfully installed Qt5.5 using a script. To do so, you must use the --script line of the normal installer.
.\qt-opensource-windows-x86-msvc2013_64-5.5.1.exe --script .\qt-installer-noninteractive.qs
Here's an example of qt-installer-noninteractive.qs file I used in the command above
function Controller() {
installer.autoRejectMessageBoxes();
installer.installationFinished.connect(function() {
gui.clickButton(buttons.NextButton);
})
}
Controller.prototype.WelcomePageCallback = function() {
gui.clickButton(buttons.NextButton);
}
Controller.prototype.CredentialsPageCallback = function() {
gui.clickButton(buttons.NextButton);
}
Controller.prototype.IntroductionPageCallback = function() {
gui.clickButton(buttons.NextButton);
}
Controller.prototype.TargetDirectoryPageCallback = function() {
gui.currentPageWidget().TargetDirectoryLineEdit.setText("C:/Qt/Qt5.5.1");
gui.clickButton(buttons.NextButton);
}
Controller.prototype.ComponentSelectionPageCallback = function() {
var widget = gui.currentPageWidget();
widget.deselectAll();
widget.selectComponent("qt.55.win64_msvc2013_64");
// widget.selectComponent("qt.55.qt3d");
// widget.selectComponent("qt.55.qtcanvas3d");
// widget.selectComponent("qt.55.qtquick1");
// widget.selectComponent("qt.55.qtscript");
// widget.selectComponent("qt.55.qtwebengine");
// widget.selectComponent("qt.55.qtquickcontrols");
// widget.selectComponent("qt.55.qtlocation");
gui.clickButton(buttons.NextButton);
}
Controller.prototype.LicenseAgreementPageCallback = function() {
gui.currentPageWidget().AcceptLicenseRadioButton.setChecked(true);
gui.clickButton(buttons.NextButton);
}
Controller.prototype.StartMenuDirectoryPageCallback = function() {
gui.clickButton(buttons.NextButton);
}
Controller.prototype.ReadyForInstallationPageCallback = function()
{
gui.clickButton(buttons.NextButton);
}
Controller.prototype.FinishedPageCallback = function() {
var checkBoxForm = gui.currentPageWidget().LaunchQtCreatorCheckBoxForm
if (checkBoxForm && checkBoxForm.launchQtCreatorCheckBox) {
checkBoxForm.launchQtCreatorCheckBox.checked = false;
}
gui.clickButton(buttons.FinishButton);
}
The tricky part was to found the id of the components! I was able to found the right id qt.55.win64_msvc2013_64 by adding the flag --verbose and installing it normally with the UI and stopping at the last page; all the ids that you selected for installation are there.
There is slightly more information in this answer if you need more details.
EDIT (29-11-2017): For installer 3.0.2-online, the "Next" button in the "Welcome" page is disabled for 1 second so you must add a delay
gui.clickButton(buttons.NextButton, 3000);
EDIT (10-11-2019): See Joshua Wade's answer for more traps and pitfalls, like the "User Data Collection" form and "Archive" and "Latest releases" checkboxes.
I am trying to use a JVM option in a forked test, which has been set externally to SBT prior to its launch. I'm also setting additional JVM options like so:
javaOptions in ThisBuild ++= List("-Xmx3072m")
to my understanding, based on the SBT documentation the JVM options provided to the SBT process should be available to the forked process:
By default, a forked process uses the same Java and Scala versions being used for the build and the working directory and JVM options of the current process.
However, I can't seem to retrieve those "external" JVM options in the forked tests, i.e. System.getProperty("foo") will always return null. Given that I am trying to pass along a password, I can't set it directly in the build file. My questions therefore are:
is there an SBT task / key to access the JVM options passed to the JVM in which SBT is running? That way I would attempt to add the key to the javaOptions
is there any other means by which to pass external Java Options to a forked test?
You may control your options with testGrouping. Below copy'n'paste from one of my projects. It properly handles hierarchical projects and root project without tests too. Options are merged from javaOptions in run and test.options file. This allow modify arguments without project reloading. That project has load time more then minute. So I use test.options to fast switch between production and debug mode with -Xrunjdwp:transport=dt_...
testGrouping in Test <<= (definedTests in Test, javaOptions in run, baseDirectory in LocalRootProject) map { (tests, javaOptions, baseDirectory) ⇒
if (tests.nonEmpty) {
val testOptionsFile = baseDirectory / "test.options"
val externalOptions = if (testOptionsFile.exists()) {
val source = scala.io.Source.fromFile(testOptionsFile)
val options = source.getLines().toIndexedSeq
source.close()
options
} else Nil
tests map { test ⇒
new Tests.Group(
name = test.name,
tests = Seq(test),
// runPolicy = Tests.InProcess)
runPolicy = Tests.SubProcess(javaOptions = javaOptions ++ externalOptions))
}
} else {
Seq(new Tests.Group(
name = "Empty",
tests = Seq(),
runPolicy = Tests.InProcess))
}
},