My question is: how can I make node available to a shell script on Cloud functions for Firebase when I use an npm package like xml2json?
Context:
I am moving a piece of software from short-living Docker containers to Cloud functions for Firebase. The software processes large XML files and converts parts to JSON.
Due to the memory consumption, I use xmllint and xml2json via the shell.
var cmd = './xmllint --xpath "//OrderData" ' + this.filename + ' | ' + __dirname + '/node_modules/xml2json/bin/xml2json';
exec(cmd, function(error, stdout, stderr) { ... }
Xmllint works as expected (distributed in the npm package that I added to package.json in the functions directory)
"#niekoost/convertini": "file:niekoost-convertini-1.0.1.tgz"
But the pipe through xml2json produces the problem:
The first line of xml2json is causing my issue
#!/usr/bin/env node
This leads to an error in the Cloud Functions log in the Firebase console
Error: Command failed: ./xmllint --xpath "//OrderData"
/tmp/kdjf6kv9hku86xw4h392j4i.xml | /user_code/node_modules/xml2json/bin/xml2json
/usr/bin/env: node: No such file or directory
Where is node located on Cloud functions for Firebase? Can I make it available to /usr/bin/env? Or should I fork the xml2json npm package and change that code.
After some additional searches I have found a solution for my situation, without having to change the npm package that is provided for xml2json.
Since xml2json is used in a command that is being executed by 'exec', we only need to make the path to the Node binary available in the environment of that exec. This path can be derived from 'process.execPath':
var exec = require('child_process').exec;
var process = require('process');
// get path where the node binary is located from within the code that
// is being executed by the node runtime provided by Firebase
var nodePath = process.execPath;
nodePath = nodePath.split('/');
nodePath = nodePath.splice(0, nodePath.length-1).join('/');
// add path to the environment
process.env.PATH += ':' + nodePath;
var cmd = './xmllint --xpath "//OrderData" /xyz.xml | /user_code/node_modules/xml2json/bin/xml2json';
// call the command and provide the extended env
exec(cmd, {env: process.env}, function(error, stdout, stderr) {
...
});
Related
import { exec } from "https://deno.land/x/exec/mod.ts";
await exec(`git clone https://github.com/vuejs/vue.git`)
when i use git clone https://github.com/vuejs/vue.git in .sh file, It has message in terminal,
but don't have in deno
First, I think it is important to echo what jsejcksn commented:
The exec module is not related to Deno. All modules at https://deno.land/x/... are third-party code. For working with your shell, see Creating a subprocess in the manual.
Deno's way of doing this without a 3rd party library is to use Deno.run.
With that said, if you take a look at exec's README you'll find documentation for what you're looking for under Capture the output of an external command:
Sometimes you need to capture the output of a command. For example, I do this to get git log checksums:
import { exec } from "https://deno.land/x/exec/mod.ts";
let response = await exec('git log -1 --format=%H', {output: OutputMode.Capture});
If you look at exec's code you'll find it uses Deno.run under the hoods. If you like exec you can use it but you might find you like using Deno.run directly instead.
Looking at https://deno.land/manual/examples/file_system_events
which is the code below.
const watcher = Deno.watchFs("./src");
for await (const event of watcher) {
console.log(">>>> event", event);
// { kind: "create", paths: [ "/foo.txt" ] }
}
However, when I try to --allow-read permissions I get the error: Is a directory
deno run --allow-read src/ main.ts
error: Is a directory (os error 21)
How do I ensure that the explicit permission --allow-read is permitted for the specified /src folder?
I know that I can use -A to --allow-all, however, I want to be explicit to the allowed permission.
I guess I found the problem. Well first of all you need to use a = to add the allowed paths, like so:
deno run --allow-read=src/ main.ts
But it'll still won't work and it seems to be a bug/enhancement.
On your script you need to provide the absolute path and then it will be effective:
const watcher = Deno.watchFs("<ABSOLUTE_PATH>/src");
For me it's an issue on the Deno.watchFs() method and I opened up one on Github:
https://github.com/denoland/deno/issues/5742
I have a Meteor app and want to call a server method from the command line, so that I can write a bash script to perform scheduled operations.
Is there any way to either call a method directly, or submit a form which will then trigger server-side code?
I've tried using curl to call a method, but either it's not possible or I'm missing something basic. This doesn't work:
curl "http://localhost:3000/Meteor.call('myMethod')"
nor does:
curl -s -d "http://localhost:3000/imports/api/test.js" > out.html
where test.js:
var test = function(){
console.log('hello');
}
I thought of using a form but I can't think how to create a submit event because the Meteor client uses template events that then call server methods.
I'll be very grateful for any help! This feels like it should be a simple thing but has me stumped.
Edit: I've also tried phantomjs and slimerjs as run through casperjs.
phantomjs is no longer maintained and generates an error:
TypeError: Attempting to change the setter of an unconfigurable property.
https://github.com/casperjs/casperjs/issues/1935
slimerjs errors with Firefox 60 and I can't figure out how to 'downgrade' back to the supported 59, and the option to disable automatic updates of Firefox no longer seems to exist. The error is:
c is undefined
https://github.com/laurentj/slimerjs/issues/694
You could make use of the node ddp package to call the Meteor method in an own js file that you created with a specific script. From there you can pipe all outs to wherever you want.
Let's assume the following Meteor method:
Meteor.methods({
'myMethod'() {
console.log("hello console")
return "hello result"
}
})
The upcoming steps will let you call this method from another shell, assuming your Meteor application is running.
1. Install ddp in your global npm directory
$ meteor npm install -g ddp
2. Create the script to call your method in your test directory
$ mkdir -p ddptest
$ cd ddptest
$ touch ddptest.js
Place the ddp script code into the file with the editor or command of your choice.
(The follwing code is freely taken from the package's readme. Feel free to configure to your needs.)
ddptest/ddptest.js
var DDPClient = require(process.env.DDP_PATH);
var ddpclient = new DDPClient({
// All properties optional, defaults shown
host : "localhost",
port : 3000,
ssl : false,
autoReconnect : true,
autoReconnectTimer : 500,
maintainCollections : true,
ddpVersion : '1', // ['1', 'pre2', 'pre1'] available
// uses the SockJs protocol to create the connection
// this still uses websockets, but allows to get the benefits
// from projects like meteorhacks:cluster
// (for load balancing and service discovery)
// do not use `path` option when you are using useSockJs
useSockJs: true,
// Use a full url instead of a set of `host`, `port` and `ssl`
// do not set `useSockJs` option if `url` is used
url: 'wss://example.com/websocket'
});
ddpclient.connect(function(error, wasReconnect) {
// If autoReconnect is true, this callback will be invoked each time
// a server connection is re-established
if (error) {
console.log('DDP connection error!');
console.error(error)
return;
}
if (wasReconnect) {
console.log('Reestablishment of a connection.');
}
console.log('connected!');
setTimeout(function () {
/*
* Call a Meteor Method
*/
ddpclient.call(
'myMethod', // namyMethodme of Meteor Method being called
['foo', 'bar'], // parameters to send to Meteor Method
function (err, result) { // callback which returns the method call results
console.log('called function, result: ' + result);
ddpclient.close();
},
function () { // callback which fires when server has finished
console.log('updated'); // sending any updated documents as a result of
console.log(ddpclient.collections.posts); // calling this method
}
);
}, 3000);
});
The code assumes that your app runs on localhost:3000, note that there is no conncection close on errors or undesired behavior.
As you can see at the top, the file imports your globally installed ddp package. Now in order to get it's path without using additional tools, just pass an environment variable (process.env.DDP_PATH) and let your shell handle the path resolving.
In order to get the installation path you can use npm root with the global flag.
Finally call your script via:
$ DDP_PATH=$(meteor npm root -g)/ddp meteor node ddptest.js
Which will give you the following output:
connected!
updated
undefined
called function, result: hello result
And logs hello console to the open session that is running your meteor app.
Edit: A note on using this in production
If you want to use this script in production you have to use the shell commands without the meteor command but using your installation of node and npm.
If you get in trouble with paths use process.execPath to find your node binary and npm root -g to find your global npm modules.
You can check out this documentation: Command Line | meteor shell.
While your meteor app is running, you can execute meteor shell to start an interactive console. In the console, you can do Meteor.call(...).
So if you want to write a script with using meteor shell, you might need to pipe the script file for meteor shell. Like,
$ meteor shell < script_file
See also the answer of "How can I pipe a command into the meteor shell?"
I have two very simple files to test my integration setup. First the r script inside test.R:
print('hello')
Then the index.js calling that file:
var R = require("r-script");
var out = R("./test.R")
.data()
.callSync();
And this is what my console is telling me:
$ node index.js
/home/user/index.js:3
.data()
^
TypeError: Cannot read property 'data' of undefined
Currently, I am running Ubuntu 18.04 and if I run only Rscript test.R it works with no problem. Any help?
I fix my similar issue with https://github.com/joshkatz/r-script/issues/19
If you still feeling lazy to pull the git, you can directly update the source code ./node_modules/r-script/index.js , then edit the function init as
function init(path) {
var obj = new R(path);
_.bindAll(obj, "data", "call", "callSync");
return obj;
}
I hope you may find this helpful.
$ cd node_modules/r-script
$ rm node_modules
$ npm install underscore#1.8.3
in my case, downgrade underscore works
$ wget --quiet http://download.qt-project.org/official_releases/qt/5.2/5.2.1/qt-opensource-windows-x86-msvc2012_64_opengl-5.2.1.exe
$
As seen above, I first downloaded the Qt package for Visual Studio in a Cygwin Bash shell.
A sidenote: The Qt library packaged within Cygwin is not useful for me because I need to use the Visual Studio C++ compiler.
First I set the correct permissions on the file
$ chmod 755 qt-opensource-windows-x86-msvc2012_64_opengl-5.2.1.exe
If I start it like this
$ ./qt-opensource-windows-x86-msvc2012_64_opengl-5.2.1.exe
a graphical window (GUI) is shown but that is not what I want as I would later like to have the installation procedure written into a Bash script that I could run in Cygwin.
If I add the option --help, like this
$ ./qt-opensource-windows-x86-msvc2012_64_opengl-5.2.1.exe --help
a new terminal window is opened with the following text
Usage: SDKMaintenanceTool [OPTIONS]
User:
--help Show commandline usage
--version Show current version
--checkupdates Check for updates and return an XML file describing
the available updates
--updater Start in updater mode.
--manage-packages Start in packagemanager mode.
--proxy Set system proxy on Win and Mac.
This option has no effect on Linux.
--verbose Show debug output on the console
--create-offline-repository Offline installer only: Create a local repository inside the
installation directory based on the offline
installer's content.
Developer:
--runoperation [OPERATION] [arguments...] Perform an operation with a list of arguments
--undooperation [OPERATION] [arguments...] Undo an operation with a list of arguments
--script [scriptName] Execute a script
--no-force-installations Enable deselection of forced components
--addRepository [URI] Add a local or remote repo to the list of user defined repos.
--addTempRepository [URI] Add a local or remote repo to the list of temporary available
repos.
--setTempRepository [URI] Set a local or remote repo as tmp repo, it is the only one
used during fetch.
Note: URI must be prefixed with the protocol, i.e. file:///
http:// or ftp://. It can consist of multiple
addresses separated by comma only.
--show-virtual-components Show virtual components in package manager and updater
--binarydatafile [binary_data_file] Use the binary data of another installer or maintenance tool.
--update-installerbase [new_installerbase] Patch a full installer with a new installer base
--dump-binary-data -i [PATH] -o [PATH] Dumps the binary content into specified output path (offline
installer only).
Input path pointing to binary data file, if omitted
the current application is used as input.
I don't know how to continue from here. Do you know how I could install the Qt 5.2.1 library (for Visual Studio) from the Bash shell in Cygwin?
Update: The advantage of writing the build script for a Cygwin environment is that commands like git, wget, and scp are available. This Stackoverflow answer describes how to invoke the MSVC compiler from a Cygwin bash script. Note, that the Qt application I'm building is not going to have any dependency on Cygwin.
I didn't test with Cygwin but I successfully installed Qt5.5 using a script. To do so, you must use the --script line of the normal installer.
.\qt-opensource-windows-x86-msvc2013_64-5.5.1.exe --script .\qt-installer-noninteractive.qs
Here's an example of qt-installer-noninteractive.qs file I used in the command above
function Controller() {
installer.autoRejectMessageBoxes();
installer.installationFinished.connect(function() {
gui.clickButton(buttons.NextButton);
})
}
Controller.prototype.WelcomePageCallback = function() {
gui.clickButton(buttons.NextButton);
}
Controller.prototype.CredentialsPageCallback = function() {
gui.clickButton(buttons.NextButton);
}
Controller.prototype.IntroductionPageCallback = function() {
gui.clickButton(buttons.NextButton);
}
Controller.prototype.TargetDirectoryPageCallback = function() {
gui.currentPageWidget().TargetDirectoryLineEdit.setText("C:/Qt/Qt5.5.1");
gui.clickButton(buttons.NextButton);
}
Controller.prototype.ComponentSelectionPageCallback = function() {
var widget = gui.currentPageWidget();
widget.deselectAll();
widget.selectComponent("qt.55.win64_msvc2013_64");
// widget.selectComponent("qt.55.qt3d");
// widget.selectComponent("qt.55.qtcanvas3d");
// widget.selectComponent("qt.55.qtquick1");
// widget.selectComponent("qt.55.qtscript");
// widget.selectComponent("qt.55.qtwebengine");
// widget.selectComponent("qt.55.qtquickcontrols");
// widget.selectComponent("qt.55.qtlocation");
gui.clickButton(buttons.NextButton);
}
Controller.prototype.LicenseAgreementPageCallback = function() {
gui.currentPageWidget().AcceptLicenseRadioButton.setChecked(true);
gui.clickButton(buttons.NextButton);
}
Controller.prototype.StartMenuDirectoryPageCallback = function() {
gui.clickButton(buttons.NextButton);
}
Controller.prototype.ReadyForInstallationPageCallback = function()
{
gui.clickButton(buttons.NextButton);
}
Controller.prototype.FinishedPageCallback = function() {
var checkBoxForm = gui.currentPageWidget().LaunchQtCreatorCheckBoxForm
if (checkBoxForm && checkBoxForm.launchQtCreatorCheckBox) {
checkBoxForm.launchQtCreatorCheckBox.checked = false;
}
gui.clickButton(buttons.FinishButton);
}
The tricky part was to found the id of the components! I was able to found the right id qt.55.win64_msvc2013_64 by adding the flag --verbose and installing it normally with the UI and stopping at the last page; all the ids that you selected for installation are there.
There is slightly more information in this answer if you need more details.
EDIT (29-11-2017): For installer 3.0.2-online, the "Next" button in the "Welcome" page is disabled for 1 second so you must add a delay
gui.clickButton(buttons.NextButton, 3000);
EDIT (10-11-2019): See Joshua Wade's answer for more traps and pitfalls, like the "User Data Collection" form and "Archive" and "Latest releases" checkboxes.