I would like to use phantomjs' screen capture capability within a firebase function to return a PDF screenshot of a URL. Looking for examples of someone having done this.
I've used PhantomJS in Cloud Functions without major issues. The only concern was that I had to increase the memory for the containers, since the in-memory rendering turns out to be a memory hog.
I ended up using node-webshot, which is a wrapper around PhantomJS:
"node-webshot": "^1.0.2",
Capturing the actual screenshot was simple enough:
const functions = require('firebase-functions');
const webshot = require('node-webshot');
exports.screenshotTweet = functions.https.onRequest((req, res) => {
var stream = webshot(req.query.url);
res.writeHead(200, {'Content-Type': 'image/jpeg'});
stream.on('data', function(data) {
res.write(data.toString('binary'), 'binary');
});
stream.on('end', function() {
res.end();
})
});
Related
I'm building an app for shopify and need to add the GDPR webhooks. My back end is handled using next.js and I'm writing a webhook handler to verify them. The docs havent been very helpful because they dont show how to do it with node. This is my verification function.
export function verifiedShopifyWebhookHandler(
next: (req, res, body) => Promise
): NextApiHandler {
return async (req, res) => {
const hmacHeader = req.headers['x-shopify-hmac-sha256'];
const rawBody = await getRawBody(req);
const digest = crypto.createHmac('sha256', process.env.SHOPIFY_API_SECRET).update(rawBody).digest('base64');
if (digest === hmacHeader) {
return next(req, res, rawBody);
}
const webhookId = req.headers['x-shopify-webhook-id'];
return res.status(401).end();
};
}
But I get this Error: error - InternalServerError: stream is not readable
I think it has to do with now Next.js parses the incoming requests before they are sent to my api. Any ideas?
I discovered the answer. Next.js was pre parsing the body in the context which made it so that I couldn't use the raw body parser to parse it. By setting this:
export const config = {
api: {
bodyParser: false
}
};
above the api function in the api file it prevented next from parsing it and causing the issue. I found the answer because people had the same issue integrating swipe and using the bodyParser.
I'm moving a MERN project into React + MongoDB Stitch after seeing it allows for easy user authentication, quick deployment, etc.
However, I am having a hard time understanding where and how can I call a site scraping function. Previously, I web scraped in Express.js with cheerio like:
app.post("/api/getTitleAtURL", (req, res) => {
if (req.body.url) {
request(req.body.url, function(error, response, body) {
if (!error && response.statusCode == 200) {
const $ = cheerio.load(body);
const webpageTitle = $("title").text();
const metaDescription = $("meta[name=description]").attr("content");
const webpage = {
title: webpageTitle,
metaDescription: metaDescription
};
res.send(webpage);
} else {
res.status(400).send({ message: "THIS IS AN ERROR" });
}
});
}
});
But obviously with Stitch no Node & Express is needed. Is there a way to fetch another site's content without having to host a node.js application just serving that one function?
Thanks
Turns out you can build Functions in MongoDB Stitch that allows you to upload external dependencies.
However, there're limitation, for example, cheerio didn't work as an uploaded external dependency while request worked. A solution, therefore, would be to create a serverless function in AWS's lambda, and then connect mongoDB stitch to AWS lambda (mongoDB stitch can connect to many third party services, including many AWS lambda cloud services like lambda, s3, kinesis, etc).
AWS lambda allows you to upload any external dependencies, if mongoDB stitch allowed for any, we wouldn't need lambda, but stitch still needs many support. In my case, I had a node function with cheerio & request as external dependencies, to upload this to lambda: make an account, create new lambda function, and pack your node modules & code into a zip file to upload it. Your zip should look like this:
and your file containing the function should look like:
const cheerio = require("cheerio");
const request = require("request");
exports.rss = function(event, context, callback) {
request(event.requestURL, function(error, response, body) {
if (!error && response.statusCode == 200) {
const $ = cheerio.load(body);
const webpageTitle = $("title").text();
const metaDescription = $("meta[name=description]").attr("content");
const webpage = {
title: webpageTitle,
metaDescription: metaDescription
};
callback(null, webpage);
return webpage;
} else {
callback(null, {message: "THIS IS AN ERROR"})
return {message: "THIS IS AN ERROR"};
}
});
};
and in mongoDB, connect to a third party service, choose AWS, enter the secret keys you got from making an IAM amazon user. In rules -> actions, choose lambda as your API, and allow for all actions. Now, in your mongoDB stitch functions, you can connect to Lambda, and that function should look like this in my case:
exports = async function(requestURL) {
const lambda = context.services.get('getTitleAtURL').lambda("us-east-1");
const result = await lambda.Invoke({
FunctionName: "getTitleAtURL",
Payload: JSON.stringify({requestURL: requestURL})
});
console.log(result.Payload.text());
return EJSON.parse(result.Payload.text());
};
Note: this slowed down performances big time though, generally, it took twice extra time for the call to finish.
Gremlin.createClient() is working in version 2.6.0 but it is not working in version 3.3.4,I know it is deprecated from 3.3.4.I want to connect to server and execute query.The below code is executed in version 2.6. I want to excute same query in 3.3.4.
const Gremlin = require('gremlin');
const client = Gremlin.createClient(8182, 'localhost');
client.execute('g.V()', { }, (err, results) => {
if (err) {
return console.error(err)
}
console.log(results);
});
How can i wirte in version 3.3.4?.
TinkerPop no longer recommends the use of scripts if possible. It is best to simply write Gremlin in the language of your choosing, which for your case is Javascript:
const g = traversal().withRemote(new DriverRemoteConnection('ws://localhost:8182/gremlin'));
g.V().hasLabel('person').values('name').toList()
.then(names => console.log(names));
That said, you should be able to still submit scripts this way:
const gremlin = require('gremlin');
const client = new gremlin.driver.Client('ws://localhost:8182/gremlin', { traversalSource: 'g' });
const result1 = await client.submit('g.V(vid)', { vid: 1 });
const vertex = result1.first();
Please see the full reference documentation for more information.
I changed the location of my cloud functions from "us-central1" to "europe-west1" but I can't change the location of my functions on the client side which is a compulsory step for it to work according to the documentation.
(IDE tells me that no argument is expected on 'functions' when i do:
firebase.initializeApp(config).functions("europe-west1");
As an attempt to solve my problem I updated the three dependancies below with no result.
firebase-tools#latest
firebase-functions#latest
firebase-admin#latest
The problem is still here.
You should visit the following documentation page.
https://firebase.google.com/docs/web/setup
https://firebase.google.com/docs/functions/manage-functions#modify-region
https://firebase.google.com/docs/functions/locations
client side
Use firebase.app.App.
https://firebase.google.com/docs/reference/js/firebase.app.App#functions
Not admin.app.App. The firebase-admin only use on the server side.
https://firebase.google.com/docs/reference/admin/node/admin.app.App
Set the specified regions for a client app.
var firebase = require("firebase/app");
require("firebase/functions");
var config = {
// ...
};
firebase.initializeApp(config).;
var functions = firebase.app().functions('europe-west1');
server side(Cloud Functions)
Set the specified regions for each function.
const functions = require('firebase-functions');
exports.webhookEurope = functions
.region('europe-west1')
.https.onRequest((req, res) => {
res.send("Hello");
});
If you are changing the specified regions for a function that's handling production traffic, you can prevent event loss by performing these steps in order:
Rename the function, and change its region or regions as desired.
Deploy the renamed function, which results in temporarily running the same code in both sets of regions.
Delete the previous function.
I finally managed to fix my situation, by reinstalling ionic.
Plus .functions("europe-west1") has to be put on every call, not only in app.module.ts
With Typescript, you set the region like this:
export const myFunction = functions.region("europe-west2").firestore.document("users/{userId}")
Got it working this way!
const firebaseApp = firebase.initializeApp(firebaseConfig);
async signup(provider, info) {
if (!this.isValid) {
return;
}
try {
switch (provider) {
case this.EMAIL:
return await firebaseAuth().createUserWithEmailAndPassword(
info.email,
info.password
).then(authUser => {
const addUser = firebaseApp.functions('europe-west1').httpsCallable('addUser');
addUser({email: info.email,
name: info.name})
return authUser
})
default:
}
} catch (error) {
return error;
}
}
Google Cloud Speech to Text documentation dictates that you can access it by:
const client = new speech.SpeechClient();
const [operation] = await client.longRunningRecognize({
config: {
encoding: 'LINEAR16',
sampleRateHertz: 16000,
languageCode: 'en-US'
},
audio: {
uri: `gs://${bucket}/${name}`
}
});
const [response] = await operation.promise();
response.results.forEach(result => {
console.log(`Transcription: ${result.alternatives[0].transcript}`);
});
Now, I wanna run this code in a Firebase Cloud Function. Unfortunately, Cloud Functions run on a version of Node that does not yet support async and await functions.
Some things I've tried:
Trying TypeScript, which supports async and await: Ran into a bunch of problems with some of the other APIs I'm using.
Upgrading all my functions to Node 8 (beta), which supports async and await: Again, ran into quite a bit of bugs from the Firebase side doing this.
"Translating" the code manually (is this even a thing?): I tried to treat the code to expect a promise.
That didn't work too well either, this is how it looks:
exports.onStorageObjectFinalize = functions.storage.object()
.onFinalize((object) => {
const client = new speech.SpeechClient();
return client.longRunningRecognize({
config: {
encoding: 'LINEAR16',
sampleRateHertz: 16000,
languageCode: 'en-US'
},
audio: {
uri: `gs://${object.bucket}/${object.name}`
}
})
.then(r1 => {
const [operations] = r1;
return operations.promise();
})
.then(r2 => {
const [response] = r2;
// response.results...
return true;
});
});
Edit: When the above function runs, it says there's no operations.promise(). In fact, after taking a look at the whole operations object, the structure doesn't look like its the same function. I did found there's a promise property in operations._callOptions, so I tried returning operations._callOptions.promise() but I got a strange error: TypeError: #<CallSettings> is not a promise at client.longRunningRecognize.then.r1.
Did I mess the translation code up or would this never work anyways?
Any other things I can try or are TypeScript and Node 8 my only two options here?
Thanks, much appreciated.