Batch Geocode using Axios - here-api

Testing the HERE Batch GeoCode life-cycle through node application. We have similar working with Azure Mappings but they are crazy expensive.
Seems as if the initial post request is succeeding. But is stuck on "submitted" status during status check. And failing during result check with 404. Using axius to make the queries - with the documented examples.
const getStatus = async requestId => {
const url = statusURL(requestId);
const res = await axios.get(url);
const response = res.data.Response;
return response;
};
const getResult = async requestId => {
const url = resultURL(requestId);
const config = { headers: { 'Content-type': 'text/plain' } };
const res = await axios.get(url, config);
const response = res.data.Response;
return response;
};
const requestGeo = async input => {
const url = requestURL;
const res = await axios.post(url, input, {
headers: { 'Content-type': 'text/plain' },
});
const requestId = res.data.Response.MetaInfo.RequestId;
return requestId;
};
getStatus(requestId)
.then(res => {
console.log(res);
})
.catch(e => {
console.log(e);
});
const input = `recId|street|city|postalCode|country
1|425 Randolph St|Chicago||USA
2|31 St James Ave|Boston|02116|USA
3|Invalidenstrasse 117|Berlin|10115|DEU`;
requestGeo(input)
.then(console.log)
.catch(e => {
console.log(e);
});

If you don't specify the "&action=run" parameter in your initial request, then the job is being checked, stored and set as "submitted". This does not mean that it will be executed.
Alternatively you can send an "action=start"-request to start the job.
Having applied one of these two options, the job will be scheduled for execution and flagged as "accepted".

Related

Firebase cloud function getting 404 timeout error

I'm trying to run a firebase cloud function. It reads 5k documents and it's taking more than one minute to run in insomnia. When it finally crashes showing a 404 timeout error, how can I fix it? I'm from Brazil and the function region is from the US by default, but I don't believe changing it will help.
I tried making the code cleaner, but didn't help.
This is one of the cloud functions:
exports.getTodosPedidos = functions.https.onRequest(
async (req: RequestType, res: ResponseType) => {
try {
const idProjeto = req.query.idProjeto;
if (!idProjeto) {
onFailureResponseStructure(res, "ID do Projeto não fornecido.", 404);
}
const projeto =
idProjeto &&
(await admin.firestore().collection("projetos").doc(idProjeto.toString()).get());
if (projeto != "" && !projeto?.createTime) {
onFailureResponseStructure(res, "O projeto fornecido não existe.", 405);
}
const snapshot = await admin.firestore().collection("pedidos").get();
const result = snapshot.docs.map(doc => ({ ...doc.data(), id: doc.id }));
let idsDosPedidos = result.map(elemento => elemento.id);
let somaDosArquivos = idsDosPedidos.map(async pedidoId => {
let bucketRef = await admin
.storage()
.bucket()
.getFiles({
prefix: `${idProjeto}/arquivos/${pedidoId}/`,
});
return bucketRef[0].map(elemento => parseInt(elemento.metadata.size));
});
const resultado = await Promise.all(somaDosArquivos);
onSuccessResponseStructure(res, { resultado, idsDosPedidos });
} catch (error: any) {
onFailureResponseStructure(res, error.message, 500);
}
}
);
const onSuccessResponseStructure = (res: ResponseType, data: any) => {
return res.status(200).json({
status: "OK",
data,
messages: [],
statusCode: "200",
});
};
const onFailureResponseStructure = (
res: ResponseType,
messages: string,
statusCode: number
) => {
return res.status(statusCode).json({ status: "ERRO", data: [], messages, statusCode });
};
I'm not entirely familiar with firebase. But other cloud services allow you to increase functions timeout lengths. It appears firebase does as well...https://firebase.google.com/docs/functions/manage-functions#set_timeout_and_memory_allocation
Though for long running tasks you may want to investigate other services.

Can i dispatch many actions in getServersideprops?

In my social media app in Home page i want to dispatch 3 actions from my api:
posts , users , userDetails
But this may show an error(500) on vercel because the request takes a lot of time to get all these data.
vercel Error
this error will not appear again after refreshing the page !!!
i think that's because the request takes a lot of time to get all the data.
-> getServersideProps Code
export const getServerSideProps = wrapper.getServerSideProps(
store => async (context) =>
{
const {req} = context
const session = await getSession({ req });
await store.dispatch(fetchPostsAction());
await store.dispatch(fetchUsersAction(4));
await store.dispatch(LoggedInUserAction({email:session.user.email}));
})
-> fetchPostsAction Code
"post/list",
async (_, { rejectWithValue, getState, dispatch }) => {
try
{
let link = `${URL}/api/posts`;
const { data } = await axios.get(link,{
headers: { "Accept-Encoding": "gzip,deflate,compress" }
});
console.log("#2 got the data",data)
return data;
} catch (error) {
if (!error?.response) throw error;
return rejectWithValue(error?.response?.data);
}
}
);
-> extraReducer Code
builder.addCase(createpostAction.pending, (state, action) => {
state.createPostLoading = true;
});
builder.addCase(createpostAction.fulfilled, (state, action) => {
state.postLists = [...state.postLists, action.payload.post].sort((a, b) => b.createdAt > a.createdAt ? 1 : -1)
state.createPostLoading = false;
state.isCreated = true;
state.appErr = null;
state.serverErr = null;
});
builder.addCase(createpostAction.rejected, (state, action) => {
state.createPostLoading = false;
state.appErr =
action?.payload?.message || action?.payload?.error?.message;
state.serverErr = action?.error?.message;
});
-> get posts from api Code
handler.get(async (req, res) =>
{
await db.connect();
try {
const posts = await Post.find().populate({
path: 'user',
model: 'User',
}).populate({
path:'comments',
options: {sort: {'createdAt' : -1} }
}).sort('-createdAt')
res.status(200).json({
success:true,
posts
});
} catch (err) {
res.status(500).json(err.message)
}
await db.disconnect();
})
so what is the best way to fetch all these data in next js ?
I hope there is a way to solve this problem

FCM very slow and unreliable when sending to a group of recipients through Cloud Function

I have the following Function that:
Listens for document (text message) creation
Grab IDs of members of a group chat
Get the FCM Tokens for each member
With a for-loop, send messages to group members
exports.sendChatMessage = functions.firestore
.document("chats/{mealID}/messages/{messageID}")
.onCreate((snap, context) => {
const data = snap.data();
const mealID = context.params.mealID;
const senderID = data.senderID;
const senderName = data.senderName;
const messageContent = data.content;
var docRef = db.collection("chats").doc(mealID);
docRef
.get()
.then((doc) => {
if (doc.exists) {
const docData = doc.data();
const mealName = docData.name;
const userStatus = docData.userStatus;
var users = docData.to;
var eligibleUsers = users.filter(
(user) => userStatus[user] == "accepted"
);
eligibleUsers.push(docData.from);
// get fcmTokens from eligibleUsers and send the messagme
db.collection("users")
.where("uid", "in", eligibleUsers)
.get()
.then((snapshot) => {
var fcmTokens = [];
var thumbnailPicURL = "";
// get thumbnailpic of the sender and collect fcmTokens
snapshot.forEach((doc) => {
if (doc.data().uid == senderID) {
thumbnailPicURL =
doc.data().thumbnailPicURL == null
? "https://i.imgur.com/8wSudUk.png"
: doc.data().thumbnailPicURL;
} else {
fcmTokens.push(doc.data().fcmToken);
}
});
// send the message fcmTokens
fcmTokens.forEach((token) => {
if (token != "") {
const fcmMessage = {
message: {
token: token,
notification: {
title: mealName,
body: senderName + ": " + messageContent,
image: thumbnailPicURL,
},
apns: {
payload: {
aps: {
category: "MESSAGE_RECEIVED",
},
MEAL_ID: mealID,
},
},
},
};
tokenManger.sendFcmMessage(fcmMessage);
}
});
return true;
});
} else {
// doc.data() will be undefined in this case
console.log("No such document!");
return false;
}
})
.catch((error) => {
console.log("Error getting document:", error);
return false;
});
return true;
});
My send function comes from a helper file that uses the HTTP V1 protocol to build the send-request:
const { google } = require("googleapis");
const https = require("https");
const MESSAGING_SCOPE = "https://www.googleapis.com/auth/firebase.messaging";
const SCOPES = [MESSAGING_SCOPE];
const PROJECT_ID = MY_PROJECT_ID;
const HOST = "fcm.googleapis.com";
const PATH = "/v1/projects/" + PROJECT_ID + "/messages:send";
exports.getAccessToken = () => {
return new Promise(function (resolve, reject) {
const key = require("./service-account.json");
var jwtClient = new google.auth.JWT(
key.client_email,
null,
key.private_key,
SCOPES,
null
);
jwtClient.authorize(function (err, tokens) {
if (err) {
reject(err);
return;
}
resolve(tokens.access_token);
});
});
};
//send message
exports.sendFcmMessage = (fcmMessage) => {
this.getAccessToken().then(function (accessToken) {
var options = {
hostname: HOST,
path: PATH,
method: "POST",
headers: {
Authorization: "Bearer " + accessToken,
},
// … plus the body of your notification or data message
};
var request = https.request(options, function (resp) {
resp.setEncoding("utf8");
resp.on("data", function (data) {
console.log("Message sent to Firebase for delivery, response:");
console.log(data);
});
});
request.on("error", function (err) {
console.log("Unable to send message to Firebase");
console.log(err);
});
request.write(JSON.stringify(fcmMessage));
request.end();
});
};
It worked all fine in the emulator but once deployed, there're significant delays (~3 mins):
I also noticed that the console says the cloud function finishes execution BEFORE sendFcmMessage logs success messages.
I did some research online, it appears that it might have something to do with the usage of Promise but I wasn't sure if that's the sole reason or it has something to do with my for-loop.
The Problem
To summarize the issue, you are creating "floating promises" or starting other asynchronous tasks (like in sendFcmMessage) where you aren't returning a promise because they use callbacks instead.
In a deployed function, as soon as the function returns its result or the Promise chain resolves, all further actions should be treated as if they will never be executed as documented here. An "inactive" function might be terminated at any time, is severely throttled and any network calls you make (like setting data in database or calling out to FCM) may never be executed.
An indicator that you haven't properly chained the promises is when you see the function completion log message ("Function execution took...") before other messages you are logging. When you see this, you need to look at the code you are running and confirm whether you have any "floating promises" or are using callback-based APIs. Once you have changed the callback-based APIs to use promises and then made sure they are all chained together properly, you should see a significant boost in performance.
The fixes
Sending the message data to FCM
In your tokenManger file, getAccessToken() could be reworked slightly and sendFcmMessage should be converted to return a Promise:
exports.getAccessToken = () => {
return new Promise(function (resolve, reject) {
const key = require("./service-account.json");
const jwtClient = new google.auth.JWT(
key.client_email,
null,
key.private_key,
SCOPES,
null
);
jwtClient.authorize(
(err, tokens) => err ? reject(err) : resolve(tokens.access_token)
);
});
};
//send message
exports.sendFcmMessage = (fcmMessage) => {
// CHANGED: return the Promise
return this.getAccessToken().then(function (accessToken) {
const options = {
hostname: HOST,
path: PATH,
method: "POST",
headers: {
Authorization: "Bearer " + accessToken,
},
// … plus the body of your notification or data message
};
// CHANGED: convert to Promise:
return new Promise((resolve, reject) => {
const request = https.request(options, (resp) => {
resp.setEncoding("utf8");
resp.on("data", resolve);
resp.on("error", reject);
});
request.on("error", reject);
request.write(JSON.stringify(fcmMessage));
request.end();
});
});
};
However, the above code was built for googleapis ^52.1.0 and google-auth-library ^6.0.3. The modern versions of these modules are v92.0.0 and v7.11.0 respectively. This means you should really update the code to use these later versions like so:
// Import JWT module directly
const { JWT } = require('google-auth-library');
// FIREBASE_CONFIG is a JSON string available in Cloud Functions
const PROJECT_ID = JSON.parse(process.env.FIREBASE_CONFIG).projectId;
const FCM_ENDPOINT = `https://fcm.googleapis.com/v1/projects/${PROJECT_ID}/messages:send`;
const FCM_SCOPES = ["https://www.googleapis.com/auth/firebase.messaging"];
exports.sendFcmMessage = (fcmMessage) => {
const key = require("./service-account.json"); // consider moving outside of function (so it throws an error during deployment if its missing)
const client = new JWT({
email: key.client_email,
key: key.private_key,
scopes: FCM_SCOPES
});
return client.request({ // <-- this uses `gaxios`, Google's fork of `axios` built for Promise-based APIs
url: FCM_ENDPOINT,
method: "POST",
data: fcmMessage
});
}
Better yet, just use the messaging APIs provided by the Firebase Admin SDKs that handle the details for you. Just feed it the message and tokens as needed.
import { initializeApp } from "firebase-admin/app";
import { getMessaging } from "firebase-admin/messaging";
initializeApp(); // initializes using default credentials provided by Cloud Functions
const fcm = getMessaging();
fcm.send(message) // send to one (uses the given token)
fcm.sendAll(messagesArr) // send to many at once (each message uses the given token)
fcm.sendMulticast(message) // send to many at once (uses a `tokens` array instead of `token`)
The Cloud Function
Updating the main Cloud Function, you'd get:
exports.sendChatMessage = functions.firestore
.document("chats/{mealID}/messages/{messageID}")
.onCreate((snap, context) => {
const mealID = context.params.mealID;
const { senderID, senderName, content: messageContent } = snap.data();
const docRef = db.collection("chats").doc(mealID);
/* --> */ return docRef
.get()
.then((doc) => {
if (!doc.exists) { // CHANGED: Fail fast and avoid else statements
console.log(`Could not find "chat:${mealID}"!`);
return false;
}
const { userStatus, to: users, name: mealName, from: fromUser } = doc.data();
const eligibleUsers = users.filter(
(user) => userStatus[user] == "accepted"
);
eligibleUsers.push(fromUser);
// get fcmTokens from eligibleUsers and send the message
/* --> */ return db.collection("users")
.where("uid", "in", eligibleUsers) // WARNING: This will only work for up to 10 users! You'll need to break it up into chunks of 10 if there are more.
.get()
.then(async (snapshot) => {
const fcmTokens = [];
let thumbnailPicURL = "";
// get thumbnailpic of the sender and collect fcmTokens
snapshot.forEach((doc) => {
if (doc.get("uid") == senderID) {
thumbnailPicURL = doc.get("thumbnailPicURL"); // update with given thumbnail pic
} else {
fcmTokens.push(doc.get("fcmToken"));
}
});
const baseMessage = {
notification: {
title: mealName,
body: senderName + ": " + messageContent,
image: thumbnailPicURL || "https://i.imgur.com/8wSudUk.png", // CHANGED: specified fallback image here
},
apns: {
payload: {
aps: {
category: "MESSAGE_RECEIVED",
},
MEAL_ID: mealID,
},
}
}
// log error if fcmTokens empty?
// ----- OPTION 1 -----
// send the message to each fcmToken
const messagePromises = fcmTokens.map((token) => {
if (!token) // handle "" and undefined
return; // skip
/* --> */ return tokenManger
.sendFcmMessage({
message: { ...baseMessage, token }
})
.catch((err) => { // catch the error here, so as many notifications are sent out as possible
console.error(`Failed to send message to "fcm:${token}"`, err);
})
});
await Promise.all(messagePromises); // wait for all messages to be sent out
// --------------------
// ----- OPTION 2 -----
// send the message to each fcmToken
await getMessaging().sendAll(
fcmTokens.map((token) => ({ ...baseMessage, token }))
);
// --------------------
return true;
})
.catch((error) => {
console.log("Error sending messages:", error);
return false;
});
})
.catch((error) => {
console.log("Error getting document:", error);
return false;
});
});
I found out that the culprit is my queries to db. Like #samthecodingman commented, I was creating floating Promises.
Originally, I have codes like:
db.collection("users")
.where("uid", "in", eligibleUsers)
.get()
.then((snapshot) => {...}
All I needed to do is to return that call:
return db.collection("users")
.where("uid", "in", eligibleUsers)
.get()
.then((snapshot) => {...}
Although it's still not instant delivery, it's much faster now.

next-redux-wrapper Wrapper.getStaticProps not working with

This is my backend controller, I am getting the rooms data
const allRooms = catchAsyncErrors(async (req, res) => {
const resPerPage = 4;
const roomsCount = await Room.countDocuments();
const apiFeatures = new APIFeatures(Room.find(), req.query)
.search()
.filter()
let rooms = await apiFeatures.query;
let filteredRoomsCount = rooms.length;
apiFeatures.pagination(resPerPage)
rooms = await apiFeatures.query;
res.status(200).json({
success: true,
roomsCount,
resPerPage,
filteredRoomsCount,
rooms
})
})
This is my redux actions I am sending the payload and data
export const getRooms = (req, currentPage = 1, location = '', guests, category) => async (dispatch) => {
try {
const { origin } = absoluteUrl(req);
let link = `${origin}/api/rooms?page=${currentPage}&location=${location}`
if (guests) link = link.concat(`&guestCapacity=${guests}`)
if (category) link = link.concat(`&category=${category}`)
const { data } = await axios.get(link)
dispatch({
type: ALL_ROOMS_SUCCESS,
payload: data
})
} catch (error) {
dispatch({
type: ALL_ROOMS_FAIL,
payload: error.response.data.message
})
}
}
This is my reducer function, dispatching the room data
export const allRoomsReducer = (state = { rooms: [] }, action) => {
switch (action.type) {
case ALL_ROOMS_SUCCESS:
return {
rooms: action.payload.rooms
}
case ALL_ROOMS_FAIL:
return {
error: action.payload
}
case CLEAR_ERRORS:
return {
...state,
error: null
}
default:
return state
}
}
Here I want to get the data using wrapper.getStaticProps but I am getting an error, but when i am using wrapper.getServerSideProps I get the data.
export const getStaticProps = wrapper.getStaticProps(store=> async ({ req, query, }) => {
await store.dispatch(getRooms(req, query.page, query.location, query.guests, query.category))
})
It seems that in ({ req, query, }) => , query is undefined.
Going by the documentation of next, the context object for getStaticProps has no property query. It is only available in getServerSideProps.
See also this info:
Only runs at build time
Because getStaticProps runs at build time, it does not receive data that’s only available during request time, such as query parameters or HTTP headers as it generates static HTML.

Handling multipart/form-data POST with Express in Cloud Functions

I've been trying to handle POSTs (multipart/form-data) with a Firebase function and Express but it just doesn't work. Tried this in local server and it works just fine. Everything's the same except it's not contained in a Firebase function.
Besides screwing up the request object it seems it also screws up the way busboy works.
I've tried different solutions presented here but they just don't work. As one user mentions, the callbacks passed to busboy (to be called when a 'field' is found or when it finishes going through the data) are never called and the function just hangs.
Any ideas?
Here's my function's code for reference:
const functions = require('firebase-functions');
const express = require('express');
const getRawBody = require('raw-body');
const contentType = require('content-type')
const Busboy = require('busboy');
const app = express();
const logging = (req, res, next) => {
console.log(`> request body: ${req.body}`);
next();
}
const toRawBody = (req, res, next) => {
const options = {
length: req.headers['content-length'],
limit: '1mb',
encoding: contentType.parse(req).parameters.charset
};
getRawBody(req, options)
.then(rawBody => {
req.rawBody = rawBody
next();
})
.catch(error => {
return next(error);
});
};
const handlePostWithBusboy = (req, res) => {
const busboy = new Busboy({ headers: req.headers });
const formData = {};
busboy.on('field', (fieldname, value) => {
formData[fieldname] = value;
});
busboy.on('finish', () => {
console.log(`> form data: ${JSON.stringify(formData)}`);
res.status(200).send(formData);
});
busboy.end(req.rawBody);
}
app.post('/', logging, toRawBody, handlePostWithBusboy);
const exchange = functions.https.onRequest((req, res) => {
if (!req.path) {
req.url = `/${req.url}`
}
return app(req, res)
})
module.exports = {
exchange
}
The documentation Doug referred to in his answer is good. An important caveat though is that rawBody does not work in the emulator. The workaround is to do:
if (req.rawBody) {
busboy.end(req.rawBody);
}
else {
req.pipe(busboy);
}
As described in this issue:
https://github.com/GoogleCloudPlatform/cloud-functions-emulator/issues/161#issuecomment-376563784
Please read the documentation for handling multipart form uploads.
... if you want your Cloud Function to process multipart/form-data, you can use the rawBody property of the request.
Because of the way Cloud Functions pre-processes some requests, you can expect that some Express middleware will not work, and that's what you're running into.
I've combined the previous two answers into a easy-to-use async function.
const Busboy = require('busboy');
const os = require('os');
const path = require('path');
const fs = require('fs');
module.exports = function extractMultipartFormData(req) {
return new Promise((resolve, reject) => {
if (req.method != 'POST') {
return reject(405);
} else {
const busboy = new Busboy({ headers: req.headers });
const tmpdir = os.tmpdir();
const fields = {};
const fileWrites = [];
const uploads = {};
busboy.on('field', (fieldname, val) => (fields[fieldname] = val));
busboy.on('file', (fieldname, file, filename) => {
const filepath = path.join(tmpdir, filename);
const writeStream = fs.createWriteStream(filepath);
uploads[fieldname] = filepath;
file.pipe(writeStream);
const promise = new Promise((resolve, reject) => {
file.on('end', () => {
writeStream.end();
});
writeStream.on('finish', resolve);
writeStream.on('error', reject);
});
fileWrites.push(promise);
});
busboy.on('finish', async () => {
const result = { fields, uploads: {} };
await Promise.all(fileWrites);
for (const file in uploads) {
const filename = uploads[file];
result.uploads[file] = fs.readFileSync(filename);
fs.unlinkSync(filename);
}
resolve(result);
});
busboy.on('error', reject);
if (req.rawBody) {
busboy.end(req.rawBody);
} else {
req.pipe(busboy);
}
}
});
};

Resources