Firebase Collection Stream to BigQuery Transform Function - firebase

the firebase extension "Stream Collections to BigQuery" allows for configuring a Transform Function for converting the Firestore Data Json to explicit BigQuery table fields.
https://firebase.google.com/products/extensions/firebase-firestore-bigquery-export
Can anyone point me to an example function or detailed docs for such functions?
Thanks, Ben

The transform Function should be an HTTP Cloud Function with the following logic (get the input object from the request, transform it, send it back in the response) as shown in the below CF skeleton:
exports.bqTransform = functions.https.onRequest(async (req, res) => {
const inputPayload = req.body // JS Object
// ...
// Transform the object
// ...
const outputPayload = {...} // JS Object
res.send(outputPayload);
});
As explained in the doc, the inputPayload object (i.e. req.body) contains a data property (which is an array) which contains a representation of the Firestore document, has shown below:
{
data: [{
insertId: int;
json: {
timestamp: int;
event_id: int;
document_name: string;
document_id: int;
operation: ChangeType;
data: string; // <= String containing the stringified object representing the Firestore document data
},
}]
}
The transformation implemented in your code shall create an object with the same structure (outputPayload in our skeleton example above) where the data[0].json property is adapted according to your transformation requirements.
Here is a very simple example in which we totally change the content of the Firestore record with some static data.
exports.bqTransform = functions.https.onRequest(async (req, res) => {
const inputPayload = req.body;
const inputData = inputPayload.data[0];
const outputPayload = [{
insertId: inputData.insertId,
json: {
timestamp: inputData.json.timestamp,
event_id: inputData.json.event_id,
document_name: inputData.json.document_name,
document_id: inputData.json.document_id,
operation: inputData.json.operation,
data: JSON.stringify({ createdOn: { _seconds: 1664983515, _nanoseconds: 745000000 }, array: ["a1", "a2"], name: "Transformed Name" })
},
}]
res.send({ data: outputPayload });
});

Related

RTK-Query meta response always empty object

When using RTK-Query to call an API, I need to access a custom header value. I've tried various options for accessing the response headers from the documentation, but all attempts result in an empty object returned:
meta: {
"request": {},
"response":{}
}
I can see in the network tab, that the headers are provided in the response. This is part of a refactor from using raw Axios calls where the header was available from the response object as the headers property.
I've tried accessing the headers through the meta parameter on the transformResponse function of the createApi
transformResponse: (response, meta, arg) => {
console.log(`Transform -> meta: ${JSON.stringify(meta)}`)
// dataKey: meta.response.headers['x-company-data-key'],
return {
...response,
// dataKey
}
}
I've also tried accessing the headers via the meta property of the action parameter from the extraReducers function in the feature slice:
extraReducers: (builder) => {
builder.addMatcher(companyApi.endpoints.getSomeData.matchFulfilled, (state, action) => {
console.log(`action meta: ${JSON.stringify(action.meta)}`)
state.result = action.payload.result
// state.dataKey = action.meta.response.headers['x-company-data-key']
})
}
Both instances result in the meta object that looks like this:
meta: {
"request": {},
"response": {}
}
The API's base query looks like this:
baseQuery: fetchBaseQuery({
baseUrl: process.env.REACT_APP_API_ENDPOINT,
prepareHeaders: ( (headers, { getState }) => {
const realm = getState().companyAuth.realm
if (realm) {
const token = readToken(realm)
if (token)
headers.set('Authorization', `Bearer ${token.access_token}`)
}
return headers
})
}),
and finally, the endpoints definition for this API endpoint:
getCompanyData: builder.query({
query: (params) => {
const { location, page, pageSize } = params
return {
url: `/companies/${location}`,
params: { page, pageSize }
}
}
})
Based on what I could find in the documentation, GitHub issues, and PRs, it seems that the meta property should automatically add the headers, status, and other additional HTTP properties. I'm not sure what I've missed where the response object is completely empty. Is there an additional setting I need to add to the baseQuery?
I should note, that the response data is working properly. I just need the additional information that is returned in the custom header.

Has anyone received this error when inserting a calendar event using the Google Calendar API? Related to BodyResponseCallback<Schema$Event> ts(2769)

I am coding in Typescript and trying to insert an event but I am getting this error that the body of what I am inserting "is not assignable to parameter of type 'BodyResponseCallback<Schema$Event>'." I have tried a few different things and even also seen tutorials do it this way but I don't think the error should be acceptable unless if it really is some bug in the source code? Does anyone see what may be the issue here? Thanks in advance. Please see the code and error message below.
router.post('/create-event', (req: CustomRequest, res, next) => {
try {
void (async () => {
const { description, startDateTime, endDateTime } = req.body
oAuth2Client.setCredentials({
refresh_token:
(hidden)
})
const calendar = google.calendar('v3')
const response = await calendar.events.insert({
auth: oAuth2Client,
calendarId: 'primary',
requestBody: {
description: description,
colorId: '7',
start: {
dateTime: new Date(startDateTime),
},
end: {
dateTime: new Date(endDateTime),
},
// see req.body available properties that could help with timeagent
},
})
res.send(response)
})()
} catch (error) {
console.log(error)
next(error)
}
})
The "auth: oAuth2Client," is underlined in red and says:
(alias) const oAuth2Client: OAuth2Client
import oAuth2Client
No overload matches this call.
Overload 2 of 6, '(callback: BodyResponseCallback<Schema$Event>): void', gave the following error.
Argument of type '{ auth: OAuth2Client; calendarId: string; requestBody: { description: string; colorId: string; start: { dateTime: Date; }; end: { dateTime: Date; }; }; }' is not assignable to parameter of type 'BodyResponseCallback<Schema$Event>'.
Object literal may only specify known properties, and 'auth' does not exist in type 'BodyResponseCallback<Schema$Event>'.ts(2769)
I solved this by adding { responseType: 'json' } as the second argument to the insert function. It seems like an issue with types in the SDK.

how to normalize with redux tool and creactentityAdapter

i using redux tool kit to build react native app and i try to normalize my data like this
const postEntitiy = new schema.Entity('post');
const postAdapter = createEntityAdapter({
selectId: (post) => post._id,
});
const normalized = normalize(response.data, postEntitiy);
this is my resopose.data
Array [
Object {
"__v": 5,
"_id": "6020b367cb94a91c9cd48c34",
"comments": Array [],
"date": "2021-02-08T03:43:35.742Z",
"likes": Array [
Object {
"_id": "60216bd341b3744ce4b13bee",
"user": "601f2d46017c85357800da96",
},
],",
},
]
and this is the error it throw
The entity passed to the `selectId` implementation returned undefined., You should probably provide
your own selectId implementation.,
The entity that was passed:, Object {
"undefined": Object {
"0": Object {
"__v": 5,
"_id": "6020b367cb94a91c9cd48c34",
"comments": Array [],
"date": "2021-02-08T03:43:35.742Z",
"likes": Array [
Object {
"_id": "60216bd341b3744ce4b13bee",
"user": "601f2d46017c85357800da96",
},
],
},
]
As this is written the postAdapter and the normalize are independent of each other. What's happening here is that normalize is attempting the map the data into entities, but it ends up with the key as undefined because it is using the default key of id which has an undefined value. Additionally you have an array of entities rather than just one.
The postAdapter then tries to find the selectId on this data which has already been normalized. But the postAdapter is not looking at a single post object -- it is looking at the nested data created by normalize. It is looking at the data structure Object { "undefined": Object { "0": Object {... from your error. The "undefined": is caused by the bad id attribute and the "0": is because it is an array.
How you do write this correctly kind of depends on what it is that you are trying to do. It's possible that normalize is not really necessary here. It looks like the likes in response.data are already returning a string id for the like and the user instead of the deeply-nested data that the normalizr package is designed to flatten. If you want to use normalize and have it work properly then you need to set the idAttribute on the options argument (docs link).
const postEntity = new schema.Entity('post', {}, {idAttribute: '_id'});
const normalized = data.map( o => normalize(o, postEntity));
idAttribute also accepts a function. You can create a reusable id extractor function that pulls the _id property from any object with {_id: string;}.
const myIdExtractor = (object : {_id: string;}) => object._id;
const postEntity = new schema.Entity('post', {}, {idAttribute: myIdExtractor});
const normalized = data.map( o => normalize(o, postEntity));
That same id extractor function also works with createEntityAdapter.
const postAdapter = createEntityAdapter({
selectId: myIdExtractor,
});

Query works at the console but not in code

My DynamoDB table alexas has this item with key "abc" as seen in the DynamoDB console below:
However, the following query returns no result:
const params = { TableName: "alexas",
KeyConditionExpression: "deviceId = :deviceId",
ExpressionAttributeValues: { ":deviceId": "abc"}
}
const docClient = new AWS.DynamoDB.DocumentClient();
docClient.query(params, (err, data) => {
if (err) {
console.error(err);
} else {
console.log(data);
}
});
The above code returns null for err and in data:
{ Items: [], Count: 0, ScannedCount: 0 }
I am new to the DynamoDB style of expressions. Is there anything wrong with my code which I took from here.
If instead of query, I used the scan method and just have TableName in params, I get the items in my table. This confirms that I am performing the operations on the correct table that has data.
The query returned no data because the key value does not match.
The item's deviceId is the string "abc" and not abc. Note the extra quotation marks.
The item was inserted using the DynamoDB console's Create editor and there is no need to include "" if the value is already expected to be of type string.
DynamoDB's Scan operation doesn't take a KeyConditionExpression - only the Query operation takes this parameter. Scan always scans the entire table, and has a FilterExpression to post-filter these results (however please note that you still pay for scanning the entire table).
For example, here is the official documentation of Scan: https://docs.aws.amazon.com/amazondynamodb/latest/APIReference/API_Scan.html
Check QueryAPI
const params = { TableName: "alexas",
KeyConditionExpression: "deviceId = :deviceId",
ExpressionAttributeValues: {
":devideId":{
S: "abc", // here
}
}
}
const docClient = new AWS.DynamoDB.DocumentClient();
docClient.query(params, (err, data) => {
if (err) {
console.error(err);
} else {
console.log(data);
}
});
ExpressionAttributeValues needs to be passed in a different manner.
Update:
Try using Exp attribute names, (I'm not sure if this will make a difference)
var params = {
TableName: "alexas",
KeyConditionExpression: "#d = :dId",
ExpressionAttributeNames:{
"#d": "email"
},
ExpressionAttributeValues: {
":dId": "abc"
}
};

Add timestamp in Firestore documents

I'm newbie to Firestore. Firestore docs says...
Important: Unlike "push IDs" in the Firebase Realtime Database, Cloud Firestore auto-generated IDs do not provide any automatic ordering. If you want to be able to order your documents by creation date, you should store a timestamp as a field in the documents.
Reference: https://firebase.google.com/docs/firestore/manage-data/add-data
So do I have to create key name as timestamp in document? Or created is suffice to fulfill above statement from Firestore documentation.
{
"created": 1534183990,
"modified": 1534183990,
"timestamp":1534183990
}
firebase.firestore.FieldValue.serverTimestamp()
Whatever you want to call it is fine afaik. Then you can use orderByChild('created').
I also mostly use firebase.database.ServerValue.TIMESTAMP when setting time
ref.child(key).set({
id: itemId,
content: itemContent,
user: uid,
created: firebase.database.ServerValue.TIMESTAMP
})
Use firestore Timestamp class, firebase.firestore.Timestamp.now().
Since firebase.firestore.FieldValue.serverTimestamp() does not work with add method from firestore. Reference
For Firestore
ref.doc(key).set({
created: firebase.firestore.FieldValue.serverTimestamp()
})
REALTIME SERVER TIMESTAMP USING FIRESTORE
import firebase from "firebase/app";
const someFunctionToUploadProduct = () => {
firebase.firestore().collection("products").add({
name: name,
price : price,
color : color,
weight :weight,
size : size,
createdAt : firebase.firestore.FieldValue.serverTimestamp()
})
.then(function(docRef) {
console.log("Document written with ID: ", docRef.id);
})
.catch(function(error) {
console.error("Error adding document: ", error);
});
}
All you need is to import 'firebase' and then call
firebase.firestore.FieldValue.serverTimestamp() wherever you need it. Be careful with the spelling though, its "serverTimestamp()". In this example it provides the timestamp value to 'createdAt' when uploading to the firestore's product's collection.
That's correct, like most database, Firestore doesn't store creation times. In order to sort objects by time:
Option 1: Create timestamp on client (correctness not guaranteed):
db.collection("messages").doc().set({
....
createdAt: firebase.firestore.Timestamp.now()
})
The big caveat here is that Timestamp.now()uses the local machine time. Therefore, if this is run on a client machine, you have no guarantee the timestamp is accurate. If you're setting this on the server or if guaranteed order isn't so important, it might be fine.
Option 2: Use a timestamp sentinel:
db.collection("messages").doc().set({
....
createdAt: firebase.firestore.FieldValue.serverTimestamp()
})
A timestamp sentinel is a token that tells the firestore server to set the time server side on first write.
If you read the sentinel before it is written (e.g., in a listener) it will be NULL unless you read the document like this:
doc.data({ serverTimestamps: 'estimate' })
Set up your query with something like this:
// quick and dirty way, but uses local machine time
const midnight = new Date(firebase.firestore.Timestamp.now().toDate().setHours(0, 0, 0, 0));
const todaysMessages = firebase
.firestore()
.collection(`users/${user.id}/messages`)
.orderBy('createdAt', 'desc')
.where('createdAt', '>=', midnight);
Note that this query uses the local machine time (Timestamp.now()). If it's really important that your app uses the correct time on the clients, you could utilize this feature of Firebase's Realtime Database:
const serverTimeOffset = (await firebase.database().ref('/.info/serverTimeOffset').once('value')).val();
const midnightServerMilliseconds = new Date(serverTimeOffset + Date.now()).setHours(0, 0, 0, 0);
const midnightServer = new Date(midnightServerMilliseconds);
The documentation isn't suggesting the names of any of your fields. The part you're quoting is just saying two things:
The automatically generated document IDs for Firestore don't have a natural time-based ordering like they did in Realtime Database.
If you want time-based ordering, store a timestamp in the document, and use that to order your queries. (You can call it whatever you want.)
This solution worked for me:
Firestore.instance.collection("collectionName").add({'created': Timestamp.now()});
The result in Cloud Firestore is:
Cloud Firestore Result
Try this one for Swift 4 Timestamp(date: Date())
let docData: [String: Any] = [
"stringExample": "Hello world!",
"booleanExample": true,
"numberExample": 3.14159265,
"dateExample": Timestamp(Date()),
"arrayExample": [5, true, "hello"],
"nullExample": NSNull(),
"objectExample": [
"a": 5,
"b": [
"nested": "foo"
]
]
]
db.collection("data").document("one").setData(docData) { err in
if let err = err {
print("Error writing document: \(err)")
} else {
print("Document successfully written!")
}
}
The way it worked with me, is just taking the timestamp from the snapshot parameter snapshot.updateTime
exports.newUserCreated = functions.firestore.document('users/{userId}').onCreate(async (snapshot, context) => {
console.log('started! v1.7');
const userID = context.params['userId'];
firestore.collection(`users/${userID}/lists`).add({
'created_time': snapshot.updateTime,
'name':'Products I ♥',
}).then(documentReference => {
console.log("initial public list created");
return null;
}).catch(error => {
console.error('Error creating initial list', error);
process.exit(1);
});
});
I am using Firestore to store data that comes from a Raspberry PI with Python. The pipeline is like this:
Raspberry PI (Python using paho-mqtt) -> Google Cloud IoT -> Google Cloud Pub/Sub -> Firebase Functions -> Firestore.
Data in the device is a Python Dictionary. I convert that to JSON.
The problem I had was that paho-mqtt will only send (publish) data as String and one of the fields of my data is timestamp. This timestamp is saved from the device because it accurately says when the measurement was taken regardless on when the data is ultimately stored in the database.
When I send my JSON structure, Firestore will store my field 'timestamp' as String. This is not convenient. So here is the solution.
I do a conversion in the Cloud Function that is triggered by the Pub/Sub to write into Firestore using Moment library to convert.
Note: I am getting the timestamp in python with:
currenttime = datetime.datetime.utcnow()
var moment = require('moment'); // require Moment
function toTimestamp(strDate){
return parsedTime = moment(strDate, "YYYY-MM-DD HH:mm:ss:SS");
}
exports.myFunctionPubSub = functions.pubsub.topic('my-topic-name').onPublish((message, context) => {
let parsedMessage = null;
try {
parsedMessage = message.json;
// Convert timestamp string to timestamp object
parsedMessage.date = toTimestamp(parsedMessage.date);
// Get the Device ID from the message. Useful when you have multiple IoT devices
deviceID = parsedMessage._deviceID;
let addDoc = db.collection('MyDevices')
.doc(deviceID)
.collection('DeviceData')
.add(parsedMessage)
.then ( (ref) => {
console.log('Added document ID: ', ref.id);
return null;
}).catch ( (error) => {
console.error('Failed to write database', error);
return null;
});
} catch (e) {
console.error('PubSub message was not JSON', e);
}
// // Expected return or a warning will be triggered in the Firebase Function logs.
return null;
});
Firestone method does not work. Use Timestamp from java.sql.Timestamp and don't cast to string.. Then firestone formats it properly. For example to mark a now() use:
val timestamp = Timestamp(System.currentTimeMillis())
multiple ways to store time in Firestore
firebaseAdmin.firestore.FieldValue.serverTimestamp() method. The actual timestamp will be computed when the doc is written to the Firestore.
while storing it looks like this:
firebaseAdmin.firestore.Timestamp.now() method.
while storing it looks like this:
For both the methods, next time you fetch data it will return Firestore Timestamp object:
So, you first need to convert it to native js Date object and then you can perform methods on it like toISOString().
export function FStimestampToDate(
timestamp:
| FirebaseFirestore.Timestamp
| FirebaseFirestore.FieldValue
): Date {
return (timestamp as FirebaseFirestore.Timestamp).toDate();
}
Store as unix timestamp Date.now, it'll be stored as number i.e. 1627235565028 but you won't be able to see it as readable Date in firestore db.
To query on this Firestore field, you need to convert the date to timestamp and then query.
Store as new Date().toISOString() i.e. "2021-07-25T17:56:40.373Z" but you won't be able to perform date range query on this.
I prefer the 2nd or 3rd way.
According to the docs, you can "set a field in your document to a server timestamp which tracks when the server receives the update".
Example:
import { updateDoc, serverTimestamp } from "firebase/firestore";
const docRef = doc(db, 'objects', 'some-id');
// Update the timestamp field with the value from the server
const updateTimestamp = await updateDoc(docRef, {
timestamp: serverTimestamp() // this does the trick!
});
Sharing what worked for me after googling for 2 hours, for firebase 9+
import { serverTimestamp } from "firebase/firestore";
export const postData = ({ name, points }: any) => {
const scoresRef = collection(db, "scores");
return addDoc(scoresRef, {
name,
points
date: serverTimestamp(),
});
};
Swift 5.1
...
"dateExample": Timestamp(date: Date()),
...
The newest version from Firestore you should use it as follow
import { doc, setDoc, Timestamp } from "firebase/firestore";
const docData = {
...
dateExample: Timestamp.fromDate(new Date("December 10, 1815"))
};
await setDoc(doc(db, "data", "one"), docData);
or for sever timestamp
import { updateDoc, serverTimestamp } from "firebase/firestore";
const docRef = doc(db, 'objects', 'some-id');
const updateTimestamp = await updateDoc(docRef, {
timestamp: serverTimestamp()
});

Resources