findAll() returns empty with WHERE option - sqlite

First question on StackOverflow, long time reader first time poster or whatever people say.
I'm developing a Discord bot in my free time using Discord.js, and I'm using Sequelize to interface with a local SQLite database. I can insert data into it just fine-- however, I can't seem to delete any of the records I add. Relevant piece of code is below, which I believe to be self-contradictory:
const query3 = await Towers.findAll({
attributes: ['channelID']
});
console.log(JSON.stringify(query3)); //returns the one Tower
console.log(query3[0].channelID === channel); //returns true(!)
const query2 = await Towers.findAll({
attributes: ['channelID'],
where: {channelID: channel}
});
console.log(JSON.stringify(query2)); //returns empty
//DELETE FROM Towers WHERE channelID = channel;
const query = await Towers.destroy({
where: {channelID: channel}
});
console.log(query); //returns 0, expected behavior given query2 returns empty
I'm attempting to delete a record from a table named Towers by passing a channel ID to it, which is expected to be unique. However, when I make any query on the database with a WHERE clause, the query returns an empty set-- even when, in this example, I sanity-checked and verified that the value I'm attempting to remove is present in the table. This occurs for both findAll() and findOne() as long as a WHERE clause is present.
(For posterity, I've double and triple checked that channelID was spelled correctly and with the correct capitalization in all instances.)
I'm happy to provide any more information if needed!
EDIT: As requested, the model definition...
const Towers = sequelize.define('Towers', {
serverID: {
type: Sequelize.INTEGER,
allowNull: false,
},
channelID: {
type: Sequelize.INTEGER,
unique: true,
allowNull: false,
},
pattern: Sequelize.STRING,
height: Sequelize.INTEGER,
delay: Sequelize.BOOLEAN,
});
channel in the snippet in the original post is defined as parseInt(interaction.options.getChannel('channel').id).

To anyone who happens to have the same issue I did, the answer is a doozy.
I wanted to store Discord server and channel ID's as integers, even though they're returned to you as strings when calling the API. As it turns out, Discord snowflakes are higher than float64 precision, which JS uses. When parsing the strings into integers to insert them into my table, the value changed from the intended number, and I was creating erroneous records.
In my case (with the actual numbers obfuscated) interaction.options.getChannel('channel').id returned "837512533934092340", while parseInt(interaction.options.getChannel('channel').id returned 837512533934092300. The number I was adding to the table was somehow 40 less!
I'm not sure if this could be fixed by using BigInt, but since it's going into a different structure anyway, I just shrugged and changed the serverId and channelId types to Sequelize.STRING in the model definition and removed the parseInt calls. Works like a charm now.
Good opportunity to shake my fist at JS though.

Related

QML Firebase startAt returns undefined

I am working on a 'typeahead’ type function which will check my Database with the current typed text to provide search suggestions of users using Felgo.
Here is the link for Felgos Firebase documentation
As to not search every entry I am looking to use the startAt and limitTo for a lower data use.
However when applying the startAt my searches only return undefined, I have tried testing this by changing my startAt from a variable to explicit data but this still only returns undefined.
My function is below:
function searchUsers(searchString) {
db.getValue("public/nameList/", {
orderByChild: true,
startAt: searchString, //searchString is a variable with my .currentText to search.
limitToFirst: 10,
}, function(success, key, value) {
if(success) {
searchArr = []
searchArr = value
console.debug("Read user value for key", key, "from DB:", value)
}
})
}
I have also tried by passing my var searchString through JSON.stringify(searchString) and also return undefined!
Removing the startAt: query entirely returns the entire result of nameList as expected, but no matter how I try to implement my startAt it always returns undefined.
A sample of my nameList JSON is:
nameList: {
"EddieLaw245" : 530343772383,
"EddieLawrence91" : 530343772385,
"EdwardL91" : 530343772386,
"EdwardLaw" : 530343772384,
"Edwardlawrence91" : 530343772380,
"JoBrownLondon" : 530343772381,
"KatiePrescottHair" : 543592635596,
"Tracey-Sweeting" : 530343772382
}
So with the above example, When I type E it should remove the last 3 entries, and so on.
The problem is that you're specifying orderByChild: true. If we look at the documentation of that:
orderByChild: If present, the queried object will have its properties ordered by values at sub-paths defined by the value of this property. Ordering by child properties makes the filter properties startAt, endAt and equalTo filter by the child property values
It may not be immediately clear from this, but orderByChild allows you to order the results on a property value under each of those nodes. So your code tries to order the child nodes on the value of a property true, which isn't possible (and should actually generate a compile-time error in the library) as the nodes under nameList don't have any child properties of their own. They merely have a key and a value.
What you're looking for is orderByKeys, which orders the child nodes on their keys. So:
db.getValue("public/nameList/", {
orderByKeys: true,
startAt: searchString,
limitToFirst: 10,
}
You'll typically also want to specify an endAt value, to ensure your type-ahead only shows values that start with the search string. If you only allow ASCII values in the keys, the simplest way to do this is:
startAt: searchString,
endAt: searchString + "~",
The ~ here is no magic operator, but merely the last ASCII characters. If you want to allow a broader character set, you'll need to use the last character in that character set - for example \uF7FF is the last code point for Unicode.
Update from OP
Though I'm certian Franks correct with typical Firebase usage; I suspect due to the Felgo plugin I am using the full solution has a slight adjustment;
db.getValue("public/nameList/", {
"orderByKey": true,
"startAt": searchString,
"endAt": searchString+"~",
"limitToFirst": 10,
}, function(success, key, value) {....}
})
Notes on the above - my filters/queries are surrounded by quotation marks "startAt", also instead of orderByKeys, I have used orderByKey

Retrieve values from firebase database in conversation flow

I am trying to grab information from my firebase database after a particular intent is invoked in my conversation flow.
I am trying to make a function which takes a parameter of user ID, which will then get the highscore for that user, and then say that users highscore back to them.
app.intent('get-highscore', (conv) => {
var thisUsersHighestscore = fetchHighscoreByUserId(conv.user.id);
conv.ask('your highest score is ${thisUsersHighestScore}, say continue to keep playing.');
});
function fetchHighscoreByUserId(userId){
var highscoresRef = database.ref("highscores");
var thisUsersHighscore;
highscoresRef.on('value',function(snap){
var allHighscores= snap.val();
thisUsersHighscore = allHighscores.users.userId.highscore;
});
return thisUsersHighscore;
}
An example of the data in the database:
"highscores" : {
"users" : {
"1539261356999999924819020" : {
"highscore" : 2,
"nickname" : "default"
},
"15393362381293223232222738" : {
"highscore" : 78,
"nickname" : "quiz master"
},
"15393365724084067696560" : {
"highscore" : "32",
"nickname" : "cutie pie"
},
"45343453535534534353" : {
"highscore" : 1,
"nickname" : "friendly man"
}
}
}
It seems like it is never setting any value to thisUsersHighScore in my function.
You have a number of issues going on here - both with how you're using Firebase, how you're using Actions on Google, and how you're using Javascript. Some of these issues are just that you could be doing things better and more efficiently, while others are causing actual problems.
Accessing values in a structure in JavaScript
The first problem is that allHighscores.users.userId.highscore means "In an object named 'allHighscores', get the property named 'users', from the result of that, get the property named 'userId'". But there is no property named "userId" - there are just a bunch of properties named after a number.
You probably wanted something more like allHighscores.users[userId].highscore, which means "In an object named 'allHighscores', get the property named 'users', fromt he result of that, get the property named by the value of 'userId'".
But if this has thousands or hundreds of thousands of records, this will take up a lot of memory. And will take a lot of time to fetch from Firebase. Wouldn't it be better if you just fetched that one record directly from Firebase?
Two Firebase Issues
From above, you should probably just be fetching one record from Firebase, rather than the whole table and then searching for the one record you want. In firebase, this means you get a reference to the path of the data you want, and then request the value.
To specify the path you want, you might do something like
var userRef = database.ref("highscores/users").child(userId);
var userScoreRef = userRef.child( "highscore" );
(You can, of course, put these in one statement. I broke them up like this for clarity.)
Once you have the reference, however, you want to read the data that is at that reference. You have two issues here.
You're using the on() method, which fetches the value once, but then also sets up a callback to be called every time the score updates. You probably don't need the latter, so you can use the once() method to get the value once.
You have a callback function setup to get the value (which is good, since this is an async operation, and this is the traditional way to handle async operations in Javascript), but you're returning a value outside of that callback. So you're always returning an empty value.
These suggest that you need to make fetchHighScoreByUserId() an asynchronous function as well, and the way we have to do this now is to return a Promise. This Promise will then resolve to an actual value when the async function completes. Fortunately, the Firebase library can return a Promise, and we can get its value as part of the .then() clause in the response, so we can simplify things a lot. (I strongly suggest you read up on Promises in Javascript and how to use them.) It might look something like this:
return userScoreRef.once("value")
.then( function(scoreSnapshot){
var score = scoreSnapshot.val();
return score;
} );
Async functions and Actions on Google
In the Intent Handler, you have a similar problem as above. The call to fetchHighScoreByUserId() is async, so it doesn't finish running (or returning a value) by the time you call conv.ask() or return from the function. AoG needs to know to wait for an async call to finish. How can it do that? Promises again!
AoG Intent Handlers must return a Promise if there is an asyc call involved.
Since the modified fetchHighScoreByUserId() returns a Promise, we will leverage that. We'll also set our response in the .then() part of the Promise chain. It might look something like this:
app.intent('get-highscore', (conv) => {
return fetchHighscoreByUserId(conv.user.id)
.then( function(highScore){
conv.ask(`Your highest score is ${highScore}. Do you want to play again?`);
} );
});
Two asides here:
You need to use backticks "`" to define the string if you're trying to use ${highScore} like that.
The phrase "Say continue if you want to play again." is a very poor Voice User Interface. Better is directly asking if they want to play again.

Firestore: how to perform a query with inequality / not equals

I want select from Firestore collection just articles written NOT by me.
Is it really so hard?
Every article has field "owner_uid".
Thats it: I JUST want to write equivalent to "select * from articles where uid<>request.auth.uid"
TL;DR: solution found already: usages for languages/platforms: https://firebase.google.com/docs/firestore/query-data/queries#kotlin+ktx_5
EDIT Sep 18 2020
The Firebase release notes suggest there are now not-in and != queries. (Proper documentation is now available.)
not-in finds documents where a specified field’s value is not in a specified array.
!= finds documents where a specified field's value does not equal the specified value.
Neither query operator will match documents where the specified field is not present. Be sure the see the documentation for the syntax for your language.
ORIGINAL ANSWER
Firestore doesn't provide inequality checks. According to the documentation:
The where() method takes three parameters: a field to filter on, a comparison operation, and a value. The comparison can be <, <=, ==, >, or >=.
Inequality operations don't scale like other operations that use an index. Firestore indexes are good for range queries. With this type of index, for an inequality query, the backend would still have to scan every document in the collection in order to come up with results, and that's extremely bad for performance when the number of documents grows large.
If you need to filter your results to remove particular items, you can still do that locally.
You also have the option of using multiple queries to exclude a distinct value. Something like this, if you want everything except 12. Query for value < 12, then query for value > 12, then merge the results in the client.
For android it should be easy implement with Task Api.
Newbie example:
FirebaseFirestore db = FirebaseFirestore.getInstance();
Query lessQuery = db.collection("users").whereLessThan("uid", currentUid);
Query greaterQuery = db.collection("users").whereGreaterThan("uid", currentUid);
Task lessQuery Task = firstQuery.get();
Task greaterQuery = secondQuery.get();
Task combinedTask = Tasks.whenAllSuccess(lessQuery , greaterQuery)
.addOnSuccessListener(new OnSuccessListener<List<Object>>() {
#Override
public void onSuccess(List<Object> list) {
//This is the list of "users" collection without user with currentUid
}
});
Also, with this you can combine any set of queries.
For web there is rxfire
This is an example of how I solved the problem in JavaScript:
let articlesToDisplay = await db
.collection('articles')
.get()
.then((snapshot) => {
let notMyArticles = snapshot.docs.filter( (article) =>
article.data().owner_uid !== request.auth.uid
)
return notMyArticles
})
It fetches all documents and uses Array.prototype.filter() to filter out the ones you don't want. This can be run server-side or client-side.
Updating the answer of Darren G, which caused "TypeError: Converting circular structure to JSON". When we perform the filter operation, the whole firebase object was added back to the array instead of just the data. We can solve this by chaining the filter method with the map method.
let articles = []
let articlesRefs = await db.collection('articles').get();
articles = articlesRefs.docs
.filter((article) => article.data.uid !== request.auth.uid) //Get Filtered Docs
.map((article) => article.data()); //Process Docs to Data
return articles
FYI: This is an expensive operation because you will fetching all the articles from database and then filtering them locallly.
Track all user id in a single document (or two)
filter unwanted id out
Use "where in"
var mylistofidwherenotme = // code to fetch the single document where you tracked all user id, then filter yourself out
database.collection("articles").where("blogId", "in", mylistofidwherenotme)
let query = docRef.where('role','>',user_role).where('role','<',user_role).get()
This is not functioning as the "not equal" operation in firestore with string values
You can filter the array of objects within the javascript code.
var data=[Object,Object,Object] // this is your object array
var newArray = data.filter(function(el) {
return el.gender != 'Male';
});

How to use startAt() in Firebase query?

Let's suppose above firebase database schema.
What I want to is retrieve messages which after "15039996197" timestamp. Each of message object has a createdAt property.
How can I get messages only after specific timestamp? In this case, last two messages are what I want to retrieve.
I tried firebaseb.database().ref(`/rooms/$roomKey/messages`).startAt('15039996197') but failed. It return 0. How can I do this?
The case with your query is that it's expecting that the message node should have a number value to start with, in that case you want a child node with the name createdAt. So in that case you must specify that you want to order by createdAt, thus you need to do this
firebase.database().ref(`/rooms/$roomKey/messages`).orderByChild('createdAt').startAt('15039996197').on(//code);
This way it'll return all nodes inside message that has a child named createdAt an it starts at 15039996197. Ordering your query may be a little bad for performance, for that i sugest taking a look at .indexOn rule.
For more information take a look here.
Hope this helps.
Firebase Data Retrieval works node by node.
So whatever data you want to get, the entire node is traversed.
So in your case to get any message your complexity would be O(number of messages).
You would want to restructure the way you are storing the data and put createdAt in Node instead of Child.
If your database structure looks like that, you can use:
firebase.database()
.ref(`/rooms/$roomKey/messages`)
.orderByChild('createdAt')
.startAt('15039996197').on('value', snapshot => { /* your code here */ });
But if you work with a lot of data entries, you can also name the item key with the timestamp, instead of storing the timestamp in your data. This saves a little data on your database.
firebase.database().ref(`${rooms}/${roomKey}/${timestamp}`).set("value");
To retrieve the data in that case instead of orderByChild('createdAt'), you'll use orderByKey(). Because firebase keys are of the string type, you need to make shure to parse the timestamp in the .startAt() to a string.
It will then look something like this:
firebase.database()
.ref(`/rooms/$roomKey/messages`)
.orderByKey()
.startAt(`${15039996197}`).on('value', snapshot => { /* your code here */ });
You can do something like that:
firebase.database().ref('/rooms/$roomKey/messages')
.orderByChild('createdAt')
.startAt('150399')
.endAt('1503999\uf8ff')
.on('value', function (snapshot) {
var key = snapshot.key,
data = snapshot.val();
console.log(key + ': ' + JSON.stringify(data))
});
Be sure to set endAt().

How to parse a collection's sub-object to find a unique result from many possibilities?

In my user's schema, I have a TokAuth Array with token sub-objects (like multiple mails addresses).
So in a method, when I search the tokens in the current user :
var id = Meteor.userId();
var usercurrent = Meteor.users.findOne({_id: id}, {fields: {"TokAuth": 1}});
var userToken = usercurrent.TokAuth.token;
I got in console.log(userToken)
[ 'fyAyXkXYrQdAlNpjuQfJ8RLU2TpfVGLnptlBs-m1h7xk',
I20170224-20:36:23.202(1)? 'YTwtUbhNTgiEfzFbJq7mESnOoOHeLYxWlqEeJJIG_GiV',
I20170224-20:36:23.206(1)? 'ViA4ydDITJtHDi2c_sArkNtpRYTjFqGL1ju2v00_-rFJ',
I20170224-20:36:23.206(1)? '51ImZcxRADLJr-FPCUL7EFGnTZYjHSZk3XxdqtBV2_fd',
I20170224-20:36:23.207(1)? 'S5aEvqjJ5zTUJqLFCPY1aZ1ZhsQppZTJtYKULM9aS2B3',
I20170224-20:36:23.207(1)? 'mhBs3oxHf2SxZfu2vCZhtiyPfg25fKMY8bKMZD8fx6IG',
I20170224-20:36:23.207(1)? '-rv0FiP-lxoqe8INyCJASV6rZpbgy3euEqB9sO9HsZSV',
I20170224-20:36:23.207(1)? 'zacr6_VBjHTsArov1LmQyZFLwI40fx4J7sygpLosTrli' ]
Beside, I've got a var who is equal to the last token in the userToken sub-object (that's of course expected : not to be the last one, but to be in the sub-object).
console.log (editAuth);
zacr6_VBjHTsArov1LmQyZFLwI40fx4J7sygpLosTrli
So how can I parse userToken to find a match with editAuth? If userToken was just a String, it will be simple but here...
Thanks
Is there a reason you are storing all the tokens as an array as opposed to just updating a single string each time?
That aside, you can check if an array contains a value by using the handy underscore function _.contains
Example:
_.contains( userToken, editAuth ); //returns true or false
In this case, you are simply trying to search for a string within an array of strings. #Sean already provided one solution.
If you are using the meteor ecmascript package then you can just simply use the native Array.includes method.
userToken.includes(editAuth);
On a side note, after using ECMAScript 2015+ for some time now, I find that I can use the native API for almost everything that I used to use underscore or lodash for. Check it out!

Resources