I'm having a bit of trouble with a Firebase query, mainly due to the size of the dataset I am querying.
What I would like to achieve is:
Find all tshirts where brandStartsWith = 'A' and salesRank is between 1 and 100
I've started to pad this out, but I am running into an issue whereby I can't seem to get the data due to having over 300,000 records within t-shirts.
If call it within React when the page loads, after a while I get the following error in console:
Uncaught RangeError: Invalid string length
Here is the code I am using to get me started, but I'm not sure where to go. Looking at the solutions on this question it seems I need to download the data per my query below, and then sort it on the client side. Something I cant seem to do
firebase.database().ref('tshirts')
.orderByChild('brandStartsWith')
.equalTo('A')
.once('value', function (snapshot) {
console.log(snapshot.val())
})
You're going to need to create a combined key as you can only do one where clause at a time.
{
"tShirts" : {
"brandStartsWith" : 'A',
"salesRank" : 5
"brandStartsWith_salesRank" = 'A_00005' //pad for as many sales ranks as you have
}, {
"brandStartsWith" : 'B',
"salesRank" : 108
"brandStartsWith_salesRank" = 'B_00108' //pad for as many sales ranks as you have
}, {
"brandStartsWith" : 'C',
"salesRank" : 52
"brandStartsWith_salesRank" = 'C_00052' //pad for as many sales ranks as you have
}
}
This will allow you to do this query:
firebase.database().ref('tshirts')
.orderByChild('brandStartsWith_salesRank')
.startAt('A_00001')
.endAt('A_00100')
.once('value', function (snapshot) {
console.log(snapshot.val())
})
Don't forget to update your rules to .index brandStartsWith_salesRank
Related
I'm running a database on sequelize and sqlite and I use soft-deletes to basically archive the data.
I'm aware that with .findAll(paranoid: false) I can find all rows including the soft deleted ones. However I would like to find ONLY the soft-deleted ones.
Is there any way to achieve this? Or is there perhaps a way to do "set operations" with two data results, like finding the relative complement of one in the other?
For this, you can add the following condition to where.
deletedAt: { [Op.not]: null }
For example like this:
const projects = await db.Project.findAndCountAll({
paranoid: false,
order: [['createdAt', 'DESC']],
where: { employer_id: null, deletedAt: { [Op.not]: null } },
limit: parseInt(size),
offset: (page - 1) * parseInt(size),
});
I have an Elastic Search cluster with a lot of nice data, that I have created some nice Kibana dashboards for.
For the next level I decided to take a look at scripted fields to make some of the dashboards even nicer.
I want to translate some of the numeric fields into more easily understandable text values. As an example of what I want to do and what I have tried I will use the http response status code field, that most will understand quite easily but also illustrates the problem.
We log the numeric status code (200, 201, 302, 400, 404, 500 etc.) I can create a data table visualization that tells me the count for each of these status codes. But I would like to display the text reason in my dashboard.
I can create a painless script with a lot of IF statements like this:
if (doc['statuscode'].value == 200) {return "OK";}
if (doc['statuscode'].value == 201) {return "Created";}
if (doc['statuscode'].value == 400) {return "Bad Request";}
return doc['statuscode'].value;
But that isn't very nice I think.
But since I will most likely have about 150 different values and that list won't change very often, so I can live with maintaining a static map. But I haven't found any examples of implementing a map or dictionary in painless scripting.
I was thinking of implementing something like this:
Map reasonMap;
reasonMap[200] = 'OK';
reasonMap[201] = 'Created';
def reason = reasonMap[doc['statuscode'].value];
if (reason != null)
{
return reason;
}
return doc['statuscode'].value;
I haven't been able to make this code work though. The question is also if this will perform well enough for a map with up to 150 values.
Thanks
EDIT
After some trial and error... and a lot of googling, this is what I came up with that works (notice that the key needs to start with a character and not a number):
def reasonMap =
[
's200': 'OK',
's201': 'Created'
];
def key = 's' + doc['statuscode'].value
def reason = reasonMap[key];
if (reason != null)
{
return reason;
}
return doc['statuscode'].value;
Should it be
def reason = reasonMap[doc['statuscode']value];
It will perform well with a Map of 150 values.
Im trying to show the total number of people in each geography when they hover over using crossfilter, but my current code is only showing the total of all geographies. So what is the equivalent in crossfilter to the sql query: SELECT COUNT(*) GROUP BY dma
This is my code so far
//geography that is being hovered over, getting dma name and removing everything that is after the comma
sel_geog = layer.feature.properties.dma_1;
sel_geog = sel_geog.split(",")[0];
console.log(sel_geog);
//crossfilter to get total number of people of each geography
var dmaDim = voter_data.dimension(function(d) {return d.dma == sel_geog}),
dma_grp = dmaDim.groupAll().reduceCount().value();
console.log(dma_grp);
Crossfilter isn't meant to be used in a way where you are building new dimensions and groups for each user interaction. It's meant to build dimensions and groups before interactions take place and then update them quickly when filtering based on user interactions.
It's not really clear from this question what your data looks like or what you are trying to do, but you probably want to create dimensions and group for your dma property and then build your map based on that:
var voter_data = crossfilter(my_data);
var dmaDim = voter_data.dimension(function(d) { return d.dma; });
var dmaGroup = dmaDim.group();
At this point dmaGroup.all() will be an array of objects that looks like { key: 'dmaKey', value: 10 } where 10 is the count of all records where d.dma === 'dmaKey'. There are lots of ways you can aggregate differently with Crossfilter, but that may get you started.
EDIT: I accidentally misrepresented the problem when trying to pare-down the example code. A key part of my code is that I am attempting to sort the array after adding elements to it. The hang appears on sort, not insert. The following abstracted code will consistently hang:
<?=
local('a' = array)
#a->insert('test1' = map('a'='1'))
#a->insert('test2' = map('b'='2')) // comment-out to make work
#a->sort
#a
?>
I have a result set for which I want to insert a pair of values into an array for each unique key, as follows:
resultset(2) => {
records => {
if(!$logTypeClasses->contains(field('logTypeClass'))) => {
local(i) = pair(field('logTypeClass'), map('title' = field('logType'), 'class' = field('logTypeClass')))
log_critical(#i)
$logTypeClasses->insert(#i) // Lasso hangs on this line, will return if commented-out
}
}
}
Strangely, I cannot insert the #i local variable into thread variable without Lasso hanging. I never receive an error, and the page never returns. It just hangs indefinitely.
I do see the pairs logged correctly, which leads me to believe that the pair-generating syntax is correct.
I can make the code work as long as the value side of the pair is not a map with values. In other words, it works when the value side of the pair is a string, or even an empty map. As soon as I add key=value parameters to the map, it fails.
I must be missing something obvious. Any pointers? Thanks in advance for your time and consideration.
I can verify the bug with the basic code you sent with sorting. The question does arise how exactly one sorts pairs. I'm betting you want them sorted by the first element in the pair, but I could also see the claim that they should be sorted by last element in the pair (by values instead of by keys)
One thing that might work better is to keep it as a map of maps. If you need the sorted data for some reason, you could do map->keys->asArray->sort
Ex:
local(data) = map('test1' = map('a'=2,'b'=3))
#data->insert('test2' = map('c'=33, 'd'=42))
local(keys) = #data->keys->asArray
#keys->sort
#keys
Even better, if you're going to just iterate through a sorted set, you can just use a query expression:
local(data) = map('test1' = map('a'=2,'b'=3))
#data->insert('test2' = map('c'=33, 'd'=42))
with elm in #data->eachPair
let key = #elm->first
let value = #elm->second
order by #key
do { ... }
I doubt you problem is the pair with map construct per se.
This test code works as expected:
var(testcontainer = array)
inline(-database = 'mysql', -table = 'help_topic', -findall) => {
resultset(1) => {
records => {
if(!$testcontainer->contains(field('name'))) => {
local(i) = pair(field('name'), map('description' = field('description'), 'name' = field('name')))
$testcontainer->insert(#i)
}
}
}
}
$testcontainer
When Lasso hangs like that with no feedback and no immediate crash it is usually trapped in some kind of infinite loop. I'm speculating that it might have to do with Lasso using references whenever possible. Maybe some part of your code is using a reference that references itself. Or something.
I trying to learn captures in Lasso 9, but I am struggling to figure out how to access the #1 local variable from within a conditional that's inside an array->forEach capture. Maybe my approach is all wrong. Is there a reference to the parent capture that I need to use? Following is the working code:
define paramstovars() => {
local(p = web_request->params)
#p->foreach => {
local(i = #1)
if(#i->type == 'pair') => {
var(#i->first->asstring = #i->second->asstring)
}
}
}
Following is the code I am trying to get working without relying on a redundant local variable definition:
define paramstovars() => {
local(p = web_request->params)
#p->foreach => {
if(#1->type == 'pair') => {
var(#1->first->asstring = #1->second->asstring)
}
}
}
In this second example, I receive an error that Position was out of range: 1 max is 0 (Error Code -1) on the line calling var().
Obvious security concerns with this custom method aside, what's the most efficient way to make #1 available inside nested conditionals?
#1 is replaced within each capture — so yes, you will need to assign it to another local in order to use it in deeper captures. If you need to work with the local again try using query expressions instead:
with i in web_request->params do {
if(#i->type == 'pair') => {
var(#i->first->asstring = #i->second->asstring)
}
}
Also, I wouldn't recommend settings variables in this fashion — it posses a security risk. It would be better to store the parameters in a single variable and then potentially set specific variables from that. There's a set of tags that does something similar here: getparam / postparam
It is my experience that #1 is consumed when called first time. At least I have never ben able to call it twice in the same capture.
If I need the value more than once I make it a local first. As you do in your example 1.
Some experiments later.
You can call #1 several times, contrary to what I wrote, but what trips your effort is that you have a capture inside the capture (the conditional).
The second capture will have it's own input params.
Here's a working, tested example to do what you want to do:
local(
myarray = array(1, 2 = 'two', 3 = 'four', 4),
mypairs = map
)
#myarray -> foreach => {
if(#1-> isa(::pair)) => {
#mypairs -> insert(#1 -> first -> asstring = #1 -> second -> asstring)
}(#1)
}
#my pairs
The result will be map(2 = two, 3 = four)
The trick is the sending of the foreach param to the conditional capture : `{some code}(#1)
Now, with all that worked out. I recommend that you take a look at Ke Carltons latest addition to tagswap. It will solve the same problem way better than creating a bunch of dynamic vars as you are attempting to do:
www.lassosoft.com/tagswap/detail/web_request_params