Crossfilter for Date for string values - crossfilter

I've a JSON model that contains strings instead of dates (the model is generated via T4TS, so I cannot change that).
The code is currently using an expanded model extending the original json, where the dates are recalculated on new fields.
I was wondering if it would be possible to apply the filters on the fields being string without adding that additional step of extending the model.
private makeNumeric(label: string, property: string) {
return {
label: label,
key: property,
prepareDimension: (crossfilter) => (CrossfilterUtils.makeNumeric(crossfilter, property)),
prepareGroup: (dimension) => {
if (!this.values[property]) {
var group = CrossfilterUtils.makeNumericGroup(dimension);
this.values[property] = group;
}
return this.values[property];
},
valuesAreOrdinal: false
};
}
I haven't used the crossfilter library much before and by looking at the documentation I can't seem to reconcile it with the code (heritage code, to put it that way).
The incoming date format looks like this: "2020-10-22T07:26:00Z"
The typescript model I'm working with is like this:
interface MyModel {
...
CreatedDate?: string;
}
Any idea?

The usual pattern in JavaScript is to loop through the data and do any conversions you need:
data.forEach(function(d) {
d.date = new Date(d.date);
d.number = +d.number;
});
const cf = crossfilter(data);
However, if this is not allowed due to TS, you can also make the conversions when creating your dimensions and groups:
const cf = crossfilter(data);
const dateDim = cf.dimension(d => new Date(d.date));
const monthGroup = dateDim.group(date => d3.timeMonth(date))
.reduceSum(d => +d.number);
I find this a little less robust because you have to remember to do this everywhere. It's a little harder to reason about the efficiency since you have to trust that crossfilter uses the accessors sparingly, but I don't recall seeing this be a problem in practice.

Related

knockout write binding without observable

In knockout, say we have something like this:
var person = {
name: "The H. Dude",
telecom: [
"mailto:dude#host.com",
"tel:+1-987-654-3210"
]
}
and I have a data binding elements like this:
<label>Name: <input type="text" data-bind="value: name"/></label>
<label>Phone:<input type="text" data-bind="value: telecom.find(url => url.startsWith('tel:'))"/></label>
<label>Email:<input type="text" data-bind="value: telecom.find(url => url.startsWith('mailto:'))"/></label>
This works alright. However, this would hit the user over the head with the URL scheme prefix.
So, what we would like to do is something like this:
data-bind="value: telecom.find(url => url.startsWith('tel:')).substring('tel:'.length)"
and
data-bind="value: telecom.find(url => url.startsWith('mailto:')).substing('mailto:'.length)"
respectively.
And that works just fine for a read-only property, which I might just display. But when I type a new phone number in, the expression ends in a function call, which can't be written to, and of course substring function doesn't know how to work backwards to prepend the "tel:" or "mailto:" prefix before the user-entered value gets written to the object.
I have worked deep down in an XForms engine and before that I had made my own UI framework with a path language similar to XPath, used precisely for two-way data-binding. There I had invented the notion of a "conversion" which was a pair of functions, in this case you might say:
telPrefixConversion = {
prefix: "tel:",
forward: function(value) { return value.substring(this.prefix.length); }
backward: function(value) { return prefix + value; }
}
And I'm thinking this would be super-useful in knockout too. Then on the binding I could just say
data-bind="{value: telecom.find(url => url.startsWith('mailto:')), conversion: telPrefixConversion}"
and now knockout-3.5.0.debug.js line 2842 could do this:
if (twoWayBindings[key] && (writableVal = getWriteableValue(val))) {
// For two-way bindings, provide a write method in case the value
// isn't a writable observable.
var writeKey = typeof twoWayBindings[key] == 'string' ? twoWayBindings[key] : key;
propertyAccessorResultStrings.push("'" + writeKey + "':function(_z){" + writableVal + "=_z}");
}
that last line could change to
propertyAccessorResultStrings.push("'" + writeKey + "':function(_z){" + conversion.backward(writableVal) + "=_z}");
Now I can already think of a way to do that by using a computed observable instead, but it seems heavy weight for something like that. This conversion principle is very powerful as it can also convert complex objects into UI string representations and on the way back it goes into the complex object again.
I am so tempted to implement that in knockout too, since I have done it twice already on other data binding UI tools. Should I go ahead? Or am I missing a simple feature like this?
ATTEMPT 1 - Computed Observable: I have since used computed observables to solve my immediate need, but I found out that this would work when you have more than one telephone number in some kind of repeated groups. For example, let's say you have a list of friends with name, email, and phone number adding computed scratchpad properties just to convert a value to and from string representation is not good.
The fist answer here also suggests computed observable, but both my initial attempt and what is suggested in that answer is too special. We want to have the ability to do it anywhere, regardless if it is a property of one or the other object no matter whether they are also repeated in an array.
So I am thinking of something like this:
class Conversion {
constructor(object, property) {
let self = this;
object[property] = ko.computed({
read: function() {
return self.m2v(ko.util.unwrapObservable(this[property]));
},
write: function(value) {
this[property](m2v(value)); // <<<< infinite recursion????
},
owner: object
});
}
model2view(modelValue) { throw new Error("undefined abstract function called"); }
view2model(viewValue) { throw new Error("undefined abstract function called"); }
}
These model2view and view2model functions can then be overridden to deal with my prefixes, or with date formats as in this other question, etc.
The problem I am stuck on is this:
we replaced the actual value with the observable
when assigning the view value to the observable we would be entering this cycle
We would still need some new property where we store the value for the view separate from the property that holds the actual model value. And that's what I'm trying to avoid by supplying this function pair instead to some other binding option somehow.
ATTEMPT 2 - Extender
I found the observable extenders might almost do the trick:
ko.extenders.prefixConversion = function(target, prefix) {
const read = function() { return target().substring(prefix.length); };
const result = ko.pureComputed({
read: read,
write: function(value) { target(prefix + value); }
});
result(read());
return result;
}
this is used together with the following initializer, and also with these telecom things being objects with url elements, not just the urls as plain strings, example:
o = { ...
telecom: [
{ url: "tel:+1-987-654-3210" },
{ url: "mailto:dude#host.com" }] ... }
then turning this into observables:
telecom
.filter(t => t.url.match('^(tel|mailto):'))
.forEach(t => {
const prefix = t.url.substring(0, value.indexOf(':')+1);
t.url = ko.observable(t.url).extend({prefixConversion: prefix});
});
nice and general works for different prefixes. And here is how we ultimately bind them:
<label>Email:<input data-bind="value: telecom.find(t => t.url.startsWith('mailto:')).url"/></label>
<label>Phone:<input data-bind="value: telecom.find(t => t.url().startsWith('tel:')).url"/></label>
Now this causes the telecom values to be wiped out, because it stores the values back with removed prefixes. It almost works, I can see how the read and write functions produce the correct value, write adds the prefix, and read removes it, but then somewhere, somehow, that value without the prefix gets written into the observable._state._latestValue and then we lose a hold of this altogether.
I don't know if this is a bug somewhere. I don't know even how that prefix-less value ever got written into the observable state.
But in the end, I find even this is too costly an approach anyway. I think the issue applies strictly to binding modes view and text and perhaps textInput, i.e., wherever the ultimate representation is a string. It only applies to the UI surface presentations, it is not about the represented object and its properties. No code should ever get the tel: and mailto: prefix removed, this is only for the user, therefore, this conversion could be bound to the binding handler.
MY OWN SOLUTION
So I am resolving this the same way that I did it with the XForms framework once. I added v2t and t2v functions: v2t means value to text, and t2v means text to value. For example:
<input ...
data-bind="..."
data-t2v="return t.indexOf('tel:') == 0 ? t : 'tel:' + t;"
data-v2t="var p = v.substring(0,4); return p == 'tel:' ? v.substring(4) : v;"
.../>
These attributes get converted to functions during first initialization (or lazily when needed):
if(!(element.t2v === null || element.t2v)) {
const att = element.getAttribute("t2v");
element.t2v = att ? new Function("t", att) : null;
}
if(!(element.v2t === null || element.v2t)) {
const att = element.getAttribute("v2t");
element.v2t = att ? new Function("v", att) : null;
}
then in a few select places, I check and use the element.t2v and .v2t properties respectively before reading and writing respectively. Example:
element.value = element.v2t ? element.v2t(element, value) : value;
This is a solution I have been using for a decade and I think it is right. It is not used when the UI text interactions are more involved. Anything that needs keystroke-event granularity or is in other ways complex, such as requiring inputs from other objects would be handled differently.
I have this implemented and working now. But it is a change to knockout and the question is not moot, because perhaps the knockout people might have a different solution for that.
For example, I note that there is also ko.utils.domData, which I don't understand yet, but which could potentially provide an avenue for this. But my solution is tested in a different data binding framework for a long time and now implemented in knockout, and I think it is the right solution as, again, the test to value and value to text conversions are about the widget and how it renders model data values, not about the data.
You could achieve this without having to modify Knockout, using the built-in writable computed functionality.
Suppose you had a list of people, like so:
vm.person = [
{
name: ko.observable("The H. Dude"),
telecom: [
"mailto:dude#host.com",
"tel:+1-987-654-3210"
]
},
{
name: ko.observable("The I. Gal"),
telecom: [
"mailto:gal#host.com",
"tel:+1-987-654-3211"
]
}
];
Then you could define on your viewmodel a constructor function that returns a writable computed:
vm.getTelecom = function(personIndex, telIndex, prefix) {
return ko.pureComputed({
read() {
return vm.person[personIndex].telecom[telIndex].substring(prefix.length);
},
write(newVal) {
vm.person[personIndex].telecom[telIndex] = prefix + newVal;
}
});
}
And bind that to your inputs:
<!-- ko foreach: person -->
<div>
<label>Name: <input type="text" data-bind="textInput: name"/></label>
<label>Phone:<input type="tel" data-bind="textInput: $parent.getTelecom($index(), 1, 'tel:')"/></label>
<label>Email:<input type="email" data-bind="textInput:$parent.getTelecom($index(), 0, 'mailto:')"/></label>
</div>
<!-- /ko -->
Working demo: https://jsfiddle.net/thebluenile/ujb50vwn/
It feels to me like you're trying to do too much in the view, where knockout would prefer you to add more logic to your viewmodel.
Here's an example of a simple Person View Model that does two things:
Handle the model's (not ideal) format for email and phone explicitly both on the way in and out
Separate the binding values from the model values. The model values have the prefixes, the binding values hide that from the user.
ko.extenders.prefixConversion = function(target, prefix) {
const newTarget = ko.pureComputed({
read: () => target().substring(prefix.length),
write: value => target(prefix + value)
});
return newTarget;
}
const Prefix = {
mail: "mailto:",
phone: "tel:"
};
const PersonVM = person => {
const name = ko.observable(person.name);
const email = ko.observable(
person.telecom.find(str => str.startsWith(Prefix.mail))
);
const phone = ko.observable(
person.telecom.find(str => str.startsWith(Prefix.phone))
);
const toJS = ko.pureComputed(() => ({
name: name(),
telecom: [
email(),
phone()
]
}));
return {
name,
email: email.extend({ prefixConversion: Prefix.mail }),
phone: phone.extend({ prefixConversion: Prefix.phone }),
toJS
};
}
const personVM = PersonVM({
name: "The H. Dude",
telecom: [
"mailto:dude#host.com",
"tel:+1-987-654-3210"
]
});
ko.applyBindings(personVM);
<script src="https://cdnjs.cloudflare.com/ajax/libs/knockout/3.4.2/knockout-min.js"></script>
<input data-bind="textInput: name" type="text">
<input data-bind="textInput: email" type="text">
<input data-bind="textInput: phone" type="text">
<pre data-bind="text: JSON.stringify(toJS(), null, 2)"></pre>

How to call endpoint.select() in RTK query with an argument to retrieve cached data (within another selector)?

I have an endpoint which accepts a parameter and I'm trying to access the cached data using endpoint.select() in a redux slice. The problem is i cant figure out how to pass in the cache key. I've done the following:
export const selectProductsResult = (storeId) =>
storeApi.endpoints.listProductsByStore.select(storeId);
This works fine if I use it within a component like this:
const currentStoreProducts = useSelector(selectProductResult(currentStoreId))
What I don't understand is how I can use this in another selector, for example this does not work:
const selectCurrentProducts = createSelector((selectCurrentStoreId), currentStoreId
=> selectProductResult(currentStoreId)
If I try to use this in a component like so:
const currentProducts = useSelector(selectCurrentProducts)
The value obtained is a memoized function. I've played around quite a bit and can't seem to build the desired selector.
The call to someEndpoint.select() generates a new selector function that is specialized to look up the data for that cache key. Loosely put, imagine it looks like this:
const createEndpointSelector = (cacheKey) => {
return selectorForCacheKey = () => {
return state.api.queries['someEndpointName'][cacheKey];
}
}
const selectDataForPikachu = createEndpointSelector('pikachu');
So, you need to call someEndpoint.select() with the actual cache key itself, and it returns a new selector that knows how to retrieve the data for that cache key:
const selectDataForPikachu = apiSlice.endpoints.getPokemon.select('pikachu');
// later, in a component
const pikachuData = useSelector(selectDataForPikachu);

How to combine data model types with document ids?

I'm working with Firestore and Typescript.
For the data models I have types definitions. For example User could be this:
interface User {
name: string;
age: number;
}
The users are stored in the database in the users collection under a unique name/id.
In Firebase when you query a collection, the ids of the documents are available on the document reference, and do not come with the data. In a common use-case for front-end, you want to retrieve an array of records with their ids, because you probably want to interact with them and need to identify each.
So I made a query similar to the code below, where the id is merged into the resulting array:
async function getUsers(): Promise<any[]> {
const query = db.collection("users")
const snapshot = await query.get();
const results = snapshot.docs.map(doc => {
return { ...doc.data(), id: doc.id };
});
}
Now the problem is, that I have a User type, but it can't be used here because it does not contain an id field.
A naïve solution could be to create a new type:
interface UserWithId extends User {
id: string
}
And write the function like:
async function getUsers(): Promise<UserWithId[]> {}
But this doesn't feel right to me, because you would have to potentially do this for many types.
A better solution I think would be to create a generic type:
type DatabaseRecord<T> = {
id: string,
data: T
}
Thus keeping data and ids separate in the returning results:
const results = snapshot.docs.map(doc => {
return { data: doc.data(), id: doc.id };
});
... and use the function signature:
async function getUsers(): Promise<DatabaseRecord<User>[]> {}
I would favour the second over the first solution, because creating new types for each case feels silly. But I am still not sure if that is the best approach.
This seems like such a common scenario but I didn't manage to find any documentation on this. I have seen developers simply write the id in the model data, essentially duplicating the document name in its data, but that to me seems like a big mistake.
I can imagine that if you don't use Typescript (of Flow) that you just don't care about the resulting structure and simply merge the id with the data, but this is one of the reasons I really love using type annotation in JS. It forces you think more about your data and you end up writing cleaner code.

About normalize in redux real-world example

In the example's src(UserPage.js):
const mapStateToProps = (state, ownProps) => {
// We need to lower case the login due to the way GitHub's API behaves.
// Have a look at ../middleware/api.js for more details.
const login = ownProps.params.login.toLowerCase()
const {
pagination: { starredByUser },
entities: { users, repos }
} = state
const starredPagination = starredByUser[login] || { ids: [] }
const starredRepos = starredPagination.ids.map(id => repos[id])
const starredRepoOwners = starredRepos.map(repo => users[repo.owner])
return {
login,
starredRepos,
starredRepoOwners,
starredPagination,
user: users[login]
}
}
I notice that there is many templates like xxx.ids.map(id => someEntities[id]),I am not sure why use this pattern to work.IMO,I would use something like import { map } from 'lodash'; someList && map(someList, item => {...}) in the container component and just pass the entities in the mapStateToProps.
So,could someone explains it's purpose?Thanks.
The standard suggestion for normalizing data in Redux is to store data items in an object, with IDs as the keys and the items as the values. However, an object doesn't have an inherent order to it. (Technically, the order of iteration for object keys should be consistent, but it's a bad practice to rely on that as the sole means of ordering.)
Because of that, it's also standard to store an array of just the IDs as well. A typical example might look like:
{
byId : {
qwerty : { },
abcd : { },
aj42913 : { }
},
items : ["qwerty", "aj42913", "abcd"],
sorted : ["abcd", "aj42913", "qwerty"],
selected : ["qwerty", "abcd"]
}
In this example, items contains all item IDs, probably in the order they were insert. sorted contains the IDs in some sort of sorted order, while selected contains a subset of the IDs.
This allows the items themselves to only be stored once, while multiple representations of those items can be saved using various arrays of IDs.
From there, you can pull together a list of the actual items by mapping over whatever array of IDs you care about, and retrieving the items by their IDs.
So, ultimately the answer is that relying just on the keys of the byId object doesn't give you any kind of ordering, and doesn't allow defining subsets of the data.

Correctly structuring this LinqToSql code

Normally I use stored procedures / work in SQL so apologies if I get the terminology slightly off here..
I have a database, with 3 seperate tables, and I need to search multiple fields in each of the 3 tables.
Im sure that I am not doing this the mose effective way, initially I am trying to do it in simple seteps to understand it.
I have the following;
var foo1 = entities.table1.Where(a => a.bodyText.Contains(searchString) || a.pageTitle.Contains(searchString));
var foo2 = entities.table2.Where(b => b.newsArticle.Contains(searchString) || b.newsArticle.Contains(searchString));
var foo3 = entities.table3.Where(c => c.ImageDescriptionContains(searchString));
I need to combine all these results into a single repeater for display.
At this point all 3 sets of data are in seperate, unique collections of anonymous data. So whats the best way of converting these into a single coherent bindable source?
I was thinking of itereating through each list in turn, pulling out the fields I need to display and putting them in a new class, then binding a lsit of these classes to the repeater.
But it all seems a bit clunky to me.
Is there a way of doing the search across all 3 tables in one go, and returning just the fields I need from each table, with a common name (i.e. in SQL I could write
select b.newsArticle as myText,
or
select newsArticle, ''
to return the news article and an empty string).
This would combine:
var foos = foo1.ToList();
foos.AddRange(foo2);
foos.AddRange(foo3);
To get just what you want:
var myExtractedValues = foos.Select(x => new {
Article = !string.IsNullOrEmpty(x.newsArticle))
? x.newsArticle
: string.Empty});
I have used an anonymous type here but you could swap the new {} with a type of your own.
I reverse the operator on the IsNullOrEmpty but that is just a personal preference (I prefer how is reads.)
To get all the results in one go you'll need to define a common class that will be used by all three queries to store the result. This class may be as well anonymous but I'll name it just for clarity.
class Data
{
public string Text{ get; set;}
}
Now, in your code you'll fetch instances of Data from database and you can use Union:
using( var entities = new YourDataContext)
{
var foo1 = entities.table1
.Where(a => a.bodyText.Contains(searchString) ||
a.pageTitle.Contains(searchString))
.Select(a => new Data{ Text = a.bodyText});
var foo2 = entities.table2
.Where(b => b.newsArticle.Contains(searchString) ||
b.newsArticle.Contains(searchString))
.Select(b => new Data{ Text = b.newsArticle});
var foo3 = entities.table3
.Where(c => c.ImageDescription.Contains(searchString))
.Select(c => new Data{ Text = c.ImageDescription});
return foo1.Union(foo2).Union(foo3);
}

Resources