I have written an api in ASP.NET which uses Entity Framework 6.
Here is the code
cr = context.Responses.FirstOrDefault(s => s.RegistrationId ==registrationId);
if (cr == null)
{
cr = new Responses()
{
Answer = answer,
RegistrationId = registrationId,
CreationTime = DateTime.Now
};
context.Responses.Add(cr);
}
else
{
cr.Answer = answer;
}
context.SaveChanges();
This is my result in database
But while doing the database inserts, it inserts the same data twice with same creation time which happens often. Why is this so? What is the best way to avoid these duplicate inserts?
first of all it should be
cr = context.Responses.FirstOrDefault(s => s.RegistrationId == registrationId );
This is possible this error originates from the UI. suppose you are filling out responses by form, and somebody presses the submit twice, then you would have two lines in the db representing the same response. The correct way to resolve this is to have the form (via javascript and such) generate the guid, and immediately disable the submit button after the click. Another way is to declare in the database that the combinations of result columns are unique, thus no two "same" lines can exist by defintion.
I'm using Azure Mobile App with Xamarin.Forms to create an offline capable mobile app.
My solution is based on https://adrianhall.github.io/develop-mobile-apps-with-csharp-and-azure/chapter3/client/
Here is the code that I use for offline sync :
public class AzureDataSource
{
private async Task InitializeAsync()
{
// Short circuit - local database is already initialized
if (client.SyncContext.IsInitialized)
{
return;
}
// Define the database schema
store.DefineTable<ArrayElement>();
store.DefineTable<InputAnswer>();
//Same thing with 16 others table
...
// Actually create the store and update the schema
await client.SyncContext.InitializeAsync(store, new MobileServiceSyncHandler());
}
public async Task SyncOfflineCacheAsync()
{
await InitializeAsync();
//Check if authenticated
if (client.CurrentUser != null)
{
// Push the Operations Queue to the mobile backend
await client.SyncContext.PushAsync();
// Pull each sync table
var arrayTable = await GetTableAsync<ArrayElement>();
await arrayTable.PullAsync();
var inputAnswerInstanceTable = await GetTableAsync<InputAnswer>();
await inputAnswerInstanceTable.PullAsync();
//Same thing with 16 others table
...
}
}
public async Task<IGenericTable<T>> GetTableAsync<T>() where T : TableData
{
await InitializeAsync();
return new AzureCloudTable<T>(client);
}
}
public class AzureCloudTable<T>
{
public AzureCloudTable(MobileServiceClient client)
{
this.client = client;
this.table = client.GetSyncTable<T>();
}
public async Task PullAsync()
{
//Query name used for incremental pull
string queryName = $"incsync_{typeof(T).Name}";
await table.PullAsync(queryName, table.CreateQuery());
}
}
The problem is that the syncing takes a lot of time even when there isn't anything to pull (8-9 seconds on Android devices and more than 25 seconds to pull the whole database).
I looked at Fiddler to find how much time takes the Mobile Apps BackEnd to respond and it is about 50 milliseconds per request so the problem doesn't seem to come from here.
Does anyone have the same trouble ? Is there something that I'm doing wrong or tips to improve my sync performance ?
Our particular issue was linked to our database migration. Every row in the database had the same updatedAt value. We ran an SQL script to modify these so that they were all unique.
This fix was actually for some other issue we had, where not all rows were being returned for some unknown reason, but we also saw a substantial speed improvement.
Also, another weird fix that improved loading times was the following.
After we had pulled all of the data the first time (which, understandably takes some time) - we did an UpdateAsync() on one of the rows that were returned, and we did not push it afterwards.
We've come to understand that the way offline sync works, is that it will pull anything that has a date newer than the most recent updated at. There was a small speed improvement associated with this.
Finally, the last thing we did to improve speed was to not fetch the data again, if it already had cached a copy in the view. This may not work for your use case though.
public List<Foo> fooList = new List<Foo>
public void DisplayAllFoo()
{
if(fooList.Count == 0)
fooList = await SyncClass.GetAllFoo();
foreach(var foo in fooList)
{
Console.WriteLine(foo.bar);
}
}
Edit 20th March 2019:
With these improvements in place, we are still seeing very slow sync operations, used in the same way as mentioned in the OP, also including the improvements listed in my answer here.
I encourage all to share their solutions or ideas on how this speed can be improved.
One of the reasons for the slow Pull() is when more than (10) rows get the same UpdatedAt value. This happens when you update the rows at once, for example running an SQL command.
One way to overcome this is to modify the default trigger on the tables. To ensure every row gets a unique UpdateAt, we did something like this:
ALTER TRIGGER [dbo].[TR_dbo_Items_InsertUpdateDelete] ON [dbo].[TableName]
AFTER INSERT, UPDATE, DELETE
AS
BEGIN
DECLARE #InsertedAndDeleted TABLE
(
Id NVARCHAR(128)
);
DECLARE #Count INT,
#Id NVARCHAR(128);
INSERT INTO #InsertedAndDeleted
SELECT Id
FROM inserted;
INSERT INTO #InsertedAndDeleted
SELECT Id
FROM deleted
WHERE Id NOT IN
(
SELECT Id
FROM #InsertedAndDeleted
);
--select * from #InsertedAndDeleted;
SELECT #Count = Count(*)
FROM #InsertedAndDeleted;
-- ************************ UpdatedAt ************************
-- while loop
WHILE #Count > 0
BEGIN
-- selecting
SELECT TOP (1) #Id = Id
FROM #InsertedAndDeleted;
-- updating
UPDATE [dbo].[TableName]
SET UpdatedAt = Convert(DATETIMEOFFSET, DateAdd(MILLISECOND, #Count, SysUtcDateTime()))
WHERE Id = #Id;
-- deleting
DELETE FROM #InsertedAndDeleted
WHERE id = #Id;
-- counter
SET #Count = #Count - 1;
END;
END;
I am implementing a website with a dynamodb + nodejs backend. I use Vogels.js in server side to query dynamodb and show results on a webpage. Because my query returns a lot of results, I would like to return only N (such as 5) results back to a user initially, and return the next N results when the user asks for more.
Is there a way I can run two vogels queries with the second query starts from the place where the first query left off ? Thanks.
Yes, vogels fully supports pagination on both query and scan operations.
For example:
var Tweet = vogels.define('tweet', {
hashKey : 'UserId',
rangeKey : 'PublishedDateTime',
schema : {
UserId : Joi.string(),
PublishedDateTime : Joi.date().default(Date.now),
content : Joi.string()
}
});
// Fetch the 5 most recent tweets from user with id 555:
Tweet.query(555).limit(5).descending().exec(function (err, data) {
var paginationKey = data.LastEvaluatedKey;
// Fetch the next page of 5 tweets
Tweet.query(555).limit(5).descending().startKey(paginationKey).exec()
});
Yes it is possible, DynamoDB has some thing called "LastEvaluatedKey" which will server your purpose.
Step 1) Query your table with option "Limit" = number of records
refer: http://docs.aws.amazon.com/amazondynamodb/latest/APIReference/API_Query.html
Step 2) If your query has more records than the "Limit value", DynamoDB will return a "LastEvaluatedKey" which you can pass in your next query as "ExclusiveStartKey" to get next set of records until there are no records left
refer: http://docs.aws.amazon.com/amazondynamodb/latest/developerguide/QueryAndScan.html#QueryAndScan.Query
Note: Be aware that to get previous set of records you might have to store all the "LastEvaluatedKeys" and implement this at application level
I'm trying to test out Firebase to allow users to post comments using push. I want to display the data I retrieve with the following;
fbl.child('sell').limit(20).on("value", function(fbdata) {
// handle data display here
}
The problem is the data is returned in order of oldest to newest - I want it in reversed order. Can Firebase do this?
Since this answer was written, Firebase has added a feature that allows ordering by any child or by value. So there are now four ways to order data: by key, by value, by priority, or by the value of any named child. See this blog post that introduces the new ordering capabilities.
The basic approaches remain the same though:
1. Add a child property with the inverted timestamp and then order on that.
2. Read the children in ascending order and then invert them on the client.
Firebase supports retrieving child nodes of a collection in two ways:
by name
by priority
What you're getting now is by name, which happens to be chronological. That's no coincidence btw: when you push an item into a collection, the name is generated to ensure the children are ordered in this way. To quote the Firebase documentation for push:
The unique name generated by push() is prefixed with a client-generated timestamp so that the resulting list will be chronologically-sorted.
The Firebase guide on ordered data has this to say on the topic:
How Data is Ordered
By default, children at a Firebase node are sorted lexicographically by name. Using push() can generate child names that naturally sort chronologically, but many applications require their data to be sorted in other ways. Firebase lets developers specify the ordering of items in a list by specifying a custom priority for each item.
The simplest way to get the behavior you want is to also specify an always-decreasing priority when you add the item:
var ref = new Firebase('https://your.firebaseio.com/sell');
var item = ref.push();
item.setWithPriority(yourObject, 0 - Date.now());
Update
You'll also have to retrieve the children differently:
fbl.child('sell').startAt().limitToLast(20).on('child_added', function(fbdata) {
console.log(fbdata.exportVal());
})
In my test using on('child_added' ensures that the last few children added are returned in reverse chronological order. Using on('value' on the other hand, returns them in the order of their name.
Be sure to read the section "Reading ordered data", which explains the usage of the child_* events to retrieve (ordered) children.
A bin to demonstrate this: http://jsbin.com/nonawe/3/watch?js,console
Since firebase 2.0.x you can use limitLast() to achieve that:
fbl.child('sell').orderByValue().limitLast(20).on("value", function(fbdataSnapshot) {
// fbdataSnapshot is returned in the ascending order
// you will still need to order these 20 items in
// in a descending order
}
Here's a link to the announcement: More querying capabilities in Firebase
To augment Frank's answer, it's also possible to grab the most recent records--even if you haven't bothered to order them using priorities--by simply using endAt().limit(x) like this demo:
var fb = new Firebase(URL);
// listen for all changes and update
fb.endAt().limit(100).on('value', update);
// print the output of our array
function update(snap) {
var list = [];
snap.forEach(function(ss) {
var data = ss.val();
data['.priority'] = ss.getPriority();
data['.name'] = ss.name();
list.unshift(data);
});
// print/process the results...
}
Note that this is quite performant even up to perhaps a thousand records (assuming the payloads are small). For more robust usages, Frank's answer is authoritative and much more scalable.
This brute force can also be optimized to work with bigger data or more records by doing things like monitoring child_added/child_removed/child_moved events in lieu of value, and using a debounce to apply DOM updates in bulk instead of individually.
DOM updates, naturally, are a stinker regardless of the approach, once you get into the hundreds of elements, so the debounce approach (or a React.js solution, which is essentially an uber debounce) is a great tool to have.
There is really no way but seems we have the recyclerview we can have this
query=mCommentsReference.orderByChild("date_added");
query.keepSynced(true);
// Initialize Views
mRecyclerView = (RecyclerView) view.findViewById(R.id.recyclerView);
mManager = new LinearLayoutManager(getContext());
// mManager.setReverseLayout(false);
mManager.setReverseLayout(true);
mManager.setStackFromEnd(true);
mRecyclerView.setHasFixedSize(true);
mRecyclerView.setLayoutManager(mManager);
I have a date variable (long) and wanted to keep the newest items on top of the list. So what I did was:
Add a new long field 'dateInverse'
Add a new method called 'getDateInverse', which just returns: Long.MAX_VALUE - date;
Create my query with: .orderByChild("dateInverse")
Presto! :p
You are searching limitTolast(Int x) .This will give you the last "x" higher elements of your database (they are in ascending order) but they are the "x" higher elements
if you got in your database {10,300,150,240,2,24,220}
this method:
myFirebaseRef.orderByChild("highScore").limitToLast(4)
will retrive you : {150,220,240,300}
In Android there is a way to actually reverse the data in an Arraylist of objects through the Adapter. In my case I could not use the LayoutManager to reverse the results in descending order since I was using a horizontal Recyclerview to display the data. Setting the following parameters to the recyclerview messed up my UI experience:
llManager.setReverseLayout(true);
llManager.setStackFromEnd(true);
The only working way I found around this was through the BindViewHolder method of the RecyclerView adapter:
#Override
public void onBindViewHolder(final RecyclerView.ViewHolder holder, int position) {
final SuperPost superPost = superList.get(getItemCount() - position - 1);
}
Hope this answer will help all the devs out there who are struggling with this issue in Firebase.
Firebase: How to display a thread of items in reverse order with a limit for each request and an indicator for a "load more" button.
This will get the last 10 items of the list
FBRef.child("childName")
.limitToLast(loadMoreLimit) // loadMoreLimit = 10 for example
This will get the last 10 items. Grab the id of the last record in the list and save for the load more functionality. Next, convert the collection of objects into and an array and do a list.reverse().
LOAD MORE Functionality: The next call will do two things, it will get the next sequence of list items based on the reference id from the first request and give you an indicator if you need to display the "load more" button.
this.FBRef
.child("childName")
.endAt(null, lastThreadId) // Get this from the previous step
.limitToLast(loadMoreLimit+2)
You will need to strip the first and last item of this object collection. The first item is the reference to get this list. The last item is an indicator for the show more button.
I have a bunch of other logic that will keep everything clean. You will need to add this code only for the load more functionality.
list = snapObjectAsArray; // The list is an array from snapObject
lastItemId = key; // get the first key of the list
if (list.length < loadMoreLimit+1) {
lastItemId = false;
}
if (list.length > loadMoreLimit+1) {
list.pop();
}
if (list.length > loadMoreLimit) {
list.shift();
}
// Return the list.reverse() and lastItemId
// If lastItemId is an ID, it will be used for the next reference and a flag to show the "load more" button.
}
I'm using ReactFire for easy Firebase integration.
Basically, it helps me storing the datas into the component state, as an array. Then, all I have to use is the reverse() function (read more)
Here is how I achieve this :
import React, { Component, PropTypes } from 'react';
import ReactMixin from 'react-mixin';
import ReactFireMixin from 'reactfire';
import Firebase from '../../../utils/firebaseUtils'; // Firebase.initializeApp(config);
#ReactMixin.decorate(ReactFireMixin)
export default class Add extends Component {
constructor(args) {
super(args);
this.state = {
articles: []
};
}
componentWillMount() {
let ref = Firebase.database().ref('articles').orderByChild('insertDate').limitToLast(10);
this.bindAsArray(ref, 'articles'); // bind retrieved data to this.state.articles
}
render() {
return (
<div>
{
this.state.articles.reverse().map(function(article) {
return <div>{article.title}</div>
})
}
</div>
);
}
}
There is a better way. You should order by negative server timestamp. How to get negative server timestamp even offline? There is an hidden field which helps. Related snippet from documentation:
var offsetRef = new Firebase("https://<YOUR-FIREBASE-APP>.firebaseio.com/.info/serverTimeOffset");
offsetRef.on("value", function(snap) {
var offset = snap.val();
var estimatedServerTimeMs = new Date().getTime() + offset;
});
To add to Dave Vávra's answer, I use a negative timestamp as my sort_key like so
Setting
const timestamp = new Date().getTime();
const data = {
name: 'John Doe',
city: 'New York',
sort_key: timestamp * -1 // Gets the negative value of the timestamp
}
Getting
const ref = firebase.database().ref('business-images').child(id);
const query = ref.orderByChild('sort_key');
return $firebaseArray(query); // AngularFire function
This fetches all objects from newest to oldest. You can also $indexOn the sortKey to make it run even faster
I had this problem too, I found a very simple solution to this that doesn't involved manipulating the data in anyway. If you are rending the result to the DOM, in a list of some sort. You can use flexbox and setup a class to reverse the elements in their container.
.reverse {
display: flex;
flex-direction: column-reverse;
}
myarray.reverse(); or this.myitems = items.map(item => item).reverse();
I did this by prepend.
query.orderByChild('sell').limitToLast(4).on("value", function(snapshot){
snapshot.forEach(function (childSnapshot) {
// PREPEND
});
});
Someone has pointed out that there are 2 ways to do this:
Manipulate the data client-side
Make a query that will order the data
The easiest way that I have found to do this is to use option 1, but through a LinkedList. I just append each of the objects to the front of the stack. It is flexible enough to still allow the list to be used in a ListView or RecyclerView. This way even though they come in order oldest to newest, you can still view, or retrieve, newest to oldest.
You can add a column named orderColumn where you save time as
Long refrenceTime = "large future time";
Long currentTime = "currentTime";
Long order = refrenceTime - currentTime;
now save Long order in column named orderColumn and when you retrieve data
as orderBy(orderColumn) you will get what you need.
just use reverse() on the array , suppose if you are storing the values to an array items[] then do a this.items.reverse()
ref.subscribe(snapshots => {
this.loading.dismiss();
this.items = [];
snapshots.forEach(snapshot => {
this.items.push(snapshot);
});
**this.items.reverse();**
},
For me it was limitToLast that worked. I also found out that limitLast is NOT a function:)
const query = messagesRef.orderBy('createdAt', 'asc').limitToLast(25);
The above is what worked for me.
PRINT in reverse order
Let's think outside the box... If your information will be printed directly into user's screen (without any content that needs to be modified in a consecutive order, like a sum or something), simply print from bottom to top.
So, instead of inserting each new block of content to the end of the print space (A += B), add that block to the beginning (A = B+A).
If you'll include the elements as a consecutive ordered list, the DOM can put the numbers for you if you insert each element as a List Item (<li>) inside an Ordered Lists (<ol>).
This way you save space from your database, avoiding unnecesary reversed data.
I'm making a simple app that informs a client that other clients clicked a button. I'm storing the clicks in a Firebase (db) using:
db.push({msg:data});
All clients get notified of other user's clicks with an on, such as
db.on('child_added',function(snapshot) {
var msg = snapshot.val().msg;
});
However, when the page first loads I want to discard any existing data on the stack. My strategy is to call db.once() before I define the db.on('child_added',...) in order to get the initial number of children, and then use that to discard that number of calls to db.on('child_added',...).
Unfortunately, though, all of the calls to db.on('child_added',...) are happening before I'm able to get the initial count, so it fails.
How can I effectively and simply discard the initial data?
For larger data sets, Firebase now offers (as of 2.0) some query methods that can make this simpler.
If we add a timestamp field on each record, we can construct a query that only looks at new values. Consider this contrived data:
{
"messages": {
"$messageid": {
"sender": "kato",
"message": "hello world"
"created": 123456 // Firebase.ServerValue.TIMESTAMP
}
}
}
We could find messages only after "now" using something like this:
var ref = new Firebase('https://<your instance>.firebaseio.com/messages');
var queryRef = ref.orderBy('created').startAt(Firebase.ServerValue.TIMESTAMP);
queryRef.on('child_added', function(snap) {
console.log(snap.val());
});
If I understand your question correctly, it sounds like you only want data that has been added since the user visited the page. In Firebase, the behavior you describe is by design, as the data is always changing and there isn't a notion of "old" data vs "new" data.
However, if you only want to display data added after the page has loaded, try ignoring all events prior until the complete set of children has loaded at least once. For example:
var ignoreItems = true;
var ref = new Firebase('https://<your-Firebase>.firebaseio.com');
ref.on('child_added', function(snapshot) {
if (!ignoreItems) {
var msg = snapshot.val().msg;
// do something here
}
});
ref.once('value', function(snapshot) {
ignoreItems = false;
});
The alternative to this approach would be to write your new items with a priority as well, where the priority is Firebase.ServerValue.TIMESTAMP (the current server time), and then use a .startAt(...) query using the current timestamp. However, this is more complex than the approach described above.