Restful api data syncing only changed models on big collections - http

Question

I'm trying to find a best practice method to make my api respond with a 204. Lets consider the following simplified steps:
Client GET: Server response 200 full collection of 10 records
No changes are made
Client GET: server response 304 data not changed
Changes are made to record 5. Record 11 is added. Record 2 is deleted
Client GET: server response 200 with the new collection of 10 records
For 10 records this is not a big issue. However when a collection is lets say a few thousands records you don't want to refresh your entire locally stored collection. In this case it's easier to change the 3 updated models (delete record 2, update record 5, add record 11) So I want something like this
Client GET: Server response 200 full collection (paginated or not)
No changes are made
Client GET: server response 304 data not changed
Changes are made to record 5. Record 11 is added. Record 2 is deleted
Client GET: server response 204 with only information about the 3 changed records
In the above case the request cycle is optimized, but the problem is: how do I get the server to respond like this. I must send some information from the client in either a header or body. I was thinking about exploiting the Last-modified-since header. This will be the date change on the server. The client stores this date when status is 200 or 204. When the client sends this header to the server, the server will only respond with records that are changed or deleted since then. For example:
{
"total": 113440,
"from": 0,
"till": 2,
"count": 3,
"removed": [1,2],
"parameters": [{"id"}, {"title"}],
"collection": [
{
"id": 3,
"title": "Updated record"
},
{
"id": 4,
"title": "New record"
},
{
"id": 5,
"title": "Another new record"
}
],
}
Downside is that the server will be more complex because it needs to keep track of the deleted data and the last updated records.
Keep in mind that I did think of sending silent push updates but I don't want to do this since the user is not always happy with background data traffic.
What do you guys think about this solutions and do you have a similar or better solution keeping the following in mind?
lower the amount of needed requests
make the api descriptive and cellular (api being it's own documentation allowing clientside generators)
be as live as possible
effectively deal with huge collections (ex: pagination, only fetch
updated records, caching etc)

Answer 1

You could send up a If-Modified-Since header with your GET requests for the collection. The endpoint could then return only those contents that have changed since the provided date.
Note that this is not the behaviour specified by the HTTP spec for conditional requests. Carefully consider whether this is the approach you want to take. I would not suggest it.
In the spec as written, there is no way I'm aware of to retrieve a subset of a collection conditionally. The closest I can think of is to have the collection contain only links to the individual elements. Make the client call a GET on each individual element, using If-None-Match or If-Modified-Since so you only pass stale entities over the wire. You'd still have to make a server hit for the collection entity, though you could also use a conditional header on that request.

Answer 2

IMHO this calls for a query interface, where you tell the rest service in the GET parameters what you want, simar to how you would do it in SQL or in a solr or elasticsearch query.
Why hide something in HTTP headers, that the client is explicitly querying, right?