Cosmos Db library Microsoft.Azure.DocumentDB.Core (2.1.0) - Actual REST invocations - .net-core

We are attempting to Wiremock (https://github.com/WireMock-Net/WireMock.Net) CosmosDb invocations - so we can build integrationtests in our .net core 2.1 microservice.
By looking at the WireMock instance Request/Response entries, we can observe the following:
1) GET towards "/"
We mock the returning metadata of databases
THIS IS OK
2) GET towards collection (in our case: "/dbs/Chunker/colls/RHTMLChunks")
Returns metadata about the collections
THIS IS OK
3) POST a Query that results in one document being returned towards the documents endpoint on the collection (in our case: "/dbs/Chunker/colls/RHTMLChunks/docs")
I have tried to emulate what we get when we do the exact same query towards the CosmosDb instance in Postman, including headers and response.
However I observe that the lib does the query again, and again, and again....
(I can see this by pausing in Visual Studio, then look at the RequestLog in WireMock)
Does anyone know what should be returned. I have set up WireMock to return the following json payload:
{
"_rid": "q0dcAOelSAI=",
"Documents": [
{
"id": "gL20020621z2D34-1",
"ChunkSize": 658212,
"TotalChunks": 2,
"Metadata": {
"Active": true,
"PublishedDate": "",
},
"ChunkId": 1,
"Markup": "<h1>hello</h1>",
"MainDestination": "gL20020621z2D34",
"_rid": "q0dcAOelSAIHAAAAAAAAAA==",
"_self": "dbs/q0dcAA==/colls/q0dcAOelSAI=/docs/q0dcAOelSAIHAAAAAAAAAA==/",
"_etag": "\"0100e92a-0000-0000-0000-5ba96cf70000\"",
"_attachments": "attachments/",
"_ts": 1537830135
}
],
"_count": 0
}
Problems:
1) Can not find .pdb belonging to Microsoft.Azure.DocumentDB.Core v2.1.0
2) What payload/headers should be returned, so the library will NOT blow up, and retry when we invoke:
var response = await documentQuery.ExecuteNextAsync<DocumentDto>(); // this hangs forever
Please help :)

We're working on open sourcing the C# code base and some other fun improvements to make this easier. In the mean time, I'd advocate for using the emulator for local testing/etc., although I understand mocking is still a lot faster an nicer - it'll just be hard :)
My best pointer is actually our Node.js code base since that's public already. The query code is relatively hard to follow, but basically, you create a query, we look up all the partitions we need to talk to, then we send a request for each partition and keep querying until we don't get back a continuation token anymore (or maxBufferedItem Count/etc. goes over the limit, and we pause until goes back down, etc.)
Effectively, we send out N number of requests for each partition, where N is the number of pages of results, and can vary per partition and query. You'd likely be able to mock a single partition, single page response relatively easy, but a full partition response isn't gonna be fun.
As I mentioned in the beginning, we've got some cool stuff coming, hopefully before the end of the year, which will make offline mocking easier, as well as open sourcing it finally. You might be better off with the emulator until then.

Related

bitbucket API Rest - Getting branches from a repository

I am looking for the list of endpoints available for bitbucket API regarding get the branches from a repository and a specific branch.
I was expecting to see something like:
GET /2.0/repositories/{workspace}/{repo_slug}/branches/
GET /2.0/repositories/{workspace}/{repo_slug}/branches/{branch}
So I can get the commits from a specific branch. 
I know I can get commits but with this endpoint, its scope is under repository perspective. 
Do you know if there are endpoints to work with branches and recursively into its hierarchy?
I looked over the documentation for API 2.0 but I did not see what I was looking for so that is why I am posting this question here.
In addition, I see some time ago that was not possible according to this answer, but it belongs to the version 1.0 of the API. Is it still true? 
When hitting this endpoint:
https://api.bitbucket.org/2.0/repositories/<workspace>/<repository-name>`
# GET /2.0/repositories/{workspace}/{repo_slug}
You get as a result a JSON document. In the links attribute you got a key called branches. It is something like this:
{
"scm": "git",
"has_wiki": false,
"links": {
"watchers": {
"href": "https://api.bitbucket.org/2.0/repositories/{workspace}/{repo_slug}/watchers"
},
"branches": {
"href": "https://api.bitbucket.org/2.0/repositories/{workspace}/{repo_slug}/refs/branches"
},
....
So you can hit the endpoint and get the branches:
https://api.bitbucket.org/2.0/repositories/{workspace}/{repo_slug}/refs/branches
# GET /2.0/repositories/{workspace}/{repo_slug}/refs/branches
And get a specific branch with
https://api.bitbucket.org/2.0/repositories/{workspace}/{repo_slug}/refs/branches/<branch-name>
# GET /2.0/repositories/{workspace}/{repo_slug}/refs/branches/<branch-name>

Evernote IOS SDK fetchResourceByHashWith throws exception

Working with Evernote IOS SDK 3.0
I would like to retrieve a specific resource from note using
fetchResourceByHashWith
This is how I am using it. Just for this example, to be 100% sure about the hash being correct I first download the note with a single resource using fetchNote and then request this resource using its unique hash using fetchResourceByHashWith (hash looks correct when I print it)
ENSession.shared.primaryNoteStore()?.fetchNote(withGuid: guid, includingContent: true, resourceOptions: ENResourceFetchOption.includeData, completion: { note, error in
if error != nil {
print(error)
seal.reject(error!)
} else {
let hash = note?.resources[0].data.bodyHash
ENSession.shared.primaryNoteStore()?.fetchResourceByHashWith(guid: guid, contentHash: hash, options: ENResourceFetchOption.includeData, completion: { res, error in
if error != nil {
print(error)
seal.reject(error!)
} else {
print("works")
seal.fulfill(res!)
}})
}
})
Call to fetchResourceByHashWith fails with
Optional(Error Domain=ENErrorDomain Code=0 "Unknown error" UserInfo={EDAMErrorCode=0, NSLocalizedDescription=Unknown error})
The equivalent setup works on Android SDK.
Everything else works so far in IOS SDK (chunkSync, auth, getting notebooks etc.. so this is not an issue with auth tokens)
would be great to know if this is an sdk bug or I am still doing something wrong.
Thanks
This is a bug in the SDK's "EDAM" Thrift client stub code. First the analysis and then your workarounds.
Evernote's underlying API transport uses a Thrift protocol with a documented schema. The SDK framework includes a layer of autogenerated stub code that is supposed to marshal input and output params correctly for each request and response. You are invoking the underlying getResourceByHash API method on the note store, which is defined per the docs to accept a string type for the contentHash argument. But it turns out the client is sending the hash value as a purely binary field. The service is failing to parse the request, so you're seeing a generic error on the client. This could reflect evolution in the API definition, but more likely this has always been broken in the iOS SDK (getResourceByHash probably doesn't see a lot of usage). If you dig into the more recent Python version of the SDK, or indeed also the Java/Android version, you can see a different pattern for this method: it says it's going to write a string-type field, and then actually emits a binary one. Weirdly, this works. And if you hack up the iOS SDK to do the same thing, it will work, too.
Workarounds:
Best advice is to report the bug and just avoid this method on the note store. You can get resource data in different ways: First of all, you actually got all the data you needed in the response to your fetchNote call, i.e. let resourceData = note?.resources[0].data.body and you're good! You can also pull individual resources by their own guid (not their hash), using fetchResource (use note?.resources[0].guid as the param). Of course, you may really want to use the access-by-hash pattern. In that case...
You can hack in the correct protocol behavior. In the SDK files, which you'll need to build as part of your project, find the ObjC file called ENTProtocol.m. Find the method +sendMessage:toProtocol:withArguments.
It has one line like this:
[outProtocol writeFieldBeginWithName:field.name type:field.type fieldID:field.index];
Replace that line with:
[outProtocol writeFieldBeginWithName:field.name type:(field.type == TType_BINARY ? TType_STRING : field.type) fieldID:field.index];
Rebuild the project and you should find that your code snippet works as expected. This is a massive hack however and although I don't think any other note store methods will be impacted adversely by it, it's possible that other internal user store or other calls will suddenly start acting funny. Also you'd have to maintain the hack through updates. Probably better to report the bug and don't use the method until Evernote publishes a proper fix.

Restful api data syncing only changed models on big collections

I'm trying to find a best practice method to make my api respond with a 204. Lets consider the following simplified steps:
Client GET: Server response 200 full collection of 10 records
No changes are made
Client GET: server response 304 data not changed
Changes are made to record 5. Record 11 is added. Record 2 is deleted
Client GET: server response 200 with the new collection of 10 records
For 10 records this is not a big issue. However when a collection is lets say a few thousands records you don't want to refresh your entire locally stored collection. In this case it's easier to change the 3 updated models (delete record 2, update record 5, add record 11) So I want something like this
Client GET: Server response 200 full collection (paginated or not)
No changes are made
Client GET: server response 304 data not changed
Changes are made to record 5. Record 11 is added. Record 2 is deleted
Client GET: server response 204 with only information about the 3 changed records
In the above case the request cycle is optimized, but the problem is: how do I get the server to respond like this. I must send some information from the client in either a header or body. I was thinking about exploiting the Last-modified-since header. This will be the date change on the server. The client stores this date when status is 200 or 204. When the client sends this header to the server, the server will only respond with records that are changed or deleted since then. For example:
{
"total": 113440,
"from": 0,
"till": 2,
"count": 3,
"removed": [1,2],
"parameters": [{"id"}, {"title"}],
"collection": [
{
"id": 3,
"title": "Updated record"
},
{
"id": 4,
"title": "New record"
},
{
"id": 5,
"title": "Another new record"
}
],
}
Downside is that the server will be more complex because it needs to keep track of the deleted data and the last updated records.
Keep in mind that I did think of sending silent push updates but I don't want to do this since the user is not always happy with background data traffic.
What do you guys think about this solutions and do you have a similar or better solution keeping the following in mind?
lower the amount of needed requests
make the api descriptive and cellular (api being it's own documentation allowing clientside generators)
be as live as possible
effectively deal with huge collections (ex: pagination, only fetch
updated records, caching etc)
You could send up a If-Modified-Since header with your GET requests for the collection. The endpoint could then return only those contents that have changed since the provided date.
Note that this is not the behaviour specified by the HTTP spec for conditional requests. Carefully consider whether this is the approach you want to take. I would not suggest it.
In the spec as written, there is no way I'm aware of to retrieve a subset of a collection conditionally. The closest I can think of is to have the collection contain only links to the individual elements. Make the client call a GET on each individual element, using If-None-Match or If-Modified-Since so you only pass stale entities over the wire. You'd still have to make a server hit for the collection entity, though you could also use a conditional header on that request.
IMHO this calls for a query interface, where you tell the rest service in the GET parameters what you want, simar to how you would do it in SQL or in a solr or elasticsearch query.
Why hide something in HTTP headers, that the client is explicitly querying, right?

Most efficient method of pulling in weather data for multiple locations

I'm working on a meteor mobile app that displays information about local places of interest and one of the things that I want to show is the weather in each location. I've currently got my locations stored with latlng coordinates and they're searchable by radius. I'd like to use the openweathermap api to pull in some useful 'current conditions' information so that when a user looks at an entry they can see basic weather data. Ideally I'd like to limit the number of outgoing requests to keep the pages snappy (and API requests down)
I'm wondering if I can create a server collection of weather data that I update regularly, server-side (hourly?) that my clients then query (perhaps using a mongo $near lookup?) - that way all of my data is being handled within meteor, rather than each client going out to grab the latest data from the API. I don't want to have to iterate through all of the locations in my list and do a separate call out to the api for each as I have approx. 400 locations(!). I'm afraid I'm new to API requests (and meteor itself) so apologies if this is a poorly phrased question.
I'm not entirely sure if this is doable, or if it's even the best approach - any advice (and links to any useful code snippets!) would be greatly appreciated!
EDIT / UPDATE!
OK I haven't managed to get this working yet but I some more useful details on the data!
If I make a request to the openweather API I can get data back for all of their locations (which I would like to add/update to a collection). I could then do regular lookup, instead of making a client request straight out to them every time a user looks at a location. The JSON data looks like this:
{
"message":"accurate",
"cod":"200",
"count":50,
"list":[
{
"id":2643076,
"name":"Marazion",
"coord":{
"lon":-5.47505,
"lat":50.125561
},
"main":{
"temp":292.15,
"pressure":1016,
"humidity":68,
"temp_min":292.15,
"temp_max":292.15
},
"dt":1403707800,
"wind":{
"speed":8.7,
"deg":110,
"gust":13.9
},
"sys":{
"country":""
},
"clouds":{
"all":75
},
"weather":[
{
"id":721,
"main":"Haze",
"description":"haze",
"icon":"50d"
}
]
}, ...
Ideally I'd like to build my own local 'weather' collection that I can search using mongo's $near (to keep outbound requests down, and speed up), but I don't know if this will be possible because the format that the data comes back in - I think I'd need to structure my location data like this in order to use a geo search:
"location": {
"type": "Point",
"coordinates": [-5.47505,50.125561]
}
My questions are:
How can I build that collection (I've seen this - could I do something similar and update existing entries in the collection on a regular basis?)
Does it just need to live on the server, or client too?
Do I need to manipulate the data in order to get a geo search to work?
Is this even the right way to approach it??
EDIT/UPDATE2
Is this question too long/much? It feels like it. Maybe I should split it out.
Yes easily possible. Because your question is so large I'll give you a high level explanation of what I think you need to do.
You need to create a collection where you're gonna save the weather data in.
A request worker that requests new data and updates the collection on a set interval. Use something like cron-tick for scheduling the interval.
Requesting data should only happen server side and I can recommend the request npm package for that.
Meteor.publish the weather collection and have the client subscribe to that, with optionally a filter for it's location.
You should now be getting the weather data on your client and should be able to get freaky with it.

Can I create an index in a mongo collection with Meteor? [duplicate]

I am trying to create a two column unique index on the underlying mongodb in a meteor app and having trouble. I can't find anything in the meteor docs. I have tried from the chrome console. I have tried from term and even tried to point mongod at the /db/ dir inside .meteor . I have tried
Collection.ensureIndex({first_id: 1, another_id: 1}, {unique: true}); variations.
I want to be able to prevent duplicate entries on a meteor app mongo collection.
Wondering if anyone has figured this out?
I answered my own question, what a noob.
I figured it out.
Start meteor server
Open 2nd terminal and type meteor mongo
Then create your index...for example I did these for records of thumbsup and thumbsdown type system.
db.thumbsup.ensureIndex({item_id: 1, user_id: 1}, {unique: true})
db.thumbsdown.ensureIndex({item_id: 1, user_id: 1}, {unique: true})
Now, just gotta figure out a bootstrap install setup that creates these when pushed to prod instead of manually.
Collection._ensureIndex(index, options)
Searching inside Meteor source code, I found a bind to ensureIndex called _ensureIndex.
For single-key basic indexes you can follow the example of packages/accounts-base/accounts_server.js that forces unique usernames on Meteor:
Meteor.users._ensureIndex('username', {unique: 1, sparse: 1});
For multi-key "compound" indexes:
Collection._ensureIndex({first_id:1, another_id:1}, {unique: 1});
The previous code, when placed on the server side, ensures that indexes are set.
Warning
Notice _ensureIndex implementation warning:
We'll actually design an index API later. For now, we just pass
through to Mongo's, but make it synchronous.
According to the docs "Minimongo currently doesn't have indexes. This will come soon." And looking at the methods available on a Collection, there's no ensureIndex.
You can run meteor mongo for a mongo shell and enable the indexes server-side, but the Collection object still won't know about them. So the app will let you add multiple instances to the Collection cache, while on the server-side the additional inserts will fail silently (errors get written to the output). When you do a hard page refresh, the app will re-sync with server
So your best bet for now is probably to do something like:
var count = MyCollection.find({first_id: 'foo', another_id: 'bar'}).count()
if (count === 0)
MyCollection.insert({first_id: 'foo', another_id: 'bar'});
Which is obviously not ideal, but works ok. You could also enable indexing in mongodb on the server, so even in the case of a race condition you won't actually get duplicate records.
The Smartpackage aldeed:collection2 supports unique indices, as well as schema-validation. Validation will both occure on server and client (reactivly), so you can react on errors on the client.
Actually why not use upsert on the server with a Meteor.method and you could also send also track it with a ts:
// Server Only
Meteor.methods({
add_only_once = function(id1,id2){
SomeCollection.update(
{first_id:id1,another_id:id2},{$set:{ts:Date.now()}},{upsert:True});
}
});
// Client
Meteor.call('add_only_once',doc1._id, doc2._id);
// actual code running on server
if(Meteor.is_server) {
Meteor.methods({
register_code: function (key,monitor) {
Codes.update({key:key},{$set:{ts:Date.now()}},{upsert:true});
}
...

Resources