I am trying to get a full list of every group in Azure AD. I am currently able to get 999 records with the following uri:
https://graph.microsoft.com/v1.0/groups?$top=999
According to the documentation from Microsoft there are only a couple OData query parameters available, none of which appear to be able to navigate to the next page. It also states the maximum page size is 999. I have tried using the $skip parameter to skip a certain number of records, but it is not supported:
{"error":{"code":"Request_BadRequest","message":"'$skip' is not supported by the service.",...
Is there any way to get a full list of all AAD groups? We have several thousand that I would need to get.
Some queries against Microsoft Graph return multiple pages of data either due to server-side paging or due to the use of the $top query parameter to specifically limit the page size in a request. When more than one query request is required to retrieve all the results, Microsoft Graph returns an #odata.nextLink property in the response that contains a URL to the next page of results.
For example, the following URL requests all the users in an organization with a page size of 5, specified with the $top query parameter:
https://graph.microsoft.com/v1.0/groups?$top=5
If the result contains more results, Microsoft Graph will return an #odata.nextLink property similar to the following along with the first page of results:
You can retrieve the next page of results by sending the URL value of the #odata.nextLink
ref doc - https://learn.microsoft.com/en-us/graph/paging
With $top, you can customize the result size within the range of 1 and 1000. Because of your question, I guess that 1000 is exclusive, so the range goes from 1 to 999 (inclusive). Read more about the query parameter $top here. I got the range information from List messages.
The response you get from List groups seems to not contain #odata.nextLink which you would normally expect in such a case, so GET https://graph.microsoft.com/v1.0/groups does not support pagination. That would also explain why you get an error if you try to use $skip. You can read more about $skip here.
In order to get the full list of all groups, I would stop using the query parameter $top.
Related
At first sight, it's clear what the continuation token does in Cosmos DB: attaching it to the next query gives you the next set of results. But what does "next set of results" mean exactly?
Does it mean:
the next set of results as if the original query had been executed completely without paging at the time of the very first query (skipping the appropriate number of documents)?
the next set of results as if the original query had been executed now (skipping the appropriate number of documents)?
Something completely different?
Answer 1. would seem preferable but unlikely given that the server would need to store unlimited amounts of state. But Answer 2. is also problematic as it may result in inconsistencies, e.g. the same document may be served multiple times across pages, if the underlying data has changed between the page queries.
Cosmos DB query executions are stateless at the server side. The continuation token is used to recreate the state of the index and track progress of the execution.
"Next set of results" means, the query is executed again on from a "bookmark" from the previous execution. This bookmark is provided by the continuation token.
Documents created during continuations
They may or may not be returned depending on the position of insert and query being executed.
Example:
SELECT * FROM c ORDER BY c.someValue ASC
Let us assume the bookmark had someValue = 10, the query engine resumes processing using a continuation token where someValue = 10.
If you were to insert a new document with someValue = 5 in between query executions, it will not show up in the next set of results.
If the new document is inserted in a "page" that is > the bookmark, it will show up in next set of results
Documents updated during continuations
Same logic as above applies to updates as well
(See #4)
Documents deleted during continuations
They will not show up in the next set of results.
Chances of duplicates
In case of the below query,
SELECT * FROM c ORDER BY c.remainingInventory ASC
If the remainingInventory was updated after the first set of results and it now satisfies the ORDER BY criteria for the second page, the document will show up again.
Cosmos DB doesn’t provide snapshot isolation across query pages.
However, as per the product team this is an incredibly uncommon scenario because queries over continuations are very quick and in most cases all query results are returned on the first page.
Based on preliminary experiments, the answer seems to be option #2, or more precisely:
Documents created after serving the first page are observable on subsequent pages
Documents updated after serving the first page are observable on subsequent pages
Documents deleted after serving the first page are omitted on subsequent pages
Documents are never served twice
The first statement above contradicts information from MSFT (cf. Kalyan's answer). It would be great to get a more qualified answer from the Cosmos DB Team specifying precisely the semantics of retrieving pages. This may not be very important for displaying data in the UI, but may be essential for data processing in the backend, given that there doesn't seem to be any way of disabling paging when performing a query (cf. Are transactional queries possible in Cosmos DB?).
Experimental method
I used Sacha Bruttin's Cosmos DB Explorer to query a collection with 5 documents, because this tool allows playing around with the page size and other request options.
The page size was set to 1, and Cross Partition Queries were enabled. Different queries were tried, e.g. SELECT * FROM c or SELECT * FROM c ORDER BY c.name.
After retrieving page 1, new documents were inserted, and some existing documents (including documents that should appear on subsequent pages) were updated and deleted. Then all subsequent pages were retrieved in sequence.
(A quick look at the source code of the tool confirmed that ResponseContinuationTokenLimitInKb is not set.)
I have setup an api that will query our Cosmos db and return the JSON results back to the front end app. There is a user defined limit on the number of results. If the number of results exceed the limit then I pass back the token to the front end and they can call for the next group of rows. The issue is I would like to provide a count of the Total Number of Matches back to the application. I have looked at the query statistics but don't see where there is a total count.
On the call to CreateDocumentQuery, i'm setting MaxItemCount to the limit, and RequestContinuation either null or the continuationToken. Looking at QueryMetrics I found RetrievedDocumentCount, but that does not seem to have the correct value.
Thanks,
J
x-ms-max-item-count request header controls how many documents should be returned to user.
Default value is 100
if your query returns 150 documents, your request will return first 100 documents and it will return a continuation token in response header(x-ms-continuation). If there is a token, you need to send another request with the given token to get the rest of the data.
SDK should be doing that for you automatically. Can you share some of your code. I might have a better answer then.
You can check out my post about this too.
https://h-savran.blogspot.com/2019/04/introduction-to-continuation-tokens-in.html
I have a Collection of Posts and a Collection of Users. Posts have certain attributes which are irrelevant for this, but they also have one attribute called tags right now its an array with certain words.
A User can follow certain Tags so he has a attribute followedTags which is also an Array right now that contains the Tags he follows.
Now one of the use cases for this is a User Feed to Show only Posts with Tags he follows, the Problem is that I only found methods to query this for ONE Attribute at a time(in Arrays). Since I dont wanna run 20 Querys for 1 Feed (For example if the user follows 20 Tags) I thought maybe I could make a smart change in the data modell itself, any suggestions?
Problem is that I only found methods to query this for ONE Attribute at a time(in Arrays)
If you try to chain multiple whereArrayContains() methods, you're most likely getting the following error:
Caused by: java.lang.IllegalArgumentException: Invalid Query. Queries only support having a single array-contains filter.
So unfortunately Firestore can only allow a single call to whereArrayContains() method in query.
Since I dont wanna run 20 Querys for 1 Feed (For example if the user follows 20 Tags)
If you have a reasonable number tags, you can create each tag as a separate property. In this case, it is allowed to call Firestore Query's whereEqualTo() multiple times.
If this is not the case, then you should consider augmenting your database structure to allow a reverse lookup by creating each tag as a seprate object (document) in a tagCollection. Under each document you can create a new collection named tagPost in which you should add all the posts that are labeled with a specific tag.
I need to get the Marketo Leads who have changes on "progressionStatus" field (inside membership) with the API.
I can get all the leads related to a Program (with Get Leads by ProgramID API) without issues, but my need is to get those Leads with changes on "progressionStatus" column.
I was thinking to use the CreatedAt / UpdatedAt fields of the Program, so then, get all the leads related to those programs. But I didn't get the accurate results that I want.
Also, I tried to use the GET Lead changes API and use "fields" parameter to "progressionstatus" but that field don't exist.
It is possible to resolve this?
Thanks in advance.
You can get the list of Leads with progression status change by querying against the Get Lead Activities endpoint.
The Get Lead Changes endpoint could sound as a good candidate, but that endpoint only returns changes on the lead fields. Progression status change is not stored on the lead directly, so at the end that won't work. On the other hand the Get Leads by ProgramId endpoint returns –amongst others– the actual value of progressionStatus (program status of the lead in the parent program) but not the “change” itself, so you cannot process the resultset based on that.
The good news is that the progression status change is an activity type and luckily we have the above mentioned Get Lead Activities endpoint (which is also mentioned as the Query in the API docs) available to query just that. This endpoint also allows for filtering by activityTypeIds to narrow down the resultset to a single activity type.
Basically you have to call the GET /rest/v1/activities.json enpoint and pass the values of activityTypeIds and nextPageToken as query parameters (next to the access token obviously). So, first you need to get the internal Id of the activity type called “Change Status in Progression”. You can do that by querying the GET /rest/v1/activities/types.json endpoint and look for a record with that name. (I don't know if this Id changes from instance to instance, but in ours it is the #104). Also, to obtain a nextPageToken you have to make a call to GET /rest/v1/activities/pagingtoken.json as well, where you have to specify the earliest datetime to retrieve activities from. See more about Paging Tokens.
Once you have all of these bits at hand, you can make your request like that:
GET https://<INSTANCE_ID>.mktorest.com/rest/v1/activities.json?activityTypeIds=<TYPE_ID>&nextPageToken=<NEXTPAGE_TOKEN>&access_token=<ACCESS_TOKEN>
The result it gives is an array with items like below, which is easy to process further.
{
"id":712630,
"marketoGUID":"712630",
"leadId":824864,
"activityDate":"2017-12-01T08:51:13Z",
"activityTypeId":104,
"primaryAttributeValueId":1104,
"primaryAttributeValue":"PROGRAM_NAME",
"attributes":[
{"name":"Acquired By","value":true},
{"name":"New Status ID","value":33},
{"name":"Old Status ID","value":32},
{"name":"Reason","value":"Filled out form"},
{"name":"Success","value":false},
{"name":"New Status","value":"Filled-out Form"},
{"name":"Old Status","value":"Not in Program"}
]
}
Knowing the leadIds in question, you can make yet another request to fetch the actual lead records.
I want to develop an app that pulls the singers of any song that we query for. So if someone types in Carry On from the Some Nights album, the app is supposed to pull out who all sang that song. Thanks.
You can search for this using the Freebase Search API and Search Metaschema like this:
https://www.googleapis.com/freebase/v1/search?query=Carry+On&filter=(all+/music/release_track/release:"Some+Nights")&output=(/music/release_track/release+/music/release_track/recording./music/recording/artist)
There are three parts to this API request: the query, the filter and the output parameter. The query is simply the name of the track that you're looking for:
query=Carry+On
The filter parameter constrains the results to only tracks which are part of an album release named "Some Nights"
filter=(all+/music/release_track/release:"Some+Nights")
The output parameter tells the API which properties to return in the response. In this case we want to know which release the track is part of and which artist recorded the track.
output=(/music/release_track/release+/music/release_track/recording./music/recording/artist)
You'll notice that this query actually returns 8 matching tracks right now. This is because there were many different releases of the album which all contained recordings of that track (and not necessarily the exact same recording).
For what you're building it sounds like you should be able to just take the first result. You can constrain the search API to only return the first result by adding a limit parameter to the request:
limit=1