I’m currently looking at improving the performance of a query that’s using Array_Contains in CosmosDB, for some reason it’s not using the index I’m yet to figure out why.
SELECT COUNT(1) as COUNT FROM c
WHERE ARRAY_CONTAINS(c.statusList, '2c89f4c1b3ac44828dsdfdb6f6c4bd') = false
Attached are the query stats.
Does anyone have any idea what’s the issue might be or how to improve the query performance?
FYI this performance is on index v2, which was newly created collection. Also does anyone how we can figure out when index version a old collection is running?
Here are the performance stats.
ARRAY_CONTAINS normally will use the index when using an aggregate such as Count() when the filter = true. However it doesn't use it when the filter is = false.
This was a known issue and a fix is being worked on now. Likely will not be fully deployed until June. Should require no changes on your end when it does. The query will just improve automatically.
Related
I am trying to get the CostCenter, MainAccount and Department starting from the LedgerDimension field in the LedgerJournalTrans table.
I found this but I am lost.
http://ax2009developer.blogspot.ro/2014/02/how-create-customize-look-up-for.html
In fact, for this task, I have implemented only queries in AOT. Is there any way to join some tables and get there without taking the X++ approach?
Financial dimensions in AX 2012 are far more complicated than in previous versions.
You should start with this white paper: http://download.microsoft.com/download/4/E/3/4E36B655-568E-4D4A-B161-152B28BAAF30/Implementing_the_Account_and_Financial_Dimensions_Framework_AX2012.pdf
You'll find the tables involved and their relations.
By the way, I recommend you not to build your own queries. As the model is really versatile, it will be first tricky to build your query, then they will not be performant.
You need to use the APIs as they are already built and also as they use the system global object cache to cache data, as the model is not set for fast queries.
Unfortunately, I don't believe there is an easy way to do what you want with queries only and X++ is the way to go.
You could, in theory, create a view that you would use in your query objects. It would have tables DimensionAttribute, DimensionAttributeValueSet DimensionAttributeValueSetItem, and DimensionAttributeValue I think. And multiple instances of each in some cases.
Then in your view, you'd set ranges with your different Attribute names. This is fairly complex, but you could repeatedly use it on any query. I could see value in it for sure, but if you've not worked much with dimensions, you have some learning to do to get that working.
Simple SQLite SELECT query on Windows Phone is very slow on a high-end device (Lumia 930).
select * from tableName
It's fetching around 15000 records (yeah, I need them all) and normally I'd expect it to not be this slow. However, it takes around 12-13 seconds to get all the records. I'm using SQLite.Net-PCL client.
What could be causing it? Is it true that it's due to the very slow wrapper? Is there a workaround, any way to improve it?
EDIT: I tried using SQLite PCL from Microsoft Open Technologies and I manually mapped property by property and I got much better results. So it seems that the count of rows, count of columns and the reflection, all combined, cause things to slow down. I am now working on trying to expose a similar functionality through SQLite.NET-PCL, the library which I'm using, to see how that would go.
EDIT2: I marked Peter's answer as answer to my question as I was able to improve performance dramatically by manually mapping type by type using Prepare call and stepping through row by row.
SQLite can easily return 15,000 records from a simple table in a fraction of a second on a Windows Phone (tested on a Lumia 920).
There are other things causing your poor performance. If you have a huge number of columns, that might be a problem. Depending on how the SQLite wrapper is implemented (I don't know), two possible culprits are use of Reflection to fill your result objects or per-row Async overhead. But again, I don't know how that wrapper is implemented specifically.
The way to speed it up (other than to return less data) is to write your code in C++ and wrap it in a WinRT component to be called by your managed app.
Depends on what information you want from your entities you may try using Query<>() method that alows you to write raw SQL query and then you can select only fields that you are interested in and map it to lighter entities if that is possible, even if you getting all fields for your entitiy Query<>() still should be quicker.Also check if you are using lates SQLite driver for WP
I'm creating my main project functionality right now so it's kind of a big decision to make in my project, I want efficient & scalable solution. I use different API's to fetch users products ultimately for 1 collection to display products information inside a table with possible merge by SKU TITLE from different sources.
I have thought of 2 approaches (In both approaches we add Meteor.userId() to collection insert so each users has it's own products:
1) to create each API it's own collection and fetch the products to it, after or in middle of the API query where I insert it to sourceXProducts also add the logic of merge products by sku and add it to main usersProducts Only the fields I need, and we have the collection of the sourceXproducts if we ever need anything we didn't really include to main usersProducts we can query it and get it so we basically keep all the information possible (because it can come handy)
source1Products = new Meteor.Collection('source1Products');
source2Products = new Meteor.Collection('source2Products');
usersProducts = new Meteor.Collection('usersProducts');
Pros: Honestly I'm not sure, It makes it organized also the way I learned Meteor it seems to be used a lot.
Cons: Meteor collection joins is not supported in core yet, So I have to use a meteor package such as: meteor-publish-composite which seems good but this way might hit performance
2) Create 1 collection and just insert everything the API resonse has and additional apiSource field so we can choose products from X user X api.
usersProducts = new Meteor.Collection('usersProducts');
Pros: No joins, possibly better performance
Cons: Not organized, It can become a large collection maybe it's not good for mongodb
3) Your ideas? :)
First, you should improve the question. You do not tell us anything precise about your schema. What are the entities you have and what type of relations are there and what type of joins do you think you will be doing. How often you will be doing them?
Second, you should rethink your schema and think in the terms of a non-relational database. I see many people coming from SQL world and then they simply design their schema in the same way. Wrong. MongoDB is not SQL and things you learned there you should not try to just reuse here. You should start using features like subdocuments and arrays which can help you solve many basic things you would do in SQL with joins. So, knowing your schema would help us help you design the schema. For example, see this answer and especially the comments for the discussion for a similar type of question you are asking here.
Third, have you evaluated various solutions which exist out there? There are many, but you have not shown us that you tried any of them and how it worked for you. What were pros and cons of them, for you and your project?
Fourth, if you are lazy to evaluate, you can just use peerlibrary:peerdb and peerlibrary:related. They are simply perfect. You should trust me. I am their author.
Search is the most used feature on our website and the search query is the most CPU intensive, complex and frequent query that executes on our db, causing heavy CPU usages on the db server. To reduce the load on the db we have been looking at various caching strategies. For now, we intend to use the ASP.NET Cache.
The idea is to have an in-memory db of the most frequently/recently created/accessed objects in the cache and then query the in-memory db using linq to come up with search results. My initial thought was to Cache a List of the Users and then query or modify this List using linq. But given the complexities of multiple threads accessing or trying to modify List I was looking at other options.
Which is when I thought that instead of Caching a List, cache the individual User objects with its Id as the key and try and query the Cache. At http://msdn.microsoft.com/en-us/library/system.web.caching.cache.aspx I see that the Cache has an extension method AsQueryable but I am not sure what does this mean. Cache is a key value pair so with AsQueryable will I able to query the keys and get a set of User Objects or will I able to query the User objects and get my desired result?
Before you start this you really need to have some measurability in place around it -- there is no way to figure out if your changes help or hurt without having some good, solid data to make that judgement on. Performance, especially performance at scale isn't something you can think or guess through. You have to know your way through it.
As for your solution, I think you might well make the problem worse or at least create another problem here. Your database server is theoretically designed to handle arbitrary user queries across vast information sets efficiently. Linq is awesome but it is not really meant to be an ad-hoc search engine -- it doesn't have the sorts of indexing capabilities one really expects from search engines. Just because it can expose things as an IQueryable doesn't mean you should treat it that way. And even if you've got a way to efficently search the cache, you've got another problem to get past -- how do you identify what is most frequently used? And how do you manage the ASP.NET cache to not start ejecting things when it gets low on memory?
You would probably be better served here by:
Starting with some good old fashioned database tuning -- why are your queries so slow and expensive? Are you missing an index somewhere?
Looking at caching the results page output, especially if your search URLs are GET-able as that is pretty easy to manage. This is a great short term solution if the site is melting.
Look at building the search bits properly. Using LIKE %whatever% is not a proper search. Full text indexes in your database is a good start. Something like lucene.net is probably better.
No, cannot use AsQueryable to query User objects and get the desired result I was looking for. So now I will be using a static List for the time being though I know I will have to change sooner rather than later.
In looking up how to perform an equivalent to SELECT TOP 5 with LINQ-to-SQL, all the answers I've seen suggest using .Take(), like so:
var myObject = (
from myObjects in repository.GetAllMyObjects()
select myObject)
.Take(10);
I don't yet understand most of how LINQ works behind-the-scenes but to my understanding of C-like languages this would resolve by first assigning a temporary array containing ALL records, then copying the first 10 elements in the array to var. Not such a problem if you're working on a small dataset or without any performance constraints but it seems horribly inefficient to me if you're, for example, selecting the most recent 5 log entries from a table which can contain millions of records.
Is my understanding of how this works wrong? If so, could someone explain what actually happens? Otherwise, what (if any) better (ie more efficient) way is there of only selecting x records through LINQ-to-SQL?
[edit]
I have the hypothetical myObject class sending LINQ-to-SQL output to the debug output as per the suggestion in the accepted answer. I ended up using the DebuggerWriter from here: http://www.u2u.info/Blogs/Kris/Lists/Posts/Post.aspx?ID=11
Your assumption is incorrect. With Linq to SQL, it evaluates to an Expression<Func<...>> which can be evaluated and the proper SQL generated. You do not need to worry about it loading all the records.
Also, see this following question. You can attach a stream to your DataContext and see the SQL generated.
How to get the TSQL Query from LINQ DataContext.SubmitChanges()
LINQ uses deferred execution, and, for LINQ-to-SQL, expression trees.
No query will be executed until you enumerate the result of the Take call, so you don't need to worry about anything.
I just went through this last week! I opened the SQL profiler on my dev data base and stepped through the code. It was very interesting to see the generated SQL for the various queries. I recommend you do the same. It may not be an exact answer to your question but it was certainly enlightening to see how your various components generate entirely different SQL statements depending on the contents of the call.
I believe the "deferred query resolution" or something (?) reading on MSDN would be enlightening as well.