I can't seem to find any good examples but what is the best way to do a query like this
tbl.Query(SELECT * FROM PRODUCTS WHERE ProductId IN (#Ids)", idlist)
I see that Dapper ORM, you can just put in a list and it knows what to do. Can you do this with Massive or do I have to build out the parameters like #0,#1,#2 and turn the list into an array idList.ToArray()?
It doesn't look like this is possible. When I tried (ids was an array in my case) I got the error:
No mapping exists from object type System.Int32[] to a known managed provider native type.
Which tends to make me think that Massive is largely letting the underlying SQL provider deal with the parameterization of the query. So yes, you are going to have to do build the parameters like you said. You could do something like this:
productTable.Query(string.Format("SELECT * FROM PRODUCTS WHERE ProductId IN ({0})",
string.Join(",", ids.Select((id, ix) => "#" + ix).ToArray())),
ids.Cast<object>().ToArray());
Related
We are trying to optimize a project that is consumig a lot of memory resources. All of our query is done using this kind of sintaxes:
$qb->select(array('e'))
->from('MyBundle:Event', 'e');
This is converted in a query selecting every field of the table, like this:
SELECT t0.id AS id1,
t0.field1 AS field12,
t0.field2 AS field23,
t0.field3 AS field34,
t0.field4 AS field45,
FROM event t0
It's a good ideia for performance to use Partial Object Syntax for hydrating only some predefined fields? I really don't know if it will affect performance and I will have a lot of disadvantages because other fields will be null. What do you use to do in your select queries with Doctrine?
Regards.
My two cents
I suppose that hydration (Object Hydration and lazy loading, of course) is good until you don't know how many and what fields to pull from DB tables and put into objects. If you know that you have to retrieve all fields, is better to get them once and work with them, instead of do every time a query that is time-consuming.
However, as a good practice, when I have retrieved and used my objects I unset them explicitly (not if they are last instructions of my function that will return and implicitly unset them)
Update
$my_obj_repo = $this->getDoctrine()->getManager()->getRepository('MyBundleName:my_obj');
$my_obj = $my_obj_repo->fooHydrateFunction(12); //here I don't pull out from db all data
//do stuff with this object, like extracting data or manipulating data
if($my_obj->getBarField() == null) //this is the only field I've load with fooHydrateFunction, so no lazy loading
{
$my_obj->setBarField($request->query->get('bar');
$entity_manager->persist($my_obj);
$entity_manager->flush();
}
//here my object isn't necessary anymore
unset($my_obj); //this is a PHP instruction
//continue with my business logic
So far, I've been using classic ADO.NET model for database access. I have to tell you that I'm quite happy with it. But I have also been hearing much about Entity Framework recently so I thought I could give it a try. Actually the main reason which pushed me was the need to find a way to build the WHERE clause of my Stored Procedures. With the classic way I have to do either of the following:
Build the WHERE clause in the client side based on the user inputs and send it as a VARCHAR2 argument to the Stored Procedure, concatenate the WHERE clausewith the main part of the SQL and pass the whole string to EXECUTE_IMMEDIATE function. I personally hate to have to do so.
Inside the Stored Procedure construct lots and lots of SQL statements, which means I have to take all the possible combinations that WHERE clause might be composed of into account. This seems worse than the first case.
I know that EF has made it possible to use Stored Procedures as well. But will it be possible to build the WHERE part dynamically? Can EF rescue me somehow?
yes, you can use Dynamic queries in Linq.
Dynamic Query LIbrary
from scott gu example
var query = Northwind.Products.Where("Lastname LIKE "someValue%");
or some complex query
var query =
db.Customers.
Where("City = #0 and Orders.Count >= #1", "London", 10).
OrderBy("CompanyName").
Select("new(CompanyName as Name, Phone)");
or from this answer Where clause dynamically.
var pr = PredicateBuilder.False<User>();
foreach (var name in names)
{
pr = pr.Or(x => x.Name == name && x.Username == name);
}
return query.AsExpandable().Where(pr);
My situation
I have a c# object which contains some lists. One of these lists are for example a list of tags, which is a list of c# "SystemTag"-objects. I want to instantiate this object the most efficient way.
In my database structure, I have the following tables:
dbObject - the table which contains some basic information about my c# object
dbTags - a list of all available tabs
dbTagConnections - a list which has 2 fields: TagID and ObjectID (to make sure an object can have several tags)
(I have several other similar types of data)
This is how I do it now...
Retrieve my object from the DB using an ID
Send the DB object to a "Object factory" pattern, which then realise we have to get the tags (and other lists). Then it sends a call to the DAL layer using the ID of our C# object
The DAL layer retrieves the data from the DB
These data are send to a "TagFactory" pattern which converts to tags
We are back to the Object Factory
This is really inefficient and we have many calls to the database. This especially gives problems as I have 4+ types of lists.
What have I tried?
I am not really good at SQL, but I've tried the following query:
SELECT * FROM dbObject p
LEFT JOIN dbTagConnection c on p.Id= c.PointId
LEFT JOIN dbTags t on c.TagId = t.dbTagId
WHERE ....
However, this retreives as many objects as there are tagconnections - so I don't see joins as a good way to do this.
Other info...
Using .NET Framework 4.0
Using LINQ to SQL (BLL and DAL layer with Factory patterns in the BLL to convert from DAL objects)
...
So - how do I solve this as efficient as possible? :-) Thanks!
At first sight I don't see your current way of work as "inefficient" (with the information provided). I would replace the code:
SELECT * FROM dbObject p
LEFT JOIN dbTagConnection c on p.Id= c.PointId
LEFT JOIN dbTags t on c.TagId = t.dbTagId
WHERE ...
by two calls to the DALs methods, first to retrieve the object main data (1) and one after that to get, only, the data of the tags related (2) so that your factory can fill-up the object's tags list:
(1)
SELECT * FROM dbObject WHERE Id=#objectId
(2)
SELECT t.* FROM dbTags t
INNER JOIN dbTag Connection c ON c.TagId = t.dbTagId
INNER JOIN dbObject p ON p.Id = c.PointId
WHERE p.Id=#objectId
If you have many objects and the amount of data is just a few (meaning that your are not going to manage big volumes) then I would look for a ORM based solution as the Entity Framework.
I (still) feel comfortable writing SQL queries in the DAOs to have under control all queries being sent to the DB server, but finally it is because in our situation is a need. I don't see any inconvenience on having to query the database to recover, first, the object data (SELECT * FROM dbObject WHERE ID=#myId) and fill the object instance, and then query again the DB to recover all satellite data that you may need (the Tags in your case).
You have be more concise about your scenario so that we can provide valuable recommendations for your particular scenario. Hope this is useful you you anyway.
We used stored procedures that returned multiple resultsets, in a similar situation in a previous project using Java/MSSQL server/Plain JDBC.
The stored procedure takes the ID corresponding to the object to be retrieved, return the row to build the primary object, followed by multiple records of each one-to-many relationship with the primary object. This allowed us to build the object in its entirety in a single database interaction.
Have you thought about using the entity framework? You would then interact with your database in the same way as you would interact with any other type of class in your application.
It's really simple to set up and you would create the relationships between your database tables in the entity designer - this will give you all the foreign keys you need to call related objects. If you have all your keys set up in the database then the entity designer will use these instead - creating all the objects is as simple as selecting 'Create model from database' and when you make changes to your database you simply right-click in your designer and choose 'update model from database'
The framework takes care of all the SQL for you - so you don't need to worry about that; in most cases..
A great starting place to get up and running with this would be here, and here
Once you have it all set up you can use LINQ to easily query the database.
You will find this a lot more efficient than going down the table adapter route (assuming that's what you're doing at the moment?)
Sorry if i missed something and you're already using this.. :)
As far I guess, your database exists already and you are familiar enough with SQL.
You might want to use a Micro ORM, like petapoco.
To use it, you have to write classes that matches the tables you have in the database (there are T4 generator to do this automatically with Visual Studio 2010), then you can write wrappers to create richer business objects (you can use the ValueInjecter to do it, it is the simpler I ever used), or you can use them as they are.
Petapoco handles insert / update operations, and it retrieves generated IDs automatically.
Because Petapoco handles multiple relationships too, it seems to fit your requirements.
I've been tasked with creating an application that allows users the ability to enter data into a web form that will be saved and then eventually used to populate pdf form fields.
I'm having trouble trying to think of a good way to store the field values in a database as the forms will be dynamic (based on pdf fields).
In the app itself I will pass data around in a hash table (fieldname, fieldvalue) but I don't know the best way to convert the hash to db values.
I'm using MS SQL server 2000 and asp.net webforms. Has anyone worked on something similar?
Have you considered using a document database here? This is just the sort of problem they solve alot better than traditional RDBMS solutions. Personally, I'm a big fan of RavenDb. Another pretty decent option is CouchDb. I'd avoid MongoDb as it really isn't a safe place for data in it's current implementation.
Even if you can't use a document database, you can make SQL pretend to be one by setting up your tables to have some metadata in traditional columns with a payload field that is serialized XML or json. This will let you search on metadata while staying out of EAV-land. EAV-land is a horrible place to be.
UPDATE
I'm not sure if a good guide exists, but the concept is pretty simple. The basic idea is to break out the parts you want to query on into "normal" columns in a table -- this lets you query in standard manners. When you find the record(s) you want, you can then grab the CLOB and deserialize it as appropriate. In your case you would have a table that looked something like:
SurveyAnswers
Id INT IDENTITY
FormId INT
SubmittedBy VARCHAR(255)
SubmittedAt DATETIME
FormData TEXT
A few protips:
a) use a text based serialization routine. Gives you a fighting chance to fix data errors and really helps debugging.
b) For SQL 2000, you might want to consider breaking the CLOB (TEXT field holding your payload data) into a separate table. Its been a long time since I used SQL 2000, but my recollection is using TEXT columns did bad things to tables.
The solution for what you're describing is called Entity Attribute Value (EAV) and this model can be a royal pain to deal with. So you should limit as much as possible your usage of this.
For example are there fields that are almost always in the forms (First Name, Last Name, Email etc) then you should put them in a table as fields.
The reason for this is because if you don't somebody sooner or later is going to realize that they have these names and emails and ask you to build this query
SELECT
Fname.value fname,
LName.Value lname,
email.Value email,
....
FROM
form f
INNER JOIN formFields fname
ON f.FormId = ff.FormID
and AttributeName = 'fname'
INNER JOIN formFields lname
ON f.FormId = ff.FormID
and AttributeName = 'lname'
INNER JOIN formFields email
ON f.FormId = ff.FormID
and AttributeName = 'email'
....
when you could have written this
SELECT
common.fname,
common.lname,
common.email,
....
FROM
form f
INNER JOIN common c
on f.FormId = c.FormId
Also get off of SQL 2000 as soon as you can because you're going to really miss the UNPIVOT clause
Its also probably not a bad idea to look at previous SO EAV questions to give you an idea of problems that people have encountered in the past
I'd suggest mirroring the same structure:
Form
-----
form_id
User
created
FormField
-------
formField_id
form_id
name
value
I just started learning LINQ to SQL, and so far I'm impressed with the easy of use and good performance.
I used to think that when doing LINQ queries like
from Customer in DB.Customers where Customer.Age > 30 select Customer
LINQ gets all customers from the database ("SELECT * FROM Customers"), moves them to the Customers array and then makes a search in that Array using .NET methods. This is very inefficient, what if there are hundreds of thousands of customers in the database? Making such big SELECT queries would kill the web application.
Now after experiencing how actually fast LINQ to SQL is, I start to suspect that when doing that query I just wrote, LINQ somehow converts it to a SQL Query string
SELECT * FROM Customers WHERE Age > 30
And only when necessary it will run the query.
So my question is: am I right? And when is the query actually run?
The reason why I'm asking is not only because I want to understand how it works in order to build good optimized applications, but because I came across the following problem.
I have 2 tables, one of them is Books, the other has information on how many books were sold on certain days. My goal is to select books that had at least 50 sales/day in past 10 days. It's done with this simple query:
from Book in DB.Books where (from Sale in DB.Sales where Sale.SalesAmount >= 50 && Sale.DateOfSale >= DateTime.Now.AddDays(-10) select Sale.BookID).Contains(Book.ID) select Book
The point is, I have to use the checking part in several queries and I decided to create an array with IDs of all popular books:
var popularBooksIDs = from Sale in DB.Sales where Sale.SalesAmount >= 50 && Sale.DateOfSale >= DateTime.Now.AddDays(-10) select Sale.BookID;
BUT when I try to do the query now:
from Book in DB.Books where popularBooksIDs.Contains(Book.ID) select Book
It doesn't work! That's why I think that we can't use thins kinds of shortcuts in LINQ to SQL queries, like we can't use them in real SQL. We have to create straightforward queries, am I right?
You are correct. LINQ to SQL does create the actual SQL to retrieve your results.
As for your shortcuts, there are ways to work around the limitations:
var popularBooksIds = DB.Sales
.Where(s => s.SalesAmount >= 50
&& s.DateOfSale >= DateTime.Now.AddDays(-10))
.Select(s => s.Id)
.ToList();
// Actually should work.
// Forces the table into memory and then uses LINQ to Objects for the query
var popularBooksSelect = DB.Books
.ToList()
.Where(b => popularBooksIds.Contains(b.Id));
Yes, query gets translated to a SQL string, and the underlying SQL can be different depending on what you are trying to do... so you have to be careful in that regard. Checkout a tool called linqpad, you can try your query in it and see the executing SQL.
Also, it runs when iterating through the collection or calling a method on it like ToList().
Entity framework or linq queries can be tricky sometimes. Sometimes you are surprised at the efficiency of the sql query generated and sometimes the query is so complicated and inefficient that you would smack your forehead.
Best idea is that if you have any suspicions about a query, run an sql profiler at the backend that would monitor all the queries coming in. That way you know exactly what is being passed on to the sql server and correct any inefficiencies if need be.
http://damieng.com/blog/2008/07/30/linq-to-sql-log-to-debug-window-file-memory-or-multiple-writers
This will help you to see what and when queries are being run. Also, Damiens blog is full of other linq to sql goodness.
You can generate an EXISTS clause by using the .Any method. I have had more success that way than trying to generate IN clauses, because it likes to retrieve all the data and pass it all back in as parameters to a query
In linq to sql, IQueryable expression fragments can be combined to create a single query, it will try to keep everything as an IQueryable for as long as it can, before you do something that cannot be expressed in SQL. When you call ToList you are directly asking it to resolve that query into an IEnumerable stored in memory.
In most cases you are better off not selecting the book ids in advance. Keep the fragment for popular books in a single place in the code and use it when necessary, to build on another query. An IQueryable is just an expression tree, which is resolved into SQL at some other point.
If you think your application will perform better by storing the popular books elsewhere (memcache or whatever), then you may consider pulling them out before hand, and checking against that later. This will mean each book id will be passed in as a sproc parameter and used in an IN clause.