I try to tune my query but I have no idea what I can change:
A screenshot of both tables: http://abload.de/image.php?img=1plkyg.jpg
The relation is: 1 UserPM (a Private Message) has 1 Sender (User, SenderID -> User.SenderID) and 1 Recipient (User, RecipientID -> User.UserID) and 1 User has X UserPMs as Recipient and X UserPMs as Sender.
The intial load takes around 200ms, it only takes the first 20 rows and display them. After this is displayed a JavaScript PageMethod gets the GetAllPMsAsReciepient method and loads the rest of the data
this GetAllPMsAsReciepient method takes around 4.5 to 5.0 seconds each time to run on around 250 rows
My code:
public static List<UserPM> GetAllPMsAsReciepient(Guid userID)
{
using (RPGDataContext dc = new RPGDataContext())
{
DateTime dt = DateTime.Now;
DataLoadOptions options = new DataLoadOptions();
//options.LoadWith<UserPM>(a => a.User);
options.LoadWith<UserPM>(a => a.User1);
dc.LoadOptions = options;
List<UserPM> pm = (
from a in dc.UserPMs
where a.RecieverID == userID
&& !a.IsDeletedRec
orderby a.Timestamp descending select a
).ToList();
TimeSpan ts = DateTime.Now - dt;
System.Diagnostics.Debug.WriteLine(ts.Seconds + "." + ts.Milliseconds);
return pm;
}
}
I have no idea how to tune this Query, I mean 250 PMs are nothing at all, on other inboxes on other websites I got around 5000 or something and it doesn't need a single second to load...
I try to set Indexes on Timestamp to reduce the Orderby time but nothing happend so far.
Any ideas here?
EDIT
I try to reproduce it on LinqPad:
Without the DataLoadOptions, in LinqPad the query needs 300ms, with DataLoadOptions around 1 Second.
So, that means:
I could save around 60% of the time, If I can avoid to load the User-table within this query, but how?
Why Linqpad needs only 1 second on the same connection, from the same computer, where my code is need 4.5-5.0 seconds?
Here is the execution plan: http://abload.de/image.php?img=54rjwq.jpg
Here is the SQL Linqpad gives me:
SELECT [t0].[PMID], [t0].[Text], [t0].[RecieverID], [t0].[SenderID], [t0].[Title], [t0].[Timestamp], [t0].[IsDeletedRec], [t0].[IsRead], [t0].[IsDeletedSender], [t0].[IsAnswered], [t1].[UserID], [t1].[Username], [t1].[Password], [t1].[Email], [t1].[RegisterDate], [t1].[LastLogin], [t1].[RegisterIP], [t1].[RefreshPing], [t1].[Admin], [t1].[IsDeleted], [t1].[DeletedFrom], [t1].[IsBanned], [t1].[BannedReason], [t1].[BannedFrom], [t1].[BannedAt], [t1].[NowPlay], [t1].[AcceptAGB], [t1].[AcceptRules], [t1].[MainProfile], [t1].[SetShowHTMLEditorInRPGPosts], [t1].[Age], [t1].[SetIsAgePublic], [t1].[City], [t1].[SetIsCityShown], [t1].[Verified], [t1].[Design], [t1].[SetRPGCountPublic], [t1].[SetLastLoginPublic], [t1].[SetRegisterDatePublic], [t1].[SetGBActive], [t1].[Gender], [t1].[IsGenderVisible], [t1].[OnlinelistHidden], [t1].[Birthday], [t1].[SetIsMenuHideable], [t1].[SetColorButtons], [t1].[SetIsAboutMePublic], [t1].[Name], [t1].[SetIsNamePublic], [t1].[ContactAnimexx], [t1].[ContactRPGLand], [t1].[ContactSkype], [t1].[ContactICQ], [t1].[ContactDeviantArt], [t1].[ContactFacebook], [t1].[ContactTwitter], [t1].[ContactTumblr], [t1].[IsContactAnimexxPublic], [t1].[IsContactRPGLandPublic], [t1].[IsContactSkypePublic], [t1].[IsContactICQPublic], [t1].[IsContactDeviantArtPublic], [t1].[IsContactFacebookPublic], [t1].[IsContactTwitterPublic], [t1].[IsContactTumblrPublic], [t1].[IsAdult], [t1].[IsShoutboxVisible], [t1].[Notification], [t1].[ShowTutorial], [t1].[MainProfilePreview], [t1].[SetSound], [t1].[EmailNotification], [t1].[UsernameOld], [t1].[UsernameChangeDate]
FROM [UserPM] AS [t0]
INNER JOIN [User] AS [t1] ON [t1].[UserID] = [t0].[RecieverID]
WHERE ([t0].[RecieverID] = #p0) AND (NOT ([t0].[IsDeletedRec] = 1))
ORDER BY [t0].[Timestamp] DESC
If you want to get rid of the LoadWith, you can select your field explicitly :
public static List<Tuple<UserPM, User> > GetAllPMsAsReciepient(Guid userID)
{
using (var dataContext = new RPGDataContext())
{
return (
from a in dataContext.UserPMs
where a.RecieverID == userID
&& !a.IsDeletedRec
orderby a.Timestamp descending
select Tuple.Create(a, a.User1)
).ToList();
}
}
I found a solution:
At first it seems that with the DataLoadOptions is something not okay, at second its not clever to load a table with 30 Coloumns when you only need 1.
To Solve this, I make a view which covers all nececeery fields and of course the join.
It reduces the time from 5.0 seconds to 230ms!
Related
I am new to go and trying to retrieve data from a sqlite database.
Im using github.com/mattn/go-sqlite3 as sqlite driver.
The query I m sending does not return any results even though it should. I tried the query my programme is generating manually and it returns data as it should when I use the query manually as well as when I send it via my programme.
Here is my code:
for index := range Array {
id, _ := strconv.Atoi(Array[index])
rand.Seed(time.Now().UnixNano())
RandomNr := rand.Intn(100)
fmt.Printf("index: %d - randomnr: %d \n", id, RandomNr)
rows, errROW := db.Query("SELECT user.id,user.name,stage.link,player.url,player.characterchance,player.chance FROM user,stage,player WHERE user.id = '%d' AND '%d' <= user.chance AND stage.user = user.id AND stage.domain = player.id AND player.chance > 0 ORDER BY player.id ASC \n",id, RandomNr)//.Scan(&idr, &name, &link, &url, &characterchance, &chance)
//this is what the finished query looks like and it returns the rows as its supposed to
//rows, errROW := db.Query("SELECT user.id,user.name,stage.link,player.url,player.characterchance,player.chance FROM user,stage,player WHERE user.id = '203' AND '33' <= user.chance AND stage.user = user.id AND stage.domain = player.id AND player.chance > 0 ORDER BY player.id ASC")
if errROW != nil {
fmt.Println("errROW")
log.Println(errROW)
}
defer rows.Close()
if rows.Next() == false {
fmt.Println("no rows ")
}
for rows.Next() {
fmt.Println("where are the rows")
var id int
var name string
var link string
var url string
var characterchance int
var chance int
rows.Scan(&id, &name, &link, &url, &characterchance, &chance)
fmt.Println(id,name,link,url,characterchance,chance)
}
rows.Close()
}
}
This query can return multiple and single rows. I also tried retrieving the data via QueryRow as a single row which also did not return any result.
Any help would be much appreciated.
UPDATE:
I added
if rows.Next() == false
as an attempt to find the problem. Removing it yields the same result. Furthermore I do not get an error message from scan. The for rows.next() loop does not even get executed.
when you do:
if rows.Next() == false
you are scrolling to the first row
and
for rows.Next()
moves to the next row
basically, you are skipping the first row in your result set in the example code you provided.
also, you are ignoring the error in Scan.
This looks like it would print something if the query returns at least 2 rows (since first row is being skipped)
Ok I figured out what the problem was:
In my query I used: %d as a placeholder for my variable when I should have used $1,$2 etc. Using this the query returns results as expected.
It seems strange to me that this behaviour is allowed returns no error from go or sqlite and even works when you just printout the query and use it with sqlite3 manually. Coming from C and just starting out with go this can obviously be the cause for some headaches.
My app allows user to upload an XML file, which I pass as an XDocument into my method. All the values are attribute strings, and I am using Linq to XML and Linq to SQL.
The dateCutoff query is supposed to get the latest date from SQL table - InsDate is nullable.
The where clause in the inspections XML query is supposed to get inspection elements with the inspection_date attribute value later than the dateCutoff value. I am using DateTime.Parse and Date.CompareTo, but am coming up empty.
What am I missing? Any help is much appreciated.
public IEnumerable<XElement> getInspections(XDocument xDoc)
{
IEnumerable<XElement> inspections = null;
using (InspectionDataContext db = new InspectionDataContext())
{
// get the latest date already in Inspections table
DateTime? dateCutoff = (from d in db.Inspections
select d.InsDate).Max();
if (dateCutoff.HasValue)
{
dateCutoff = dateCutoff.Value.Date;
}
// get only the inspections later than the dateCutoff
inspections = from i in xDoc.Descendants("inspections")
where DateTime.Parse(i.Element("inspection").Attribute("inspection_date").Value).Date.CompareTo(dateCutoff) == 1
select i;
}
return inspections;
}
I'm gong to make a few assumptions here because it's not totally clear as written.
1. You want to return "inspection" elements (not the "inspections" container elements).
2. You want to return only elements whose dates are greater than the cutoff
3. If the cutoff is null, you want to return all inspections.
In that case, you'd do something like this:
var inspections = xDoc.Descendents("inspection");
if (!dateCutoff.HasValue)
return inspections;
return inspections.Where(i => (DateTime)i.Attribute("inspection_date") > dateCutoff.Value )
Here is what works:
DateTime? dateCutoff;
using (InspectionDataContext db = new InspectionDataContext())
{
// get the latest date already in Inspections table
DateTime? dateCutoffQ = (from d in db.Inspections
select d.InsDate).Max();
if (dateCutoffQ.HasValue)
{
dateCutoff = dateCutoffQ.Value.Date;
}
else
{
dateCutoff = DateTime.Now.AddYears(-20).Date;
}
string date1 = dateCutoff.ToString();
// get only the inspections later than the dateCutoff
inspectionList = from i in xDoc.Descendants("inspection")
where DateTime.Parse(i.Attribute("inspection_date").Value).Date > dateCutoff
select i;
I have following linq queries:
var itembind = (from q in dsSerach.Tables[0].AsEnumerable()
select new
{
PatternID = q.Field<int>("PatternID"),
PatternName = q.Field<string>("PatternName") + " " + q.Field<string>("ColorID") + q.Field<string>("BookID"),
ColorID = q.Field<string>("ColorID"),
BookID = q.Field<string>("BookID"),
CoverImage = (from img1 in objJFEntities.ProductImages.ToList()
where img1.PatternName.ToLower() == q.Field<string>("PatternName").ToLower()
select new CoverImage
{
URL = "Images/MediumPatternImages/" +
q.Field<string>("PatternName") + "_" + q.Field<string>("ColorID") + q.Field<string>("BookID") + q.Field<string>("ImageExtension"),
ID = q.Field<int>("ProductImageID")
}).FirstOrDefault(),
TotalCount = q.Field<int>("TotalCount")
}).Distinct();
var patterns = (from r in itembind
group r by new { r.PatternID, r.ColorID } into g
select new SearchPattern
{
PatternID = g.Key.PatternID,
PatternName = string.Join(",", g.OrderBy(s => s.ColorID).OrderBy(s => s.BookID)
.Select(s => String.Format("<a href='{0:s}' title='{1:s}'>{2:s}</a><br />",
new object[] { String.Format("Product.aspx?ID={0}&img={1}", g.Key.PatternID, s.CoverImage.ID), s.PatternName, s.PatternName })).FirstOrDefault()),
CoverImage = g.Count() > 1 ? (from img1 in objJFEntities.ProductImages.ToList()
where img1.ProductImageID == g.Select(i => i.CoverImage.ID).FirstOrDefault() && img1.ColorID.ToString() == g.Key.ColorID
select new CoverImage
{
URL = "Images/MediumPatternImages/" +
img1.PatternName + "_" + img1.ColorID + img1.BookID + img1.ImageExtension,
ID = img1.ProductImageID
}).FirstOrDefault() : g.Select(i => i.CoverImage).FirstOrDefault()
}).ToList();
these queries are taking more then 1 minute to execute for the 1000 records only.
The dsSearch is a dataset filled with records returned from my procedure in SQL.
Am using entity framework. The site is deployed with IIS7.0. The SQL server 2008 is in use.
I got "Error Message:Timeout expired. The timeout period elapsed prior to completion of the operation or the server is not responding." ,
"Cannot open database "DB" requested by the login. The login failed." & "The underlying provider failed on Open." kind of error very frequently site.
Please tell me how to optimize such a query.
EDIT:
Here is the procedure
http://pastie.org/7160934
In the first query you are doing a objJFEntities.ProductImages.ToList() , with the ToList() call you are fetching every entry from the database, and later filter the results in memory.
Rolfvm is correct in pointing out that objJFEntities.ProductImages causes the problem, but the analysis is a bit different. You fetch the entire ProductImages table into memory for each iteration of the query when you enumerate over it. So one optimization would be to fetch the images first in a collection and use that collection in the query statement
var localImages = objJFEntities.ProductImages.ToList();
...
CoverImage = (from img1 in localImages....
But then, your query seems to do far too much. You build the first part itembind without executing it. Then you build the second part (var patterns = (from r in itembind) and execute it by ToList(). But in the second part you never use the CoverImage from the first part. So creating these is a waste of resources. (Or you skimmed the code, hiding another use of the first part).
I want to know that,
If we have LIST object created at server side which contains large amount of data entries like employess master data(10,000), & I want to give search option to search valid employee ID or name.
So I have tried to compare that entered text with that list of large entries in loop, which is obvious degrading performance.
So is there any option to better performace?
Thanks in advance.
Try this:
public List<Employee> SearchEmployee(string search, int pageNo, int pageLength)
{
MasterDataContext db = new MasterDataContext();
var searchResult = (from e in db.Employess
where (search == null ||
e.Name.ToLower().Contains(search.ToLower()))
select e).ToList();
int pageStart = (pageNo - 1) * pageLength;
var pageResult = from c in searchResult.Skip(pageStart).Take(pageLength)
orderby c.CardNo
select c;
return pageResult;
}
I hope it helps.
I'm trying to do a search on a table in my database where it returns the top 50 rows with a firstname like the search term being passed to the function, but its always returning the same 50 results
My sql looks like this:
Select TOP(50) *
FROM [database].[dbo].[records]
WHERE (A_1STNAME LIKE '" + #searchTerm + "%')
ORDER BY A_RECID
When I run this query in Visual Studios query window, it works as expected but when I run it through my ASP.NET application, it always returns the same 50 results, and only one of them has a first name close to the searchTerm I passed to it.
here is the page code that runs the function:
protected void Page_Load(object sender, EventArgs e)
{
_migrated_data data = new _migrated_data();
DataSet ds = data.Search(Request.QueryString.Get("query"), "A_RECID", 50);
if (ds.Tables.Count > 0 && ds.Tables[0].Rows.Count > 0)
{
rpt_Data.DataSource = ds.Tables[0].DefaultView;
rpt_Data.DataBind();
}
}
and here is the search method of _migrated_data:
public DataSet Search(String #pSearchTerm, String #pSortBy, int #pRowCount)
{
DataSet ds = new DataSet();
OleDbConnection objOleDBConn;
OleDbDataAdapter objOleDBDa;
objOleDBConn = new OleDbConnection(ClearingHouse_OLEDDB);
objOleDBConn.Open();
string lSQL = "SELECT TOP(50) * FROM [database].[dbo].[records]";
lSQL += " WHERE (A_1STNAME LIKE #searchTerm ) ORDER BY #sortBy";
SqlCommand t = new SqlCommand(lSQL);
if (pSearchTerm != null && pSearchTerm != "")
{
t.Parameters.AddWithValue("#searchTerm", #pSearchTerm + "%");
}
if (pSortBy != null && pSortBy != "")
{
t.Parameters.AddWithValue("#sortBy", #pSortBy);
}
else
{
t.Parameters.AddWithValue("#sortBy", "A_RECID");
}
objOleDBDa = new OleDbDataAdapter(t.CommandText, objOleDBConn);
objOleDBDa.SelectCommand.CommandType = CommandType.Text;
objOleDBDa.Fill(ds);
objOleDBConn.Close();
return ds;
}
Using locals to view the final CommandText of t, I get the sql results I gave above.
Any help is greatly appriciated :)
AS Rob Rodi said, first of all, to get the results begining with a value you will need % char at the end of the therm:
Select TOP(50) *
FROM [database].[dbo].[records]
WHERE (A_1STNAME LIKE '" + #searchTerm + "%')
ORDER BY A_RECID
Second, probably your searchTerm is empty, so it's grabbing always the same results.
Debug your app and watch the variable to see if it's receiving some value.
You are having SQL injection there. Also it looks like the %-signs are missing.
The fact, that the query works in a query window and not in your app means, that the error is not in your query! Did you think about that? Probably, you should post code of your ASP.NET app.
It sounds like your parameter isn't being correctly passed to your data layer. Easiest way to find out if that's the case is to turn on Sql Profiler and check to see if it's being passed. If it is, your sql's to blame. If it's not, it's your application. You can then use the debugger on your application to work your way up the stack to see where the problem lies.