I am trying to compare two data tables using Linq. Very simple tables, just one column but have about 44,000 rows. I use the following but when I trace it, when it gets to if (dr.Any()), it just sits there and next line, or exception, is never executed:
public static DataTable GetTableDiff(DataTable dt1, DataTable dt2)
{
DataTable dtDiff = new DataTable();
try
{
var dr = from r in dt1.AsEnumerable() where !dt2.AsEnumerable().Any(r2 => r["FacilityID"].ToString().Trim().ToLower() == r2["FacilityID"].ToString().Trim().ToLower()) select r;
if (dr.Any())
dtDiff = dr.CopyToDataTable();
}
catch (Exception ex)
{
}
return dtDiff;
}
I set max request length in web.config to make sure that is not an issue, but no change:
<system.web>
<compilation debug="true" targetFramework="4.5" />
<httpRuntime targetFramework="4.5" maxRequestLength="1048576" />
I don't think 44,000 rows is too big, is it?
Join tables O(N1+N2) instead of doing O(N1*N2) search (currently for each row in dt1 you are scanning all rows in dt2):
var diff = from r1 in dt1.AsEnumerable()
join r2 in dt2.AsEnumerable()
on r1.Field<string>("FacilityID").Trim().ToLower()
equals r2.Field<string>("FacilityID").Trim().ToLower() into g
where !g.Any() // get only rows which do not have joined rows from dt2
select r1;
With join you will also calculate each key (facility id) only once.
Another option is creating simple row comparer:
public class FacilityIdComparer : IEqualityComparer<DataRow>
{
public bool Equals(DataRow x, DataRow y) => GetFacilityID(x) == GetFacilityID(y);
public int GetHashCode(DataRow row) => GetFacilityID(row)?.GetHashCode() ?? 0;
private string GetFacilityID(DataRow row)
=> row.Field<string>("FacilityID")?.Trim().ToLower();
}
Then getting new rows is one liner with LINQ Except method:
var diff = dt2.AsEnumerable().Except(dt1.AsEnumerable(), new FacilityIdComparer());
and it will work for searching intersections as well
I would use a different, more leightweight, approach because you just take rows from one table and you want only those with a new FacilityId:
public static DataTable GetTableDiff(DataTable dtNew, DataTable dtOld)
{
DataTable dtDiff = dtNew.Clone(); // no data only columns and constraints
var oldFacilityIds = dtOld.AsEnumerable().Select(r => r.Field<string>("FacilityID").Trim());
var oldFacilityIDSet = new HashSet<string>(oldFacilityIds, StringComparer.CurrentCultureIgnoreCase);
var newRows = dtNew.AsEnumerable()
.Where(r => !oldFacilityIDSet.Contains(r.Field<string>("FacilityID").Trim()));
foreach (DataRow row in newRows)
dtDiff.ImportRow(row);
return dtDiff;
}
I am working on MVC using WebGrid and WebGridColumns.
I have a list of data on my Model, but some of the columns on my data are null so I want to specify a different value for null columns.
Here is a sample of my code on View:
var objData = Model.myListofUsers;
var webMyGrid= new WebGrid();
webMyGrid.Bind(objData);
//Build a list of columns
List<WebGridColumn> webColumns= new List<WebGridColumn>();
webColumns.Add(new WebGridColumn { ColumnName = "strName", Header = "Name"});
webColumns.Add(new WebGridColumn { ColumnName = "strPhone", Header = "Phone"});
Not all Names have Phone numbers so I want to specify if there is no Phone number, I want to declare some words like "User have no phone" instead on those fields.
Format property of WeGridColumn could help you.
webColumns.Add(new WebGridColumn { ColumnName = "strName", Header = "Name", Format= (item) => String.IsNullOrEmpty(item.strName)? "Your Message": item.strName});
My app allows user to upload an XML file, which I pass as an XDocument into my method. All the values are attribute strings, and I am using Linq to XML and Linq to SQL.
The dateCutoff query is supposed to get the latest date from SQL table - InsDate is nullable.
The where clause in the inspections XML query is supposed to get inspection elements with the inspection_date attribute value later than the dateCutoff value. I am using DateTime.Parse and Date.CompareTo, but am coming up empty.
What am I missing? Any help is much appreciated.
public IEnumerable<XElement> getInspections(XDocument xDoc)
{
IEnumerable<XElement> inspections = null;
using (InspectionDataContext db = new InspectionDataContext())
{
// get the latest date already in Inspections table
DateTime? dateCutoff = (from d in db.Inspections
select d.InsDate).Max();
if (dateCutoff.HasValue)
{
dateCutoff = dateCutoff.Value.Date;
}
// get only the inspections later than the dateCutoff
inspections = from i in xDoc.Descendants("inspections")
where DateTime.Parse(i.Element("inspection").Attribute("inspection_date").Value).Date.CompareTo(dateCutoff) == 1
select i;
}
return inspections;
}
I'm gong to make a few assumptions here because it's not totally clear as written.
1. You want to return "inspection" elements (not the "inspections" container elements).
2. You want to return only elements whose dates are greater than the cutoff
3. If the cutoff is null, you want to return all inspections.
In that case, you'd do something like this:
var inspections = xDoc.Descendents("inspection");
if (!dateCutoff.HasValue)
return inspections;
return inspections.Where(i => (DateTime)i.Attribute("inspection_date") > dateCutoff.Value )
Here is what works:
DateTime? dateCutoff;
using (InspectionDataContext db = new InspectionDataContext())
{
// get the latest date already in Inspections table
DateTime? dateCutoffQ = (from d in db.Inspections
select d.InsDate).Max();
if (dateCutoffQ.HasValue)
{
dateCutoff = dateCutoffQ.Value.Date;
}
else
{
dateCutoff = DateTime.Now.AddYears(-20).Date;
}
string date1 = dateCutoff.ToString();
// get only the inspections later than the dateCutoff
inspectionList = from i in xDoc.Descendants("inspection")
where DateTime.Parse(i.Attribute("inspection_date").Value).Date > dateCutoff
select i;
Well, I have a problem with creating expression tree for SelectMany.. especially at the typeArguments part..
So, I have a database with tables like below:
[Group] (one to many) [GroupDetail] (many to one) [Item] (one to many) [ItemDetail]
GroupDetail.group is a Group
GroupDetail.item is an Item
ItemDetail.item is an Item
Item.itemDetail is a collection of ItemDetail
Group.groupDetail is a collection of GroupDetail
so you can see that group detail is simply a many to many link for Group and Item
and (one to many) is a one to many relation..
For example, the data is like below:
Group, GroupDetail, Item, ItemDetail
------------------------------------
gr1, grDt1, ItemA, PartsAA
gr1, grDt1, ItemA, PartsAB
gr1, grDt2, ItemB, PartsBA
gr1, grDt2, ItemB, PartsBB
gr2, grDt3, ItemC, PartsCA
gr2, grDt4, ItemA, PartsAA
gr2, grDt4, ItemA, PartsAB
gr3, grDt4, ItemD, PartsDA
gr3, grDt5, ItemE, PartsEA
I want to select items and each of it's detail by a group search
and return it as a collection of some sort of view class..
Similar like this function below:
public IQueryable<ItemGroupDetailView> getViewQ(IQueryable<GroupDetail> myQ)
{
return myQ.SelectMany(
m => m.item.itemDetail,
(m, n) => new ItemGroupDetailView
{
groupName = m.group.name,
groupDetailCount = m.group.groupDetail.Count,
item = new ItemView
{
itemName = n.item.name,
itemDetailCount = n.item.itemDetail.Count
},
itemDetail = new ItemDetailView
{
itemDetailName = n.name
}
}
);
}
simply like that above BUT I want it to be a dynamic exp tree instead, so maybe I can just use it like:
Filter filter = new Filter("gr1","ItemA"); // just a filter
var myQ = getSearchQ(filters); // it gets all the where etc, everything is fine here..
var viewQ = getViewQ(myQ); // simply to convert the data to the view,.. where all the errors are
var finalQ = ApplyLimit(ApplyGrouping(ApplySorting(ApplySelect(myQ))); // paging, sorting, grouping, etc..
// run the select.. get the count etc..
now I want to make it dynamic but I seems to get it wrong on the SelectMany part
This is roughly how I do the SelectMany things:
step 1: I bind the property/field assignment.. it came from some sort of list-string-configuration-kinda-thing that map the assignment
PropertyInfo pInfo;
MemberExpression mExp;
// parse getproperty reflect etc...
List<MemberAssignment> memberAssginments = new List<MemberAssignment>();
memberAssginments.Add(Expression.Bind(pInfo, mExp);
step 2: then the usual member init
MemberInitExpression mie =
Expression.MemberInit(Expression.New
(typeof(ItemGroupDetailView)), memberAssginments);
so I get this:
new ItemGroupDetailView
{
groupName = m.group.name,
groupDetailCount = m.group.groupDetail.Count,
item = new ItemView
{
itemName = n.item.name,
itemDetailCount = n.item.itemDetail.Count
},
itemDetail = new ItemDetailView
{
itemDetailName = n.name
}
}
step 3: then get the expression collectionSelector & resultSelector
ParamterExpression m = Expression.Parameter(typeof(GroupDetail),"m");
ParamterExpression n = Expression.Parameter(typeof(ItemDetail),"n");
Expression<Func<GroupDetail, ItemDetail, ItemGroupDetailView>> exp2 =
Expression.Lambda<Func<GroupDetail, ItemDetail, ItemGroupDetailView>>
(mie, new ParameterExpression[] { m, n });
I think I get what I need, exp2 (resultSelector):
(m, n) => new ItemGroupDetailView
{
groupName = m.group.name,
groupDetailCount = m.group.groupDetail.Count,
item = new ItemView
{
itemName = n.item.name,
itemDetailCount = n.item.itemDetail.Count
},
itemDetail = new ItemDetailView
{
itemDetailName = n.name
}
}
and with similar way I get the other clause, exp1 (collectionSelector)
MemberExpression mEx = .... reflect get property/field etc..
Expression<Func<GroupDetail, IEnumerable<ItemDetail>>> exp1 =
Expression.Lambda<Func<GroupDetail, IEnumerable<ItemDetail>>>(mEx, m);
so I get this:
m => m.item.itemDetail
step 4: then get the selectMany MethodCallExpression itself
MethodCallExpression selectManyExp =
Expression.Call(
typeof(Queryable),
"SelectMany",
new Type[] {
typeof(GroupDetail),
typeof(Expression<Func<GroupDetail, IEnumerable<ItemDetail>>>),
typeof(Expression<Func<GroupDetail, ItemDetail, ItemGroupDetailView>>)
},
new Expression[] { Expression.Quote(exp1), Expression.Quote(exp2) }
);
It doesnt work at all..
(No generic method 'SelectMany' on type 'System.Linq.Queryable' is compatible with the supplied type arguments and arguments. No type arguments should be provided if the method is non-generic.)
so I think the main question here is:
How do I build an expression tree for such selectMany query
How do I build an expression query that has resultSelector & collectionSelector and multiple parameters..
and why does the code below works, but the Expression.Call always error..
myQ.SelectMany(exp1, exp2);
I guess I dont understand how SelectMany or Expression Tree works.. :(
but, I need this to be dynamic because the property/field assignment binding and source, selector and result is dynamic
public IQueryable<TView> getViewQ(IQueryable<T> myQ)
{
// some code..
}
EDIT 1 :
Switching the exp1 and exp2.. now exp1 is the collectionSelector and exp2 is the resultSelector..
EDIT 2 :
Furthermore I tried several things:
first, I change the type argument like what Mike said below, but the error still the same
MethodCallExpression selectManyExp =
Expression.Call(
typeof(Queryable),
"SelectMany",
new Type[] {
typeof(GroupDetail),
typeof(ItemDetail),
typeof(ItemGroupDetailView)
},
new Expression[] { Expression.Quote(exp1), Expression.Quote(exp2) }
);
then I try some reflection this and that to check
System.Reflection.MethodInfo sminfo = null;
System.Reflection.MethodInfo sminfo2 = null;
IEnumerable<System.Reflection.MethodInfo> sminfos = typeof(Queryable)
.GetMethods(System.Reflection.BindingFlags.Static
| System.Reflection.BindingFlags.Public)
.Where(xxx => xxx.Name.Equals("SelectMany"));
foreach (System.Reflection.MethodInfo mi in sminfos)
{
if (mi.GetParameters().Count() == 3)
{
sminfo = mi;
}
}
/*
I ran this step by step to make sure that the method I get in sminfo is:
public static IQueryable<TResult> SelectMany<TSource, TCollection, TResult>(
this IQueryable<TSource> source,
Expression<Func<TSource, IEnumerable<TCollection>>> collectionSelector,
Expression<Func<TSource, TCollection, TResult>> resultSelector
);
*/
sminfo2 = sminfo.MakeGenericMethod(
new Type[] {
typeof(GroupDetail), typeof(ItemDetail), typeof(ItemGroupDetailView)
});
MethodCallExpression selectManyExp =
Expression.Call(sminfo2, new Expression[] { exp1, exp2 });
and I get different error:
(Incorrect number of arguments supplied for call to method ..)
and it tells me that the method required 3 parameters instead of 2, and the one I miss is the IQueryable<GroupDetail> source
so I get back to the Expression.Call and add the source parameter
MethodCallExpression selectManyExp =
Expression.Call(
typeof(Queryable),
"SelectMany",
new Type[] {
typeof(GroupDetail),
typeof(ItemDetail),
typeof(ItemGroupDetailView)
},
new Expression[] { myQ.Expression, exp1, exp2 }
);
return (IQueryable<ItemGroupDetailView>)myQ.Provider.CreateQuery(selectManyExp);
and it works.. :D
Sorry for the messy and long post,.. my English is bad.. :(
It looks like you've conflated the type parameters with the formal parameters. I believe your type arguments should look like below:
MethodCallExpression selectManyExp =
Expression.Call(
typeof(Queryable),
"SelectMany",
new Type[] {
typeof(GroupDetail),
typeof(ItemDetail),
typeof(ItemGroupDetailView)
},
new Expression[] { Expression.Quote(exp1), Expression.Quote(exp2) }
);
I have 2 List one stores the name of filterable columns(of type DropDown) and another store the values to load in those filterable columns.
List<string> filterableFields = new List<string>() { "A_B", "C_D", "E_F" };
List<string> AB, CD , EF;
Now at the run time I get the data from web service and I have written a function to to extract values for these filterable fields and store the values to 2nd List.
private void prepareListForFilterableColumns(XDocument records)
{
foreach (var currentField in filterableFields)
{
var values = (from xml in records.Descendants(z + "row")
let val = (string)xml.Attribute("ows_" + currentField.Replace("_", "_x0020_"))
where val != ""
orderby val
select val
).Distinct();
switch (currentField)
{
case "A_B": AB = values.ToList(); break;
case "C_D": CD = values.ToList(); break;
}
}
}
Now I was thinking that instead of hard coding the assignment in swtich case block, If I could just use the first List name "A_B" and replace "_" from it to point to my 2nd List and assign values.ToList() to it.
I understand that c# is a static language, So not sure if we can achieve this, but IF I can it will make my function generic.
Thanks a lot in advance for time and help.
Vishal
You could use a dictionary of lists of strings instead of 3 lists to store the values.
Dictionary<string, List<string>> val lists = new Dictionary<string,List<string>>();
And make the keys of the dictionary equal to the filterables: "AB", "CD",..
then, instead of AB you would use valLists["AB"] and could then reference reach list based on a string key.
The other option would be to use reflection but that would be slower and unnecessarily a bit more complicated.