Iterating over SearchResultCollection is Very Slow - asp.net

I'm running an LDAP query that returns multiple entries and stores them inside a SearchResultCollection. I'm iterating over the SearchResultCollection like so:
// results is my SearchResultCollection object
foreach (SearchResult sr in results)
{
... do things to each SearchResult in here ...
}
This seems like the most logical way to do this, but loop is incredibly slow. When I step through the loop with the debugger, I find that it's the very first step of initializing the foreach loop that takes the time - the actual iterations are instantaneous.
In addition, when I view the contents of the SearchResultCollection while debugging, the watch takes just as long to load the contents of the variable.
I have a theory that the SearchResultCollection doesn't actually contain complete SearchResult objects, but rather references to entries in the Active Directory server that are then individually fetched when I iterate over the SearchResultCollection object. Can anyone confirm this theory? And is there a better (faster) way to fetch a set of LDAP entries?

I suffered from the same problem and the methodology in the post below was much quicker then my own:
http://msdn.microsoft.com/en-us/library/ms180881(v=vs.80).aspx
... to answer you question yes it seems as if you try to process the entries directly while still in the loop by using something like:
If Not UserAccount.GetDirectoryEntry().Properties("sAMAccountName").Value Is Nothing Then sAMAccountName = UserAccount.Properties("sAMAccountName")(0).ToString()
... this has a drastic impact on performance, you can get around this by adding the collection to a Dictionary within the loop and then processing the dictionary:
Dim searchResult As SearchResult
Dim dictLatestLogonDatesTemp As New Dictionary(Of String, Date)
SearchResults1 = mySearcher.FindAll()
For Each searchResult In SearchResults1
Dim propertyKey,sAMAccountName As String
Dim dteLastLogonDate As Date = Nothing
For Each propertyKey In searchResult.Properties.PropertyNames
Dim valueCollection As ResultPropertyValueCollection = searchResult.Properties(propertyKey)
For Each propertyValue As Object In valueCollection
If LCase(propertyKey) = LCase("sAMAccountName") Then sAMAccountName = propertyValue
If LCase(propertyKey) = LCase("lastLogon") Then dteLastLogonDate = Date.FromFileTime(propertyValue)
Next propertyValue
Next propertyKey
If sAMAccountName <> Nothing Then dictLatestLogonDatesTemp.Add(sAMAccountName, dteLastLogonDate)
Next searchResult
It is a bit restrictive bacause a dictionary ony has two entries but you can comma seperate other values or use a dictionary of values in the dictionary:
Dim tempDictionary As Dictionary(Of String, Dictionary(Of String, String))
Hope this helps someone!

I would like to submit that adding them to a datatable also works very well and allows for more properties than a dictionary. I went from iterating 2000 users in the searchresultscollection taking almost 2 minutes to less that 1-2 seconds. Hope this helps others.
Private Sub getAllUsers()
Dim r As SearchResultCollection
Dim de As DirectoryEntry = New DirectoryEntry(GetCurrentDomainPath)
Dim ds As New DirectorySearcher(de)
ds.SearchScope = SearchScope.Subtree
ds.PropertiesToLoad.Add("name")
ds.PropertiesToLoad.Add("distinguishedName")
ds.PropertiesToLoad.Add("objectSID")
ds.Filter = "(&(objectCategory=person)(objectClass=user))" '(!userAccountControl:1.2.840.113556.1.4.803:=2) not disabled users
PleaseWait.Status("Loading Users...")
Application.DoEvents()
r = ds.FindAll()
Dim dt As New DataTable
dt.Columns.Add("Name")
dt.Columns.Add("SID")
For Each sr As SearchResult In r
Dim SID As New SecurityIdentifier(CType(sr.Properties("objectSID")(0), Byte()), 0)
dt.Rows.Add(sr.Properties("name")(0).ToString(), SID.ToString)
Next
With lstResults
lstResults.DataSource = dt
.DisplayMember = "name"
.ValueMember = "SID"
.Items.Sort()
End With
End Sub

There may be some ways to decrease the response time:
restrict the scope of the search
use a more restrictive search filter
use a base object closer to the object(s) being retrieved.
see also
LDAP: Mastering Search Filters
LDAP: Programming practices
LDAP: Search Best Practices

Related

VB. NET Add multiple ChildNodes in a Treeview

I have many records in a database and I need to populate my treeview dynamically like this: Below is just an example of what I need:
TreeView1.Nodes(a).ChildNodes.Add(New TreeNode("ChildNode " & b))
TreeView1.Nodes(a).ChildNodes(b).ChildNodes.Add(New TreeNode("ChildNode 2 lvl " & b))
I'm getting the records from a MySQL Db and I need to know how can I add multilevel ChildNodes into a loop For ... Next etc...
Do you have any suggestion or idea???
if you want to work with various levels of Treenodes you can use Find function
Dim TempNode As TreeNode = TreeView1.Nodes.Find("Node where I want to add SubNode", True).FirstOrDefault
TempNode.Nodes.Add("SubNode", "SubNode")
This way you can add SubNode to any Node you pick.
.Find("key",True)finds treenodes with following key and .FirstOrDefault picks first. Finally you just add new SubNode to Tempnode.
You considered you are getting it dynamically and from MySql. It may cause error like "Action beeing preformed on this control is being called from the wrong thread. Marshal to the correct thread using Contol.Invoke or Control.BeginInvoke to perform this action." Simply change TempNode.Nodes.Add("SubNode", "SubNode") to TreeView1.Invoke(Sub() TempNode.Nodes.Add("SubNode", "SubNode"))
EXAMPLE:
Dim comm As String = "SELECT * FROM YourTableName"
Dim SqlCmnd as SqlCommand = New SqlCommand(comm, YourMySqlConnection)
Dim READER As SqlDataReader
READER = SqlCmnd.ExecuteReader
While READER.Read
Dim NewNode As TreeNode = New TreeNode(READER.Item("origCategoryID"))
TreeView1.Nodes.Add(NewNode)
NewNode.Nodes.Add(READER.Item("categoryOrderID"))
End While
READER.Close()
EXAMPLE 2:
While READER.Read
If TreeView1.Nodes.Find(READER.Item("OrigCatOrderID"), True).Length > 0 Then
Dim NewNode As TreeNode = TreeView1.Nodes.Find(READER.Item("OrigCatOrderID"), True).FirstOrDefault
NewNode.Nodes.Add(READER.Item("CatOrderID"), READER.Item("CatOrderID"))
Else
TreeView1.Nodes.Add(READER.Item("OrigCatOrderID"), READER.Item("OrigCatOrderID"))
TreeView1.Nodes(READER.Item("OrigCatOrderID")).Nodes.Add(READER.Item("CatOrderID"), READER.Item("CatOrderID"))
End While

Visual basic and Json.net Web request

Basically what im trying to do is make a program that list game information for league of legends.. using there API to extract data. how this works is you Search there username and it returns an integer linked to that account, you then use that integer to search for all of the information for that account, EG account level, wins, losses, etc.
I've run into a problem i can't seem to figure out.. Please not that I'm very new to Json.net so have little experience about working with it.. Below is how the search for the user ID is found, The First section is the Username Minus Any spaces in the name the next is the ID which is the information i require.
{"chucknoland":{"id":273746,"name":"Chuck Noland","profileIconId":662,"summonerLevel":30,"revisionDate":1434821021000}}
I must be declaring the variables wrong in order to obtain the data as everything i do it returns as 0.
these are the following class i have to store the ID within
Public Class ID
Public Shared id As Integer
Public Shared name As String
End Class
Looking at a previous example seen here Simple working Example of json.net in VB.net
They where able to resolve there issue by making a container class with everything inside it.. My problem is that The data i seek i always changing.. The first set will always be different to the "Chucknoland" that's displayed in the example.. is someone able to explain how i could go about extracting this information?
Please note that the variables rRegion has the value of what server there on, Chuck Noland is on OCE, and sSearch is the Username.. Due to Problems with API keys i had to remove the API key from the code... But the URL returns the Json Provided.
'URL string used to grab Summoner ID
jUrlData = "https://oce.api.pvp.net/api/lol/" + rRegion + "/v1.4/summoner/by-name/" + sSearch +
' Create a request for URL Data.
Dim jsonRequest As WebRequest = WebRequest.Create(jUrlData)
'request a response from the webpage
Dim jsonResponse As HttpWebResponse = CType(jsonRequest.GetResponse(), HttpWebResponse)
'Get Data from requested URL
Dim jsonStream As Stream = jsonResponse.GetResponseStream()
'Read Steam for easy access
Dim jsonReader As New StreamReader(jsonStream)
'Read Content
Dim jsonResponseURL As String = jsonReader.ReadToEnd()
jUrlString = jsonResponseURL
this is the request i have to obtain the information, and this is the code i tried to use to display the ID for that json.
Private Sub Button2_Click(sender As Object, e As EventArgs) Handles Button2.Click
Dim obj As ID
obj = JsonConvert.DeserializeObject(Of ID)(jUrlString)
MsgBox(obj.id)
End Sub
Is anyone able to explain how i can go about getting this to work?
One way to handle this would be to get the item into a Dictionary where the keys are the property names.
The class you have is not quite right unless you only want name and id and not the rest of the information. But using a Dictionary you wont need it anyway. The "trick" is to skip over the first part since you do not know the name. I can think of 2 ways to do this, but there are probably more/better ways.
Since json uses string keys pretty heavily, create a Dictionary, then get the data from it for the actual item:
jstr = ... from whereever
' create a dictionary
Dim jResp = JsonConvert.DeserializeObject(Of Dictionary(Of String, Object))(jstr)
' get the first/only value item
Dim jobj = jResp.Values(0) ' only 1 item
' if you end up needing the name/key:
'Dim key As String = jResp.Keys(0)
' deserialize first item to dictionary
Dim myItem = JsonConvert.DeserializeObject(Of Dictionary(Of String, Object))(jobj.ToString)
' view results
For Each kvp As KeyValuePair(Of String, Object) In myItem
Console.WriteLine("k: {0} v: {1}", kvp.Key, kvp.Value.ToString)
Next
Output:
k: id v: 273746
k: name v: Chuck Noland
k: profileIconId v: 662
k: summonerLevel v: 30
k: revisionDate v: 1434821021000
Using String, String may also work, but it would convert numerics to string (30 becomes "30") which is usually undesirable.
While poking around I found another way to get at the object data, but I am not sure if it is a good idea or not:
' parse to JObject
Dim js As JObject = JObject.Parse(jstr)
' 1 = first item; 2+ will be individual props
Dim jT As JToken = js.Descendants(1)
' parse the token to String/Object pairs
Dim myItem = JsonConvert.DeserializeObject(Of Dictionary(Of String, Object))(jT.ToString)
Same results.

How to improve the performance of this function?

This function takes around 1.2 seconds to execute. I am unable to understand why? Is it because of the inner joins? If yes, then how can i improve the execution speed? I am using Microsoft Enterprise Library.
Public Shared Function GetDataByInterests(ByVal accountId As Integer) As Object
Dim details As New List(Of GetIdBasedOnInterest)()
Dim getIDs As New GetIdBasedOnInterest
Dim interests As String = ""
Dim db As SqlDatabase = Connection.Connection
Using cmdGeneric As DbCommand = db.GetSqlStringCommand("SELECT Interests.InterestName FROM UserInterests INNER JOIN Interests ON UserInterests.InterestID = Interests.InterestID WHERE UserInterests.AccountID=#AccountID")
db.AddInParameter(cmdGeneric, "AccountID", SqlDbType.Int, accountId)
Dim dsInterests As DataSet = db.ExecuteDataSet(cmdGeneric)
For i = 0 To dsInterests.Tables(0).Rows.Count - 1
If i = dsInterests.Tables(0).Rows.Count - 1 Then
interests = interests & dsInterests.Tables(0).Rows(i).Item(0).ToString
Else
interests = interests & dsInterests.Tables(0).Rows(i).Item(0).ToString & ","
End If
Next
End Using
getIDs.InterestName = interests
details.Add(getIDs)
Return details
End Function
Without knowing anything of the underlying tables and their indexes (and this is a check you should do immediately) there is an obvious problem in your loop.
You cancatenate strings, this, potentially could pose a strong pressure on the memory used by your program.
A string concatenation results in a new string allocated on the memory and thus, if your table contains many rows, the effect could be noticeable.
You could try to use a StringBuilder
Dim interests As new StringBuilder(1024) ' suppose an internal buffer of 1K'
...
If i = dsInterests.Tables(0).Rows.Count - 1 Then
interests.Append(dsInterests.Tables(0).Rows(i).Item(0).ToString)
Else
interests.Append(dsInterests.Tables(0).Rows(i).Item(0).ToString & ",")
End If
....
getIDs.InterestName = interests.ToString
Of course this optimization could be absolutely not important if your tables (UserInterests and Interests) are not correctly indexed on the fields InterestID and AccountID
EDIT: Another micro-optimization is to remove the internal IF test and truncate the resulting output only after the loop ends
For ....
interests.Append(dsInterests.Tables(0).Rows(i).Item(0).ToString & ",")
Next
if(interest.Length > 0) interest.Length -= 1;
EDIT As for your request, this is an example to create an unique index. The syntax could be more complex and varying depending on the Sql Server version, but basically you do this in Sql Management Studio
CREATE UNIQUE INDEX <indexname> ON <tablename>
(
<columntobeindexed>
)
Check the CREATE INDEX statement examples on MSDN
1) Time your query in SQL Server Management studio. It will be much easier to tune it there in isolation from your VB code. Also you can run the display the query plan, and it ight even suggest new indexes.
2) Check you have the relevent primary keys and indexes defined.
3) Pull common expressions out of your for loop, to avoid recomputing the same thing over and over:
4) Like Steve says, use a StringBuilder
Combining those points:
Dim theTable as ...
Dim rowCount as Integer
Dim interests As new StringBuilder(1024)
Set theTable = dsInterests.Tables(0)
rowCount = theTable.Rows.Count
For i = 0 To rowCount - 1
interests.Append(theTable.Rows(i).Item(0).ToString)
If i <> rowCount - 1 Then
interests.Append(",")
End If
Next

Using arrays with asp.net and VB

Sorry, I'm sure this is something that has been covered many times, but I can't find quite what I am after.
I have a single row data table which contains various settings which are used within my web system. I have been toying with turning this into an XML document instead of the single row datatable, would that make more sense?
Anyway, so, given that this is one record, there is a field called "locations," this field contains data as follows:
locationName1,IpAddress|locationName2,IpAddress|etc
The ipAddress is just the first 5 digits of the IP and allows me to ensure that logins to certain elements (admin section managed by staff) can only be accepted when connected from a company computer (ie using our ISP) - this is a largely unnecessary feature, but stops kids I employ logging in at home and stuff!
So, as the user logs in, I can check if the IP is valid by a simple SQL query.
SELECT ips FROM settings WHERE loginLocations LIKE '%" & Left(ipAddress, 5) & " %'
What I need to be able to do now, is get the name of the users location from the dataField array.
I've come up with a few long winded looping procedures, but is there a simple way to analyse
locationName1,IpAddress1|locationName2,IpAddress2|etc
as a string and simply get the locationName where LoginIp = IpAddressX
... or am I going about this in a totally ridiculous way and should turn it into an xml file? (which will then create a whole load of other questions for you about parsing XML!!)
u can split the string in Vb.net and the send it to a query or anything
'Split the string on the "," character
Dim parts As String() = s.Split(New Char() {","c})
In SQL Server, these are the functions of interest in extracting a sub-string:
CHARINDEX
SUBSTRING
LENGTH
You can google the details about them (e.g. CHARINDEX at http://msdn.microsoft.com/en-us/library/ms186323.aspx). With these, you'll need to use computed columns in your query to extract that location name.
If it's not too late, you may want to devise a new way to layout these elements. For instance, instead of:
locationName1,IpAddress1|locationName2,IpAddress2|etc
How about
{IpAddress1,locationName1}{IpAddress2,locationName2}etc
That way you can use CHARINDEX to locate "{" & Left(ipAddress, 5); from that position, locate ","; then from that position locate the closing "}". From there it should be straightforward to use SUBSTRING to get at the locationName. Be forewarned that this will likely be a messy query (probably built from a few sub-queries, one for each position).
In the end, SpectralGhost's idea to just read in the column and do the extraction in VB is probably the way with the least hassle.
I've decided to do it this way, but would appreciate comments on efficiency.
I've split ipAddress and locationName into two database columns, within the one row, and getting the required data by comparing strings as arrays, as per the code below.
For this particular application I'm only dealing with one record, so it's pretty simple. However, further down the line, I need to product a system for monitoring product sales. from an invoices table.
Each invoice record is a row in the database with items in the invoice held similarly, [item1],[item2] etc. There is another column for quantities [qty1],[qty2] etc, prices [price1],[price2], etc. I'll need to be able to search the database for invoices in which an item number occurs (easy SQL WHERE itemList LIKE %[invNO]%) and then compare arrays to get the quantities and individual prices of each item on that invoice. Extracting these rows as arrays and locating the relevant position in these as per the code below, will work fine, but when the whole operation is looping through several hundred or thousand rows, will this become really slow?
ipList, locationList = list as comma separated string from database record field
Dim ipArray As Array = Split(ipList, ",")
Dim locationArray As Array = Split(locationList, ",")
For i = 0 To UBound(ipArray)
If Left(ipArray(i), 5) = Left(ipAddress, 5) Then
arrayPosition = i
itemFound = "True"
Exit For
End If
Next
location = locationArray(arrayPosition)
'loop through ips then get the position and make the location equal to that
If itemFound <> "True" Then
inValidIP = "True"
End If
I strongly advise that you break those our into database columns and each location is a separate row; had it been done that way in the first place, you wouldn't have the problem you have now.
But since you have it in a single row/column, why not bring that back and simply get the value from .NET?
Dim Source As String = "TestLocationA,127.0|TestLocationB,128.0|TestLocationC,129.0"
Dim Test As String = Source
Dim ToFind As String = "127.0"
Test = Test.Substring(0, Test.IndexOf(ToFind & "|") - 1)
Test = Test.Substring(Test.LastIndexOf("|") + 1)
MsgBox(Test)
OR
Public Class Form1
Private IpLocationList As New List(Of IpLocation)
Private IpToFind As String = ""
Private Class IpLocation
Public Name As String
Public IP As String
Public Sub New(ByVal FullLocation As String)
Me.Name = FullLocation.Substring(0, FullLocation.IndexOf(","))
Me.IP = FullLocation.Substring(FullLocation.IndexOf(",") + 1)
End Sub
End Class
Private Function FindIP(ByVal IpLocationItem As IpLocation) As Boolean
If IpLocationItem.IP = IpToFind Then
Return True
Else
Return False
End If
End Function
Private Sub Button1_Click(sender As System.Object, e As System.EventArgs) Handles Button1.Click
Dim Source As String = "TestLocationA,127.0|TestLocationB,128.0|TestLocationC,129.0"
Dim LocationList As New List(Of String)
LocationList.AddRange(Split(Source, "|"))
For Each LocationItem As String In LocationList
IpLocationList.Add(New IpLocation(LocationItem))
Next
IpToFind = "127.0"
Dim result As IpLocation = IpLocationList.Find(AddressOf FindIP)
If result IsNot Nothing Then
MsgBox(result.Name)
End If
End Sub
End Class
enter code here
This has the benefit of loading the array one time and not needing to manually loop through arrays.

List of Object in httpcontext.current.cache

Is there a way to look through the cache for all objects in the cache? I'm dynamically creating objects and I need to periodically go through the list to purge out objects I'm no longer using.
var keysToClear = (from System.Collections.DictionaryEntry dict in HttpContext.Cache
let key = dict.Key.ToString()
where key.StartsWith("Something_")
select key).ToList();
foreach (var key in keysToClear)
{
HttpContext.Cache.Remove(key);
}
You can enumerate through the objects:
System.Web.HttpContext.Current.Cache.GetEnumerator()
Yes, you can either index based on the cache key, or you you can iterate over the contents:
For Each c In Cache
' Do something with c
Next
' Pardon my VB syntax if it's wrong
Here is a VB function to iterate through the Cache and return a DataTable representation.
Private Function CreateTableFromHash() As DataTable
Dim dtSource As DataTable = New DataTable
dtSource.Columns.Add("Key", System.Type.GetType("System.String"))
dtSource.Columns.Add("Value", System.Type.GetType("System.String"))
Dim htCache As Hashtable = CacheManager.GetHash()
Dim item As DictionaryEntry
If Not IsNothing(htCache) Then
For Each item In htCache
dtSource.Rows.Add(New Object() {item.Key.ToString, item.Value.ToString})
Next
End If
Return dtSource
End Function
This may be a bit late, but I used the following code to easily iterate over all cache items and perform some custom logic for removal of cache items that contain a certain string in their names.
I have provided both versions of code in VB.Net as well as C#.
VB.Net Version
Dim cacheItemsToRemove As New List(Of String)()
Dim key As String = Nothing
'loop through all cache items
For Each c As DictionaryEntry In System.Web.HttpContext.Current.Cache
key = DirectCast(c.Key, String)
If key.Contains("prg") Then
cacheItemsToRemove.Add(key)
End If
Next
'remove the selected cache items
For Each k As var In cacheItemsToRemove
System.Web.HttpContext.Current.Cache.Remove(k)
Next
C# Version
List<string> cacheItemsToRemove = new List<string>();
string key = null;
//loop through all cache items
foreach (DictionaryEntry c in System.Web.HttpContext.Current.Cache)
{
key = (string)c.Key;
if (key.Contains("prg"))
{
cacheItemsToRemove.Add(key);
}
}
//remove the selected cache items
foreach (var k in cacheItemsToRemove)
{
System.Web.HttpContext.Current.Cache.Remove(k);
}
Since you potentially want to be removing items from the Cache object, it is not terribly convenient to iterate over it (as an IEnumerable), since this doesn't allow removal during the iteration process. However, given that you cannot access items by index, it is the only solution.
A bit of LINQ can however simplify the problem. Try something like the following:
var cache = HttpContext.Current.Cache;
var itemsToRemove = cache.Where(item => myPredicateHere).ToArray();
foreach (var item in itemsToRemove)
cache.Remove(itemsToRemove.Key);
Note that each item in the iteration is of type DictionaryEntry.
Jeff, you should really look up dependencies for your cached items. That's the proper way of doing this. Logically group your cached data (items) and setup dependencies for your groups. This way when you need to expire the entire group you touch such common dependency and they're all gone.
I'm not sure I understand the List of Object part.

Resources