When I call this function, everything works, as long as I don't try to recursively call the function again. In other words if I uncomment the line:
GetChilds rsData("AcctID"), intLevel + 1
Then the function breaks.
<%
Function GetChilds(ParentID, intLevel)
Set rsData= Server.CreateObject("ADODB.Recordset")
sSQL = "SELECT AcctID, ParentID FROM Accounts WHERE ParentID='" & ParentID &"'"
rsData.Open sSQL, conDB, adOpenKeyset, adLockOptimistic
If IsRSEmpty(rsData) Then
Response.Write("Empty")
Else
Do Until rsData.EOF
Response.Write rsData("AcctID") & "<br />"
'GetChilds rsData("AcctID"), intLevel + 1
rsData.MoveNext
Loop
End If
rsData.close: set rsData = nothing
End Function
Call GetChilds(1,0)
%>
*Edited after feedback
Thanks everyone,
Other than the usual error:
Error Type: (0x80020009) Exception occurred.
I wasn't sure what was causing the problems. I understand that is probably due to a couple of factors.
Not closing the connection and attempting to re-open the same connection.
To many concurrent connections to the database.
The database content is as follows:
AcctID | ParentID
1 Null
2 1
3 1
4 2
5 2
6 3
7 4
The idea is so that I can have a Master Account with Child Accounts, and those Child Accounts can have Child Accounts of their Own. Eventually there will be Another Master Account with a ParentID of Null that will have childs of its own. With that in mind, am I going about this the correct way?
Thanks for the quick responses.
Thanks everyone,
Other than the usual error:
Error Type: (0x80020009) Exception
occurred.
I wasn't sure what was causing the problems. I understand that is probably due to a couple of factors.
Not closing the connection and attempting to re-open the same connection.
To many concurrent connections to the database.
The database content is as follows:
AcctID | ParentID
1 Null
2 1
3 1
4 2
5 2
6 3
7 4
The idea is so that I can have a Master Account with Child Accounts, and those Child Accounts can have Child Accounts of their Own. Eventually there will be Another Master Account with a ParentID of Null that will have childs of its own. With that in mind, am I going about this the correct way?
Thanks for the quick responses.
Look like it fails because your connection is still busy serving the RecordSet from the previous call.
One option is to use a fresh connection for each call. The danger there is that you'll quickly run out of connections if you recurse too many times.
Another option is to read the contents of each RecordSet into a disconnected collection: (Dictionary, Array, etc) so you can close the connection right away. Then iterate over the disconnected collection.
If you're using SQL Server 2005 or later there's an even better option. You can use a CTE (common table expression) to write a recursive sql query. Then you can move everything to the database and you only need to execute one query.
Some other notes:
ID fields are normally ints, so you shouldn't encase them in ' characters in the sql string.
Finally, this code is probably okay because I doubt the user is allowed to input an id number directly. However, the dynamic sql technique used is very dangerous and should generally be avoided. Use query parameters instead to prevent sql injection.
I'm not too worried about not using intLevel for anything. Looking at the code this is obviously an early version, and intLevel can be used later to determine something like indentation or the class name used when styling an element.
Running out of SQL Connections?
You are dealing with so many layers there (Response.Write for the client, the ASP for the server, and the database) that its not surprising that there are problems.
Perhaps you can post some details about the error?
hard to tell without more description of how it breaks, but you are not using intLevel for anything.
How does it break?
My guess is that after a certain number of recursions you're probably getting a Stack Overflow (ironic) because you're not allocating too many RecordSets.
In each call you open a new connection to the database and you don't close it before opening a new one.
Not that this is actually a solution to the recursion issue, but it might be better for you to work out an SQL statement that returns all the information in a hierarchical format, rather than making recursive calls to your database.
Come to think of it though, it may be because you have too many concurrent db connections. You continually open, but aren't going to start closing until your pulling out of your recursive loop.
try declaring the variables as local using a DIM statement within the function definition:
Function GetChilds(ParentID, intLevel)
Dim rsData, sSQL
Set ...
Edit: Ok, I try to be more explicit.
My understanding is that since rsData is not declared by DIM, it is not a local variable, but a global var. Therefore, if you loop through the WHILE statement, you reach the .Eof of the inner-most rsData recordset. You return from the recursive function call, and the next step is again a rsData.MoveNext, which fails.
Please correct me if rsData is indeed local.
If you need recursion such as this I would personally put the recursion into a stored procedure and handle that processing on the database side in order to avoid opening multiple connections. If you are using mssql2005 look into something called Common Table Expressions (CTE), they make recursion easy. There are other ways to implement recursion with other RDBMS's.
Based on the sugestions I will atempt to move the query into a CTE (common table expression) when I find a good tutorial on how to do that. For now and as a quick and dirty fix, I have changed the code as follows:
Function GetChilds(ParentID, intLevel)
'Open my Database Connection and Query the current Parent ID
Set rsData= Server.CreateObject("ADODB.Recordset")
sSQL = "SELECT AcctID, ParentID FROM Accounts WHERE ParentID='" & ParentID &"'"
rsData.Open sSQL, conDB, adOpenKeyset, adLockOptimistic
'If the Record Set is not empty continue
If Not IsRSEmpty(rsData) Then
Dim myAccts()
ReDim myAccts(rsData.RecordCount)
Dim i
i = 0
Do Until rsData.EOF
Response.Write "Account ID: " & rsData("AcctID") & " ParentID: " & rsData("ParentID") & "<br />"
'Add the Childs of the current Parent ID to an array.
myAccts(i) = rsData("AcctID")
i = i + 1
rsData.MoveNext
Loop
'Close the SQL connection and get it ready for reopen. (I know not the best way but hey I am just learning this stuff)
rsData.close: set rsData = nothing
'For each Child found in the previous query, now lets get their childs.
For i = 0 To UBound(myAccts)
Call GetChilds(myAccts(i), intLevel + 1)
Next
End If
End Function
Call GetChilds(1,0)
I have working code with the same scenario.
I use a clientside cursor
...
rsData.CursorLocation = adUseClient
rsData.Open sSQL, conDB, adOpenKeyset, adLockOptimistic
rsData.ActiveConnectcion = Nothing
...
as pointed out in other responses, this is not very efficient, I use it only in an admin interface where the code is called infrequently and speed is not as critical.
I would not use such a recursive process in a regular web page.
Either rework the code to get all data in one call from the database, or make the call once and save it to a local array and save the array in an application variable.
Related
I use airflow python operators to execute sql queries against a redshift/postgres database. In order to debug, I'd like the DAG to return the results of the sql execution, similar to what you would see if executing locally in a console:
I'm using psycop2 to create a connection/cursor and execute the sql. Having this logged would be extremely helpful to confirm the parsed parameterized sql, and confirm that data was actually inserted (I have painfully experiences issues where differences in environments caused unexpected behavior)
I do not have deep knowledge of airflow or the low level workings of the python DBAPI, but the pscyopg2 documentation does seem to refer to some methods and connection configurations that may allow this.
I find it very perplexing that this is difficult to do, as I'd imagine it would be a primary use case of running ETLs on this platform. I've heard suggestions to simply create additional tasks that query the table before and after, but this seems clunky and ineffective.
Could anyone please explain how this may be possible, and if not, explain why? Alternate methods of achieving similar results welcome. Thanks!
So far I have tried the connection.status_message() method, but it only seems to return the first line of the sql and not the results. I have also attempted to create a logging cursor, which produces the sql, but not the console results
import logging
import psycopg2 as pg
from psycopg2.extras import LoggingConnection
conn = pg.connect(
connection_factory=LoggingConnection,
...
)
conn.autocommit = True
logging.basicConfig(level=logging.DEBUG)
logger = logging.getLogger(__name__)
logger.addHandler(logging.StreamHandler(sys.stdout))
conn.initialize(logger)
cur = conn.cursor()
sql = """
INSERT INTO mytable (
SELECT *
FROM other_table
);
"""
cur.execute(sql)
I'd like the logger to return something like:
sql> INSERT INTO mytable (
SELECT ...
[2019-07-25 23:00:54] 912 rows affected in 4 s 442 ms
Let's assume you are writing an operator that uses postgres hook to do something in sql.
Anything printed inside an operator is logged.
So, if you want to log the statement, just print the statement in your operator.
print(sql)
If you want to log the result, fetch the result and print the result.
E.g.
result = cur.fetchall()
for row in result:
print(row)
Alternatively you can use self.log.info in place of print, where self refers to the operator instance.
Ok, so after some trial and error I've found a method that works for my setup and objective. To recap, my goal is to run ETL's via python scripts, orchestrated in Airflow. Referring to the documentation for statusmessage:
Read-only attribute containing the message returned by the last command:
The key is to manage logging in context with transactions executed on the server. In order for me to do this, I had to specifically set con.autocommit = False, and wrap SQL blocks with BEGIN TRANSACTION; and END TRANSACTION;. If you insert cur.statusmessage directly following a statement that deletes or inserts, you will get a response such as 'INSERT 0 92380'.
This still isn't as verbose as I would prefer, but it is a much better than nothing, and is very useful for troubleshooting ETL issues within Airflow logs.
Side notes:
- When autocommit is set to False, you must explicitly commit transactions.
- It may not be necessary to state transaction begin/end in your SQL. It may depend on your DB version.
con = psy.connect(...)
con.autocommit = False
cur = con.cursor()
try:
cur.execute([some_sql])
logging.info(f"Cursor statusmessage: {cur.statusmessage})
except:
con.rollback()
finally:
con.close()
There is some buried functionality within psycopg2 that I'm sure can be utilized, but the documentation is pretty thin and there are no clear examples. If anyone has suggestions on how to utilize things such as logobjects, or returning join PID to somehow retrieve additional information.
I am not sure why this is not working.
I am verifying the logged on person has the correct security, if they do not I want to redirect them to another page.
If they do have the correct security the rest of the code on the page will continue to execute.
When I step through the code, it does execute the response.redirect, but the page continues to load.
strSQL = "Select * from tblSecurity Where SFID = '" & Right(My.User.Name, 4) & "' and (SecurityLevel = '900' or SecurityLevel = '850')"
ds = objData.SQLExecuteDataset(strSQL, CommandType.Text)
If ds.Tables(0).Rows.Count = 0 Then
Response.Redirect("~/NotAuthorized.aspx", False)
End If
That's exactly what you told it to do.
Passing false as the second parameter makes it not terminate the current page.
dont give the table value equal to zero...give the validation like ds.Tables(0).Rows.Count > 0... it means the table is holding the record if the given login details are right.
I tried all this and more ... and I only note my solution here for posterity and my own reference later. I ran into this problem while I had about 6 VS projects open, each in their own VS instance. Once I closed all but the one project down the problem went away. I am not sure what it was about the other project(s) that caused this particular problem but apparently VS doesn't like having too many of itself open.
I have a classic ASP CRM that was built by a third party company. Currently, I have access to the source code and am able to make any changes required.
Randomly throughout the day, usually after some prolonged usage by users, most of my pages start getting an Out of Memory error.
The way that the application is built, is all the pages and scripts pull core functions from a Global.asp file. In that file are embeds to other global files as well, but the error presented shows
Out Of Memory
WhateverScriptYouTriedToRun.asp Line 0
Line 0 is the include for the global.asp file. Once the error occurs, after an unspecified amount of time the error occurence subsides for some time but then begins to reoccur again. With how the application is written, and the functions it uses, and the "diagnostics" I've already done - it seems to be a common used function that is withholding data such as recordset or something of that nature and then not releasing it properly. Other users then try to use the same function and eventually it just fills up causing the error. The only way for me to effectively clear the error is to actually restart IIS, Recycle the App Pool, and Restart the SQL Server Services.
Needless to say, myself and my users are getting annoyed....
I can't pinpoint the error due to the actual error message presented being Line 0 - but from there I have no idea where in the 20K lines of code it could be hanging up. Any thoughts or ideas on how to isolate or at least point me in the right direction to begin clearing this up? Is there a way for me to increase "memory" size for VBScript? I know there is a limitation but is it set at say...512K and you can increase it to 1GB?
Here are things I have tried:
Removing SQL Inline statements into Views
Going through several hundred scripts and ensuring that every OpenConnection & OpenRecordSet is followed by an appropriate Close.
Going through the Global File and commenting out any large SQL statements such as ApplicationLog (A function that writes the executed query into a table).
Some smaller script edits.
Common Memory Leak
You say you are closing all recordsets and connections which is good.
But are you deleting objects?
For example:
Set adoCon = new
Set rsCommon = new
'Do query stuff
'You do this:
rsCommon.close
adocon.close
'But do you do this?
Set adoCon = nothing
Set rsCommon = nothing
No garbage collection in classic ASP, so any objects not destroyed will remain in memory.
Also, ensure your closes/nothings are run in every branch. For example:
adocon.open
rscommon.open etc
'Sql query
myData = rscommon("condition")
if(myData) then
response.write("ok")
else
response.redirect("error.asp")
end if
'close
rsCommon.close
adocon.close
Set adoCon = nothing
Set rsCommon = nothing
Nothing is closed/destroyed before the redirect so it will only empty memory some of the time as not all branches of logic lead to the proper memory clearance.
Better Design
Also unfortunately it sounds like the website wasn't designed well. I always structure my classic ASP as:
<%
Option Explicit
'Declare all vars
Dim this
Dim that
'Open connections
Set adoCon...
adocon.open()
'Fetch required data
rscommon.open strSQL, adoCon
this = rsCommon.getRows()
rsCommon.close
'Fetch something else
rscommon.open strSQL, adoCon
that = rsCommon.getRows()
rsCommon.close
'Close connections and drop objects
adoCon.close
set adoCon = nothing
set rscommon = nothing
'Process redirects
if(condition) then
response.redirect(url)
end if
%>
<html>
<body>
<%
'Use data
for(i = 0 to ubound(this,2)
response.write(this(0, i) & " " & this(1, i) & "<br />")
next
%>
</body>
</html>
Hope some of this helped.
Have you looked at using a memory monitoring tool to see how much memory fragmentation is happening? My guess at a possible cause is that some object of a size is trying to be created but there isn't enough room in the memory to store it as one contiguous chunk. Imagine needing room to store an object that would take 100 MB and while there may be several hundred megabytes free, the largest contiguous chunk is 90MB then this doesn't fit.
Debug Diagnostic Tool v1.1 would be a tool where Bernard's articles may help in understanding how to use the tool.
Another thought is the question of how much string concatenation is there in the code? I remember where I used to work had problems with doing a lot of string concatenation operations that sucked up memory that may be another idea to consider.
Yeah, I could see some shock at that kind of number the first few times you see it but then if you understand what the code is doing it may make sense for why so much space gets reserved right off the bat at times.
I haven't used that debug tool specifically but I did have a tool that took a snapshot of memory when pages were hung so I couldn't tell if there was a performance impact of the tool or not. Course in my case I used a similar tool in 2004 so it has been a few years since I've had to research this kind of issue.
Just going to throw this in here, but this problem has taken a long time to solve. Here's a breakdown of what we did:
We took all the inline SQL and made SQL Views, every SELECT statement is now handled with a VIEW first.
I took every single SQL INSERT and UPDATE (as much as I could without breaking the system) and put them into Stored Procedures.
#2 was the one item that really made the biggest difference
Went through several thousand scripts, and ensured that variables were properly disposed of, and all the DB Open Connections were followed correctly with a Close Connection and same with Open/Close RecordSet.
One of the slow killers was doing something like:
ID = Request.QueryString("ID)
at the top of the page. Before redirecting, or closing a page, there is always a:
Set ID = Nothing
or the complete removal of the inference.
I am working on a "classic" ASP application with a SQL Server 2000 database.
We have a stored procedure (let's call it SP0) that calls other stored procedures (let's say SP0.1, SP0.2 ...) which themselves call another stored procedure called SPX.
All those procedures generate errors when something goes wrong using RAISERROR().
We want to be able to launch SP0 with a parameter #errorsInResultSet which will change its behaviour : instead of "re-raising" the errors as it does so far, each sub-procedure will log the errors in a temporary table #detectedProblems and return it at the end.
Adding errors to the temporary table is not a problem, but I can not figure out how to ignore the errors generated by the nested stored procedures.
I have done this so far :
EXEC #rc = [SP0.1] #errorsAsResultSet = #errorsAsResultSet
IF (0 <> ##ERROR) OR (0 <> #rc)
BEGIN
IF (#errorsAsResultSet <> 0x1)
BEGIN
RAISERROR('SP0.1: Error for table Tests in db %s.%s', 16, 1, ##SERVERNAME, #db)
END
GOTO FAILURE
END
This works fine, but it still generate errors from the lowest SPX, which prevent it from being executed by ADO in classic ASP.
How can I ignore the errors ?
If you're happy that the errors are being logged and it's safe to continue, you can use ON ERROR RESUME NEXT on the line before the SP call. This will prevent the page from throwing errors.
To turn back on errors later in the page, you can use ON ERROR GOTO 0
In the end, it looks like there is no way to "hide" messages generated by PRINT or RAISERROR statements in SP0.1, SP0.2 from the calling Stored Procedure, which means that the execution is always interpreted as "erroneous" by ASP.
In the end, I rewrote a new Stored Procedure with a special parameter to configure how to report errors.
I have an ASP.NET site (VB.NET) that I'm trying to clean up. When it was originally created it was written with no error handling, and I'm trying to add it in to improve the User Experience.
Try
If Not String.IsNullOrEmpty(strMfgName) And Not String.IsNullOrEmpty(strSortType) Then
If Integer.TryParse(Request.QueryString("CategoryID"), i) And String.IsNullOrEmpty(Request.QueryString("CategoryID"))
MyDataGrid.DataSource = ProductCategoryDB.GetMfgItems(strMfgName, strSortType, i)
Else
MyDataGrid.DataSource = ProductCategoryDB.GetMfgItems(strMfgName, strSortType)
End If
MyDataGrid.DataBind()
If CType(MyDataGrid.DataSource, DataSet).Tables("Data").Rows.Count > 0 Then
lblCatName.Text = CType(MyDataGrid.DataSource, DataSet).Tables("Data").Rows(0).Item("mfgName")
End If
If MyDataGrid.Items.Count < 2 Then
cboSortTypes.Visible = False
table_search.Visible = False
End If
If MyDataGrid.PageCount < 2 Then
MyDataGrid.PagerStyle.Visible = False
End If
Else
lblCatName.Text &= "<br /><span style=""fontf-size: 12px;"">There are no items for this manufacturer</span>"
MyDataGrid.Visible = False
table_search.Visible = False
End If
Catch
lblCatName.Text &= "<br /><span style=""font-size: 12px;"">There are no items for this manufacturer</span>"
MyDataGrid.Visible = False
table_search.Visible = False
End Try
Now, this is trying to avoid generating a 500 error by catching exceptions. There can be three items on the query string, but only two matter here. In my test environment and in Visual Studio when I run this site, it doesn't matter if that item is on the query string. In production, it does matter. If that third item isn't present (SubCategoryID) on the query string, then the "There are no items for this manufacturer" displays instead of the data from the database.
In the two different environments I am seeing two different code execution paths, despite the same URLs and the same code base.
The site is running on Server 2003 with IIS 6.
Thoughts?
EDIT:
In response to the answer below, I doubt it's a connection error (though I see what you're getting to), as when I add the SubCategoryID to the query string, the site works correctly (displaying data from the database).
Also, if please let me know if you have any suggestions for how to test this scenario, without deploying the code back to production (it's been rolled back).
I think you should try to print out the exception details in your catch block to see what the problem is. It could anything for example a connection error to your database.
The error could be anything, and you should definitely consider printing this out or logging it somewhere, rather than making the assumption that there's no data. You're also outputting the same error message to the UI for two different code paths, which makes things harder to debug, especially without knowing if an exception occurred, and if so, what it was.
Generally, it's also better not to have a catch for all exceptions in cases like this, especially without logging the error. Instead, you should catch specific exceptions and handle these appropriately, and any real exceptions can get passed up the stack, ideally to a global error handler which can log it and/or send out some kind of error notification.
I discovered the reason yesterday. In short it was because when I copied my files from my computer into my dev-test environment, I missed a file, which ironically caused it to work, rather than not. So in the end it would have functioned the same in both environments.