I'm accessing an sqlite3 with a Python script, using
input_data = None
connection = sqlite3.connect(db_path)
db = connection.cursor()
read_string = 'select viewer_id, email, device_group, created_date, path_found, analyzed from ' + db_table + ' where analyzed < ' + str(cfg_history_number_of_days) + ';'
db.execute(read_string)
input_data = db.fetchall()
connection.close()
and this works great when there's data in the 'analyze' field. When 'analyze' s empty, the script considers the field to have a value much greater than my string value, and this test fails. How can I change my script so that I can determine whether the field is blank in the database?
Any comparison with NULL fails. To make NULL values behave as if they were a small string, replace them with a small string:
SELECT ... FROM db_table WHERE ifnull(analyzed, '') < ...;
Related
I am using ASP.NET and a SQL Server database.
I have options to search values from gridview base on some criteria. The criteria are mostly independent from each other.
For example:
"where ProductType = " + Convert.ToInt32(recordType.persoRecord) +
" and AccountNumber like '%" + SearchValue + "%'";
or
"where fileid=" + File_ID + ShowSearch +
" and lower(j.CardHolderName) like '%" + SearchValue.Trim().ToLower() + "%'
There are a lot of options to search by user. I have millions of rows of data in this table, in order to fetch the data and bind it fast to gridview, I have created a stored procedure.
It works fine for fetching and binding but for searching, it's hard to manage. Due to I don't have much time, i want to configure the stored procedure to 'if there's a searching' fetch the searched data only.
Here is my stored procedure:
SET ANSI_NULLS ON
GO
SET QUOTED_IDENTIFIER ON
GO
ALTER PROCEDURE [dbo].[SetJobsGrid]
(#FileID varchar(50),
#PageIndex int = 1,
#PageSize int = 1,
#DynamicQuery NVARCHAR(MAX),
#SearchFlag bit,
#RecordCount int output)
AS
BEGIN
DECLARE #SearchQuerySQL as nvarchar(MAX)
SET NOCOUNT ON;
select ROW_NUMBER()OVER (
order by j.creationdate asc,
j.id desc
) AS RowNumber
, j.id as 'Serial #', j.jobname as 'File Record Name',
j.RecordNumber as 'Record Number',i.issuername as'Issuer Name',p.ProductName as 'Product Name',p.productNumber as 'Product Number',
(j.AccountNumber) as AccountNumber ,j.CardHolderName as 'Card Holder Name', j.CardholdersBranchName as 'Card Holder Branch Name',
j.ShipmentBranchName as 'Shipment Branch Name', j.EmbossingCardholderName as 'Embossing Card Holder Name', j.MaskedPAN as 'PAN',
j.creationdate as 'File Record Creation Date', j.status as 'File Record Status', j.chipdatastatuserror as 'ChipDataStatus Erro',
j.chipdatastatuserrormessage as 'Error Message', j.chipdatastatus as 'Data Prepared',j.isduplicaterecord as 'isduplicate',j.isduplicatefromsamefile as 'IsDuplicateFromSameFile',
j.validationerrors , j.isworkordercreatedForCard,j.isworkordercreatedForPin,j.isworkordercreatedForCarrier,j.PersoMachineId,j.PinMachineId,j.CarrierMachineId
INTO #Results
FROM jobs j join issuer i on j.issuerid=i.id join Product p on p.id=j.productid WHERE fileid = #FileID
IF(#SearchFlag = 1)
begin
select #SearchQuerySQL = 'SELECT ' + #RecordCount + ' = COUNT(*) FROM #Results ' + #DynamicQuery
EXEC(#SearchQuerySQL)
select #SearchQuerySQL = 'SELECT * FROM #Results ' + #DynamicQuery + ' and RowNumber BETWEEN(#PageIndex -1) * #PageSize + 1 AND(((#PageIndex -1) * #PageSize + 1) + #PageSize) - 1'
EXEC(#SearchQuerySQL)
end
ELSE
begin
SELECT #RecordCount = COUNT(*)
FROM #Results
SELECT * FROM #Results
WHERE RowNumber BETWEEN(#PageIndex -1) * #PageSize + 1 AND(((#PageIndex -1) * #PageSize + 1) + #PageSize) - 1
end
DROP TABLE #Results
END
When the SearchFlag is set to true from ASP.NET, I want to fetch only searched value. #DynamicQuery set from asp for example:
WHERE AccountNumber LIKE '%" + SearchValue + "%'"
or with many different case.
When I run this stored procedure as in the above, I get an exception:
Conversion failed when converting the varchar value 'select ' to data type int
Regards
To make the stored procedure use your dynamic where statement, it is better to use sp_executesql
example:
EXEC sp_executesql
N'select * from Employees where Id = #param1',
N'#param1 int'
,#param1 = 1
for more information about dynamic query refer to the following site
SQL Server Dynamic SQL
Too long for a comment but you need to understand why the error occurs to fix it. You have this snippet in your code
select #SearchQuerySQL = 'SELECT ' + #RecordCount +
First, if you intend to assign a scalar value (e.g., #SearchQuerySQL) then you should use SET, not SELECT. That's another topic you can research at your leisure. The assignment expression that follows is where you intended to do string concatenation. Unfortunately, the interpretation of the plus sign operator varies according to datatypes involved.
What happens when the database engine encounters an operator that involves 2 different datatypes. Like any other language, one (or both) must be converted to the same type in order to perform the expressed operation. How does the engine do that? There are rules for datatype precedence. In this case, your int parameter has higher precedence and so those strings in that expression are converted to int. That fails with the error that you encountered.
If you want to write dynamic sql, you need to have an advanced understanding of tsql. You should also consider searching the internet first before trying to reinvent the wheel. Maybe Aaron's article on dynamic pagination might help - but it might be a bit much for you at this point.
And while you're mucking about with things, add some comments within the procedure declaration about the usage you intend to support. No one should have to read your code to understand what it does and how it should be used.
Ok I am new to sqlite and python in general so please be nice =)
I have a simple dictionary -
time = data[0]['timestamp']
price = data[0]['price']
myprice = {'Date':time,'price':price}
myprice looks like this (time is a timestamp) -
{'Date': 1553549093, 'price': 1.7686}
I now want to add the data to sqlite3 database...so I created this -
#Create database if not exist...
db_filename = 'mydb_test.db'
connection = sqlite3.connect(db_filename)
#Get a SQL cursor to be able to execute SQL commands...
cursor = connection.cursor()
#Create table
sql = '''CREATE TABLE IF NOT EXISTS TEST
(PID INTEGER PRIMARY KEY AUTOINCREMENT,
DATE TIMESTAMP,
PRICE FLOAT)'''
#Now lets execute the above SQL
cursor.execute(sql)
#Insert data in sql
sql2 = ("INSERT INTO GBPCAD VALUES (?,?)", [(myprice['Date'],myprice['price'])])
cursor.execute(sql2)
cursor.commit()
connection.close()
But when executing this code I get ValueError: operation parameter must be str
What am I doing wrong?
Pass the arguments of the insert statement in execute():
sql2 = "INSERT INTO GBPCAD (DATE, PRICE) VALUES (?,?)"
cursor.execute(sql2, (myprice['Date'], myprice['price']))
Also include in the insert statement the names of the columns.
I have a page where users select multiple search criteria to retrieve data from a SQL Server 2014 view. The view is grabbing data from a table on a linked server (I am not able to put the view directly on that server, and the table I am reading from has over 800 million rows so copying that data onto the local server isn't going to happen).
Of course, I can't index the view either (on linked server) so I'm trying to find a way to stop the timeouts from happening when the query is run. Is it possible to do something like this in a stored procedure?
SELECT
cast(trees as varchar(3)) as Trees
, MIN(fruitnumber) AS FN_Start
, MAX(fruitnumber) AS FN_End
, COUNT(CASE WHEN fruitType = 'apple' THEN 1 ELSE NULL END) AS apple
, COUNT(CASE WHEN fruitType = 'banana' THEN 1 ELSE NULL END) AS banana
FROM
view_fruitReport
WHERE
(orchard = #orchard) and
and here's where it gets wonky. Users select the orchard from a dropdown (not a combobox because we use IE11 and ajaxtoolkit combo box still doesn't work there) so only one selection possible but.
They are able to add criteria to listboxes. Unlimited criteria. And they don't need to select any of the criteria, they can just search by orchard.
So the rest of the WHERE clause is built based on what they have added to the listboxes.
Like this:
' check if items selected in both listboxes'
If trees_Listbox.Items.Count > 0 Then
If fruitminListBox.Items.Count > 0 Then
'cycle through items in fruitnum listbox to create an "in" clause for sql query'
For Each item As ListItem In trees_Listbox.Items
whereString += String.Join(",", item) + ", "
Next
whereString = Left(whereString, Len(whereString) - 2) + ")"
selectQry += "(" + wherecls + whereString + ")"
whereFNcls = "(fruitNumber between "
For Each itemFNmin As ListItem In fruitminListBox.Items
'create a "between" clause for the min and max FN values entered by user.'
whereOEcls += itemFNmin.Value + " and " + fruitmaxListBox.Items(i).ToString + ") or (fruitNumber between " '(fruitnumber between number and number) or '
i += 1
Next
'trim off the last text portion of the whereOEcls'
whereOEcls = Left(whereOEcls, Len(whereFNcls) - 25)
selectQry += " and (" + whereFNcls + ") GROUP BY trees ORDER BY trees"
fruityData.SelectCommand = selectQry
WeeklyGridView.Visible = True
Else
'see if FN is empty but trees is selected'
For Each item As ListItem In trees_Listbox.Items
whereString += String.Join(",", item) + ", "
Next
whereString = Left(whereString, Len(whereString) - 2)
selectQry += wherecls + whereString + ") GROUP BY trees ORDER BY trees"
fruityData.SelectCommand = selectQry
WeeklyGridView.Visible = True
End If
Else
Essentially ending up with a where clause that could look like this:
WHERE (orchard = #orchard)
and trees in (100,200,300,400)
and fruitnumber between (itemFNmin.Value and itemFNmax.Value)
or fruitnumber between (itemFNmin.Value and itemFNmax.Value)
etc etc etc
Which works except it makes things very ugly and I am certain is a poor way of doing this.
I have no clue if/how I can make these lists of variables pass to a stored procedure as multiple arrays or tables etc.
Probably anything is better than having them tied to a view, whose linked server table isn't even an indexed table (not my fault haha)
For your first question: You can return the Count of each fruit type, but it will have performance implications as it requires a subquery for each one. This also requires that you hard code each possible fruit type in the query. I assume that the fruit types can change or have other types added to them, so this isn't the most desirable in terms of maintenance either. You can't dynamically add columns to a query unless you build SQL in your proc and make use of sp_executesql, which is more convoluted than doing in line SQL in your .Net code.
SELECT
cast(trees as varchar(3)) as Trees
, MIN(fruitnumber) AS FN_Start
, MAX(fruitnumber) AS FN_End
, CASE
WHEN fruitType = 'apple' THEN (SELECT COUNT(fruitType) FROM view_fruitReport WHERE fruitType = 'apple') ELSE NULL
END AS [apple]
, CASE
WHEN fruitType = 'banana' THEN (SELECT COUNT(fruitType) FROM view_fruitReport WHERE fruitType = 'banana') ELSE NULL
END AS [banana]
FROM
view_fruitReport
WHERE
(orchard = #orchard)
For your second question, you can pass in lists/tables into a stored procedure. One method is to pass some sort of delimited string and parse it using T-SQL. I recommend a different approach, however, which is
Table Value Parameters. This is a parameter that acts as a table that you can join with in your stored procedure.
Here is an example for implementing a Table Value Parameter for the Trees column.
You will first need to declare a SQL Type:
CREATE TYPE [dbo].[Trees] AS TABLE (Trees INT)
Then you can reference it in your stored procedure as a parameter that acts as a table. Note that you can't use WITH(NOLOCK) with these and must specify READONLY in the paramter:
CREATE PROCEDURE [dbo].[up_getOrchardInfo]
(
#Trees As [dbo].[Trees] READONLY
, #Orchard INT
)
AS
BEGIN
SELECT
cast(trees as varchar(3)) as Trees
, MIN(fruitnumber) AS FN_Start
, MAX(fruitnumber) AS FN_End
, COUNT(CASE WHEN fruitType = 'apple' THEN 1 ELSE NULL END) AS apple
, COUNT(CASE WHEN fruitType = 'banana' THEN 1 ELSE NULL END) AS banana
FROM
view_fruitReport AS F
INNER JOIN #Trees AS T
ON F.Trees = T.Trees
WHERE
(orchard = #orchard)
END
GO
The above example will filter by the Trees passed in. Note that if want to return everything for the Orchard if #Trees is Null or the count is 0 you will need to include that conditional logic in your stored procedure.
IF (#Trees IS NULL OR (SELECT COUNT(1) FROM #Trees = 0))
BEGIN
--No Join to #Trees
END
ELSE
BEGIN
--Query from above.
END
Finally, on the .Net side you will need to pass in a DataTable object as a Parameter on the SqlCommand with the Type of Structured:
Dim sqlCommand As New SqlCommand("up_getOrchardInfo", sqlConnection.SqlConnection)
sqlCommand.CommandType = CommandType.StoredProcedure
Dim sqlTreesParameter As New SqlParameter("#Trees", SqlDbType.Structured)
sqlOrchardParameter.Direction = ParameterDirection.Input
Dim tblExample As New DataTable
tblExample.Columns.Add("Trees", New Integer().GetType())
Dim drExample As DataRow = tblExample.NewRow()
drExample.Item("Trees") = 100
tblExample.Rows.Add(drExample)
'Adjust if Orchard is a VarChar/String'
Dim sqlOrchardParameter As New SqlParameter("#Orchard", SqlDbType.Int)
sqlOrchardParameter.Direction = ParameterDirection.Input
sqlOrchardParameter.Value = intYourOrchardValue
sqlCommand.Parameters.Add(sqlTreesParameter)
sqlCommand.Parameters.Add(sqlOrchardParameter)
'Execute Dataset
All of this said, you may want to consider making multiple stored procedures. One stored procedure could be optimized for returning everything when only an Orchard is passed and another for when Trees are also passed. This depends on how many parameters you're dealing with.
Maybe there's a UX answer. When you know it's not going to come back for a long time give the user a heads up and confirm if they want to wait or not. (With check box on the confirmation that says, "don't show me this again.")
But don't re-event the godawful estimated file transfer time from Windows Explorer.
Could some one explain in the following TCL source code:
set sql "SELECT PROD.KEY || ' {' || PARAMETERS || '}' \
FROM PRV_PROD_MAPPING PROD \
WHERE PROD.SERVICE_ID = :service_id \
AND (PROD.KEY || ' ' || PROD.KEY_VAL) \
IN (:keys) "
what :service_id and :keys mean. Could I see the values behind by simple trace.
How could I find where these vars are defined?
Thanks in advance!
It's not TCL its an SQL query embedded in a TCL string, specifically one that binds a variable, which is then assigned to a normal TCL variable
As glenn points out, on its own this snippet of TCL does very little. presuembly somewhere in your program you actually connect to a DB and pass it a query from this string and some other variables
If you're using TDBC, you might have
# assume the connection has already occured and is named "db"
set sql "SELECT ... WHERE a.b=:service_id IN (:keys)"
set statement [db prepare $sql]
# get the bind variables' values from the local context:
set service_id 42
set keys [join {key1 key2 key3} ,]
set resultset [$statement execute]
# or, without setting the "service_id" and "keys" variables,
# provide them as an argument to the execute subcommand
unset service_id keys
set resultset [$statement execute {service_id 24 keys "foo,bar,baz"}]
As others pointed out it you just assign a string* to a variable.
More exactly, you assign the following string to a variable called sql
SELECT PROD.KEY || ' {' || PARAMETERS || '}' FROM PRV_PROD_MAPPING PROD WHERE PROD.SERVICE_ID = :service_id AND (PROD.KEY || ' ' || PROD.KEY_VAL) IN (:keys)
The values of :service_id and :keys are bound values that are passed later (In a oraplexec statement probably).
|| is string concatenation in SQL. So PROD.KEY || ' {' || PARAMETERS || '}' could be PRODKEY {PARAMETERS}.
*Everything is a string
I've done it using traces:
sys.DBMS_SYSTEM.SET_EV(n_sid, n_serial, 10046, 12, '');
DBMS_OUTPUT.put_line('Trace started: ' || to_char(SYSDATE,'dd.mm.yyyy hh24:mi:ss'));
Set rslistings = my_conn.Execute(strSQL)
Do while NOT rslistings.Eof
description = strip(rslistings("description"))
rslistings.MoveNext
loop
In strip - NULL is being passed. However, if I attach a debugger and inspect the contents of rslistings("description"), then the actual Field object is passed through
It's quite old asp code, but it works on IIS6, just not IIS7
EDIT This only happens on the "description" field with is a text type (MySQL database)
strip doesn't do a lot:
If NOT IsNull(passedinvalue) Then
// do something
Else
// do something else
If I call strip like strip(rs("description")), it is never null as the Field object is passed in. If I assign it to another value, then pass it in (like strip(mynewvar)) then the correct value is passed in.
Edit - database bits as requested below
Set my_conn = Server.CreateObject("ADODB.Connection")
Set rs = Server.CreateObject("ADODB.Recordset")
my_conn.Open "DSN=mydb"
SQL
Set rs = my_conn.Execute("SELECT description FROM table")
the Field Collection is the default member of the Recordset object.
so is the value property for the Field object.
so the following two code statements are equivalent.
Debug.Print objRs.Fields.Item(0) ' Both statements print
Debug.Print objRs(0) ' the Value of Item(0).
it is a difference if you assign a value to a variable or use it as a parameter in a function.
#Paul: If strip doesn't check if description is NULL before working on it, you could do this --
Do while NOT rslistings.Eof
description = rslistings("description")
If NOT IsNull(description) Then
description = strip(description)
Else
description = "" ' or you could have description = " "
' if you output to screen later on
End If
rslistings.MoveNext
loop