I've got a search box that users can type terms into. I have a table setup with fulltext searching on a string column. Lets say a user types this: "word, office, microsoft" and clicks "search".
Is this the best way to deal with multiple search terms?
(pseudocode)
foreach (string searchWord in searchTerms){
select col1 from myTable where contains(fts_column, ‘searchWord’)
}
Is there a way of including the search terms in the sql and not iterating? I'm trying to reduce the amount of calls to the sql server.
FREETEXT might work for you. It will separate the string into individual words based on word boundaries (word-breaking). Then you'd only have a single SQL call.
MSDN -- FREETEXT
Well you could just build your SQL Query Dynamically...
string [] searchWords = searchTerm.Split(",");
string SQL = "SELECT col1 FROM myTable WHERE 1=2";
foreach (string word in searchWords)
{
SQL = string.Format("{0} OR contains(fts_column, '{1}')", SQL, word);
}
//EXEC SQL...
Obviously this comes with the usual warnings/disclaimers about SQL Injection etc... but the principal is that you would dynamically build up all your clauses and apply them in one query.
Depending on how your interacting with your DB, it might be feasible for you to pass the entire un-split search term into a SPROC and then split & build dynamic SQL inside the stored procedure.
You could do it similar to what you have there: just parse the search terms based on delimiter, and then make a call on each, joining the results together. Alternatively, you can do multiple CONTAINS:
SELECT Name FROM Products WHERE CONTAINS(Name, #Param1) OR CONTAINS(Name, #Param2) etc.
Maybe try both and see which is faster in your environment.
I use this class for Normalizing SQL Server Full-text Search Conditions
Related
I'm writing an R Shiny/SQLite app. In the app, I have a function that returns a column from one of the tables in my SQLite database, with the function taking the table name as an input.
Before sending the query to SQLite, the function checks that the table name equals one of the table names that the user is allowed to access. However, I am not using a parameterized query, because the term I'm changing is not a variable used for comparison but the name of the table to extract information from. (There might be a way to make this work anyway with a parameterized search, I suppose.)
My question is whether this is safe from an SQL injection? Can the query be altered on its way from the server to the database, or only from an alteration in the ui input to the server?
(Bear with me, I am new to SQLite.)
Assuming your query is being concatenated as follows:
tbl <- "yourTable"
sql <- paste0("select * from ", tbl, " where some_col = 1")
Then there should be no chance of SQL injection, assuming you check the incoming table name and verify that it matches a table name in your whitelist. Note that this step is critical here to keeping things safe. Let's say that you didn't sterilize the incoming table name. Then, consider this:
tbl <- "yourTable; delete from yourTable"
This would result in the following query being submitted for execution:
select * from yourTable; delete from yourTable where some_col = 1;
Assuming your SQLite driver allows multiple SQL statements to execute, the above hack/trick might end up deleting data from a large portion of one of your tables.
So, your approach should be safe provided that you check the table name. Note that strictly speaking the table name itself is not a parameter in a parameterized query. Rather, only the literal values in the query are parameters.
SQL query parameters cannot be used in place of a table name anyway, so comparing the table name to a list of known authorized tables is your only option.
Yes, it is safe. If you're in control of the set of values that can be interpolated into the SQL query, then you can prevent unauthorized SQL injection.
Note that some other elements of SQL queries cannot be parameters:
Any identifier, e.g. a table name, column name, or schema name.
Expressions
Lists of values in an IN ( ... ) predicate. Use one parameter per value in the list.
SQL keywords.
A query parameter can be used only in place of a single scalar value. That is, where you would use a quoted string literal, quoted date literal, or numeric literal.
The problem of SQL injection is only the user input. Nothing happens to the query on its way from the server to the database (well a malware could in theory alter it, but then even a parametrized query wouldn't help).
I.e., if you create a SQL string like this (C#):
sql = "SELECT * FROM " + tableName;
Then a user might enter a tableName like
MyTable; DROP TABLE MyTable
Guess what happens.
So, if you check the table name, you are on the safe side.
I have an SQLITE Database with the following tables
items(item_id,item_name,....);
tags(tag_id,tag_name);
items_tags(item_id,tag_id);
I'm using QSqlRelationalTableModel class to for the items table.
this class have this function QSqlTableModel::setFilter(const QString &filter);
The filter is a SQL WHERE clause without the keyword WHERE
for example: setFilter("item_name=arg");
what I want to do is to use the setFilter function to select only items with a specific tag/tags.
since I can only use the setFilter(QString &Filter) function which is basically a WHERE clause
I found this solution which I think is a bad one
I'll use this filter
setFilter(FilterString);
FilterString="item_id=arg1 or item_id=arg2 or item_id=argX or......"
the IDs will be obtained from this query
Query.prepare("select item_id from items_tags where tag_id=?");
the FilterString will be generated and incremented by a (do while loop) using Query.next() function that will loop over the resulted Query IDs
FilterString+="item=Query.value(item_id).toString();
and use like 2 if statements to determine when to add the or clause to the FilterString
this solution will definitely work but I think it's stupid because if there are many tagged items(say 5000) the Query string will contain 4999 or clause and another 5000 "id=x" so the Query will be more than 10000 characters long.
QString object can contain a lot more characters but is this Query will have a big impact on performance ?
what are the other alternatives, Even if it requires to switch to another DB like ORACLE or Mysql since SQLITE doesn't have many internal features
I am not able to search the middle part of string using fulltext search index for eg:there was a string "I like music" i was not able to search for like which is in the middle part of string..
Try LIKE operator.
SELECT
*
FROM
YourTABLE
WHERE
ColumnName LIKE '%like%'
Use Like keyword in query as follows:
select * from tablename where col like '%like%'
Here is the tutorial of like in sql:
http://www.w3schools.com/sql/sql_like.asp
Here is the MSDN:
http://msdn.microsoft.com/en-us/library/ms179859.aspx
Hope its helpful.
Well, in order to still use the fulltext index and to avoid a table scan, try this
select *
from yourTable
where contains (ColumnName,'like')
and (ColumnName not like 'like%' or ColumnName like '_%like%')
However, the query optimizer might decide that a full table scan is more effective.
Check if statistics of your table out of date, especially check the statistics relates to your fulltext index. Since you are using SQl Server 2008, so, you can query DMV to get statistics information of your index or using DBCC to see the statistics detail information
DBCC SHOW_STATISTICS ("[schema].[table]",indexname);
Check the first return result, [Updated], [rows], [rows sampled], if the statistics out of date, or [rows sampled] far less than [rows], which many cause SQL Engine decided to use table scan instead of using your index.
I've been tasked with creating an application that allows users the ability to enter data into a web form that will be saved and then eventually used to populate pdf form fields.
I'm having trouble trying to think of a good way to store the field values in a database as the forms will be dynamic (based on pdf fields).
In the app itself I will pass data around in a hash table (fieldname, fieldvalue) but I don't know the best way to convert the hash to db values.
I'm using MS SQL server 2000 and asp.net webforms. Has anyone worked on something similar?
Have you considered using a document database here? This is just the sort of problem they solve alot better than traditional RDBMS solutions. Personally, I'm a big fan of RavenDb. Another pretty decent option is CouchDb. I'd avoid MongoDb as it really isn't a safe place for data in it's current implementation.
Even if you can't use a document database, you can make SQL pretend to be one by setting up your tables to have some metadata in traditional columns with a payload field that is serialized XML or json. This will let you search on metadata while staying out of EAV-land. EAV-land is a horrible place to be.
UPDATE
I'm not sure if a good guide exists, but the concept is pretty simple. The basic idea is to break out the parts you want to query on into "normal" columns in a table -- this lets you query in standard manners. When you find the record(s) you want, you can then grab the CLOB and deserialize it as appropriate. In your case you would have a table that looked something like:
SurveyAnswers
Id INT IDENTITY
FormId INT
SubmittedBy VARCHAR(255)
SubmittedAt DATETIME
FormData TEXT
A few protips:
a) use a text based serialization routine. Gives you a fighting chance to fix data errors and really helps debugging.
b) For SQL 2000, you might want to consider breaking the CLOB (TEXT field holding your payload data) into a separate table. Its been a long time since I used SQL 2000, but my recollection is using TEXT columns did bad things to tables.
The solution for what you're describing is called Entity Attribute Value (EAV) and this model can be a royal pain to deal with. So you should limit as much as possible your usage of this.
For example are there fields that are almost always in the forms (First Name, Last Name, Email etc) then you should put them in a table as fields.
The reason for this is because if you don't somebody sooner or later is going to realize that they have these names and emails and ask you to build this query
SELECT
Fname.value fname,
LName.Value lname,
email.Value email,
....
FROM
form f
INNER JOIN formFields fname
ON f.FormId = ff.FormID
and AttributeName = 'fname'
INNER JOIN formFields lname
ON f.FormId = ff.FormID
and AttributeName = 'lname'
INNER JOIN formFields email
ON f.FormId = ff.FormID
and AttributeName = 'email'
....
when you could have written this
SELECT
common.fname,
common.lname,
common.email,
....
FROM
form f
INNER JOIN common c
on f.FormId = c.FormId
Also get off of SQL 2000 as soon as you can because you're going to really miss the UNPIVOT clause
Its also probably not a bad idea to look at previous SO EAV questions to give you an idea of problems that people have encountered in the past
I'd suggest mirroring the same structure:
Form
-----
form_id
User
created
FormField
-------
formField_id
form_id
name
value
I have a trivial issue that I can't resolve. Currently our app uses Linq to retrieve data and get a basic integer value of the row count. I can't form a query that gives back a count without a 'select i'. I don't need the select, just the count(*) response. How do I do this? Below is a sample:
return (from io in db._Owners
where io.Id == Id && io.userId == userId
join i in db._Instances on io.Id equals i.Id **select i**).Count()
;
The select i is fine - it's not actually going to be fetching any data back to the client, because the Count() call will be translated into a Count(something) call at the SQL side.
When in doubt, look at the SQL that's being generated for your query, e.g. with the DataContext.Log property.
Using the LINQ query syntax requires a select statement. There's no way around that.
That being said, the statement will get transformed into a COUNT()-based query; the select i is there only to satisfy the expression system that underlies the LINQ query providers (otherwise the type of the expression would be unknown).
Including the select will not affect the performance here because the final query will get translated into SQL. At this point it will be optimized and will be like select (*) from ......