I am running OWASP Zap as a security review for our code.
A SQL injection is being detected when we are using preparedquery and I am struggling to determine why? The basis of our code is:
DLPreparedQuery preparedQuery = this._env.Conn.Prepare("select a from Table1",
new DLDataTypes[] { DLDataTypes.BigInt, DLDataTypes.Char, DLDataTypes.BigInt, DLDataTypes.BigInt, DLDataTypes.Char, DLDataTypes.BigInt },
new string[] { "#P1", "#P2", "#P3", "#P4", "#P5", "#P6" },
DLQueryOptions.Default, -1);
object[,] objResult = (object[,])preparedQuery.Query(1, 'ABC', 'DEF', 1,'AAA',21);
We also escape all user input text with antixss using
Microsoft.Security.Application.Encoder.HtmlEncode(inputData)
Are there settings I should be looking at at SQLServer?
Thanks in Advance.
Haven’t worked with preparedquery but I’d suggest you run sql profiler until you identify the malicious statement. After identifying how statement looks like, you’ll most probably know what to do.
HtmlEncode doesn’t work here. I’d suggest you try replacing single quote ‘ with two single quotes. This is not a bullet proof solution but it’s likely to give you good results.
If your database is in full recovery mode you can try using some third party transaction log reader to scan through already executed statements and try to identify what could be the issue.
Related
If you try to Configure SQL Server Transport in Rebus you are encountered with a warning that using a connection string is obsolete and you are directed to use SqlServerTransportOptions.
However I have tried to find any example or code snippet that shows how to do configuration using SqlServerTransportOptions without any success.
Kindly if any body can share a snippet on how to do it properly.
Thanks
Have you tried
Configure.With(activator)
.Transport(t =>
{
var options = new SqlServerTransportOptions(connectionString);
t.UseSqlServer(options, queueName);
})
.Start();
? (where connectionString is your connection string, and queueName is the name of the input queue)
Generally, it should be possible with Rebus to figure things out simply by letting Intellisense and the help tooltips guide you... if it didn't work out for you in this case, I'm very curious to hear what you think is missing.
I use airflow python operators to execute sql queries against a redshift/postgres database. In order to debug, I'd like the DAG to return the results of the sql execution, similar to what you would see if executing locally in a console:
I'm using psycop2 to create a connection/cursor and execute the sql. Having this logged would be extremely helpful to confirm the parsed parameterized sql, and confirm that data was actually inserted (I have painfully experiences issues where differences in environments caused unexpected behavior)
I do not have deep knowledge of airflow or the low level workings of the python DBAPI, but the pscyopg2 documentation does seem to refer to some methods and connection configurations that may allow this.
I find it very perplexing that this is difficult to do, as I'd imagine it would be a primary use case of running ETLs on this platform. I've heard suggestions to simply create additional tasks that query the table before and after, but this seems clunky and ineffective.
Could anyone please explain how this may be possible, and if not, explain why? Alternate methods of achieving similar results welcome. Thanks!
So far I have tried the connection.status_message() method, but it only seems to return the first line of the sql and not the results. I have also attempted to create a logging cursor, which produces the sql, but not the console results
import logging
import psycopg2 as pg
from psycopg2.extras import LoggingConnection
conn = pg.connect(
connection_factory=LoggingConnection,
...
)
conn.autocommit = True
logging.basicConfig(level=logging.DEBUG)
logger = logging.getLogger(__name__)
logger.addHandler(logging.StreamHandler(sys.stdout))
conn.initialize(logger)
cur = conn.cursor()
sql = """
INSERT INTO mytable (
SELECT *
FROM other_table
);
"""
cur.execute(sql)
I'd like the logger to return something like:
sql> INSERT INTO mytable (
SELECT ...
[2019-07-25 23:00:54] 912 rows affected in 4 s 442 ms
Let's assume you are writing an operator that uses postgres hook to do something in sql.
Anything printed inside an operator is logged.
So, if you want to log the statement, just print the statement in your operator.
print(sql)
If you want to log the result, fetch the result and print the result.
E.g.
result = cur.fetchall()
for row in result:
print(row)
Alternatively you can use self.log.info in place of print, where self refers to the operator instance.
Ok, so after some trial and error I've found a method that works for my setup and objective. To recap, my goal is to run ETL's via python scripts, orchestrated in Airflow. Referring to the documentation for statusmessage:
Read-only attribute containing the message returned by the last command:
The key is to manage logging in context with transactions executed on the server. In order for me to do this, I had to specifically set con.autocommit = False, and wrap SQL blocks with BEGIN TRANSACTION; and END TRANSACTION;. If you insert cur.statusmessage directly following a statement that deletes or inserts, you will get a response such as 'INSERT 0 92380'.
This still isn't as verbose as I would prefer, but it is a much better than nothing, and is very useful for troubleshooting ETL issues within Airflow logs.
Side notes:
- When autocommit is set to False, you must explicitly commit transactions.
- It may not be necessary to state transaction begin/end in your SQL. It may depend on your DB version.
con = psy.connect(...)
con.autocommit = False
cur = con.cursor()
try:
cur.execute([some_sql])
logging.info(f"Cursor statusmessage: {cur.statusmessage})
except:
con.rollback()
finally:
con.close()
There is some buried functionality within psycopg2 that I'm sure can be utilized, but the documentation is pretty thin and there are no clear examples. If anyone has suggestions on how to utilize things such as logobjects, or returning join PID to somehow retrieve additional information.
Here's my situation: I'm in the process of a website overhaul to the public website for our company. The old site was running on server 2003 on .net framework 3.5? it may have been originally built on the 2.0 framework for that matter. ANYway, the old site had a search feature which worked really nicely for users to find pages related to topics they were interested in. It used the old MSIDXS oledb connection type... simple code as follows...
Dim odbSearch As New System.Data.OleDb.OleDbConnection("Provider=""MSIDXS"";Data Source=""Proto"";")
Dim cmdSearch As New System.Data.OleDb.OleDbCommand
cmdSearch.CommandText = String.Format("SELECT doctitle, filename, vpath, rank, characterization, size FROM scope() WHERE FREETEXT(Contents, '{0}') ORDER BY rank DESC", searchText) WHERE CONTAINS(*,'\""" & searchText & "*\""') AND scope='file:C:\...\Web_App' ORDER BY System.ItemPathDisplay DESC
This worked great. But now we're moving this to a 2008 r2 server, which doesn't have the MSIDXS indexing anymore... or it does, but doesn't work for sites? I was able to turn it on, but it never found anything, and the catalog remained empty, and everything I've read said this isn't how to do searches on sites anymore. The 'new way' that I read about was using windows search service. I've adjusted the service on the box to 'index' the website's directory, and it seems to have stuff in the catalog... however, what code I've converted always returns 0. so the new code looks like...
Dim odbSearch As New System.Data.OleDb.OleDbConnection("Provider=Search.CollatorDSO.1;Extended Properties='Application=Windows';")
cmdSearch.CommandText = String.Format("SELECT system.title, system.filename, System.ItemPathDisplay FROM SystemIndex WHERE scope='file:C:\...\Web_App'")
Dim rdrSearch As OleDbDataReader = cmdSearch.ExecuteReader()
While rdsSearch.read()
I can't get this to actually return any results. regardless of what I put in as the search criteria, it jumps right to the end while.
Can someone tell me what piece of the puzzle I'm missing?
Everything was actually correct; I wasn't getting any rows back because I had already narrowed the systemIndex to be looking at the subdirectory that I wanted, so including a scope in the where clause was causing zero results. As soon as I took that out, the search works.
It looks that the scope should be specified as 'file:C:\\...\\Web_App', or add a '#' sign at the beginning of your query string.
I have a personal website written in ASP.NET Web Pages, with a small SQL Server Compact Edition database. This is handy for me because the database is stored as a file (.sdf), and when I add, modify or delete any records and test it out locally, I can just use Publish... in WebMatrix and copy the .sdf file over.
This works great except for a random error message I get when it accesses the .sdf file.
I get...
There is a file sharing violation. A different process might be using the file.
Sometimes a page refresh will fix it, sometimes I have to refresh the page a couple of times. I suppose migrating the database to a SQL Server database will fix it, but I don't really want to do that. Is that the best option though?
Here's a link to the site, maybe you'll hit the error: http://www.garethlewisweb.com
Is there something I have to change in the code, or will I need to ask my hosting company to change something. I don't really have a lot of experience in this area.
Thanks!
Update
Here's my code
_AppStart.cshtml
#{
WebSecurity.InitializeDatabaseConnection("GarethLewisWeb", "UserProfile", "UserId", "Email", true);
// WebMail.SmtpServer = "mailserver.example.com";
// WebMail.EnableSsl = true;
// WebMail.UserName = "username#example.com";
// WebMail.Password = "your-password";
// WebMail.From = "your-name-here#example.com";
}
Default.cshtml
#{
Layout = "~/Shared/_SiteLayout.cshtml";
Page.Title = "Home";
var db = Database.Open("GarethLewisWeb");
var featuredPhotoSQL = " SELECT col " +
" FROM tablename ";
var featuredPhoto = db.QuerySingle(featuredPhotoSQL);
}
[Lots of HTML and C# code here...]
Well, you know- that is like trying to wash dishes in an oven. It won't work.
Basically - most likely you try to create multiple instances of SQL Server CE and then use the same DB and guess what - the old one still has the exclusive lock.
In general, you CAN NOT USE SQL CE ON A WEBSITE. It is not made for concurrent access, and even if you are one of the few, that is like asking for trouble. Google comes around, a fiend comes around, and occaionally your browser comes around with 2 requests that overlap and boom - dead.
You do not show any code, so I can only guess what causes the problem, but the documentation for CE is QUITE strict - it is NOT for multi threaded scenarios.
Not really an answer to this specific problem, but the migration to SQL Server solved this issue.
When we added Mode=Read Only; Temp Path=c:/Temp in the connection string then it fixed.
<connectionStrings>
<add name="XYZ" providerName="System.Data.SqlServerCe.4.0" connectionString="Data Source=|DataDirectory|\XYZ.sdf;Max Database Size = 4091;Enlist=false;encryption mode=engine default;password=!secret!;Mode = Read Only;Temp Path=c:/temp" />
</connectionStrings>
It worked for us !
There is a GetLastInsertId method in WebMatrix. However, using Reflector, I can see that it's implemented like this:
[return: Dynamic]
public object GetLastInsertId()
{
return this.QueryValue("SELECT ##Identity", new object[0]);
}
Which, as far as I can see, is a very bad way of doing it, because ##Identity returns the last insert considering every table and every session. I need this restricted to the scope and session I'm working with, so I was expecting this to use SELECT SCOPE_IDENTITY(), since that also seems to be what is most often used according to my reading.
My questions are:
Do the WebMatrix wrappers do something that makes this ok to use in my case?
If not, is there a simple way to get the inserted ID of an insert query using WebMatrix classes, or should I fall back on stuff like SqlClient for this?
I am interested in answers for SQL Server (not compact) if that makes a difference.
I had the same problem, and I read these answers but still couldn't get the code to work correctly. I possibly was misreading the solutions above, but what seemed to work for me was as follows:
var db=Database.Open(...);
var lastInsertID = db.QueryValue("INSERT name into Names; SELECT SCOPE_IDENTITY()");
To quote David Fowler, one of the devs on the Data APIs in WebMatrix: "[SCOPE_IDENITTY] doesn't work on ce". Since we wanted a data API that would work on both CE and regular SQL Server, we used ##Identity. So if you want to use SCOPE_IDENTITY specifically, you should use db.QueryValue method:
var db = Database.Open(...);
db.Insert(...);
var lastInsertId = db.QueryValue("SELECT SCOPE_IDENTITY()")
here is my solution
var db.Database.open('..');
var lastId = db.GetLastInsertId();