I have been using olapR (along with Microsoft R Client, which you need in order to use olapR) to do MDX queries on a MSOLAP database.
However, every time I execute any query it return a null object. I know these are not empty queries. Here is an example:
library(olapR)
cnnstr="Provider=MSOLAP;Password=***;Persist Security Info=True;User ID=***;Data Source=***;Initial Catalog=***"
olapCnn=OlapConnection(cnnstr)
qry <- Query(validate = TRUE)
cube(qry) <- "PER - Prestación Servicios de Salud"
columns(qry) <- c("[Measures].[Número Personas Atendidas]")
rows(qry) <- c("[Diagnóstico Principal].[Diagnósticos].[Diagnósticos].MEMBERS")
results1=executeMD(olapCnn, qry)
The problem is results1 is NULL.
If I use the command execute2D instead I get the following error:
Unknown object recieved from olapRExecuteRecordset, which is just a consequence of the object being return being NULL
Since the author has not answered here, I can tell how I solved this in case someone else has maybe the same issue.
Basically I'm using 2-3 cubes, but I had only used the connection string to one of the cubes, so when I tried to query the other cubes I got the NULL object, or unknown object received. The solution in my case was simply to create different connection strings for each of the cubes, and use the correct connection string when querying.
Related
I am using this query to get some statistics from our DB
SELECT DateTimePart("yyyy", c.RedeemedDate) AS RedeemedInYear,
DateTimePart("m", c.RedeemedDate) AS RedeemedInMonth,
AVG(c.RedPriceStoreSummary ?? c.WhitePriceStoreSummary) AS AverageBasketValue
FROM c
WHERE c.IsRedeemed = true
AND c.Brand = 'xxx'
AND c.RedeemedDate != null
AND DateTimePart("yyyy", c.RedeemedDate) = DateTimePart("yyyy", GetCurrentDateTime())
GROUP BY DateTimePart("m", c.RedeemedDate),
DateTimePart("yyyy", c.RedeemedDate)
The problem is that query crashes with the following error if there are no results
Cannot read property 'create' of undefined
If I force the query to where it gets results then everything works fine but I dont want the query to crash if there are no results I want an empty result set.
Am I missing something here?
You need to trap queries in a try block and catch if it returns a 404. This is a hold over in behavior from the previous SDK. Alternatively you can use the stream variant of these SDK functions which will not throw exceptions when data is not found, however you still want to check the http response code and you'll also need to manually deserialize the data.
I use airflow python operators to execute sql queries against a redshift/postgres database. In order to debug, I'd like the DAG to return the results of the sql execution, similar to what you would see if executing locally in a console:
I'm using psycop2 to create a connection/cursor and execute the sql. Having this logged would be extremely helpful to confirm the parsed parameterized sql, and confirm that data was actually inserted (I have painfully experiences issues where differences in environments caused unexpected behavior)
I do not have deep knowledge of airflow or the low level workings of the python DBAPI, but the pscyopg2 documentation does seem to refer to some methods and connection configurations that may allow this.
I find it very perplexing that this is difficult to do, as I'd imagine it would be a primary use case of running ETLs on this platform. I've heard suggestions to simply create additional tasks that query the table before and after, but this seems clunky and ineffective.
Could anyone please explain how this may be possible, and if not, explain why? Alternate methods of achieving similar results welcome. Thanks!
So far I have tried the connection.status_message() method, but it only seems to return the first line of the sql and not the results. I have also attempted to create a logging cursor, which produces the sql, but not the console results
import logging
import psycopg2 as pg
from psycopg2.extras import LoggingConnection
conn = pg.connect(
connection_factory=LoggingConnection,
...
)
conn.autocommit = True
logging.basicConfig(level=logging.DEBUG)
logger = logging.getLogger(__name__)
logger.addHandler(logging.StreamHandler(sys.stdout))
conn.initialize(logger)
cur = conn.cursor()
sql = """
INSERT INTO mytable (
SELECT *
FROM other_table
);
"""
cur.execute(sql)
I'd like the logger to return something like:
sql> INSERT INTO mytable (
SELECT ...
[2019-07-25 23:00:54] 912 rows affected in 4 s 442 ms
Let's assume you are writing an operator that uses postgres hook to do something in sql.
Anything printed inside an operator is logged.
So, if you want to log the statement, just print the statement in your operator.
print(sql)
If you want to log the result, fetch the result and print the result.
E.g.
result = cur.fetchall()
for row in result:
print(row)
Alternatively you can use self.log.info in place of print, where self refers to the operator instance.
Ok, so after some trial and error I've found a method that works for my setup and objective. To recap, my goal is to run ETL's via python scripts, orchestrated in Airflow. Referring to the documentation for statusmessage:
Read-only attribute containing the message returned by the last command:
The key is to manage logging in context with transactions executed on the server. In order for me to do this, I had to specifically set con.autocommit = False, and wrap SQL blocks with BEGIN TRANSACTION; and END TRANSACTION;. If you insert cur.statusmessage directly following a statement that deletes or inserts, you will get a response such as 'INSERT 0 92380'.
This still isn't as verbose as I would prefer, but it is a much better than nothing, and is very useful for troubleshooting ETL issues within Airflow logs.
Side notes:
- When autocommit is set to False, you must explicitly commit transactions.
- It may not be necessary to state transaction begin/end in your SQL. It may depend on your DB version.
con = psy.connect(...)
con.autocommit = False
cur = con.cursor()
try:
cur.execute([some_sql])
logging.info(f"Cursor statusmessage: {cur.statusmessage})
except:
con.rollback()
finally:
con.close()
There is some buried functionality within psycopg2 that I'm sure can be utilized, but the documentation is pretty thin and there are no clear examples. If anyone has suggestions on how to utilize things such as logobjects, or returning join PID to somehow retrieve additional information.
I was trying to make a website using perl dancer, below is my code. It seems to be correct but the page keeps loading and never enters the values in the database. When I cancel the page I get an error stating "request to POST /appform crashed: Can't call method "execute" on an undefined value". I can't figured out whats wrong in the code. If you have any other code please mention.
I am using SQLite for database.
There is a database campus.dband I am inserting the value in student table.
post '/appform' => sub {
my $q = CGI ->new;
my $name = $q->param ("firstname");
my $password = $q->param("password");
my $mobile_no = $q->param("mobile");
my $gender = $q->param("gender");
my $email = $q->param("email");
my $address = $q->param("address");
my $sslc = $q->param("SSLC");
my $hsc = $q->param("HSC");
my $cgpa = $q->param("cgpa");
my $languages = $q->param("lang");
my $internships = $q->param("intern");
my $preferred_loc = $q->param("country");
my $sql = "insert into student(name,mobile_no,gender,email,address,sslc,hsc,cgpa,languages,internships,preferred_loc,password,applied_job,company_applied) values ('?','?','?','?','?','?','?','?','?','?','?','?','?','?');";
my $sth = database->prepare($sql);
$sth->execute($name,$mobile_no,$gender,$email,$address,$sslc,$hsc,$cgpa,$languages,$internships,$preferred_loc,$password) or die $sth->errstr;
#$sth->execute();
$sth-> finish;
set_flash('New entry posted!');
redirect '/';
};
You're using the database keyword to get a database handle. I'm guessing that's coming from Dancer2::Plugin::Database (it would be useful if you could include information like this in your question).
The error says that you're calling execute() on an undefined value. You're calling execute() on the variable $sth. So $sth is undefined. You get $sth by calling prepare() on the database handle returned from database(). So it looks like the prepare() call is failing. You should check the return value from that call and throw an error if it fails.
The most common reason for prepare() to fail is that you're trying to compile an SQL statement that contains an error. I can't see any obvious error in your SQL, but it's worth checking it by running it manually against your database.
I see you're using bind params in your SQL statement. That's a great idea, but please note that you don't need to quote the question marks in your SQL - the database driver will handle that for you. I don't think that's what is causing your problem though.
I also see that you're using CGI.pm inside your Dancer app to get the request parameters. To be honest, I'm slightly surprised that it works - but it's a terrible idea. Dancer has its own keywords that will give you this information. Look at query_parameters(), body_parameters() and route_parameters() in the Dancer documentation.
In addition to the points made already, that your DBI prepare() call is probably failing (add error-checking to see why, e.g. my $sth = database->prepare('...') or die "DB error: " . database->errstr) and that you're using CGI.pm within a Dancer app (... don't do that, I'm surprised it would work at all - look at the Dancer documentation for how to access the params your app was sent), look also at the quick_insert convenience method provided by Dancer::Plugin::Database / Dancer2::Plugin::Database so that you don't have to write that SQL INSERT statement at all.
I have encountered an error when trying to insert thousands of rows with R/RJDBC and the dbSendUpdate command on an Oracle database.
Problem can be reproduced by creating a test table with
CREATE TABLE mytest (ID NUMBER(10) not null);
and then executing the following R script
library(RJDBC)
drv<-JDBC("oracle.jdbc.OracleDriver","ojdbc-11.1.0.7.0.jar") # place your JDBC driver here
conn <- dbConnect(drv, "jdbc:oracle:thin:#MYSERVICE", "myuser", "mypasswd") # place your connection details here
for (i in 1:10000) {
dbSendUpdate(conn,"INSERT INTO mytest VALUES (?)",i))
}
Searching the Internet provided the information, that one should close result cursors, which is obvious (e.g. see java.sql.SQLException: - ORA-01000: maximum open cursors exceeded or Unable to resolve error - java.sql.SQLException: ORA-01000: maximum open cursors exceeded).
But the help file for ??dbSendUpdate claims for not using result cursors at all:
.. that dbSendUpdate is used with DBML queries and thus doesn't return any result set.
Therefore this behavior doesn't make much sense to me :-(
Can anybody help?
Thanks, a lot!
PS: Found something interessting in the RJDBC documentation http://www.rforge.net/RJDBC/
Note that the life time of a connection, result set, driver etc. is determined by the lifetime of the corresponding R object. Once the R handle goes out of scope (or if removed explicitly by rm) and is garbage-collected in R, the corresponding connection or result set is closed and released. This is important for databases that have limited resources (like Oracle) - you may need to add gc() by hand to force garbage collection if there could be many open objects. The only exception are drivers which stay registered in the JDBC even after the corresponding R object is released as there is currently no way to unload a JDBC driver (in RJDBC).
But again, even inserting gc() within the loop will produce the same beahvoir.
Finally we realized this being a bug in package RJDBC. If you are willing to patch RJDBC you can use the patched sources available at https://github.com/Starfox899/RJDBC (as long as they are not imported into the package).
I have a super simple select in a wcf service like
BusinessModel.Candidate candidateObject
= dcMUPView.Candidates.SingleOrDefault(dev => dev.Username == username);
But when I hit this code, and then check the candidateObject, it's candidateid value is incorrect. In the DB candidateid is the identity column and primary key. In this case it is 2572884 in the DB but when I look in the candidateObject is says the value returned is something like 0x00274254. Anyone know what might cause something like this?
I think your query is returning the correct record, since 0x00274254 (base 16) = 2572884 (base 10).
If you are checking the value using the debugger, make sure you have the Hexadecimal Display option turned off (see here for more info: Visual Studio debugger - Displaying integer values in Hex).