R in U-SQL in azure data lake analytics - r

I have been experimenting for a while now with R in Azure Data Lake Analytics and I keep getting stuck on installaing extra R-packages like dlplyr and tidyr. I asked the following question, but that did not solve the issue: Installing R-packages in Azure Data Lake Analytics
Now I am trying to follow this tutorial https://blog.revolutionanalytics.com/2017/10/adla-with-r.html but get an error when executing the third script usqlscriptEx3a.usql (https://github.com/Azure/ADLAwithR-GettingStarted/tree/master/Tutorial/Exercise3). I am using te script as is and I have been debugging the R-code locally, and that seems to do the right thing, so now I do not know where to look. The error is:
An unhandled exception from user code has been reported when invoking the method 'Reduce' on the user type 'Extension.R.Reducer'
Unhandled exception from user code: "Output column 'Par' is missing from the data frame"
However Par seems to be there as it should be.
Here follows some of the code from usqlscriptEx3a.usql:
REFERENCE ASSEMBLY [ExtR];
//declare the R script as a string variable and pass it as a parameter to
the Reducer:
DECLARE #myRScript = #"
temp = inputFromUSQL
t1 = loadedNamespaces()
temp = data.frame(loadedNamespaces = t1)
t2 = data.frame(apply(temp[1],1,FUN=function(x)
{paste(unlist(packageVersion(as.character(x))),collapse='.')}) )
names(t2) = 'packageVersion'
temp$packageVersion = t2$packageVersion
t3 = sessionInfo()[1]
t3 = t3$R.version$version.string
t3 = as.character(t3)
temp$Rversion = t3
temp$Rversion[2:nrow(temp)]=''
outputToUSQL = temp
";
DECLARE #myOutputFile string = #"/TutorialMaterial/outex3a.txt";
#somedata =
SELECT * FROM
(VALUES
("Contoso", 1500.0),
("Woodgrove", 2700.0)
) AS
D( customer, amount );
#ExtendedData = SELECT
0 AS Par,
*
FROM #somedata;
#RScriptOutput = REDUCE #ExtendedData ON Par PRODUCE
Par,
loadedNamespaces string,
packageVersion string,
Rversion string
USING new Extension.R.Reducer(command:#myRScript, rReturnType:"dataframe");

It turns out, that what was missing, at least in my case, is the following line:
READONLY Par
In order to make the script usqlscriptEx3a.usql work I had to insert READONLY Par in the following lines:
#RScriptOutput = REDUCE #ExtendedData ON Par
PRODUCE Par,
loadedNamespaces string,
packageVersion string,
Rversion string
READONLY Par
USING new Extension.R.Reducer(command:#myRScript, rReturnType:"dataframe");

Related

I am trying to extract values and place them in a .db to use later, however my code has bugged and I can no longer even load SQLite

Using :
-IronPython
-AutoDesk
-Revit (PyRevit)
-Revit API
-SQLite3
My code is as follows:
try:
conn = sqlite3.connect('SQLite_Python.db')
c = conn.cursor()
print("connected")
Insert_Volume = """INSERT INTO Column_Coordinates
(x, y, z)
VALUES
(1, 2, 3)"""
count = c.execute(Insert_Volume)
conn.commit()
print("Volume values inserted", c.rowcount)
c.close()
except sqlite3.Error as error:
print("Failed to insert data into sqlite table", error)
finally:
if (conn):
conn.close()
print("The SQLite connection is closed")'''
This code used to work within PyRevit, but now does not, with the following error:
Exception : System.IO.IOException: Could not add reference to assembly IronPython.SQLite
Please advise, this is one of the early steps of a large project and therefore is delaying my work quite a bit.
I look forward to your reply.

RODBC channel error in global environnement

I've set up a connection between R and SQL using the package RODBC and managed to connect and query the database from R.
I've created a small R function which objective is to delete some lines (as parameter) in a specific table.
Here it is (nameDB is the name of my database, and values_conversion another function I did to convert some data from R format to SQL format) :
delete_SQL = function(data, table){
ch = odbcConnect(nameDB,"postgres")
names = sqlColumns(channel=ch,table,schema="public",catalog = nameDB)$COLUMN_NAME
for(i in 1:nrow(data)){
sqlQuery(channel=ch,query=paste0("DELETE FROM public.\"",table,"\" WHERE ",
paste0(names," = ",values_conversion(data[i,]),collapse = " and "),";"),errors = TRUE)
}
odbcCloseAll()
}
Query exemple : "DELETE FROM public.\"lieu_protection\" WHERE lieu_id = 3 and protection_id = 1430;"
The code inside this function works fine when I execute everything directly, but when I call the function it throws an
Error in sqlQuery(channel = ch, query = paste0("DELETE FROM public.\"", :
first argument is not an open RODBC channel
I have a similar function that's getting and returning data from SQL and which is working fine, so I guess it has something to do with the delete, but the error is about the channel, so I'm quite confused.
Thanks to anyone that can help !

Accessing the same sqlite in memory database from two different engines in sqlalchemy

I "inherited" the following sample code. This is not the actual code but the essence of it
Base = declarative_base()
engine1 = create_engine('sqlite://')
engine2 = create_engine('sqlite://')
Session1 = sessionmaker(bind=engine1, autocommit=False)
Session2 = sessionmaker(bind=engine2, autocommit=False)
ScopedSession1 = scoped_session(Session1)
ScopedSession2 = scoped_session(Session2)
scoped_session1 = ScopedSession1()
scoped_session1 = ScopedSession2()
I have a bunch of models that inherit from Base and get created using the following
Base.metadata.create_all(bind=engine1)
If I do
scoped_session1.query(MyModel).all()
I get an empty list because nothing was inserted. If I do
scoped_session_2.query(MyModel).all()
I get
OperationalError: (sqlite3.OperationalError) no such table: reasons [SQL: u'SELECT reasons.id AS reasons_id, reasons.reason_text AS reasons_reason_text \nFROM reasons']
I am guessing they create different database in memory so the tables are only created in the first (engine1). I am using in memmory databases in my test cases of my application. Needed to do some refactoring and now I am stuck with this. Is it possible for two different engines to see the same in memory database?
As we know create_engine() returns an instance of Engine object. So what if define a function which return a cached reference to a previous instance created with same argument:
import weakref
_engine_cached = weakref.WeakValueDictionary()
def get_engine(engine):
if engine not in _engine_cached:
e = create_engine(engine)
_engine_cached[engine] = e
else:
e = _engine_cached[engine]
return e
engine1 = get_engine('sqlite://')
engine2 = get_engine('sqlite://')
Now you will be able to do scoped_session_2.query(MyModel).all()

StackoverflowException while in prod on IIS

I have a serious issue trying to put my Web Application into production.
My application is coded in VB.Net (With Visual Web Developer 2010), and the environment of production is a virtual machine with Microsoft Server 2008 R2, and IIS 7.5, the application is running with Asp.net 4.0. I tried also on my own computer (Windows 7), where I installed IIS, and the same bug occurs.
To limit the charge on SQL server, I put into session a list of rights, which is loaded each time the user wants to have access to any page.
The bug occurs exactly when this list contains more than 178 objects. At 178, the session is created and the page is correctly executed ; but when I add a 179th element or more (and I tried with different elements, so this can't come from the elements themselves), the page just don't execute correctly the code and IIS crash...
It seems that when there is more than 179 elements in this list, IIS just recycle the App, and so, the session is lost...
When I debug the application with Visual Web Developer, it is running well, without bug. But when I try to put it into production, that bug occurs. So, this seems to be linked with IIS.
Another problem is that when IIS recycle my App, it doesn't create any log file. So I have nothing more precise to give you.
I need help to find a solution to this problem. I try for a week, and I found nothing... this is driving me crazy... If someone has any idea or need more information, please tell me, so I can dig deeper...
Thanks to everyone.
EDIT :
I tried something else, I bypassed the sessions, and now, the rights are directly charged from SQL. The problem still occurs. So, it doesn't come from sessions, but SQL.
So, I wonder if there is any limit while doing requests on SQL.
But SQL doesn't seems to have any problem with request that return 3000 elements, and when it returns 179 elements with a lot of joins, it just crash... Well, I have no idea what to do or where search for a solution...
EDIT2 :
But, like I said, the problem doesn't occurs when I try execute the code from Visual Web developer. And, with it or in production environment, it's the same SQL server that execute de SQL requests... So, the problem can't come from SQL finally...
EDIT3 :
I finally found something. In the Windows logs, I have a StackOverflowException... What happy news ! (And we are on stackoverflow.com, is that linked? xD).
EDIT4 :
Here is the error from the windows log. Maybe it can help anyone.
Nom du journal :Application
Source : Windows Error Reporting
Date : 29/04/2014 11:53:11
ID de l’événement :1001
Catégorie de la tâche :Aucun
Niveau : Information
Mots clés : Classique
Utilisateur : N/A
Ordinateur : SRVWEB.DOMMH01.local
Description :
Récipient d’erreurs , type 0
Nom d’événement : CLR20r3
Réponse : Non disponible
ID de CAB : 0
Signature du problème :
P1 : w3wp.exe
P2 : 7.5.7601.17514
P3 : 4ce7afa2
P4 : System.Core
P5 : 4.0.30319.18408
P6 : 5231116c
P7 : 1197
P8 : 0
P9 : System.StackOverflowException
P10 :
Fichiers joints :
Ces fichiers sont peut-être disponibles ici :
C:\ProgramData\Microsoft\Windows\WER\ReportQueue\AppCrash_w3wp.exe_e41148f8d4e7fcf353a2d93bfe4367a48b981de2_0a6131b2
Symbole d’analyse :
Nouvelle recherche de la solution : 0
ID de rapport : 125b520d-cf84-11e3-912f-00155dc81602
Statut du rapport : 4
EDIT 5 :
I put here the piece of code where I take the rights with LINQ to SQL. I don't think this will be really helpful, but there it is...
A right contains a Username, the id of the rights, the id of a family (a family is a set of "objects"), and a place. So finnaly, I get a list of "Object/Place" associated with rights for a specified user... So I can limit the access for only some "object/place" for each user.
Another thing is the familyJoker, and the placeJoker. When a right is associated with a placeJoker, wee add to the rights of this user any object/place where the object is in the family. It's the same when the family is the familyJoker. Finally, if family and place are both joker, we add all the combinations possible(the combinations possible are in the objectPlace table)
This is the code :
Public Shared Function getListeDroitsUtilisateurSession(ByVal username As String) As List(Of DroitsUtilisateurSession)
Dim db As SINDIDB = New SINDIDB
Dim familleJoker As String = FamilleMetier.famille_Joker
Dim lieuJoker As String = Lieu.lieu_Joker
Dim requeteSansJoker = From d In db.DroitsUtilisateurs
Where d.Utilisateur = username
Join r In db.Roles
On d.ID_Role Equals r.ID_Role
Join oif In db.ObjetsInFamilleMetiers
On d.ID_Famille Equals oif.ID_Famille
Join f In db.FamilleMetiers
On oif.ID_Famille Equals f.ID_Famille
Where (Not f.Nom_Famille = familleJoker)
Join ol In db.ObjetLieux
On oif.ID_Objet Equals ol.ID_Objet
Where (d.ID_Lieu = ol.ID_Lieu)
Join l In db.Lieux
On ol.ID_Lieu Equals l.ID_Lieu
Where (Not l.Valeur_Lieu = lieuJoker)
Select New DroitsUtilisateurSession With {.Role = r, .ObjetLieu = ol}
Dim requeteFamilleJoker = From d In db.DroitsUtilisateurs
Where d.Utilisateur = username
Join r In db.Roles
On d.ID_Role Equals r.ID_Role
Join f In db.FamilleMetiers
On d.ID_Famille Equals f.ID_Famille
Where (f.Nom_Famille = familleJoker)
From oCross In db.Objets
Join ol In db.ObjetLieux
On oCross.ID_Objet Equals ol.ID_Objet
Where (d.ID_Lieu = ol.ID_Lieu)
Select New DroitsUtilisateurSession With {.Role = r, .ObjetLieu = ol}
Dim requeteLieuJoker = From d In db.DroitsUtilisateurs
Where d.Utilisateur = username
Join r In db.Roles
On d.ID_Role Equals r.ID_Role
Join l In db.Lieux
On d.ID_Lieu Equals l.ID_Lieu
Where (l.Valeur_Lieu = lieuJoker)
Join oif In db.ObjetsInFamilleMetiers
On d.ID_Famille Equals oif.ID_Famille
From lCross In db.Lieux
Join ol In db.ObjetLieux
On oif.ID_Objet Equals ol.ID_Objet
Where (lCross.ID_Lieu = ol.ID_Lieu)
Select New DroitsUtilisateurSession With {.Role = r, .ObjetLieu = ol}
Dim requeteFamilleEtLieuJoker = From d In db.DroitsUtilisateurs
Where d.Utilisateur = username
Join r In db.Roles
On d.ID_Role Equals r.ID_Role
Join f In db.FamilleMetiers
On d.ID_Famille Equals f.ID_Famille
Where (f.Nom_Famille = familleJoker)
Join l In db.Lieux
On d.ID_Lieu Equals l.ID_Lieu
Where (l.Valeur_Lieu = lieuJoker)
From olCross In db.ObjetLieux
Select New DroitsUtilisateurSession With {.Role = r, .ObjetLieu = olCross}
Dim requeteTotale = requeteSansJoker.Union(requeteFamilleJoker).Union(requeteLieuJoker).Union(requeteFamilleEtLieuJoker)
Dim resultat As List(Of DroitsUtilisateurSession) = requeteTotale.ToList
Dim listeDroitsUtilisateur As List(Of DroitsUtilisateurSession) = New List(Of DroitsUtilisateurSession)
If (resultat IsNot Nothing) Then
For Each item In resultat
listeDroitsUtilisateur.Add(New DroitsUtilisateurSession With {.Role = item.Role, .ObjetLieu = item.ObjetLieu})
Next
End If
Return listeDroitsUtilisateur
End Function
EDIT 6: I found something about the limit size of the pile of w3wp applications. This can be linked to my problem...
Here is the link :
http://support.microsoft.com/kb/932909
I'm digging.
EDIT 7: So, I made a little change to my code. Now, when it have to calculate the rights, it let another thread do it. This thread has a bigger stack size. So I thought it can change something, but the same problem occurs...
And I don't said it before, but there is NO recursive code in my App.
Here is the code I implemented :
Dim ThreadD As ThreadDroits = New ThreadDroits
ThreadD.Username = User.Identity.Name
Dim t = New Thread(AddressOf ThreadD.ExecuteDroits, 419430400)
t.Start()
t.Join()
Dim listeDroits As List(Of DroitsUtilisateurSession) = ThreadD.ListeDroits
And here is the new class :
Public Class ThreadDroits
Private m_Username As String
Public Property Username As String
Get
Return m_Username
End Get
Set(ByVal value As String)
m_Username = value
End Set
End Property
Private m_listeDroits As List(Of DroitsUtilisateurSession)
Public Property ListeDroits As List(Of DroitsUtilisateurSession)
Get
Return m_listeDroits
End Get
Set(ByVal value As List(Of DroitsUtilisateurSession))
m_listeDroits = value
End Set
End Property
Public Sub ExecuteDroits()
ListeDroits = DroitsUtilisateurSession.getListeDroitsUtilisateurSession(Username)
End Sub
End Class

asp ERROR MESSAGE

How I can solve this error message
Microsoft VBScript runtime error
'800a01a8' Object required: 'lUATRef'
/cmgtest/transaction/viewPCReqForm.asp,
line 284
this is some source code that I wrote below
function checkUATReq(aUATRef)
Dim correctness,lUATRef,uatRef
correctness = False
lUATRef = aUATRef
uatRef = lUATRef.Substring(1,2)
rwriteln uatRef
'sqlCheckUATReq = "select * from PC_DETAIL where ID ='"&uatReqRef&"'"
'rwriteln uatReqRef
End function
Seems like your function isn't getting a parameter passed to it. Check whether aUATRef is getting initialized.
In VBScript, Strings aren't objects. You should use the Mid function to get a portion of a string:
uatRef = Mid(IUATRef, 1, 2)

Resources