Access parquet files in Azure DataBricks by R via RODBC - r

Question

I successfully configured connection to Azure DataBricks cluster and can query tables with
conn <- odbcConnect("AzureDatabricks")
sqlQuery(conn, "SELECT * FROM my_table")
but I need to access parquet files.
In Databricks I can do it with this code:
%sql
Select * FROM parquet.`/path/to/folder`
If I try this by R as
sqlQuery(conn, "Select * FROM parquet.`/path/to/folder`")
I receive error:
[Simba][SQLEngine] Table or view not found: SPARK.parquet./path/to/folder"
[RODBC] ERROR: Could not SQLExecDirect 'Select * FROM parquet.`/path/to/folder`
Is there way to access parquet files via RODBC?

Answer 1

You are experiencing this issue due to an error in your sql query itself. When you run Select * FROM parquet./path/to/folder, command you will not see table or view not found due to syntax error.
Example: Sample example for understanding the issue (when you run SELECT * FROM parquer.'somepath'), you will see the syntax error.
Note: After creating a Dataframe from parquet file, you have to register it as a temp table to run sql queries on it.
val sqlContext = new org.apache.spark.sql.SQLContext(sc)
val df = sqlContext.read.parquet("src/main/resources/peopleTwo.parquet")
df.printSchema
// after registering as a table you will be able to run sql queries
df.registerTempTable("people")
sqlContext.sql("select * from people").collect.foreach(println)
Reference: Spark SQL guide - Parquet files

Answer 2

You need to add UseNativeQuery=1; parameter to the odbc connection string.
Examlpe: Driver={Simba Spark ODBC Driver};Host=[serverHost];Port=443;HTTPPath=[httpPath];ThriftTransport=2;SSL=1;AuthMech=3;UID=token;PWD=[pwd];UseNativeQuery=1;
https://docs.databricks.com/integrations/jdbc-odbc-bi.html#ansi-sql-92-query-support-in-odbc