How to connect Kyvos cube (Bigdata Cube) in Excel
Open Excel -> Data -> Get Data -> From Database -> From Analysis Services
URL to enter in server Name (sample url): https://kyvospreprod.xyzcompany.com:2443/kyvos/xmlaKyvos
Related
I am open to other ways of doing this. Here are my constraints:
I have parquet files in a container in Azure Blob Storage
These parquet files will be partitioned by a product id, as well as the date (year/month/day)
I am doing this in R, and want to be able to connect interactively (not just set up a notebook in databricks, though that is something I will probably want to figure out later)
Here's what I am able to do:
I understand how to use arrow::open_dataset() to connect to a local parquet directory: ds <- arrow::open_dataset(filepath, partitioning = "product")
I can connect to, view, and download from my blob container with the AzureStor package. I can download a single parquet file this way and turn it into a data frame:
blob <- AzureStor::storage_endpoint("{URL}", key="{KEY}")
cont <- AzureStor::storage_container(blob, "{CONTAINER-NAME}")
parq <- AzureStor::storage_download(cont, src = "{FILE-PATH}", dest = NULL)
df <- arrow::read_parquet(parq)
What I haven't been able to figure out is how to use arrow::open_dataset() to reference the parent directory of {FILE-PATH}, where I have all the parquet files, using the connection to the container that I'm creating with AzureStor. arrow::open_dataset() only accepts a character vector as the "sources" parameter. If I just give it the URL with the path, I'm not passing any kind of credential to access the container.
Unfortunately, you probably are not going to be able to do this today purely from R.
Arrow-R is based on Arrow-C++ and Arrow-C++ does not yet have a filesystem implementation for Azure. There are JIRA tickets ARROW-9611,ARROW-2034 for creating one but these tickets are not in progress at the moment.
In python it is possible to create a filesystem purely in python using the FSspec adapter. Since there is a python SDK for Azure Blob Storage it should be possible to do what you want today in python.
Presumably something similar could be created for R but you would still need to create the R equivalent of the fsspec adapter and that would involve some C++ code.
If you use Azure Synapse then you can connect to your data with odbc as if it were a SQL Server database and it has support for partitioning and other files types as well. The pricing, from what I recall, is like $5/month fixed plus $5/TB queried.
Querying data would look something like this...
library(odbc)
syncon <- dbConnect(odbc(),
Driver = "SQL Server Native Client 11.0",
Server = "yourname-ondemand.sql.azuresynapse.net",
Database = "dbname",
UID = "sqladminuser",
PWD = rstudioapi::askForPassword("Database password"),
Port = 1433)
somedata <- dbGetQuery(syncon, r"---{SELECT top 100
result.filepath(1) as year,
result.filepath(2) as month,
*
FROM
OPENROWSET(
BULK 'blobcontainer/directory/*/*/*.parquet',
DATA_SOURCE='blobname',
FORMAT = 'parquet'
) as [result]
order by node, pricedate, hour}---")
the filepath keyword refers to the name of the directory in the BULK path.
Here's the MS website https://learn.microsoft.com/en-us/azure/synapse-analytics/sql/query-specific-files
You can also make views so that people who like SQL but not parquet files can query the views without having to know anything about the underlying data structure, it'll just look like a SQL Server database to them.
I have an SQLite connection set up as data source in IntelliJ IDEA Ultimate. How do you ATTACH a separate SQLite database to the connection?
Open Data Source properties -> Option tab -> Enable 'Single connection mode'. Close all sessions in 'Services' tool window, open new console for this data source and try something like this
attach database 'C:\temp\db1.sqlite' as db1;
it will be shown in schemas list as db1.
To persist an attached SQLite database, open the data source "Properties", click the "Options" tab, find the "Connection" section, find the "Startup script:" field, and enter the following:
ATTACH '/path/to/my/second/sqlite.db' AS db2;
I am trying to connect directly to a Domo data set within my datawarehouse in Domo. Is there a way I could make a direct connection to the Domo dataset in R? I simply want to have a connection directly to the dataset so as to avoid having to download a CSV and then import manually.
IBM Data Catalog doesn't allow to download the Connected Data Asset
I created a connected data from a dashdb connection -> selecting a table.
Also tried to create a connected data asset from cloudant connection -> selecting document.
Also uploaded a csv file as data asset.
None of the above enables Download Button.
Currently the download button is always disabled. We are looking to re-enable it for certain assets soon. More information will be provided when it comes.
I have been using ArcMap to access GIS data on a spatial data server. I want to figure out how to do the same within R.
I know how to read shapefiles into R. I have successfully used maptools and rgdal to open and map locally stored shapefiles (e.g.
http://www.nceas.ucsb.edu/scicomp/usecases/ReadWriteESRIShapeFiles)
My problem is when the data is not stored locally, but rather it is on an Application Server. I believe it's an Oracle database. I've been given information about the 1. Server 2. Instance (a number) 3. Database 4. User and 5. Password. Normally, I would include an example, but it's doubtful that an external user could access the servers.
For example here's how to read and plot local files in R
library(rgdal)
ogrInfo(".", "nw-rivers")
centroids.rg <- readOGR(".", "nw-centroids")
plot(centroids.rg)
The "." points to the local directory. How would I change this to access data on a server? The actual syntax of code would be helpful.
You can read data from Oracle Spatial DBs using GDAL/OGR:
http://www.gdal.org/ogr/drv_oci.html
if you have the driver in your GDAL/OGR installation. If:
require(rgdal)
ogrDrivers()
shows the Oracle driver then you can use readOGR with all the parameters in the right place.
At a guess, and by analogy with the PostGIS example, I'd say try:
dsn="OCI:userid/password#database_instance:")
ogrListLayers(dsn)
s = readOGR(dsn, layername)
but I don't have an Oracle server to test it on (if I did I'd ditch it tomorrow for PostGIS, and spend the license saving on a yacht) and you don't sound certain its an Oracle server anyway. The general principle for connecting to any spatial database is the same - check you have an OGR driver, figure out what the dsn parameter looks like, try it.
Another way is to go via ODBC, or another non-spatial R database connection. However you'll likely get back the spatial data in WKB or WKT form and have to convert to SpatialWhatevers (point, lines, polygons?).
PostGIS example is here:
https://gis.stackexchange.com/questions/64950/which-is-the-best-way-of-working-with-postgis-data-in-r