I want to import data in JSON format from MongoDB in to R. I am using Mongolite package to connect MongoDB to R, but when i use mongo$find('{}') data is getting stored as dataframe. Please check my Rcode,
library(mongolite)
mongo <- mongolite::mongo(collection = "Attributes", db = "Test", url =
"mongodb://IP:PORT",verbose = TRUE)
df1 <- mongo$find('{}')
df1 is getting stored as dataframe, but I want the data in JSON format only. Please give your suggestions for the same.
Edit -
Actual json structure converted in to list
But when i load data from MongoDB to R using mongolite package, data is getting stored as dataframe and then if i convert to list, the structure is getting changed and few extra columns are inserted in to list.
Please let me know on how to solve this issue.
Thanks
SJB
Related
I am trying to import dynamodb in R using paws.database library. I am successful in retrieving the required attribute into R(using scan operation). However the imported data is in the form of a nested list i.e. in [[]] form. My intention is to format the imported dynamodb attribute into a dataframe and later be able to plot it using ggpplot. I have tried using options such as
df <- ldply (list_a, data.frame), ldply (list_a, data.frame),
data.frame(matrix(unlist(list_a), nrow=length(list_a),
byrow=TRUE),stringsAsFactors=FALSE),
as.data.frame(do.call(cbind,list_a))
so far and was unable to convert the data in a proper dataframe format. The final error I get in ggplot is "
Error: data must be a data frame, or other object coercible by fortify(), not a list "
Could anyone please help ?
See this similar issue.
I'm also using paws. Here's what I did to work with a small DynamoDB table:
dyna <- paws::dynamodb()
Table<-dyna$scan("table_name")
newtable<-rbindlist(Table$Items,fill = TRUE)
Then I create a new dataframe by using unlist() on each column of newtable.
I've explored but didn't find any suggestions.
I want to know how to save an empty file to adlsgen2 using R and then read it back to the same code.
Thanks for the help.
Since you want to write an empty file to ADLS gen2 using R and read it from that location, first create a dataframe.
The dataframe can either be completely empty, or it can be a dataframe with column names but no rows. You can use the following code to create one.
df <- data.frame() #OR
df <- data.frame(<col_name>=<type>)
Once you have created a dataframe, you must establish connection to ADLS gen2 account. You can use the method specified in the following link to do that.
https://saketbi.wordpress.com/2019/05/11/how-to-connect-to-adls-gen2-using-sparkr-from-databricks-rstudio-while-integrating-securely-with-azure-key-vault/
After making the connection you can use read and write functions in R language using the path to ADLS gen2 storage. The following link provides numerous functions that can be used according to your requirement.
https://www.geeksforgeeks.org/reading-files-in-r-programming/
Sharing the solution to my question:
I've used SparkR to create an empty data frame and save the same in the adls after setting up the spark context.
Solution is below:
Step 1: Create a schema
fcstSchema <- structType(structField("ABC", "string",TRUE))
new_df<- data.frame(ABC=NULL,stringsAsFactors=FALSE)
n<-createDataFrame(new_df,fcstSchema)
SparkR::saveDF(n,path = "abfss://<account_name>#<datalake>.dfs.core.windows.net/<path>/",source = "delta",mode = "overwrite")
I have my final output in R dataframe. I need to write this output to a database in Azure Databricks. Can someone help me with the syntax? I used this code:
require(SparkR)
data1 <- createDataFrame(output)
write.df(data1, path="dbfs:/datainput/sample_dataset.parquet",
source="parquet", mode="overwrite")
This code runs without error, but i dont see the database in the datainput folder (mentioned in the path). Is there some other way to do it?
I believe you are looking for saveAsTable function. write.df is particularly to save the data in a file system only, not to tag the data as table.
require(SparkR)
data1 <- createDataFrame(output)
saveAsTable(data1, tableName = "default.sample_table", source="parquet", mode="overwrite")
In the above code, default is some existing database name, under which a new table will get created having name as sample_table. If you mention sample_table instead of default.sample_table then it will be saved in the default database.
I am trying to import data from an excel sheet to an Oracle table. I am able to extract the correct data, but when I run the following code:
$bulkCopy = new-object ("Oracle.DataAccess.Client.OracleBulkCopy") $oraClientConnString
$bulkCopy.DestinationTableName = $entity
$bulkCopy.BatchSize = 5000
$bulkCopy.BulkCopyTimeout = 10000
$bulkCopy.WriteToServer($dt)
$bulkcopy.close()
$bulkcopy.Dispose()
The data inserted in the table is some garbage values, consisting of 0's and 10's.
Value received from excel is stored in a data table ($dt).
Any help will be highly appreciated.
Please check the data type of the values in your data table. I have experienced this issue in .Net with the data type Double. When I change my data type to Decimal, everything was fine.
I used Rblpapi to retrieve historic data, format is a list of data.frames. I now want to write that list of data into a MySQL database, I tried
dbWriteTable(con,name="db_all",data,header=T), which gives me
"unable to find an inherited method for function ‘dbWriteTable’ for
signature ‘"MySQLConnection", "character", "list"’"
I have the feeling the problem is the list of data.frames, is there any easy way to insert this data from R into MySQL?
Did you try:
df <- do.call("rbind", data)
dbWriteTable(con, name="db_all", df, header=T)