I have two data sets:
str(a)
'data.frame': 525930 obs. of 3 variables:
$ reg_code : int 11542359 10077860 10050401 10988998 11465162 10933454 11170863 11291673 12086780 10248250 ...
$ begin_date: Date, format: "2008-10-01" "1994-06-01" ...
$ pair_id : chr "115423591" "100778601" "100504011" "109889981" ...
str(b)
'data.frame': 618655 obs. of 3 variables:
$ reg_code: int 10077860 10050401 10988998 11465162 10933454 11170863 11291673 10248250 10998100 10837319 ...
$ end_date: Date, format: "2006-03-09" "2000-11-16" ...
$ pair_id : chr "100778601" "100504011" "109889981" "114651621" ...
merge:
abc<-merge(x=df1,y=df2,by="id")
but it is throwing an error:
Error in data.frame(..., check.names = FALSE) :
arguments imply
differing number of rows:15930, 28655, 1
This might seem silly, but just to confirm, are you trying to merge based on "pair_id"? It looks like you're using "id" for the by argument.
If you're simply trying to add one to the other and they have the same columns, you can use rbind().
Related
I have ASCII files with data separated by $ signs.
There are 23 columns in the data, the first row is of column names, but there is inconsistency between the line endings, which causes R to import the data improperly, by shift the data left-wise with respect to their columns.
Header line:
ISR$CASE$I_F_COD$FOLL_SEQ$IMAGE$EVENT_DT$MFR_DT$FDA_DT$REPT_COD$MFR_NUM$MFR_SNDR$AGE$AGE_COD$GNDR_COD$E_SUB$WT$WT_COD$REPT_DT$OCCP_COD$DEATH_DT$TO_MFR$CONFID$REPORTER_COUNTRY
which does not end with a $ sign.
First row line:
7215577$8135839$I$$7215577-0$20101011$$20110104$DIR$$$67$YR$F$N$220$LBS$20110102$CN$$N$Y$UNITED STATES$
Which does end with a $ sign.
My import command:
read.table(filename, header=TRUE, sep="$", comment.char="", header=TRUE, quote="")
My guess is that the inconsistency between the line endings causes R to think that the records have one column more than the header, thus making the first column as a row.names column, which is not correct. Adding the specification row.names=NULL does not fix the issue.
If I manually add a $ sign in the file the problem is solved, but this is infeasible as the issue occurs in hundreds of files. Is there a way to specify how to read the header line? Do I have any alternative?
Additional info: the headers change across different files, so I cannot set my own vector of column names
Create a dummy test file:
cat("ISR$CASE$I_F_COD$FOLL_SEQ$IMAGE$EVENT_DT$MFR_DT$FDA_DT$REPT_COD$MFR_NUM$MFR_SNDR$AGE$AGE_COD$GNDR_COD$E_SUB$WT$WT_COD$REPT_DT$OCCP_COD$DEATH_DT$TO_MFR$CONFID$REPORTER_COUNTRY\n7215577$8135839$I$$7215577-0$20101011$$20110104$DIR$$$67$YR$F$N$220$LBS$20110102$CN$$N$Y$UNITED STATES$",
file="deleteme.txt",
"\n")
Solution using gsub:
First read the file as text and then edit its content:
file_path <- "deleteme.txt"
fh <- file(file_path)
file_content <- readLines(fh)
close(fh)
Either add a $ at the end of header row:
file_content[1] <- paste0(file_content, "$")
Or remove $ from the end of all rows:
file_content <- gsub("\\$$", "", file_content)
Then we write the fixed file back to disk:
cat(paste0(file_content, collapse="\n"), file=paste0("fixed_", file_path), "\n")
Now we can read the file:
df <- read.table(paste0("fixed_", file_path), header=TRUE, sep="$", comment.char="", quote="", stringsAsFactors=FALSE)
And get the desired structure:
str(df)
'data.frame': 1 obs. of 23 variables:
$ ISR : int 7215577
$ CASE : int 8135839
$ I_F_COD : chr "I"
$ FOLL_SEQ : logi NA
$ IMAGE : chr "7215577-0"
$ EVENT_DT : int 20101011
$ MFR_DT : logi NA
$ FDA_DT : int 20110104
$ REPT_COD : chr "DIR"
$ MFR_NUM : logi NA
$ MFR_SNDR : logi NA
$ AGE : int 67
$ AGE_COD : chr "YR"
$ GNDR_COD : logi FALSE
$ E_SUB : chr "N"
$ WT : int 220
$ WT_COD : chr "LBS"
$ REPT_DT : int 20110102
$ OCCP_COD : chr "CN"
$ DEATH_DT : logi NA
$ TO_MFR : chr "N"
$ CONFID : chr "Y"
$ REPORTER_COUNTRY: chr "UNITED STATES "
I am trying to create a table (in Snowflake db) with exactly the same column names as I keep in the R data.frame object:
'data.frame': 1 obs. of 26 variables:
$ Ship_To : chr "0002061948"
$ Del_Coll_Indicator : chr "D"
$ Currency : chr "GBP"
$ Total_Volume : num 0
$ Total_Quantity : num 0
...
There is no problem with the table creation:
dbWriteTable(con = my_db$con, name = "test5", value = df)
but all column names in the database are converted to upper cases:
'data.frame': 1 obs. of 26 variables:
$ SHIP_TO : chr "0002061948"
$ DEL_COLL_INDICATOR : chr "D"
$ CURRENCY : chr "GBP"
...
Is there any way to keep in the table original names from R's data frame?
As covered by Snowflake's SQL reference docs, when identifiers (such as column names) are unquoted at creation, Snowflake will upper case them, and treat them as case-insensitive. Any quoted identifiers will be kept as-is and treated as a case-sensitive identifier.
Alter the data frame column names (colnames(df)) to use a quoted identifier format via the dbQuoteIdentifier(my_db$con, each_column_name) DBI function. This should help preserve the casing.
I have a data file that contains several character variables that only consist of numbers. They need to remain character variables as some of them start with a 0 and when converting to integer/numeric, the leading zeros are cut-off. For some strange reason, when I use fwrite to save my data file as csv and then open it again with fread, the character variables that only consisted of numbers are suddenly integer variables. How can I keep R from doing this?
> str(Dataset_Master)
Classes ‘data.table’ and 'data.frame': 12178669 obs. of 4 variables:
$ Date_of_goods_arrival_at_the_customer: int 20160527 20160527 20160527...
$ Sales_document : chr "0505399186" "0505435949"...
$ Warehouse : chr "8150" "8150" "8150" "8150" ...
$ Sold_to_country : chr "DE" "DE" "DE" "DE" ...
- attr(*, ".internal.selfref")=<externalptr>
> ##Save document
> fwrite(Dataset_Master, "Dataset_Master_3.csv")
> ##Load data
> Dataset_Master <- fread("Dataset_Master_3.csv")
|--------------------------------------------------|
|==================================================|
> str(Dataset_Master)
Classes ‘data.table’ and 'data.frame': 12178669 obs. of 4 variables:
$ Date_of_goods_arrival_at_the_customer: int 20160527 20160527 20160527...
$ Sales_document : int 505399186 505435949 505435949...
$ Warehouse : int 8150 8150 8150 8150 8150 8150...
$ Sold_to_country : chr "DE" "DE" "DE" "DE" ...
- attr(*, ".internal.selfref")=<externalptr>
library(dtplyr)
library(xlsx)
library(lubridate)
'data.frame': 612 obs. of 7 variables:
$ Company : Factor w/ 10 levels "Harbor","HCG",..: 6 10 10 3 6 8 6 8 6 6 ...
$ Title : chr NA NA NA NA ...
$ Send.Offer.Letter :Classes ‘tbl_df’, ‘tbl’ and 'data.frame': 612 obs. of 1 variable:
..$ Send Offer Letter: Date, format: NA NA NA NA ...
..- attr(*, "spec")=List of 2
.. ..$ cols :List of 1
.. .. ..$ Send Offer Letter: list()
.. .. .. ..- attr(*, "class")= chr "collector_character" "collector"
.. ..$ default: list()
.. .. ..- attr(*, "class")= chr "collector_guess" "collector"
.. ..- attr(*, "class")= chr "col_spec"
$ Accepted.Position : chr NA NA NA NA ...
$ Application.Date : chr NA NA NA NA ...
$ Hire.Date..Start. :Classes ‘tbl_df’, ‘tbl’ and 'data.frame': 612 obs. of 1 variable:
..$ Hire Date (Start): POSIXct, format: "2008-05-20" NA NA "2008-05-13" ...
$ Rehire..Yes.or.No.: Factor w/ 23 levels "??","36500","continuing intern",..: NA NA NA NA NA NA NA NA NA NA ...
I have an extremely messy dataset (it was entered entirely freehand on excel spreadsheets) regarding new hires. Variables associated with dates are, of course, making things difficult. There was no consistency in entry format, sometimes random character strings were a part of a date (think 5/17, day tbd) etc. I finally got the dates consistently formatted into POSIXct format, but it led to the odd situation you see above where it appears there are nested variables in my columns. I have already coerced two date variables into as.character ($Accepted.Position and $Application.Date), as I have seen examples of POSIXct date formatting causing issues with write.xlsx.
When I attempt to write to xlsx, I get the following:
write.xlsx(forstack, file = "forstackover.xlsx", col.names = TRUE)
Error in .jcall(cell, "V", "setCellValue", value) :
method setCellValue with signature ([D)V not found
In addition: There were 50 or more warnings (use warnings() to see the first 50)
My dput is too long to post here, so here is the pastebin for it:
Dput forstack
Attempting to coerce $Hire.Date..Start with as.character produces the odd result which I have partially pasted here:
as.character result
I am not sure what action to take here. I found a similar discussion here:
stack question similar to this one
but this user was trying to call a specific portion of a column for ggplot2 graphing. Any help is appreciated.
I had this issue when trying to write a tibble tbl_df to xlsx using the xlsx package.
The error threw when I added the row.names = FALSE option, but no error without row.names call.
I converted the tbl_df to data.frame and it worked.
I agree with #greg dubrow's solution. I have a simpler suggestion as code.
write.xlsx(as.data.frame(forstack), file = "forstackover.xlsx", col.names = TRUE)
You can be more free with file.choose()
write.xlsx(as.data.frame(forstack), file = file.choose(), col.names = TRUE)
By the way, in my code, similar to #Lee's, it gave an error for row.names = FALSE. The error is now resolved. If we expand it a little bit more:
write.xlsx(as.data.frame(forstack), file = file.choose(), col.names = TRUE, row.names=FALSE)
For me the issue was because the data.frame was grouped so I added ungroup(forstack) prior to write.xlsx and that fixed the issue.
It's a trivial issue. But this trick works.
write.xlsx(as.data.frame(forstack), file = "forstackover.xlsx", col.names = TRUE)
This happens because when col.names or row.names are called out, the input file must be a data.frame and not a tibble.
I can't really create a code example because I'm not quite sure what the problem is and my actual problem is rather involved. That said it seems like kind of a generic problem that maybe somebody's seen before.
Basically I'm constructing 3 different dataframes and rbinding them together, which is all as expected smooth sailing but when I try to write that merged frame back to the DB I get this error:
Error in .External2(C_writetable, x, file, nrow(x), p, rnames, sep, eol, :
unimplemented type 'list' in 'EncodeElement'
I've tried manually coercing them using as.data.frame() before and after the rbinds and the returned object (the same one that fails to write with the above error message) exists in the environment as class data.frame so why does dbWriteTable not seem to have got the memo?
Sorry, I'm connecting to a MySQL DB using RMySQL. The problem I think as I look a little closer and try to explain myself is that the columns of my data frame are themselves lists (of the same length), which sorta makes sense of the error. I'd think (or like to think anyways) that a call to as.data.frame() would take care of that but I guess not?
A portion of my str() since it's long looks like:
.. [list output truncated]
$ stcong :List of 29809
..$ : int 3
..$ : int 8
..$ : int 4
..$ : int 2
I guess I'm wondering if there's an easy way to force this coercion?
Hard to say for sure, since you provided so little concrete information, but this would be one way to convert a list column to an atomic vector column:
> d <- data.frame(x = 1:5)
> d$y <- as.list(letters[1:5])
> str(d)
'data.frame': 5 obs. of 2 variables:
$ x: int 1 2 3 4 5
$ y:List of 5
..$ : chr "a"
..$ : chr "b"
..$ : chr "c"
..$ : chr "d"
..$ : chr "e"
> d$y <- unlist(d$y)
> str(d)
'data.frame': 5 obs. of 2 variables:
$ x: int 1 2 3 4 5
$ y: chr "a" "b" "c" "d" ...
This assumes that each element of your list column is only a length one vector. If any aren't, things will be more complicated, and you'd likely need to rethink your data structure anyhow.