I have the following code:
myTable[i,] = strsplit(line, split=";")[[1]]
write.csv(myTable[-1,], file="episodes_cleared.csv", sep=";", row.names=FALSE, quote=FALSE)
Unfortunately, the separator still is ',':
iEpisodeId,iPatientId,sTitle,sICPc,dStart,dEnd,bProblem
Running the code gives me:
Warning messages:
1: In write.csv(myTable[-1, ], file = "episodes_cleared.csv", sep = ";", : attempt to set 'sep' ignored
2: In write.csv(myTable[-1, ], file = "episodes_cleared.csv", sep = ";", :
attempt to set 'sep' ignored
What am I doing wrong?
First, you should provide a reproducible example.
However, if you use write.csv2 it defaults to using a semicolon as the separator.
The documentation https://stat.ethz.ch/R-manual/R-devel/library/utils/html/write.table.html says:
write.csv uses "." for the decimal point and a comma for the separator.
write.csv2 uses a comma for the decimal point and a semicolon for the separator, the Excel convention for CSV files in some Western European locales.
These wrappers are deliberately inflexible: they are designed to ensure that the correct conventions are used to write a valid file. Attempts to change append, col.names, sep, dec or qmethod are ignored, with a warning.
So it not only defaults to this seperator, it enforces it. Use
write.table(data,file="data.csv",sep=",",dec = " ")
You can also use fwrite, fast CSV writer, with option sep = ";".
Related
I have a tab-delimited file that I am trying to read into R. String variables are denoted with double quotes, however, there are additional double quotes inside some of the strings themselves:
There are also coordinates in some of the string variables that produce the same issue of quotes inside the string:
When trying to read into R, it provide the following error:
"Error in scan(file = file, what = what, sep = sep, quote = quote, dec = dec, : line 237 did not have 55 elements."
How can I read these files into R without having it split up the string? Alternatively, is there a way to save the source data differently so that this issue does not occur when reading into R?
I've been trying to read in a pipe delimited csv file containing 96 variables about some volunteer water quality data. Randomly within the file, there's single and double quotation marks as well as semi-colons, dashes, slashes, and likely other special characters
Name: Jonathan "Joe" Smith; Jerry; Emily; etc.
From the output of several variables (such as IsNewVolunteer), it seems that r is having issues reading in the data. IsNewVolunteer should always be Y or N, but numbers are appearing and when I queried those lines it appears that the data is getting shifted. Variables that are clearly not names are in the Firstname and lastname column.
The original data format makes it a little difficult to see and troubleshoot, especially due to extra variables. I would find a way to remove them, but the goal of the work with R is to provide code that will be able to run on a dataset that is frequently updated.
I've tried
read.table("dnrvisualstream.csv",sep="|",stringsAsFactors = FALSE,quote="")
But that produces the following error:
Error in scan(file = file, what = what, sep = sep, quote = quote, dec = dec, :
line 132 did not have 94 elements
However, there's nothing out of the ordinary that I've noticed about line 132. I've had more success with
read.csv("dnrvisualstream.csv",sep="|",stringsAsFactors = FALSE,quote="")
but that still produces offsets and errors as discussed above. Is there something I'm doing incorrectly? Any information would be helpful.
I think it's one of two issues:
Encoding is either UTF-8 or UTF-16:
Try this...
read.csv("dnrvisualstream.csv", sep = "|", stringsAsFactors = FALSE, quote = "", encoding = UTF-8)
or this...
read.csv("dnrvisualstream.csv", sep = "|", stringsAsFactors = FALSE, quote = "", encoding = UTF-16)
Too many separators:
If this doesn't work, right-click on your .csv file and open it in a text editor. You have multiple |||| separators in rows 2,3,4, and 21,22 that are visible in your screenshot. Press CTRL+H to find and replace:
Find: ||||
Replace: |
Save the new file and try to open in R again.
I am using fread function for importing ".dat" file [file size 3.5 GB]. The issue with the file is some fields have embedded separator[I got to know as the same file is being used for loading via SSIS ETL tool] in it.
data <- fread("xyz.dat", sep = '|', encoding = "UTF-8",showProgress = T, select = ord_shp_col, fill = TRUE, sep2 = "|")
Tried sep2 argument to handle as per the R document and even tried with only limited column, so that such columns could be skipped.
However, ending with same error again n again.
Read 0.0% of 1712440 rowsError in fread("xyz.dat", sep = "|", encoding
= "UTF-8", : Expecting 118 cols, but line 2143 contains text after processing all cols. Try again with fill=TRUE. Another reason could be
that fread's logic in distinguishing one or more fields having
embedded sep='|' and/or (unescaped) '\n' characters within unbalanced
unescaped quotes has failed. If quote='' doesn't help, please file an
issue to figure out if the logic could be improved.
Any help is highly appreciated.
I am working with importing a series of text files from many zipped folders.
Before, I was unzipping the files with WinRAR, then applying read.table as:
main <- read.table(file.path(dir, paste0("Current/17/Folder/Main.txt")),
sep = '|',
header = FALSE,
stringsAsFactors = FALSE,
skipNul = TRUE,
comment.char="",
quote = ""
)
Which worked, but required unzipping myself (which takes up a prohibitive amount of space). I stumbled on unz, which would allow me to avoid this step as follows:
main <- read.table(unz(file.path(dir, paste0("Current/17.zip")), "Folder/Main.txt"),
sep = '|',
header = FALSE,
stringsAsFactors = FALSE,
skipNul = TRUE,
comment.char="",
quote = ""
)
The latter works in most cases, but for some files, the read.table throws an error
number of items read is not a multiple of the number of columns
I discerned that this is from some odd characters in the file. Comparing the two approaches in one case, I find that the manual unzip and read approach converts these special characters to "??" (with a white background, suggesting a foreign character) (which is desirable, if it cannot read them), whereas the second approach simply throws an error when it hits them (the ?? are not reported in the output), and does not read the rest of the current line or file. I looked at the line in question in Bash with:
sed -n '10905951 p' Main.txt
And it also reports the same "??" that show up in the read.table in R.
I have tried using fileEncoding, with a few different file formats, to no avail. Many other approaches I have seen for dealing with odd characters like this require manually replacing the characters, but I am struggling to even find out what the characters are, and why WinRAR's unzip converts them to something readable while unz does not. I have also considered other read options, but read.table is preferable to me because it allows for the skipNul option, which is relevant for my data.
Any help or ideas would be appreciated.
I have seen in several cases that while read.table() is not able to read a tab delimited file (for example the annotation table of a microarray) returning the following error:
Error in scan(file, what, nmax, sep, dec, quote, skip, nlines, na.strings, :
line xxx did not have yyy elements
read.csv() works perfectly on the same file with no errors. I think also the speed of read.csv() is also higher than read.table().
Even more: read.table() is doing very crazy reading a file of me. It makes this error while reading line 100, but when I copy and paste lines 90 to 110 just after the head of the same file, it still makes error of line 100+21 (new lines copied at the beginning). If there is any problem with that line, why doesn't it report that error while reading the pasted line at the beginning? I confirm that read.csv() reads the same file with no error.
Do you have any idea of why read.table() is unable to read the same files that read.csv() works on it? Also is there any reason to use read.table() in any cases?
read.csv is a fairly thin wrapper around read.table; I would be quite surprised if you couldn't exactly replicate the behaviour of read.csv by supplying the correct arguments to read.table. However, some of those arguments (such as the way that quotation marks or comment characters are handled) could well change the speed and behaviour of the function.
In particular, this is the full definition of read.csv:
function (file, header = TRUE, sep = ",", quote = "\"", dec = ".",
fill = TRUE, comment.char = "", ...) {
read.table(file = file, header = header, sep = sep, quote = quote,
dec = dec, fill = fill, comment.char = comment.char, ...)
}
so as stated it's just read.table with a particular set of options.
As #Chase states in the comments below, the help page for read.table() says just as much under Details:
read.csv and read.csv2 are identical to read.table except for the defaults. They are intended for reading ‘comma separated value’ files (‘.csv’) or (read.csv2) the variant used in countries that use a comma as decimal point and a semicolon as field separator.
Don't use read.table to read tab-delimited files, use read.delim. (It is just a thin wrapper around read.table but it sets the options to appropriate values)
read_table() does fail sometime on tab sep'ed file and setting sep='\s+' may help assuming item in your table have no space