Repeated data.table fread and fwrite causes "Permission denied" error - r

Question

I encountered this issue using the data.table fwrite() and fread() functions for managing resources in a parallel calculation, but was also able to recreate the behavior in the below sequential example code. Calling fwrite() throws the following error:
Error in fwrite(dt, csv_path) : Permission denied: 'D:/mypath/test.csv'. Failed to open
existing file for writing. Do you have write permission to it? Is this
Windows and does another process such as Excel have it open?
The behavior seems to be related to the calling of fread() right before, as commenting out the fread() command makes the error disappear. Depending on your system, you might have to increase the number of iterations before the error occurs as it occurs at varying iteration numbers.
Does anyone have an idea why this is happening? Thanks in advance for your assistance!
Example code:
library(data.table)
dt = data.table(a = c(1, 2), b = c("a", "b"))
csv_path = "D:/mypath/test.csv"
fwrite(dt, csv_path)
for(i in 1:10000){
test = fread(csv_path)
fwrite(dt, csv_path)
}
System info
R version 4.0.0 (2020-04-24)
Platform: x86_64-w64-mingw32/x64 (64-bit)
Running under: Windows Server x64 (build 14393)
data.table_1.12.8

Answer 1

I tried your code on a Windows machine and I was not able reproduce it.
I believe the issue is related to Windows file handler, which seems to be not fast enought to close file connection before opening it again.
You can try following code to see if it is reproducible just in R:
x = "a,b\n1,a\n2,b\n"
csv_path = "D:/mypath/test.csv"
file.create(csv_path)
f = file(csv_path, "w")
cat(x, file=f)
close(f)
for (i in 1:10000) {
f = file(csv_path, "r")
test = readLines(f)
close(f)
f = file(csv_path, "w")
cat(x, file=f)
close(f)
}
What could also make sense is it see how much Sys.sleep is enough to make the problem disappear.

Answer 2

Determine the number of threads you're using for data tables with
data.table::getDTthreads()
I was receiving the same fread() error until I reduced this from 96 to 24 with
data.table::setDTthreads(threads = 24)
Other users have reported threads < 79 works. See .data.table crashes with segfault while grouping with more than 79 threads #5077.