Powershell, R, Import-Csv, select-object, Export-csv - r

Question

I'm performing several tests using different approaches for cleaning a big csv file and then importing it into R.
This time I'm playing with Powershell in Windows.
While things work well and most accurate than when using cut() with pipe(), the process is horribly slow.
This is my command:
shell(shell = "powershell",
"Import-Csv In.csv |
select-object col1, col2, etc |
Export-csv new.csv")
And these are the system.time() results:
user system elapsed
0.61 0.42 1568.51
I've seen some other posts that use C# via streaming taking couple of dozens of seconds, but I don't know C#.
My question is, how can improve the PowerShell command in order to make it faster?
Thanks,
Diego

Answer 1

There's a fair amout of overhead in reading in the csv, converting the rows to powershell objects, and the converting back to csv. Doing it through the pipeline that way also causes it to do this one record at a time. You should be able to speed that up considerably if you switch to using Get-Content with a -ReadCount parameter, and extracting your data using a regular expression in a -replace operator, e.g.:
shell(shell = "powershell",
"Get-Content In.csv -ReadCount 1000 |
foreach { $_ -replace '^(.+?,.+?),','$1' |
Add-Content new.csv")
This will reduce the number if disk reads, and the -replace will be functioning as an array operator, doing 1000 records at a time.

Answer 2

First and foremost, my first test was wrong in the sense that due some errors I had before, several other sessions of powershell remained open and delayed the whole process.
These are the real numbers:
> system.time(shell(shell = "powershell", psh.comm))
user system elapsed
0.09 0.05 824.53
Now, as I said I couldn't find a good pattern for splitting the columns of my csv file.
I maybe need to add that it is a messy file, with fields containing commas, multiline fields, summary lines, and so on.
I tried other approaches, like one very famous in stack overflow that uses embedded C# code in PowerShell for splitting csv files.
While it works faster than the more common approach I showed previously, results are not accurate for these types of messy files.
> system.time(shell(shell = "powershell", psh.comm))
user system elapsed
0.01 0.00 212.96
Both approaches showed similar RAM consumption (~40Mb), and CPU usage (~50%) most of the time.
So while the former approach took 4 times the amount of the later, the accuracy of the results, the low cost in terms of resources, and the lesser developing time make me consider it the most efficient for big and messy csv files.