Write to text file but escape special characters - r

I'm trying to use R to write some perl script to a text file. I just cannot figure out how to escape certain characters?
I've used backslash (single and double), square brackets, "\\Q...\\E", etc. but still can't make it work.
Any assistance would be appreciated. Thanks in advance!
taskFilename = "example.txt"
cat("
#!/usr/bin/perl
use strict;
use warnings;
use File::Find;
use File::Temp qw(tempfile);
my #imagedir_roots = ("/Users/Ross/Desktop/images");
my $parallel = 8;
my $exiftool_command = 'exiftool -all= -tagsfromfile # -all:all --gps:all --xmp:geotag -unsafe -icc_profile -overwrite_original';
# Create the (temporary) -# files
my #atfiles;
my #atfilenames;
for (my $i = 0; $i < $parallel; ++$i) {
my ($fh, $filename) = tempfile(UNLINK => 1);
push #atfiles, $fh;
push #atfilenames, $filename;
}
# Gather all JPG image files and distribute them over the -# files
my $nr = 0;
find(sub { print { $atfiles[$nr++ % $parallel] } "$File::Find::name\n" if (-f && /\.(?:jpg|jpeg)/i); }, #imagedir_roots);
# Process all images in parallel
printf("Processing %d JPG files...\n", $nr);
for (my $i = 0; $i < $parallel; ++$i) {
close($atfiles[$i]);
my $pid = fork();
if (!$pid) {
# Run exiftool in the background
system qq{$exiftool_command -# \"$atfilenames[$i]\"};
last;
}
}
# Wait for processes to finish
while (wait() != -1) {}
", fill = TRUE, file = taskFilename
)

I also played around with this once. If I remember correctly:
the double quotes you need to escape with \
if you want to write a \ you also need to escape it with \
if you want to write a \" you need to \\"
taskFilename = "example.txt"
cat("
#!/usr/bin/perl
use strict;
use warnings;
use File::Find;
use File::Temp qw(tempfile);
my #imagedir_roots = (\"/Users/Ross/Desktop/images\");
my $parallel = 8;
my $exiftool_command = 'exiftool -all= -tagsfromfile # -all:all --gps:all --xmp:geotag -unsafe -icc_profile -overwrite_original';
# Create the (temporary) -# files
my #atfiles;
my #atfilenames;
for (my $i = 0; $i < $parallel; ++$i) {
my ($fh, $filename) = tempfile(UNLINK => 1);
push #atfiles, $fh;
push #atfilenames, $filename;
}
# Gather all JPG image files and distribute them over the -# files
my $nr = 0;
find(sub { print { $atfiles[$nr++ % $parallel] } \"$File::Find::name\n\" if (-f && /\\.(?:jpg|jpeg)/i); }, #imagedir_roots);
# Process all images in parallel
printf(\"Processing %d JPG files...\n\", $nr);
for (my $i = 0; $i < $parallel; ++$i) {
close($atfiles[$i]);
my $pid = fork();
if (!$pid) {
# Run exiftool in the background
system qq{$exiftool_command -# \\\"$atfilenames[$i]\\\"};
last;
}
}
# Wait for processes to finish
while (wait() != -1) {}
", fill = TRUE, file = taskFilename
)

Related

How SoundCloud stream their mp3 files?

Im trying to implement same method to play .mp3 files with JwPlayer on my webiste like SoundCloud has.
I have a large mp3 files which sometime have 4 hours length. So I don't want to load it as one source file in JwPlayer.
I see soundcloud stream thier mp3 files in partial (160k pices). How they do that?
I try configure my nginx to
location /uploads/ {
mp4;
mp4_buffer_size 160k;
mp4_max_buffer_size 200k;
}
But it does not working. Mp3 still load in a one big request.
I try option 2 read file manualy by php script like
if (isset($_SERVER['HTTP_RANGE'])) {
$c_start = $start;
$c_end = $end;
list(, $range) = explode('=', $_SERVER['HTTP_RANGE'], 2);
if (strpos($range, ',') !== false) {
header('HTTP/1.1 416 Requested Range Not Satisfiable');
header("Content-Range: bytes $start-$end/$size");
exit;
}
if ($range == '0-') {
$c_start = 0;
}
else {
$range = explode('-', $range);
$c_start = (int)$range[0];
if($c_start > 2407200) {
exit;
}
$c_end = (isset($range[1]) && is_numeric($range[1])) ? $range[1] : $c_start + $end;
}
$c_end = ($c_end > $size) ? $size : $c_end;
if ($c_start > $c_end || $c_start > $size - 1 || $c_end >= $size) {
header('HTTP/1.1 416 Requested Range Not Satisfiable');
header("Content-Range: bytes $start-$end/$size");
exit;
}
$start = $c_start;
$end = $c_end;
$length = $end - $start + 1; // Calculate new content length
fseek($fp, $start);
header('HTTP/1.1 206 Partial Content');
}
header("Content-Range: bytes $start-$end/$size");
header("Content-Length: $length");
$buffer = 1024 * 8;
while(!feof($fp) && ($p = ftell($fp)) <= $end) {
if ($p + $buffer > $end) {
$buffer = $end - $p + 1;
}
set_time_limit(0); // Reset time limit for big files
echo fread($fp, $buffer);
flush(); // Free up memory. Otherwise large files will trigger PHP's memory limit.
}
fclose($fp);
It's not bad, but when I STOP playing mp3 the script load other pieces of file.
So how SoundCloud do this? When play music they load music in small pieces and when stop player the request stop too.

Creating multiple zip files with PHP

I have a problem in PHP / Laravel to create multiple zip files synchronously, I copy all commands that is generated and squeeze into the Shell, it executes normally, but when I step into the PHP run it only generates the first file = /.
Controller code.
foreach ($passwords as $p){
if($i == 0){
$command = 'zip -u -j -P '.$p.' '.$dir.'/'.$count.'.zip '.storage_path().'/app/'.$directory.'/'.$file1->getClientOriginalName();
$commands->push($command);
}else{
$command = 'zip --quiet -j -P '.$p.' '.$dir.'/'.$count.'.zip '.storage_path().'/app/'.$directory.'/'.($count+1).'.zip';
$commands->push($command);
}
$count--;
$i++;
}
foreach ($commands as $p){
echo $p.'<br/>';
}
foreach ($commands as $c){
$process = new Process($c);
$process->start();
sleep(10);
if($process->isTerminated()){
sleep(1);
}
if ($errorOutput = $process->getErrorOutput()) {
throw new RuntimeException('Process: ' . $errorOutput);
}
}
Data $commands
The script only generates the file 50.zip.
Not sure if sleep could interfere with subprocess (shell command). Please try:
foreach ($commands as $c){
$process = new Process($c);
// Set the timeout instead of sleeping
$process->setTimeout(10);
$process->start();
// Wait for the process to finish
$process->wait();
if ($errorOutput = $process->getErrorOutput()) {
throw new RuntimeException('Process: ' . $errorOutput);
}
}
wait() call uses usleep in a more fine-grained manner it might help with that.
Does it work like this?

Downloading files from the Internet using Powershell with progress

I have been working on a powershell script that uses a .txt file to download multiple files from tinyurls. I have been successful in using Jobs to make this happen simultaneously, thanks to those on this forum.
The project requires some pretty large files to be downloaded, and using the current method has no progress indicator. I figured some users might think the program died. Looking for a way give a status of where it is in the download. Here is what I came up with, but I'm lost in how to pipe this information back out to the console. Any suggestions?
#Checks to see if NT-Download folder is on the Desktop, if not found, creates it
$DOCDIR = [Environment]::GetFolderPath("Desktop")
$TARGETDIR = "$DOCDIR\NT-Download"
if(!(Test-Path -Path $TARGETDIR )){
New-Item -ItemType directory -Path $TARGETDIR
}
$filepaths = Resolve-Path "files.txt"
Get-Content "$filepaths" | Foreach {
Start-Job {
function Save-TinyUrlFile
{
PARAM (
$TinyUrl,
$DestinationFolder
)
$response = Invoke-WebRequest -Uri $TinyUrl
$filename = [System.IO.Path]::GetFileName($response.BaseResponse.ResponseUri.OriginalString)
$filepath = [System.IO.Path]::Combine($DestinationFolder, $filename)
$totalLength = [System.Math]::Floor($response.get_ContentLength()/1024)
$responseStream = $response.GetResponseStream()
$buffer = new-object byte[] 10KB
$count = $responseStream.Read($buffer,0,$buffer.length)
$downloadedBytes = $count
try
{
$filestream = [System.IO.File]::Create($filepath)
$response.RawContentStream.WriteTo($filestream)
$filestream.Close()
while ($count -gt 0)
{
[System.Console]::CursorLeft = 0
[System.Console]::Write("Downloaded {0}K of {1}K", [System.Math]::Floor($downloadedBytes/1024), $totalLength)
$targetStream.Write($buffer, 0, $count)
$count = $responseStream.Read($buffer,0,$buffer.length)
$downloadedBytes = $downloadedBytes + $count
}
"`nFinished Download"
$targetStream.Flush()
$targetStream.Close()
$targetStream.Dispose()
$responseStream.Dispose()
}
finally
{
if ($filestream)
{
$filestream.Dispose();
}
}
}
Save-TinyUrlFile -TinyUrl $args[0] -DestinationFolder $args[1]
} -ArgumentList $_, "$TARGETDIR"
}
Have a look at Write-Progress
PS C:> for ($i = 1; $i -le 100; $i++ )
{write-progress -activity "Search in Progress" -status "$i% Complete:" -percentcomplete $i;}
Far more simple way : rely on Bits:
Start-BitsTransfer -Source $tinyUrl

Is there any faster way to truncate column in Unix

I want to truncate 4th column of TSV file to given length in Unix. File has records in few millions and is of size 8GB.
I am trying this but it seems to be kind of slow.
awk -F"\t" '{s=substr($4,0,256); print $1"\t"$2"\t"$3"\t"s"\t"$5"\t"$6"\t"$7}' file > newFile
Is there any faster alternatives for same?
Thanks
Your command could be written a little more nicely (assuming you are re-building the record), which may give some performance increases:
awk 'BEGIN { FS=OFS="\t" } { $4 = substr($4,0,256) }' file > newFile
If you have access to a multi-core machine (which you probably do), you can use GNU parallel. You may want to vary the number of cores you use (I've set 4 here) and the block size that's fed to awk (I've set this to two megabytes)...
< file parallel -j 4 --pipe --block 2M -q awk 'BEGIN { FS=OFS="\t" } { $4 = substr($4,0,2) }' > newFile
Here's some testing I did on my system using a 2.7G file with 100 million lines and a block size of 2M:
time awk 'BEGIN { FS=OFS="\t" } { $4 = substr($4,0,2) }' file >/dev/null
Results:
real 1m59.313s
user 1m57.120s
sys 0m2.190s
With one core:
time < file parallel -j 1 --pipe --block 2M -q awk 'BEGIN { FS=OFS="\t" } { $4 = substr($4,0,2) }' >/dev/null
Results:
real 2m28.270s
user 4m3.070s
sys 0m41.560s
With four cores:
time < file parallel -j 4 --pipe --block 2M -q awk 'BEGIN { FS=OFS="\t" } { $4 = substr($4,0,2) }' >/dev/null
Results:
real 0m54.329s
user 2m41.550s
sys 0m31.460s
With twelve cores:
time < file parallel -j 12 --pipe --block 2M -q awk 'BEGIN { FS=OFS="\t" } { $4 = substr($4,0,2) }' >/dev/null
Results:
real 0m36.581s
user 2m24.370s
sys 0m32.230s
I’ll assume that your file has exactly one space character between fields and no whitespace at the beginning of the line.  If that is wrong, this can be enhanced. 
Otherwise, this should work:
sed 's/^\([^ ]* [^ ]* [^ ]* [^ ]\{1,256\}\)[^ ]* /\1 /'
I haven’t actually tested it with 256-character-long data (I tested it with \{1,2\} and I have no idea how its speed compares to that of awk.  BTW, on some versions, you might need to leave off the backslashes from the curly braces and use just {1,256}.
If Scott or Steve's solutions are still too slow, it may be time to break out the C. Run as ./a.out < file > newFile. Test on a small file with some long fields first; I am not 100% sure I have the math right.
#include <stdio.h>
int
main(void)
{
int field = 1;
int character = 0;
int c;
while ((c = getchar()) != EOF)
{
switch (c)
{
case '\n':
field = 1;
character = 0;
break;
case '\t':
character = 0;
field++;
break;
default:
character++;
break;
}
if (field != 4 || character < 256)
putchar(c);
}
if (ferror(stdout) || fflush(stdout) || fclose(stdout))
{
perror("write");
return 1;
}
return 0;
}

How can I perform HTTP PUT uploads to a VMware ESX Server in PowerShell?

VMware ESX, ESXi, and VirtualCenter are supposed to be able to support HTTP PUT uploads since version 3.5. I know how to do downloads, that's easy. I've never done PUT before.
Background information on the topic is here: http://communities.vmware.com/thread/117504
You should have a look at the Send-PoshCode function in the PoshCode cmdlets script module ... it uses a POST, not a PUT, but the technique is practically identical. I don't have PUT server I can think of to test against, but basically, set your $url and your $data, and do something like:
param($url,$data,$filename,[switch]$quiet)
$request = [System.Net.WebRequest]::Create($url)
$data = [Text.Encoding]::UTF8.GetBytes( $data )
## Be careful to set your content type appropriately...
## This is what you're going to SEND THEM
$request.ContentType = 'text/xml;charset="utf-8"' # "application/json"; # "application/x-www-form-urlencoded";
## This is what you expect back
$request.Accept = "text/xml" # "application/json";
$request.ContentLength = $data.Length
$request.Method = "PUT"
## If you need Credentials ...
# $request.Credentials = (Get-Credential).GetNetworkCredential()
$put = new-object IO.StreamWriter $request.GetRequestStream()
$put.Write($data,0,$data.Length)
$put.Flush()
$put.Close()
## This is the "simple" way ...
# $reader = new-object IO.StreamReader $request.GetResponse().GetResponseStream() ##,[Text.Encoding]::UTF8
# write-output $reader.ReadToEnd()
# $reader.Close()
## But there's code in PoshCode.psm1 for doing a progress bar, something like ....
$res = $request.GetResponse();
if($res.StatusCode -eq 200) {
[int]$goal = $res.ContentLength
$reader = $res.GetResponseStream()
if($fileName) {
$writer = new-object System.IO.FileStream $fileName, "Create"
}
[byte[]]$buffer = new-object byte[] 4096
[int]$total = [int]$count = 0
do
{
$count = $reader.Read($buffer, 0, $buffer.Length);
if($fileName) {
$writer.Write($buffer, 0, $count);
} else {
$output += $encoding.GetString($buffer,0,$count)
}
if(!$quiet) {
$total += $count
if($goal -gt 0) {
Write-Progress "Downloading $url" "Saving $total of $goal" -id 0 -percentComplete (($total/$goal)*100)
} else {
Write-Progress "Downloading $url" "Saving $total bytes..." -id 0
}
}
} while ($count -gt 0)
$reader.Close()
if($fileName) {
$writer.Flush()
$writer.Close()
} else {
$output
}
}
$res.Close();
In the VI Toolkit Extensions use Copy-TkeDatastoreFile. It will work with binaries.

Resources