Downloading files from the Internet using Powershell with progress - http

I have been working on a powershell script that uses a .txt file to download multiple files from tinyurls. I have been successful in using Jobs to make this happen simultaneously, thanks to those on this forum.
The project requires some pretty large files to be downloaded, and using the current method has no progress indicator. I figured some users might think the program died. Looking for a way give a status of where it is in the download. Here is what I came up with, but I'm lost in how to pipe this information back out to the console. Any suggestions?
#Checks to see if NT-Download folder is on the Desktop, if not found, creates it
$DOCDIR = [Environment]::GetFolderPath("Desktop")
$TARGETDIR = "$DOCDIR\NT-Download"
if(!(Test-Path -Path $TARGETDIR )){
New-Item -ItemType directory -Path $TARGETDIR
}
$filepaths = Resolve-Path "files.txt"
Get-Content "$filepaths" | Foreach {
Start-Job {
function Save-TinyUrlFile
{
PARAM (
$TinyUrl,
$DestinationFolder
)
$response = Invoke-WebRequest -Uri $TinyUrl
$filename = [System.IO.Path]::GetFileName($response.BaseResponse.ResponseUri.OriginalString)
$filepath = [System.IO.Path]::Combine($DestinationFolder, $filename)
$totalLength = [System.Math]::Floor($response.get_ContentLength()/1024)
$responseStream = $response.GetResponseStream()
$buffer = new-object byte[] 10KB
$count = $responseStream.Read($buffer,0,$buffer.length)
$downloadedBytes = $count
try
{
$filestream = [System.IO.File]::Create($filepath)
$response.RawContentStream.WriteTo($filestream)
$filestream.Close()
while ($count -gt 0)
{
[System.Console]::CursorLeft = 0
[System.Console]::Write("Downloaded {0}K of {1}K", [System.Math]::Floor($downloadedBytes/1024), $totalLength)
$targetStream.Write($buffer, 0, $count)
$count = $responseStream.Read($buffer,0,$buffer.length)
$downloadedBytes = $downloadedBytes + $count
}
"`nFinished Download"
$targetStream.Flush()
$targetStream.Close()
$targetStream.Dispose()
$responseStream.Dispose()
}
finally
{
if ($filestream)
{
$filestream.Dispose();
}
}
}
Save-TinyUrlFile -TinyUrl $args[0] -DestinationFolder $args[1]
} -ArgumentList $_, "$TARGETDIR"
}

Have a look at Write-Progress
PS C:> for ($i = 1; $i -le 100; $i++ )
{write-progress -activity "Search in Progress" -status "$i% Complete:" -percentcomplete $i;}

Far more simple way : rely on Bits:
Start-BitsTransfer -Source $tinyUrl

Related

How to export folderpermission of all drivers in server

Until now, my company instead of sharing a specific folder, shares all drives (C:, D:, etc)
How do I get permission from all folders in all drives to serve an audit operation?
I tried with the following code, but it is for specific drivers only
$FolderPath = Get-ChildItem -Directory -Path **"S:\"** -Recurse -Force
$Output = #()
ForEach ($Folder in $FolderPath) {
$Acl = Get-Acl -Path $Folder.FullName
ForEach ($Access in $Acl.Access) {
$Properties = [ordered]#{
'Folder Name'=$Folder.FullName;
'Group/User'=$Access.IdentityReference;
'Permissions'=$Access.FileSystemRights;
'Inherited'=$Access.IsInherited
}
$Output += New-Object -TypeName PSObject -Property $Properties
}
}
$Output | Out-GridView

How SoundCloud stream their mp3 files?

Im trying to implement same method to play .mp3 files with JwPlayer on my webiste like SoundCloud has.
I have a large mp3 files which sometime have 4 hours length. So I don't want to load it as one source file in JwPlayer.
I see soundcloud stream thier mp3 files in partial (160k pices). How they do that?
I try configure my nginx to
location /uploads/ {
mp4;
mp4_buffer_size 160k;
mp4_max_buffer_size 200k;
}
But it does not working. Mp3 still load in a one big request.
I try option 2 read file manualy by php script like
if (isset($_SERVER['HTTP_RANGE'])) {
$c_start = $start;
$c_end = $end;
list(, $range) = explode('=', $_SERVER['HTTP_RANGE'], 2);
if (strpos($range, ',') !== false) {
header('HTTP/1.1 416 Requested Range Not Satisfiable');
header("Content-Range: bytes $start-$end/$size");
exit;
}
if ($range == '0-') {
$c_start = 0;
}
else {
$range = explode('-', $range);
$c_start = (int)$range[0];
if($c_start > 2407200) {
exit;
}
$c_end = (isset($range[1]) && is_numeric($range[1])) ? $range[1] : $c_start + $end;
}
$c_end = ($c_end > $size) ? $size : $c_end;
if ($c_start > $c_end || $c_start > $size - 1 || $c_end >= $size) {
header('HTTP/1.1 416 Requested Range Not Satisfiable');
header("Content-Range: bytes $start-$end/$size");
exit;
}
$start = $c_start;
$end = $c_end;
$length = $end - $start + 1; // Calculate new content length
fseek($fp, $start);
header('HTTP/1.1 206 Partial Content');
}
header("Content-Range: bytes $start-$end/$size");
header("Content-Length: $length");
$buffer = 1024 * 8;
while(!feof($fp) && ($p = ftell($fp)) <= $end) {
if ($p + $buffer > $end) {
$buffer = $end - $p + 1;
}
set_time_limit(0); // Reset time limit for big files
echo fread($fp, $buffer);
flush(); // Free up memory. Otherwise large files will trigger PHP's memory limit.
}
fclose($fp);
It's not bad, but when I STOP playing mp3 the script load other pieces of file.
So how SoundCloud do this? When play music they load music in small pieces and when stop player the request stop too.

Creating multiple zip files with PHP

I have a problem in PHP / Laravel to create multiple zip files synchronously, I copy all commands that is generated and squeeze into the Shell, it executes normally, but when I step into the PHP run it only generates the first file = /.
Controller code.
foreach ($passwords as $p){
if($i == 0){
$command = 'zip -u -j -P '.$p.' '.$dir.'/'.$count.'.zip '.storage_path().'/app/'.$directory.'/'.$file1->getClientOriginalName();
$commands->push($command);
}else{
$command = 'zip --quiet -j -P '.$p.' '.$dir.'/'.$count.'.zip '.storage_path().'/app/'.$directory.'/'.($count+1).'.zip';
$commands->push($command);
}
$count--;
$i++;
}
foreach ($commands as $p){
echo $p.'<br/>';
}
foreach ($commands as $c){
$process = new Process($c);
$process->start();
sleep(10);
if($process->isTerminated()){
sleep(1);
}
if ($errorOutput = $process->getErrorOutput()) {
throw new RuntimeException('Process: ' . $errorOutput);
}
}
Data $commands
The script only generates the file 50.zip.
Not sure if sleep could interfere with subprocess (shell command). Please try:
foreach ($commands as $c){
$process = new Process($c);
// Set the timeout instead of sleeping
$process->setTimeout(10);
$process->start();
// Wait for the process to finish
$process->wait();
if ($errorOutput = $process->getErrorOutput()) {
throw new RuntimeException('Process: ' . $errorOutput);
}
}
wait() call uses usleep in a more fine-grained manner it might help with that.
Does it work like this?

Powershell Scheduled Task Open Web Page to Execute Script

In an effort to setup a "cron job" like scheduled task on Windows I've setup a Powershell script using code recommended via previous stackoverflow question.
I have some backups that I need to cleanup daily and delete old backups so I created a asp.net script to perform this task - the file name is BackupCleanup.aspx and I have confirmed that the ASP.net script does work when executed on its own by visiting the above url - I however cannot get it to execute using the Powershell script below.
The Powershell Script code I'm using is:
$request = [System.Net.WebRequest]::Create("http://127.0.0.1/BackupCleanup.aspx")
$response = $request.GetResponse()
$response.Close()
I have created this file with a PS1 extension, it shows properly in my os (Windows 2008) - I have tried both manually executing this task by right clicking and choosing "Run with Powershell" and also have scheduled this as a task - both to no avail.
I cannot figure out why the script does not work - any help would be GREATLY appreciated.
Here is the Powershell script I use to call up web pages using IE. Hopefully this will work for you as well.
Function NavigateTo([string] $url, [int] $delayTime = 100)
{
Write-Verbose "Navigating to $url"
$global:ie.Navigate($url)
WaitForPage $delayTime
}
Function WaitForPage([int] $delayTime = 100)
{
$loaded = $false
while ($loaded -eq $false) {
[System.Threading.Thread]::Sleep($delayTime)
#If the browser is not busy, the page is loaded
if (-not $global:ie.Busy)
{
$loaded = $true
}
}
$global:doc = $global:ie.Document
}
Function SetElementValueByName($name, $value, [int] $position = 0) {
if ($global:doc -eq $null) {
Write-Error "Document is null"
break
}
$elements = #($global:doc.getElementsByName($name))
if ($elements.Count -ne 0) {
$elements[$position].Value = $value
}
else {
Write-Warning "Couldn't find any element with name ""$name"""
}
}
Function ClickElementById($id)
{
$element = $global:doc.getElementById($id)
if ($element -ne $null) {
$element.Click()
WaitForPage
}
else {
Write-Error "Couldn't find element with id ""$id"""
break
}
}
Function ClickElementByName($name, [int] $position = 0)
{
if ($global:doc -eq $null) {
Write-Error "Document is null"
break
}
$elements = #($global:doc.getElementsByName($name))
if ($elements.Count -ne 0) {
$elements[$position].Click()
WaitForPage
}
else {
Write-Error "Couldn't find element with name ""$name"" at position ""$position"""
break
}
}
Function ClickElementByTagName($name, [int] $position = 0)
{
if ($global:doc -eq $null) {
Write-Error "Document is null"
break
}
$elements = #($global:doc.getElementsByTagName($name))
if ($elements.Count -ne 0) {
$elements[$position].Click()
WaitForPage
}
else {
Write-Error "Couldn't find element with tag name ""$name"" at position ""$position"""
break
}
}
#Entry point
# Setup references to IE
$global:ie = New-Object -com "InternetExplorer.Application"
$global:ie.Navigate("about:blank")
$global:ie.visible = $true
# Call the page
NavigateTo "http://127.0.0.1/BackupCleanup.aspx"
# Release resources
$global:ie.Quit()
$global:ie = $null
I had the same issue. I manually opened powershell and executed my script and I received "WebPage.ps1 cannot be loaded because running scripts is disabled on this system.".
You have to allow scripts to run
Execute the below in PowerShell
Set-ExecutionPolicy RemoteSigned -Scope LocalMachine

How can I perform HTTP PUT uploads to a VMware ESX Server in PowerShell?

VMware ESX, ESXi, and VirtualCenter are supposed to be able to support HTTP PUT uploads since version 3.5. I know how to do downloads, that's easy. I've never done PUT before.
Background information on the topic is here: http://communities.vmware.com/thread/117504
You should have a look at the Send-PoshCode function in the PoshCode cmdlets script module ... it uses a POST, not a PUT, but the technique is practically identical. I don't have PUT server I can think of to test against, but basically, set your $url and your $data, and do something like:
param($url,$data,$filename,[switch]$quiet)
$request = [System.Net.WebRequest]::Create($url)
$data = [Text.Encoding]::UTF8.GetBytes( $data )
## Be careful to set your content type appropriately...
## This is what you're going to SEND THEM
$request.ContentType = 'text/xml;charset="utf-8"' # "application/json"; # "application/x-www-form-urlencoded";
## This is what you expect back
$request.Accept = "text/xml" # "application/json";
$request.ContentLength = $data.Length
$request.Method = "PUT"
## If you need Credentials ...
# $request.Credentials = (Get-Credential).GetNetworkCredential()
$put = new-object IO.StreamWriter $request.GetRequestStream()
$put.Write($data,0,$data.Length)
$put.Flush()
$put.Close()
## This is the "simple" way ...
# $reader = new-object IO.StreamReader $request.GetResponse().GetResponseStream() ##,[Text.Encoding]::UTF8
# write-output $reader.ReadToEnd()
# $reader.Close()
## But there's code in PoshCode.psm1 for doing a progress bar, something like ....
$res = $request.GetResponse();
if($res.StatusCode -eq 200) {
[int]$goal = $res.ContentLength
$reader = $res.GetResponseStream()
if($fileName) {
$writer = new-object System.IO.FileStream $fileName, "Create"
}
[byte[]]$buffer = new-object byte[] 4096
[int]$total = [int]$count = 0
do
{
$count = $reader.Read($buffer, 0, $buffer.Length);
if($fileName) {
$writer.Write($buffer, 0, $count);
} else {
$output += $encoding.GetString($buffer,0,$count)
}
if(!$quiet) {
$total += $count
if($goal -gt 0) {
Write-Progress "Downloading $url" "Saving $total of $goal" -id 0 -percentComplete (($total/$goal)*100)
} else {
Write-Progress "Downloading $url" "Saving $total bytes..." -id 0
}
}
} while ($count -gt 0)
$reader.Close()
if($fileName) {
$writer.Flush()
$writer.Close()
} else {
$output
}
}
$res.Close();
In the VI Toolkit Extensions use Copy-TkeDatastoreFile. It will work with binaries.

Resources