Carica file BIG via HTTP

Question

Sto cercando di caricare immagini VM molto grandi (5-15 Gb size) su un server HTTP usando PowerShell.

ho cercato di utilizzare per che alcuni metodi (qui si collega a script with net.WebClient.UploadFile e script with Invoke-webRequest)

Funziona bene per i file meno di 2 GB, ma non per i file più grandi di questo.

Sto provando a lavorare direttamente con httpWebRequest ma non riesco a inserire FileStream in esso.

Quindi la mia domanda è: come mettere il filestream in webrequest?

O più in generale: come caricare file enorme tramite http con PowerShell?

$Timeout=10000000; 
$fileName = "0.iso"; 
$data = "C:\\$fileName"; 
$url = "http://nexus.lab.local:8081/nexus/content/sites/myproj/$fileName"; 
#$buffer = [System.IO.File]::Open("$data",[System.IO.Filemode]::Open, [System.IO.FileAccess]::Read) #Err Cannot convert argument "buffer", with value: "System.IO.FileStream", for "Write" to type "System.Byte[]": 
#$buffer = gc -en byte $data # too much space in memory 
$buffer = [System.IO.File]::ReadAllBytes($data) #Limit 2gb 
[System.Net.HttpWebRequest] $webRequest = [System.Net.WebRequest]::Create($url) 
$webRequest.Timeout = $timeout 
$webRequest.Method = "POST" 
$webRequest.ContentType = "application/data" 
#$webRequest.ContentLength = $buffer.Length; 
$webRequest.Credentials = New-Object System.Net.NetworkCredential("admin", "admin123"); 

$requestStream = $webRequest.GetRequestStream() 
$requestStream.Write($buffer, 0, $buffer.Length) 
$requestStream.Flush() 
$requestStream.Close() 

[System.Net.HttpWebResponse] $webResponse = $webRequest.GetResponse() 
$streamReader = New-Object System.IO.StreamReader($webResponse.GetResponseStream()) 
$result = $streamReader.ReadToEnd() 
return $result 
$stream.Close() 

Answer 1

Per impostazione predefinita HttpWebRequest è buffering dei dati in memoria. Basta impostare la proprietà HttpWebRequest.AllowWriteStreamBuffering su false e si sarà in grado di caricare file con quasi tutte le dimensioni. Vedi ulteriori dettagli a msdn

Answer 2

Grazie @Stoune, è stata l'ultima cosa che ha contribuito a ricevere finalmente una soluzione funzionante.

Un altro, è necessario organizzare la lettura e la scrittura dei file di flusso nel buffer di richiesta web. Ed è forse a che fare con quel pezzo di codice:

$requestStream = $webRequest.GetRequestStream() 
$fileStream = [System.IO.File]::OpenRead($file) 
$chunk = New-Object byte[] $bufSize 
    while($bytesRead = $fileStream.Read($chunk,0,$bufsize)) 
    { 
    $requestStream.write($chunk, 0, $bytesRead) 
    $requestStream.Flush() 
    } 

e ultimo aspetto script come questo:

$user = "admin" 
$pass = "admin123" 
$dir = "C:\Virtual Hard Disks" 
$fileName = "win2012r2std.vhdx" 
$file = "$dir/$fileName" 
$url = "http://nexus.lab.local:8081/nexus/content/sites/myproj/$fileName" 
$Timeout=10000000 
$bufSize=10000 

$cred = New-Object System.Net.NetworkCredential($user, $pass) 

$webRequest = [System.Net.HttpWebRequest]::Create($url) 
$webRequest.Timeout = $timeout 
$webRequest.Method = "POST" 
$webRequest.ContentType = "application/data" 
$webRequest.AllowWriteStreamBuffering=$false 
$webRequest.SendChunked=$true # needed by previous line 
$webRequest.Credentials = $cred 

$requestStream = $webRequest.GetRequestStream() 
$fileStream = [System.IO.File]::OpenRead($file) 
$chunk = New-Object byte[] $bufSize 
    while($bytesRead = $fileStream.Read($chunk,0,$bufsize)) 
    { 
    $requestStream.write($chunk, 0, $bytesRead) 
    $requestStream.Flush() 
    } 

$responceStream = $webRequest.getresponse() 
#$status = $webRequest.statuscode 

$FileStream.Close() 
$requestStream.Close() 
$responceStream.Close() 

$responceStream 
$responceStream.GetResponseHeader("Content-Length") 
$responceStream.StatusCode 
#$status 

Answer 3

Per caricare su Sonatype Nexus3 ho usato il codice qui sotto. Ci ho messo un po 'di tempo per capirlo, lavorando con upload e risposta da Nexus3 e caricando e scaricando file di grandi dimensioni (più grandi di 2 GB). Abbiamo Apache di fronte a Nexus3 che si occupa delle connessioni https. L'utilizzo dell'autenticazione di base ha fatto in modo che Nexus rispondesse correttamente, quando Apache è di fronte ad esso. Invio della pre-autenticazione tramite HEAD e utilizzo del caricamento a blocchi per file di grandi dimensioni, risolto nel caso in cui il caricamento di file di grandi dimensioni non terminasse prematuramente.

Il download di file di grandi dimensioni tramite Invoke-WebRequest non avrebbe esito positivo con errori diversi. Ora uso WebRequest via .Net e funziona, vedi Download-File.

E quando il codice veniva eseguito tramite un processo automatizzato (da System Center Orchestrator), https avrebbe esito negativo. Quindi imponiamo TLS 1.2 quando viene rilevato lo schema https.

function New-HttpWebRequest 
{ 
    <# 
    .SYNOPSIS 
    Creates a new [System.Net.HttpWebRequest] ready for file transmission. 

    .DESCRIPTION 
    Creates a new [System.Net.HttpWebRequest] ready for file transmission. 
    The method will be Put. If the filesize is larger than the buffersize, 
    the HttpWebRequest will be configured for chunked transfer. 

    .PARAMETER Url 
    Url to connect to. 

    .PARAMETER Credential 
    Credential for authentication at the Url resource. 

    .EXAMPLE 
    An example 
    #> 
    param(
     [Parameter(Mandatory=$true)] 
     [ValidateNotNullOrEmpty()] 
     [string]$Url, 

     [Parameter(Mandatory=$true)] 
     [ValidateNotNullOrEmpty()] 
     [pscredential]$Credential, 

     [Parameter(Mandatory=$true)] 
     [long]$FileSize, 

     [Parameter(Mandatory=$true)] 
     [long]$BufferSize 
    ) 

    $webRequest = [System.Net.HttpWebRequest]::Create($Url) 
    $webRequest.Timeout = 600 * 1000; 
    $webRequest.ReadWriteTimeout = 600 * 1000; 
    $webRequest.ProtocolVersion = [System.Net.HttpVersion]::Version11; 
    $webRequest.Method = "PUT"; 
    $webRequest.ContentType = "application/octet-stream"; 
    $webRequest.KeepAlive = $true; 
    $webRequest.UserAgent = "<I use a specific UserAgent>"; 
    #$webRequest.UserAgent = 'Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 5.1)'; 
    $webRequest.PreAuthenticate = $true; 
    $auth = [System.Convert]::ToBase64String([System.Text.Encoding]::UTF8.GetBytes($Credential.UserName + ":" + $Credential.GetNetworkCredential().Password)); 
    $webRequest.Headers["Authorization"] = "Basic $auth" 

    if (Get-UseChunkedUpload -FileSize $FileSize -BufferSize $BufferSize) 
    { 
     Write-Verbose "FileSize is greater than BufferSize, using chunked transfer."; 
     $webRequest.AllowWriteStreamBuffering = $false; 
     $webRequest.SendChunked = $true; 
    } 
    else 
    { 
     # Filesize is equal to or smaller than the BufferSize. The file will be transferred in one write. 
     # Chunked cannot be used in this case. 
     $webRequest.AllowWriteStreamBuffering = $true; 
     $webRequest.SendChunked = $false; 
     $webRequest.ContentLength = $FileSize; 
    } 

    return $webRequest; 
} 

function Get-BufferSize 
{ 
    <# 
    .SYNOPSIS 
    Returns a buffer size that is 1% of ByteLength, rounded in whole MB's or at least AtLeast size. 

    .DESCRIPTION 
    Returns a buffer size that is 1% of ByteLength, rounded to whole MB's or if 1% is smaller than AtLeast, then AtLeast size is returned which is 1MB by default. 

    .PARAMETER ByteLength 
    Length of the bytes for which to calculate a valid buffer size. 

    .PARAMETER AtLeast 
    The minimum required buffer size, default 1MB. 

    .EXAMPLE 
    Get-BufferSize 4283304773 

    Returns 42991616 which is 41MB. 

    .EXAMPLE 
    Get-BufferSize 4283304 

    Returns 1048576 which is 1MB. 

    .EXAMPLE 
    Get-BufferSize 4283304 5MB 

    Returns 5242880 which is 5MB. 
    #> 
    param(
     [Parameter(Mandatory=$true)] 
     [long]$ByteLength, 

     [long]$AtLeast = 1MB 
    ) 

    [long]$size = $ByteLength/100; 
    if ($size -lt $AtLeast) 
    { 
     $size = $AtLeast; 
    } 
    else 
    { 
     $size = [Math]::Round($size/1MB) * 1MB; 
    } 

    return $size; 
} 

function Get-UseChunkedUpload 
{ 
    param(
     [Parameter(Mandatory=$true)] 
     [long]$FileSize, 

     [Parameter(Mandatory=$true)] 
     [long]$BufferSize 
    ) 

    return $FileSize -gt $BufferSize; 
} 

function Configure-Tls 
{ 
    param(
     [Parameter(Mandatory=$true)] 
     [ValidateNotNullOrEmpty()] 
     [string]$Url 
    ) 

    [System.Uri]$uri = $Url; 
    if ($uri.Scheme -eq "https") 
    { 
     Write-Verbose "Using TLS 1.2"; 
     [System.Net.ServicePointManager]::SecurityProtocol = [System.Net.SecurityProtocolType]::Tls12; 
    } 
} 

function Send-PreAuthenticate 
{ 
    param(
     [Parameter(Mandatory=$true)] 
     [ValidateNotNullOrEmpty()] 
     [string]$Url, 

     [Parameter(Mandatory=$true)] 
     [ValidateNotNullOrEmpty()] 
     [pscredential]$Credential 
    ) 

    $response = $null; 
    try 
    { 
     [System.Uri]$uri = $Url; 
     $repositoryAuthority = (($uri.GetLeftPart([System.UriPartial]::Authority)).TrimEnd('/') + '/'); 
     Write-Verbose "Send-PreAuthenticate - Sending HEAD to $repositoryAuthority"; 
     $wr = [System.Net.WebRequest]::Create($repositoryAuthority); 
     $wr.Method = "HEAD"; 
     $wr.PreAuthenticate = $true; 
     $auth = [System.Convert]::ToBase64String([System.Text.Encoding]::UTF8.GetBytes($Credential.UserName + ":" + $Credential.GetNetworkCredential().Password)); 
     $wr.Headers["Authorization"] = "Basic $auth" 
     $response = $wr.GetResponse(); 
    } 
    finally 
    { 
     if ($response) 
     { 
      $response.Close(); 
      $response.Dispose(); 
      $response = $null; 
     } 
    } 
} 

function Upload-File 
{ 
    <# 
    .SYNOPSIS 
    Uploads a file to the Nexus repository. 

    .DESCRIPTION 
    Uploads a file to the Nexus repository. 
    If the file was uploaded successfully, the url via which the resource can be downloaded is returned. 

    .PARAMETER Url 
    The Url where the resource should be created. 
    Please note that underscores and dots should be encoded, otherwise the Nexus repository does not accept the upload. 

    .PARAMETER File 
    The file that should be uploaded. 

    .PARAMETER Credential 
    Credential used for authentication at the Nexus repository. 

    .EXAMPLE 
    Upload-File -Url https://nexusrepo.domain.com/repository/repo-name/myfolder/myfile%2Eexe -File (Get-ChildItem .\myfile.exe) -Credential (Get-Credential) 

    .OUTPUTS 
    If the file was uploaded successfully, the url via which the resource can be downloaded. 
    #> 
    param(
     [Parameter(Mandatory=$true)] 
     [ValidateNotNullOrEmpty()] 
     [string]$Url, 

     [Parameter(Mandatory=$true)] 
     [ValidateNotNullOrEmpty()] 
     [System.IO.FileInfo]$File, 

     [Parameter(Mandatory=$true)] 
     [ValidateNotNullOrEmpty()] 
     [pscredential]$Credential 
    ) 

    Write-Verbose "Upload-File Url:$Url" 

    Configure-Tls -Url $Url; 

    $fileSizeBytes = $File.Length; 
    #$bufSize = Get-BufferSize $fileSizeBytes; 
    $bufSize = 4 * 1MB; 
    Write-Verbose ("FileSize is {0} bytes ({1:N0}MB). BufferSize is {2} bytes ({3:N0}MB)" -f $fileSizeBytes,($fileSizeBytes/1MB),$bufSize,($bufSize/1MB)); 
    if (Get-UseChunkedUpload -FileSize $fileSizeBytes -BufferSize $bufSize) 
    { 
     Write-Verbose "Using chunked upload. Send pre-auth first."; 
     Send-PreAuthenticate -Url $Url -Credential $Credential; 
    } 

    $progressActivityMessage = ("Sending file {0} - {1} bytes" -f $File.Name, $File.Length); 
    $webRequest = New-HttpWebRequest -Url $Url -Credential $Credential -FileSize $fileSizeBytes -BufferSize $bufSize; 
    $chunk = New-Object byte[] $bufSize; 
    $bytesWritten = 0; 
    $fileStream = [System.IO.File]::OpenRead($File.FullName); 
    $requestStream = $WebRequest.GetRequestStream(); 
    try 
    { 
     while($bytesRead = $fileStream.Read($chunk,0,$bufSize)) 
     { 
      $requestStream.Write($chunk, 0, $bytesRead); 
      $requestStream.Flush(); 
      $bytesWritten += $bytesRead; 
      $progressStatusMessage = ("Sent {0} bytes - {1:N0}MB" -f $bytesWritten, ($bytesWritten/1MB)); 
      Write-Progress -Activity $progressActivityMessage -Status $progressStatusMessage -PercentComplete ($bytesWritten/$fileSizeBytes*100); 
     } 
    } 
    catch 
    { 
     throw; 
    } 
    finally 
    { 
     if ($fileStream) 
     { 
      $fileStream.Close(); 
     } 
     if ($requestStream) 
     { 
      $requestStream.Close(); 
      $requestStream.Dispose(); 
      $requestStream = $null; 
     } 
     Write-Progress -Activity $progressActivityMessage -Completed; 
    } 

    # Read the response. 
    $response = $null; 
    try 
    { 
     $response = $webRequest.GetResponse(); 
     Write-Verbose ("{0} responded with {1} at {2}" -f $response.Server,$response.StatusCode,$response.ResponseUri); 
     return $response.ResponseUri; 
    } 
    catch 
    { 
     if ($_.Exception.InnerException -and ($_.Exception.InnerException -like "*bad request*")) 
     { 
      throw ("ERROR: " + $_.Exception.InnerException.Message + " Possibly the file already exists or the content type of the file does not match the file extension. In that case, disable MIME type validation on the server.") 
     } 

     throw; 
    } 
    finally 
    { 
     if ($response) 
     { 
      $response.Close(); 
      $response.Dispose(); 
      $response = $null; 
     } 
     if ($webRequest) 
     { 
      $webRequest = $null; 
     } 
    } 
} 

function Download-File 
{ 
    param(
     [Parameter(Mandatory=$true)] 
     [ValidateNotNullOrEmpty()] 
     [string]$Url, 

     [Parameter(Mandatory=$true)] 
     [ValidateNotNullOrEmpty()] 
     [string]$FileName 
    ) 

    $SDXDownloadType = @" 
    using System.IO; 
    using System.Net; 

    public class SDXDownload 
    { 
     static public void DownloadFile(string Uri, string Filename) 
     { 
      HttpWebRequest webRequest = (HttpWebRequest)WebRequest.Create(Uri); 
      webRequest.Method = "GET"; 
      using (HttpWebResponse myHttpWebResponse = (HttpWebResponse)webRequest.GetResponse()) 
      using (Stream fileStream = File.OpenWrite(Filename)) 
      using (Stream streamResponse = myHttpWebResponse.GetResponseStream()) 
      { 
       int bufSize = 64 * 1024; 
       byte[] readBuff = new byte[bufSize]; 
       int bytesRead = streamResponse.Read(readBuff, 0, bufSize); 
       while (bytesRead > 0) 
       { 
        fileStream.Write(readBuff, 0, bytesRead); 
        bytesRead = streamResponse.Read(readBuff, 0, 256); 
       } 
      } 
     } 
    } 
"@ 

    Configure-Tls -Url $Url; 

    Add-Type -TypeDefinition $SDXDownloadType; 
    [SDXDownload]::DownloadFile($Url, $FileName); 
}