uploading files using httpSendRequest c++ - asp.net

I am trying to send a file to HTTP server via POST request (c++ and winapi), steps:
// Read file into "buff" and file size into "buffSize"
....
....
....
HINTERNET internetRoot;
HINTERNET httpSession;
HINTERNET httpRequest;
internetRoot = InternetOpen(agent_info, INTERNET_OPEN_TYPE_DIRECT, NULL, NULL, NULL);
//Connecting to the http server
httpSession = InternetConnect(internetRoot, IP,PORT_NUMBER, NULL, NULL, INTERNET_SERVICE_HTTP, NULL, NULL);
//Creating a new HTTP POST request to the default resource on the server
httpRequest = HttpOpenRequest(httpSession, TEXT("POST"), TEXT("/Post.aspx"), NULL, NULL, NULL, INTERNET_FLAG_RELOAD | INTERNET_FLAG_NO_CACHE_WRITE, NULL);
//Send POST request
HttpSendRequest(httpRequest, NULL, NULL, buff, buffSize);
//Closing handles...
In server I am recieving the file using this code (asp.net)
Stream httpStream;
try
{
httpStream = request.RequestContext.HttpContext.Request.InputStream;
}
catch (HttpException)
{
return;
}
byte[] tmp = new byte[httpStream.Length];
int bytesRead = httpStream.Read(tmp, 0, 1024 * 1024);
int totalBytesRead = bytesRead;
while (bytesRead > 0)
{
bytesRead = httpStream.Read(tmp, totalBytesRead, 1024 * 1024);
totalBytesRead += bytesRead;
}
httpStream.Close();
httpStream.Dispose();
//Save "tmp" to file...
I can send large files on local server (visual studio asp server), but I cannot send files over 1 MB to internet server. (HttpOpenRequest is failing)
is there a better way to upload files?

Caveat: My Wininet is very rusty these days.
I wonder whether you ought to be setting the "Content-Length" header yourself. Your code seems to assume that either a) you are making a HTTP/1.0 request or b) that HttpSendRequest will add the header for your (which I don't think it does).
Either way without the server being told how big the incoming request is the default configuration of IIS will reject it if it can't determine the request size itself quickly.
My guess is if you use the lpszHeaders and dwHeadersLength parameters of the HttpSendRequest function to include the appropriate "Content-Length" header the problem will be resolved.

What error do you receive? I mean what does GetLastError() returns? If you send file 800KB then it works OK? I dont really see how because HttpOpenRequest does not know about size of the data.
Maybe it timeouts? But this would mean that HttpSendRequest actually fails. It might buffer all data but since size is huge, then it takes more time than timeout allows.
use following code to query current timeouts (in ms):
InternetQueryOption(h, INTERNET_OPTION_RECEIVE_TIMEOUT, &dwReceiveTimeOut, sizeof(dwReceiveTimeOut));
InternetQueryOption(h, INTERNET_OPTION_SEND_TIMEOUT, &dwSendTimeOut, sizeof(dwSendTimeOut));
and following to set new ones:
InternetSetOption(h, INTERNET_OPTION_RECEIVE_TIMEOUT, &dwNewReceiveTimeOut, sizeof(dwNewReceiveTimeOut));
InternetSetOption(h, INTERNET_OPTION_SEND_TIMEOUT, &dwNewSendTimeOut, sizeof(dwNewSendTimeOut));

Related

ASP.NET Core 2.2 kestrel server's performance issue

I'm facing problem with kestrel server's performance. I have following scenario :
TestClient(JMeter) -> DemoAPI-1(Kestrel) -> DemoAPI-2(IIS)
I'm trying to create a sample application that could get the file content as and when requested.
TestClient(100 Threads) requests to DemoAPI-1 which in turn request to DemoAPI-2. DemoAPI-2 reads a fixed XML file(1 MB max) and returns it's content as a response(In production DemoAPI-2 is not going to be exposed to outside world).
When I tested direct access from TestClient -> DemoAPI-2 I got expected result(good) which is following :
Average : 368ms
Minimum : 40ms
Maximum : 1056ms
Throughput : 40.1/sec
But when I tried to access it through DemoAPI-1 I got following result :
Average : 48232ms
Minimum : 21095ms
Maximum : 49377ms
Throughput : 2.0/sec
As you can see there is a huge difference.I'm not getting even the 10% throughput of DemoAPI-2. I was told has kestrel is more efficient and fast compared to traditional IIS. Also because there is no problem in direct access, I think we can eliminate the possible of problem on DemoAPI-2.
※Code of DemoAPI-1 :
string base64Encoded = null;
var request = new HttpRequestMessage(HttpMethod.Get, url);
var response = await this.httpClient.SendAsync(request, HttpCompletionOption.ResponseContentRead).ConfigureAwait(false);
if (response.StatusCode.Equals(HttpStatusCode.OK))
{
var content = await response.Content.ReadAsByteArrayAsync().ConfigureAwait(false);
base64Encoded = Convert.ToBase64String(content);
}
return base64Encoded;
※Code of DemoAPI-2 :
[HttpGet("Demo2")]
public async Task<IActionResult> Demo2Async(int wait)
{
try
{
if (wait > 0)
{
await Task.Delay(wait);
}
var path = Path.Combine(Directory.GetCurrentDirectory(), "test.xml");
var file = System.IO.File.ReadAllText(path);
return Content(file);
}
catch (System.Exception ex)
{
return StatusCode(500, ex.Message);
}
}
Some additional information :
Both APIs are async.
Both APIs are hosted on different EC2 instances(C5.xlarge Windows Server 2016).
DemoAPI-1(kestrel) is a self-contained API(without reverse proxy)
TestClient(jMeter) is set to 100 thread for this testing.
No other configuration is done for kestrel server as of now.
There are no action filter, middleware or logging that could effect the performance as of now.
Communication is done using SSL on 5001 port.
Wait parameter for DemoAPI2 is set to 0 as of now.
The CPU usage of DEMOAPI-1 is not over 40%.
The problem was due to HttpClient's port exhaustion issue.
I was able to solve this problem by using IHttpClientFactory.
Following article might help someone who faces similar problem.
https://www.stevejgordon.co.uk/httpclient-creation-and-disposal-internals-should-i-dispose-of-httpclient
DEMOAPI-1 performs a non-asynchronous read of the streams:
var bytes = stream.Read(read, 0, DataChunkSize);
while (bytes > 0)
{
buffer += System.Text.Encoding.UTF8.GetString(read, 0, bytes);
// Replace with ReadAsync
bytes = stream.Read(read, 0, DataChunkSize);
}
That can be an issue with throughput on a lot of requests.
Also, I'm not fully aware of why are you not testing the same code with IIS and Kestrel, I would assume you need to make only environmental changes and not the code.

PHPmailer - Add pdf file as attachment doesn't work

My Goal:
Use AddStringAttachment() to send a auto-generated base64 string as a .pdf file to another email address.
Coding Environment:
I'm working on WordPress with a ajax call passing a base64 string to the server. The size of the string is usually around 30kbs, it can be guaranteed not exceeding over 50kbs. I have MAX_EXECUTION_TIME 120s.
What I've Been Working Through:
I succeeded:
Sending plain text body
Sending a small .txt file
I failed:
Sending base64 string using AddStringAttachment(). The server returns me a 504 Gateway Time-out error most of time, even if $mail->send() function passes through, I can only receive a corrupt .pdf file with 10kbs bigger than original size.
Sending a already exist .pdf file with AddAttachment(), The server also returns me a 504 Gateway Time-out error, and I also get a warning like Resource interpreted as Document but transferred with MIME type application/pdf
My Code:
function sendPdf() {
$mail = new PHPMailer(true);
//Server settings
$mail->SMTPDebug = 2; // Enable verbose debug output
$mail->isSMTP(); // Set mailer to use SMTP
$mail->Host = 'smtp.hostinger.com'; // Specify main and backup SMTP servers
$mail->SMTPAuth = true; // Enable SMTP authentication
$mail->Username = 'janice#popper.ga'; // SMTP username
$mail->Password = 'secret'; // SMTP password
$mail->SMTPSecure = 'tls'; // Enable TLS encryption, `ssl` also accepted
$mail->Port = 587; // TCP port to connect to
//Recipient
$mail->SetFrom('janice#popper.ga');
$mail->AddAddress( 'xxxxxxxx#gmail.com' );
$pdf_base64 = $_POST[pdfString];
//Content
$mail->isHTML(true); // Set email format to HTML
$mail->Subject= ' New Application Form ';
$mail->Body= ' New Application Form From WordPress site ';
//Attachment
//$mail->AddStringAttachment($pdf_base64, $_POST[clientName].'_Application.pdf', 'base64', 'application/pdf');
//$mail->AddAttachment(dirname(__FILE__)."/Qian_Zhong_Application.pdf", 'Qian_Zhong_Application.pdf');
$error = '';
if(!$mail->send()){
$error = 'Mail error: '.$mail->ErrorInfo;
echo $error;
}else{
echo 'Message has been sent.';
}
exit; // This is required to end AJAX requests properly.
}
The data you pass in to addStringAttachment should be raw binary, not encoded in any way, as PHPMailer will take care of that for you. It will also set the encoding and MIME type from the filename you provide, so you do not need to set them manually.
Using a debugger would allow you to watch the script as it runs so you would be able to see exactly what it’s having trouble with. Any error 500s will cause errors to be logged in your web server logs and will usually provide more info.
I would also recommend against using $_POST[clientName] like that without any filtering or validation - you should never trust user input like that.

Not getting Range header

I am trying to resume an upload from google drive, I have the id of the file when I close the connection I make this session after the Internet is connected again:
var request = (HttpWebRequest)WebRequest.Create(fileUri);
request.Method = "PUT";
request.ContentLength = 0;
request.Headers.Add("Content-Range", "bytes */" + FileByteArray.Length);
try
{
var response = request.GetResponse();
}
catch (WebException e)
{
fileRange = e.Response.Headers.Get("Range");
}
}
I am not getting the Range header, why is that?
Content-Range indicates which range of the original entity is contained in the body of either a request (like your PUT request) or a 206 partial response. Range is set by the client not the server in order to request a sub-range. I would assume that the server you are talking to will not respond with the uploaded chunk, so a Content-Range (and Range in no case) will not be present as a response header.
In your code snippet the actual upload range is missing for Content-Range (see the updated HTTP RFC). It has to have the form of:
Content-Range: bytes 42-1233/1234
which means: upload byte 42-1233 of an entity whose total size is 1234 byte.
Or when the complete length is unknown:
Content-Range: bytes 42-1233/*
So remove the check for the Range header and specify the complete upload range and you should be fine.

Connecting Qt with SSL to a jetty server

I have some problems with connecting a qt client to an embedded jetty server.
At first, I use the following components:
Qt 4.4.3 (compiled with active openssl support)
jetty 8.8.1
java 6
I know, the versions are not most recent, but because of licencing issues and customer wishes I can not use newer one.
So, the scenario is that a qt client has to send http GET and POST requests to the jetty server. As long I use simple http with the QHttp object it works fine, the problems start when I switch to SSL.
My first try was to use the QSslSocket object for the GET request:
// Load certs + private key to socket
_pSocket = new QSslSocket(this);
_pSocket->setLocalCertificate(_certificate);
_pSocket->setPrivateKey(_privatekey);
_pSocket->addDefaultCaCertificate(_cacertificate);
connect (_pSocket, SIGNAL(encrypted()), this, SLOT(_encrypted()));
_pSocket->connectToHostEncrypted("localhost", 8000);
with the following slot function for the encrypted state:
void TestClient::_encrypted() {
QString _path("/testpath/list");
QByteArray buffer("GET ");
buffer.append(_path).append(" HTTP/1.1\r\n");
_pSocket->write(buffer);
}
Here I have my first problem:
This results in the following string, which is as far as I see compliant to RFC 2616:
"GET /testpath/list HTTP/1.1\r\n"
For some reason, the jetty server has a problem with that, keeping in a loop till the client close the connection because of a time out.
But if I use the following string, it works perfect:
"GET /testpath/list\r\n"
Here is my first question: Do you now an explanation for this behaviour ? I can live with it, but I want to know the reason
My second problem is the POST request, this fails always.
These examples I already tried:
"POST /testpath/receive/\r\n{"data":"hello world ?!"}\r\n"
"POST /testpath/receive/ HTTP/1.1\r\n{"data":"hello world ?!"}\r\n"
"POST /testpath/receive/\r\n\r\n{"data":"hello world ?!"}\r\n"
"POST /testpath/receive/ HTTP/1.1\r\n\r\n{"data":"hello world ?!"}\r\n"
I have the feeling, that the body is every time empty, so my server crashes because he tries to parse an empty string as json.
At least, the following log shows that:
2013-11-19 17:11:51.671, INFO, foo.bar.RepositoryHandler, qtp11155366-16 - /testpath/receive request type : /receive
2013-11-19 17:11:51.811, ERROR, foo.bar.RepositoryHandler, qtp11155366-16 - /testpath/receive missing or unknown elements in JSON request. Check JSON against documentation
2013-11-19 17:11:51.874, WARN, org.eclipse.jetty.server.AbstractHttpConnection, qtp11155366-16 - /testpath/receive /testpath/receive
java.lang.NullPointerException: null
at foo.bar.RepositoryHandler.decodeViewingRequest(RepositoryHandler.java:366) ~[MyServer.jar:na]
at foo.bar.RepositoryHandler.handle(RepositoryHandler.java:182) ~[MyServer.jar:na]
So, all together, I think I have several major errors in my requests. But which ?
My second try was to use the QHttp object and change the QSocket it uses with a QSslSocket I already initiated.
Here's the code of the main function:
QSslSocket* _pSocket;
QHttp* _pHttp;
int _id;
QBuffer* _pBuffer;
QByteArray _data;
_pSocket = new QSslSocket(this);
_pSocket->setLocalCertificate(_certificate);
_pSocket->setPrivateKey(_privatekey);
_pSocket->addDefaultCaCertificate(_cacertificate);
QUrl url;
url.setScheme("https");
url.setHost("localhost");
url.setPort(8001);
url.setPath("/testpath/receive");
connect (_pSocket, SIGNAL(encrypted()), this, SLOT(_encrypted()));
connect(_pHttp,SIGNAL(requestFinished(int,bool)),this,SLOT(_requestFinished(int,bool)));
connect(_pHttp,SIGNAL(done(bool)),this,SLOT(_done(bool)));
_pBuffer = new QBuffer(&_data);
_pHttp->setSocket(_pSocket);
_pSocket->connectToHostEncrypted(strHost, strPort.toInt());
_id = _pHttp->get(url.toString(),_pBuffer);
And the callbacks:
void _requestFinished(int id, bool error) {
if(id = _id)
qDebug() << "data=" << _data;
}
void _encrypted() {
qDebug() << "encrypted";
}
void _done(bool error) {
logInfo() << "_done";
if(_pHttp) {
_pHttp->abort();
delete _pHttp;
_pHttp = 0;
}
if(_pBuffer) {
delete _pBuffer;
_pBuffer = 0;
}
if(_pSocket) {
_pSocket->disconnectFromHost();
delete _pSocket;
_pSocket = 0;
}
}
I think, I only have to change the position of the _pHttp->get call, perhaps in the _encrypted callback, but I'm not sure.
Some good advise ?
Thanks,
Robert
Your HTTP request is incomplete, per RFC2616.
"GET /testpath/list HTTP/1.1\r\n"
That is invalid.
Try this instead.
"GET /testpath/list HTTP/1.1\r\n" + /* request line (required) */
"Host: localhost\r\n" + /* host header (required minimum) */
"\r\n" /* terminating CR + LF (required) */
As outlined in Section 5.1.2
The most common form of Request-URI is that used to identify a
resource on an origin server or gateway. In this case the absolute
path of the URI MUST be transmitted (see section 3.2.1, abs_path) as
the Request-URI, and the network location of the URI (authority) MUST
be transmitted in a Host header field. For example, a client wishing
to retrieve the resource above directly from the origin server would
create a TCP connection to port 80 of the host "www.w3.org" and send
the lines:
GET /pub/WWW/TheProject.html HTTP/1.1
Host: www.w3.org
The Request-URI line and Host header Header are mandated.

large file send using webservice in asp.net c#?

I have written following code and it does not work. The following error comes while uploading file to the web services:
1.An operation on a socket could not be performed because the system lacked sufficient buffer space or because a queue was full
2.The underlying connection was closed: An unexpected error occurred on a send.
I have used the following code for the web service and when the file size is more than 90 mb the error comes:
LocalService.IphoneService obj = new LocalService.IphoneService();
byte[] objFile = FileToByteArray(#"D:\Brijesh\My Project\WebSite5\IMG_0010.MOV");
int RtnVal = obj.AddNewProject("demo", "demo", "demo#demo.com", "demo#demo.com", 1, 2, 29, "IMG_0010.MOV", objFile,"00.00.06");
public byte[] FileToByteArray(string fileName)
{
byte[] fileContent = null;
System.IO.FileStream fs = new System.IO.FileStream(fileName, System.IO.FileMode.Open, System.IO.FileAccess.Read);
System.IO.BinaryReader binaryReader = new System.IO.BinaryReader(fs);
long byteLength = new System.IO.FileInfo(fileName).Length;
//byteLength = 94371840;
fileContent = binaryReader.ReadBytes((Int32)byteLength);
fs.Close();
fs.Dispose();
binaryReader.Close();
return fileContent;
}
No socket will transfer 200MB in one chunk. Your will receive data in chunks mostly be between 1024 and 4096 bytes(depending on your settings).
Read this data in chunks.
Reassemble your file on the server.
Then use this received file, assembled from bytes, as you need.
For an asp.net webservice:
enable webservice to receive large amounts of data
Increase the ASP.NET limits on the maximum size of SOAP messages and
the maximum number of seconds that a request is allowed to execute by
adding the configuration element to the application's
web.config file. The following code example sets the ASP.NET limit on
the maximum size of an incoming request to 400MB and the maximum
amount of time a request is allowed to execute to 5 minutes (300
seconds).
Put this in your web.config.
<configuration>
<system.web>
<httpRuntime maxMessageLength="409600"
executionTimeoutInSeconds="300"/>
</system.web>
</configuration>
Remember you will be blocking a thread for as long this request is processed. This will not scale for a large number of users.

Resources