Response to post request to AWS "breaks the pipe", cannot read - http

I am trying to post a video file to AWS using Android, using HttpUrlConnection. I am formatting the header and body per the instructions here http://docs.aws.amazon.com/AmazonS3/latest/API/sigv4-post-example.html and here http://docs.aws.amazon.com/AmazonS3/latest/API/RESTObjectPOST.html
I seem to be able to write the request body and file data to the connection output stream, in the sense that it lets me finish the writing block (when I had some errors in my header before, this did not happen). Nonetheless, when I check to see if the file is on AWS, it is not.
The response stream seems to be the best way to figure out why. However, when I attempt to open and input stream on the connection to get the response, this results in the "broken pipe error."
So, here is the code:
URL url = new URL(action);
HttpURLConnection conn = null;
conn = (HttpURLConnection) url.openConnection();
conn.setUseCaches(false);
conn.setDoOutput(true);
conn.setRequestMethod("POST");
conn.setRequestProperty("Host",bucket);
conn.setRequestProperty("User-Agent","Java/1.7.0_60");
conn.setRequestProperty("Accept","text/html,application/xhtml+xml, application/xml,application/json;q=0.9,*/*;q=0.8");
conn.setRequestProperty("Acccept-Language", "en-US,en;q=0.5");
conn.setRequestProperty("Accept-Encoding","gzip, deflate");
conn.setRequestProperty("Accept-Charset","UTF-8");
conn.setRequestProperty("Connection", "keep-alive");
conn.setRequestProperty("Content-Type", "multipart/form-data; boundary=" + boundary);
File fileObj = new File(file);
String closingString = crlf+twohyphens + boundary + twohyphens + crlf;
//get content length and set
int totalLength = body.getBytes().length + (int)fileObj.length() + closingString.getBytes().length;
conn.setRequestProperty("Content-Length",Integer.toString(totalLength));
//set body content fields
DataOutputStream request = new DataOutputStream(conn.getOutputStream());
request.writeBytes(body);
//get mp4 data and write to request:
InputStream input = new BufferedInputStream(new FileInputStream(file));
int data = input.read();
while(data != -1) {
//TODO loading animation
request.write(data);
data = input.read();
}
input.close();
Log.d("Writing","of file to output stream complete.");
request.writeBytes(closingString);
request.flush();
request.close();
Log.d("Writing","closing string to output stream complete. Closed stream");
//Response
/******** it seems to break on the following line: *************/
InputStream responseStream = new BufferedInputStream(conn.getInputStream());
BufferedReader responseStreamReader = new BufferedReader(new InputStreamReader(responseStream));
String line = "";
StringBuilder stringBuilder = new StringBuilder();
while ((line = responseStreamReader.readLine()) != null)
{
stringBuilder.append(line).append("\n");
}
responseStreamReader.close();
String responseString = stringBuilder.toString();
Log.d("server response",responseString);
Response response = new Response(0,"test");
conn.disconnect();
Note: the "body" string includes everything in the post body content (ie the required form fields, formatted as in the example links), except the actual file content, which must go last.
Here are the logs:
D/Writing: of file to output stream complete
D/Writing: closing string to output stream complete. Closed stream
W/System.err﹕ java.net.SocketException: sendto failed: EPIPE (Broken pipe)
W/System.err﹕ at libcore.io.IoBridge.maybeThrowAfterSendto(IoBridge.java:546)
W/System.err﹕ at libcore.io.IoBridge.sendto(IoBridge.java:515)
W/System.err﹕ at java.net.PlainSocketImpl.write(PlainSocketImpl.java:504)
W/System.err﹕ at java.net.PlainSocketImpl.access$100(PlainSocketImpl.java:37)
W/System.err﹕ at java.net.PlainSocketImpl$PlainSocketOutputStream.write(PlainSocketImpl.java:266)
W/System.err﹕ at com.android.okio.Okio$1.write(Okio.java:73)
W/System.err﹕ at com.android.okio.RealBufferedSink.emitCompleteSegments(RealBufferedSink.java:116)
W/System.err﹕ at com.android.okio.RealBufferedSink.write(RealBufferedSink.java:44)
W/System.err﹕ at com.android.okhttp.internal.http.RetryableSink.writeToSocket(RetryableSink.java:77)
W/System.err﹕ at com.android.okhttp.internal.http.HttpConnection.writeRequestBody(HttpConnection.java:259)
W/System.err﹕ at com.android.okhttp.internal.http.HttpTransport.writeRequestBody(HttpTransport.java:84)
W/System.err﹕ at com.android.okhttp.internal.http.HttpEngine.readResponse(HttpEngine.java:771)
W/System.err﹕ at com.android.okhttp.internal.http.HttpURLConnectionImpl.execute(HttpURLConnectionImpl.java:379)
W/System.err﹕ at com.android.okhttp.internal.http.HttpURLConnectionImpl.getResponse(HttpURLConnectionImpl.java:323)
W/System.err﹕ at com.android.okhttp.internal.http.HttpURLConnectionImpl.getInputStream(HttpURLConnectionImpl.java:190)
W/System.err﹕ at android.synq.fm.connectivity.WebService.post(WebService.java:244)
W/System.err﹕ at android.synq.fm.connectivity.AwsPoster.doInBackground(AwsPoster.java:36)
W/System.err﹕ at android.synq.fm.connectivity.AwsPoster.doInBackground(AwsPoster.java:14)
W/System.err﹕ at android.os.AsyncTask$2.call(AsyncTask.java:288)
W/System.err﹕ at java.util.concurrent.FutureTask.run(FutureTask.java:237)
W/System.err﹕ at android.os.AsyncTask$SerialExecutor$1.run(AsyncTask.java:231)
W/System.err﹕ at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1112)
W/System.err﹕ at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:587)
W/System.err﹕ at java.lang.Thread.run(Thread.java:818)
W/System.err﹕ Caused by: android.system.ErrnoException: sendto failed: EPIPE (Broken pipe)
W/System.err﹕ at libcore.io.Posix.sendtoBytes(Native Method)
W/System.err﹕ at libcore.io.Posix.sendto(Posix.java:176)
W/System.err﹕ at libcore.io.BlockGuardOs.sendto(BlockGuardOs.java:278)
W/System.err﹕ at libcore.io.IoBridge.sendto(IoBridge.java:513)
Please note that the log line:
W/System.err﹕ at android.synq.fm.connectivity.WebService.post(WebService.java:244)
corresponds to the following line in the code:
InputStream responseStream = new BufferedInputStream(conn.getInputStream());
So, my question is two-fold. Why does this not seem to successfully upload the file, even though it lets it be written to the connection output stream? And why does my attempt to get the request response "break the pipe"? Is there a special way to get POST responses from AWS that is not covered in the docs or that I somehow missed? (Please note this is my first time working with POST requests and AWS.)
Thanks in advance.

Related

Sometimes FileNotFoundException at Url

Sorry for bad English.
My problem is so similar this issue "FileNotFoundException at URL"
but sometimes it gives error and doesnt continue but sometimes it works perfectly. This problem is in first connection.What can be the problem that causes it works sometimes and doesnt work sometimes?
10-10 02:19:41.128 9667-9819/? W/System.err﹕ java.io.FileNotFoundException: http://cdn59.my.mail.ru/v/59908302.mp4?slave[]=s%3Ahttp%3A%2F%2Fvideo-cephgw1.i%3A8080%2Frados%2F59908302-v&p=f&expire_at=1476068400&touch=1475880462&reg=76&sign=dd01f023e682705a105440ab5e93f5cb38cfeadd
10-10 02:19:41.128 9667-9819/? W/System.err﹕ at com.android.okhttp.internal.http.HttpURLConnectionImpl.getInputStream(HttpURLConnectionImpl.java:186)
10-10 02:19:41.128 9667-9819/? W/System.err﹕ at makgun.kturk.MRDownloader$1.doInBackground(MRDownloader.java:105)
10-10 02:19:41.128 9667-9819/? W/System.err﹕ at makgun.kturk.MRDownloader$1.doInBackground(MRDownloader.java:39)
10-10 02:19:41.128 9667-9819/? W/System.err﹕ at android.os.AsyncTask$2.call(AsyncTask.java:288)
10-10 02:19:41.128 9667-9819/? W/System.err﹕ at java.util.concurrent.FutureTask.run(FutureTask.java:237)
10-10 02:19:41.128 9667-9819/? W/System.err﹕ at android.os.AsyncTask$SerialExecutor$1.run(AsyncTask.java:231)
10-10 02:19:41.128 9667-9819/? W/System.err﹕ at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1112)
10-10 02:19:41.128 9667-9819/? W/System.err﹕ at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:587)
10-10 02:19:41.128 9667-9819/? W/System.err﹕ at java.lang.Thread.run(Thread.java:841)
MRDownloader:
#Override
protected String doInBackground(String... urls) {
try {
String path = Environment.getExternalStorageDirectory().getAbsolutePath() + "/" +name;
File file = new File(path);
int total;
if (file.exists())total=Integer.parseInt(String.valueOf(file.length()));
else total=0;
//New Added
if (urls[0].startsWith("//"))urls[0]="http:"+urls[0];
//New Added finished
URL url = new URL(urls[0]);
HttpURLConnection urlConnection = (HttpURLConnection)url.openConnection();
OutputStream output = new FileOutputStream(path, true);
urlConnection.addRequestProperty("Range", "bytes=" + file.length() + "-");
if(!Cookie.isEmpty())
urlConnection.addRequestProperty("Cookie", "video_key=" +Cookie);
urlConnection.setRequestProperty("User-Agent","Mozilla/5.0");
urlConnection.setRequestProperty("Accept","*/*");
urlConnection.setConnectTimeout(7000);
urlConnection.setReadTimeout(7000);
urlConnection.connect();
int lenghtOfFile = urlConnection.getContentLength();
PD.setMax(lenghtOfFile+total);
InputStream input = new BufferedInputStream(urlConnection.getInputStream());
byte data[] = new byte[1024];
// int total=Integer.parseInt(String.valueOf(file.length()));
int count;
Log.d("MakgunLENGHT-OF-FILE", Integer.toString(lenghtOfFile));
while ((count = input.read(data)) != -1) {
total += count;
PD.setProgress(total);
output.write(data, 0, count);
if (pause){output.flush();output.close();input.close();if (PD.isShowing()) PD.dismiss();break;}
}
output.flush();
output.close();
input.close();
} catch (MalformedURLException e) {
e.printStackTrace();
} catch (IOException e1) {
e1.printStackTrace();
}
return null;
}
You can use this link to try it. TryLink.
When you request this link it generates "video_key" cookie and response body includes all of information what you need like stream URL of video. I dont copy all of java file so if you need I can add more. Thanks.
Nowadays, I have had time to understand this and finally I find out its why and when happens. I want to share this for other users. Its why is that it happens whenever server send bad request and for this site sometimes send "Service Unavaible 503" and this code failures. But adding a condition to repeat max 10 times it works like a charm. At least one click is enough and it repeats max 10 times and generally it gets 2xx response code from server after 2-3 times tried. If 2xx code received it breaks the loop. And it doesnt reapet any more. This solved my problem.
int x = 1;
while (true) {
//otherStuffs...
if (responseCode == 503) {
Log.d("makgunResponseCode503", "Now ResponseCode is 503 repeating " + x + " times");
x++;
}
else
{Log.d("makgunResponseCode", "ResponseCode is: " + responseCode);break}
if (x > 10) break;
}

"A Connection attempt Failed.." Error during at Soap Request in Asp.net

I have connected a Third Party Web service with my web application, it is working fine
in localhost . But after i uploaded the project in server it is showing the following error.
A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond 100.100.100.100:9090
Here is my Soap Request:
StringBuilder soapmsg = new StringBuilder();
soapmsg.Append("<?xml version='1.0' encoding='UTF-8'?>");
soapmsg.Append("<soap:Envelope xmlns='http://www.opentravel.org/OTA/2003/05' xmlns:soap='http://schemas.xmlsoap.org/soap/envelope/' xmlns:wsse='http://schemas.xmlsoap.org/ws/2002/12/secext'>");
soapmsg.Append("<soap:Body><OTA_AirLowFareSearchRQ EchoToken='' SequenceNmbr='0' TransactionIdentifier='0' Version='1.001' xmlns='http://www.opentravel.org/OTA/2003/05' DirectFlightsOnly='false'>");
soapmsg.Append("<POS xmlns='http://www.opentravel.org/OTA/2003/05'><Source AgentSine='' PseudoCityCode='NPCK' TerminalID='1'><RequestorID ID='AFFILIATE'/></Source><YatraRequests><YatraRequest AffiliateID='YTFABTRAVEL' MidOfficeAgentID='' AgentEmailID='' Destination='' AgentTypeCode='' Description='' DoNotCache='false' DoNotHitCache='true' IsYatraAgent='' LastResultFromCache='true' RequestType='"+di+"' YatraRequestTypeCode='SMPA'/></YatraRequests></POS>");
soapmsg.Append("<OriginDestinationInformation>");
soapmsg.Append("<DepartureDateTime WindowAfter='" + flexi + "' WindowBefore='" + flexi + "'>" + date + "</DepartureDateTime>");
soapmsg.Append("<OriginLocation CodeContext='IATA' LocationCode='" + origin + "'>'" + origin + "'</OriginLocation>");
soapmsg.Append("<DestinationLocation CodeContext='IATA' LocationCode='" + destination + "'>'" + destination + "'</DestinationLocation>");
soapmsg.Append("</OriginDestinationInformation>");
soapmsg.Append("</OTA_AirLowFareSearchRQ></soap:Body></soap:Envelope>");
HttpWebRequest request = (HttpWebRequest)WebRequest.CreateDefault(new Uri(#"http://100.100.100.100:9090/services/spm/spm"));
request.ContentType = "text/xml; charset=utf-8";
request.Method = "POST";
request.Accept = "text/xml";
request.Headers.Add("SOAPAction", #"http://100.100.100.100:9090/services/spm/spm");
StreamWriter writer = new StreamWriter(request.GetRequestStream(), Encoding.UTF8);
writer.Write(soapmsg);
writer.Flush();
writer.Close();
using (HttpWebResponse response = request.GetResponse() as HttpWebResponse)
{
XNamespace ns = "http://www.opentravel.org/OTA/2003/05";
XDocument xd = XDocument.Load(response.GetResponseStream());
}
**The following exception i got after i tried to connect the web service:**
SocketException (0x274c): A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond 100.100.100.100:9090]
System.Net.Sockets.Socket.DoConnect(EndPoint endPointSnapshot, SocketAddresssocketAddress) +251System.Net.ServicePoint.ConnectSocketInternal(Boolean connectFailure, Socket s4, Socket s6, Socket& socket, IPAddress& address, ConnectSocketState state, IAsyncResult asyncResult, Int32 timeout, Exception& exception) +279
I have searched a lot. I am new to web service.
Please let me know if you need more information.
Thanks in advance,

app_offline.htm file being used to take webservice offline - possible to read contents of file?

I'm using an app_offline.htm file as described here : http://weblogs.asp.net/scottgu/archive/2005/10/06/426755.aspx to take an old asmx web service offline.
All works fine, and the client end gets a HTTP 503 exception like :
Exception : System.Net.WebException
The request failed with HTTP status 503: Service Unavailable.
Source : System.Web.Services
Stack trace :
at System.Web.Services.Protocols.SoapHttpClientProtocol.ReadResponse(SoapClientMessage message, WebResponse response, Stream responseStream, Boolean asyncCall)
at System.Web.Services.Protocols.SoapHttpClientProtocol.Invoke(String methodName, Object[] parameters)
My question : is it possible for the client application to read the contents of the app_offline.htm file which would have been returned ? The basic HTML in that file has helpful text like : "The application is currently undergoing maintenance". I can see the contents of this file being returned in the response using Fiddler.
It would be useful to be able to parse this html response to provide more info to the user. (i.e. so it is possible to distinguish between a 503 error due to system maintenance, and other 503s due to system being overloaded etc).
EDIT : BluesRockAddict's response sounded good, but the stream seems to be unavailable at this time. e.g. :
// wex is the caught System.Net.WebException
System.Net.WebResponse resp = wex.Response;
byte[] buff = new byte[512];
Stream st = resp.GetResponseStream();
int count = st.Read(buff, 0, 512);
The last line above which attempts to read the stream gives :
Exception : System.ObjectDisposedException
Cannot access a closed Stream.
Source : mscorlib
Stack trace :
at System.IO.__Error.StreamIsClosed()
at System.IO.MemoryStream.Read(Byte[] buffer, Int32 offset, Int32 count)
credit goes to BluesRockAddict, adding to his answer, this is how you can read the content of html page.
catch (WebException ex)
{
if (((HttpWebResponse)ex.Response).StatusCode == HttpStatusCode.ServiceUnavailable)
{
using (Stream stream = ex.Response.GetResponseStream())
{
using(StreamReader reader = new StreamReader(stream))
{
var message = reader.ReadToEnd();
}
}
}
}
You should use WebException.Response to retrieve the message:
using (WebClient wc = new WebClient())
{
try
{
string content = wc.DownloadString(url);
}
catch (WebException ex)
{
if (((HttpWebResponse)ex.Response).StatusCode == HttpStatusCode.ServiceUnavailable)
{
message = ex.Response
}
}
}

BizTalk Custom Pipeline Component System.OutOfMemoryException

I'm currently developing a BizTalk Custom Send Pipeline that accepts an xml file and converts it to Excel. Unfortunately, after deploying the pipeline, I'm receiving a System.OutOfMemoryException. I've included the code for Execute Method of IComponent interface. All suggestions are welcome.
public Microsoft.BizTalk.Message.Interop.IBaseMessage Execute(IPipelineContext pContext, Microsoft.BizTalk.Message.Interop.IBaseMessage inmsg)
{
MemoryStream outMemStream = new MemoryStream();
try
{
if (inmsg.BodyPart.Data != null)
{
// Read the source message coming from the messaging engine and convert it to memory stream
byte[] buffer = new byte[16 * 1024];
using (MemoryStream ms = new MemoryStream())
{
int read;
while ((read = inmsg.BodyPart.Data.Read(buffer, 0, buffer.Length)) > 0)
{
ms.Write(buffer, 0, read);
}
buffer = ms.ToArray();
}
if (buffer != null)
{
var binaryWriter = new BinaryWriter(outMemStream);
binaryWriter.Write(buffer);
}
OpenXMLOffice oOffice = new OpenXMLOffice();
outMemStream.Position = 0;
oOffice.XMLToExcel(outMemStream, TemporaryFileLocation);
inmsg.BodyPart.Data.Position = 0;
inmsg.BodyPart.Data = outMemStream;
pContext.ResourceTracker.AddResource(outMemStream);
}
return inmsg;
}
catch (Exception ex)
{
throw new ApplicationException(String.Format("Error converting XML to Excel:{0} - Stack Trace: {1}", ex.Message, ex.StackTrace));
}
}
Here is the most recent error received:
Log Name: Application
Source: BizTalk Server
Date: 2/14/2012 9:29:00 AM
Event ID: 5754
Task Category: BizTalk Server
Level: Error
Keywords: Classic
User: N/A
Computer: IASDev-PC
Description:
A message sent to adapter "FILE" on send port "ExcelSendPort" with URI "C:\SeleneFTPFile\Excel\%MessageID%.xml" is suspended.
Error details: There was a failure executing the send pipeline: "IAS.SeleneFTPFile.ExcelEncodePipeline, IAS.SeleneFTPFile, Version=1.0.0.0, Culture=neutral, PublicKeyToken=2add433e7764165f" Source: "Excel File Encoder" Send Port: "ExcelSendPort" URI: "C:\SeleneFTPFile\Excel\%MessageID%.xml" Reason: Error converting XML to Excel:Exception of type 'System.OutOfMemoryException' was thrown. - Stack Trace: at System.IO.MemoryStream.set_Capacity(Int32 value)
at System.IO.MemoryStream.EnsureCapacity(Int32 value)
at System.IO.MemoryStream.Write(Byte[] buffer, Int32 offset, Int32 count)
at IAS.SeleneFTPFile.Components.ExcelPipeline.EncodeExcel.Execute(IPipelineContext pContext, IBaseMessage inmsg)
MessageId: {ED37CDD1-EF0C-46E7-9519-061AF3D4F8A4}
InstanceID: {B0E448B3-3DAD-4E52-8F87-07C5D5AA5224}
You could try pre-allocating the MemoryStream buffer (as suggested here). The error message states that it can't allocate enough memory for the line when performing the buffered write (ms.Write(buffer, 0, read);).
using (MemoryStream ms = new MemoryStream(buffer.Length))
The other issue you have is that your buffer could overflow - producing this exact error when the MemoryStream is written to (ms.Write(buffer, 0, read)).
byte[] buffer = new byte[2 * 1024 * 1024]; // try increasing to 2MB buffer
Even I face the same issue. I can find that the reason is the stream position does not advance and remains at 0 for the inmsg.BodyPart.Data stream, even after the below statement executes:
read = inmsg.BodyPart.Data.Read(buffer, 0, buffer.Length)

The remote server returned an error: (400) Bad Request while consuming a WCF Service

Please view the code given below. While the debug reaches the request.GetResponse() statement the error has been thrown.
Uri uri = new Uri(address);
string data = "<s:Envelope xmlns:s='http://schemas.xmlsoap.org/soap/envelope/'><s:Body><HasRole xmlns='http://tempuri.org/'><userName>" + sid + "</userName><role>" + role + "</role></HasRole></s:Body></s:Envelope>";
data.Replace("'", "\"");
// Create a byte array of the data we want to send
byte[] byteData = UTF8Encoding.UTF8.GetBytes(data);
if (uri.Scheme == Uri.UriSchemeHttps)
{
HttpWebRequest request = (HttpWebRequest)HttpWebRequest.Create(uri);
request.Method = "POST";// WebRequestMethods.Http.Post;
request.ContentLength = byteData.Length;
request.ContentType = "application/soap+xml; charset=UTF-8"; // "text/xml; charset=utf-8";
//request.ContentType = "application/x-www-form-urlencoded";
//Stream requestStream = request.GetRequestStream();
using (Stream writer = request.GetRequestStream())
{
writer.Write(byteData, 0, byteData.Length);
}
//writer.Close();
using (HttpWebResponse response = request.GetResponse() as HttpWebResponse)
{
// Get the response stream
StreamReader reader = new StreamReader(response.GetResponseStream());
string tmp = reader.ReadToEnd();
Response.Close();
Response.Write(tmp);
}
I would double check the URL. If the URL looks ok on the client side, I recommend looking at access logs on your server to see what URL is being hit. 4xx errors mean a resource was not found. If the endpoint was correct, but the request was fubared, you would get a 5xx error code. (Assuming that your server side frameworks uses standard HTTP Response Codes).
As has been mentioned you should use the 'Add Service Reference' to access the WCF service from a .NET client. However, if you're emulating trying to connect from a non .NET client, your soap envelope is missing the header information.
<s:Header>
<Action s:mustUnderstand="1" xmlns="http://schemas.microsoft.com/ws/2005/05/addressing/none">
specify your action namespace here (e.g. http://tempuri.org/ISomeService/Execute)
</Action>
</s:Header>

Resources