I am building a HTTP proxy with netty, which supports HTTP pipelining. Therefore I receive multiple HttpRequest Objects on a single Channel and got the matching HttpResponse Objects. The order of the HttpResponse writes is the same than I got the HttpRequest. If a HttpResponse was written, the next one will be written when the HttpProxyHandler receives a writeComplete event.
The Pipeline should be convenient:
final ChannelPipeline pipeline = Channels.pipeline();
pipeline.addLast("decoder", new HttpRequestDecoder());
pipeline.addLast("encoder", new HttpResponseEncoder());
pipeline.addLast("writer", new HttpResponseWriteDelayHandler());
pipeline.addLast("deflater", new HttpContentCompressor(9));
pipeline.addLast("handler", new HttpProxyHandler());
Regarding this question only the order of the write calls should be important, but to be sure I build another Handler (HttpResponseWriteDelayHandler) which suppresses the writeComplete event until the whole response was written.
To test this I enabled network.http.proxy.pipelining in Firefox and visited a page with many images and connections (a news page). The problem is, that the browser does not receive some responses in spite of the logs of the proxy consider them as sent successfully.
I have some findings:
The problem only occurs if the connection from proxy to server is faster than the connection from proxy to browser.
The problem occurs more often after sending a larger image on that connection, e.g. 20kB
The problem does not occur if only 304 - Not Modified responses were sent (refreshing the page considering browser cache)
Setting bootstrap.setOption("sendBufferSize", 1048576); or above does not help
Sleeping a timeframe dependent on the responses body size in before sending the writeComplete event in HttpResponseWriteDelayHandler solves the problem, but is a very bad solution.
I found the solution and want to share it, if anyone else has a similar problem:
The content of the HttpResponse is too big. To analyze the content the whole HTML document was in the buffer. This must be splitted in Chunks again to send it properly. If the HttpResponse is not chunked I wrote a simple solution to do it. One needs to put a ChunkedWriteHandler next to the logic handler and write this class instead of the response itself:
public class ChunkedHttpResponse implements ChunkedInput {
private final static int CHUNK_SIZE = 8196;
private final HttpResponse response;
private final Queue<HttpChunk> chunks;
private boolean isResponseWritten;
public ChunkedHttpResponse(final HttpResponse response) {
if (response.isChunked())
throw new IllegalArgumentException("response must not be chunked");
this.chunks = new LinkedList<HttpChunk>();
this.response = response;
this.isResponseWritten = false;
if (response.getContent().readableBytes() > CHUNK_SIZE) {
while (CHUNK_SIZE < response.getContent().readableBytes()) {
chunks.add(new DefaultHttpChunk(response.getContent().readSlice(CHUNK_SIZE)));
}
chunks.add(new DefaultHttpChunk(response.getContent().readSlice(response.getContent().readableBytes())));
chunks.add(HttpChunk.LAST_CHUNK);
response.setContent(ChannelBuffers.EMPTY_BUFFER);
response.setChunked(true);
response.setHeader(HttpHeaders.Names.TRANSFER_ENCODING, HttpHeaders.Values.CHUNKED);
}
}
#Override
public boolean hasNextChunk() throws Exception {
return !isResponseWritten || !chunks.isEmpty();
}
#Override
public Object nextChunk() throws Exception {
if (!isResponseWritten) {
isResponseWritten = true;
return response;
} else {
HttpChunk chunk = chunks.poll();
return chunk;
}
}
#Override
public boolean isEndOfInput() throws Exception {
return isResponseWritten && chunks.isEmpty();
}
#Override
public void close() {}
}
Then one can call just channel.write(new ChunkedHttpResponse(response) and the chunking is done automatically if needed.
Related
In all the gRPC bidistream examples that I have seen follow a pattern that when (inbound) requestObserver receives onComplete it invokes the onComplete method of the (outbound) responseObserver. However, this is not done for onError.
Wondering what happens if I don't invoke responseObserver.onComplete() does it lead to memory leak? Why we don't do it for onError?
public StreamObserver<Point> recordRoute(final StreamObserver<RouteSummary> responseObserver) {
return new StreamObserver<Point>() {
#Override
public void onNext(Point point) {
// does something here
}
#Override
public void onError(Throwable t) {
logger.log(Level.WARNING, "recordRoute cancelled");
}
#Override
public void onCompleted() {
responseObserver.onCompleted();
}
};
}
Wondering what happens if I don't invoke responseObserver.onComplete() does it lead to memory leak?
An RPC is not complete/done until the response stream is also "completed" so yes there will be resource leak if you don't eventually call responseObserver.onCompleted(). In this particular example it just so happens that the response stream is terminated when the request stream is "complete" but there could be cases where the response stream is "completed" only after more processing is done or more data is sent on the response stream.
Why we don't do it for onError?
onError() is a terminating error from the stream which means the call is terminated. onError() on the response stream is not needed and most probably won't do anything.
I have implemented a custom receiver for Microsoft ASP.NET WebHooks by implementing WebHookHandler.
public class Web_WebHookHandler : WebHookHandler
{
public Web_WebHookHandler()
{
this.Receiver = CustomWebHookReceiver.ReceiverName;
}
public override Task ExecuteAsync(string generator, WebHookHandlerContext context)
{
SendNotification();
return Task.FromResult(true);
}
private void SendNotification()
{
Task.Factory.StartNew(() => {
// doing some processing
});
}
}
Whenever some event gets fired, it hits my above receiver 3 times. I have tried everything but nothing made any difference. Please help me sort it out.
Try adding bellow code in the ExecuteAsync before return. .i.e.
context.Response = new System.Net.Http.HttpResponseMessage (System.Net.HttpStatusCode.Gone);
return Task.FromResult(true);
Actually webhooks dispatcher inspects response from your receiver and retries if proper response is not sent back. So in order to tell dispatcher that request has been processed and everything is okay, you need to set context.Response and also return Task.FromResult(true).
Otherwise it will keep trying for 3 times atleast.
I'm trying to add custom http headers to a webview client (for authorization).
It seems to work in some cases, I'am able to login to a webpage without entering username and password, and I get redirected to another page. But when the page is calling other resources to get elements populated with data an error is thrown and OnReceivedHttpError is invoked. The error I'm getting is 401 unauthorized and when i look through the headers on the IWebResourceRequest i can't see the authorization headers at all.
Am I missing something or have anyone had same problems ?
Using Xamarin Forms 2.3.3.180 and targeting API 21 (Android 5.0 Lollipop), compile with Android 7.1 Nougat.
I've tried in postman to add headers to request and it works perfectly.
Renderer:
public class MyWebViewRenderer : WebViewRenderer
{
private MyWebViewClient _webViewClient;
protected override void OnElementChanged(ElementChangedEventArgs<Xamarin.Forms.WebView> e)
{
base.OnElementChanged(e);
if(_webViewClient == null)
_webViewClient = new MyWebViewClient();
Control.SetWebViewClient(_webViewClient);
Control.LongClickable = false;
Control.HapticFeedbackEnabled = false;
Control.Settings.JavaScriptEnabled = true;
var data = Encoding.GetEncoding("ISO-8859-1").GetBytes("username:password");
var base64string = Base64.EncodeToString(data, Base64Flags.NoWrap);
var headers = new Dictionary<string, string>();
headers.Add("Authorization", $"Basic {base64string}")
Control.LoadUrl(Control.Url, headers);
}
}
WebViewClient:
public override bool ShouldOverrideUrlLoading(WebView view, string url)
{
WebView.SetWebContentsDebuggingEnabled(true);
var data = Encoding.GetEncoding("ISO-8859-1").GetBytes("username:password");
var base64string = Base64.EncodeToString(data, Base64Flags.NoWrap);
var headers = new Dictionary<string, string>();
headers.Add("Authorization", $"Basic {base64string}")
view.LoadUrl(url, headers);
return true;
}
public override WebResourceResponse ShouldInterceptRequest(WebView view, IWebResourceRequest urlResource)
{
//headers does not always contains authorization header, so let's add it.
if (!urlResource.RequestHeaders.ContainsKey("authorization") && !urlResource.RequestHeaders.ContainsKey("Authorization"))
{
var data = Encoding.GetEncoding("ISO-8859-1").GetBytes("username:password");
var base64string = Base64.EncodeToString(data, Base64Flags.NoWrap);
urlResource.RequestHeaders.Add("Authorization", $"{base64string}");
}
return base.ShouldInterceptRequest(view, urlResource);
}
public override void OnReceivedHttpError(WebView view, IWebResourceRequest request, WebResourceResponse errorResponse)
{
base.OnReceivedHttpError(view, request, errorResponse);
}
If you only need the headers on the get requests, the code below will work. However POST requests are a different issue. I needed to do a similar thing (with all requests, not just GET), and all I can say is that there's not straightforward solution, at least not one that I've found (and I've tried everything short of writing my own network driver). I've tried so many methods (ShouldOverrideUrlLoading, ShouldInterceptRequest, custom LoadUrl and PostUrl etc.) and none of them give a 100% solution. There is a lot of misinformation about this so I think some clarification is needed since I've spent two days on this without success.
So here's what I've learned:
If you only need the headers in the GET requests, that's trivial. Simply create an implementation of WebViewClient and override ShouldOverrideUrlLoading like this:
[assembly: ExportRenderer(typeof(Xamarin.Forms.WebView), typeof(App.Android.HybridWebViewRenderer))]
namespace App.Android
{
public class HybridWebViewRenderer : WebViewRenderer
{
public HybridWebViewRenderer(Context context) : base(context)
{
}
protected override void OnElementChanged(ElementChangedEventArgs<Xamarin.Forms.WebView> e)
{
base.OnElementChanged(e);
Control.SetWebViewClient(new CustomWebViewClient());
}
}
public class CustomWebViewClient : WebViewClient
{
public override bool ShouldOverrideUrlLoading(Android.Webkit.WebView view, string url)
{
Dictionary<string, string> headers = new Dictionary<string, string>
{
["Name"] = "value"
};
view.LoadUrl(url, headers);
return true;
}
}
}
If, however, you need the headers in other requests (specifically POST requests) there really isn't a perfect solution. Many answers tell you to override ShouldInterceptRequest but this is unlikely to help. ShouldInterceptRequest provides an IWebResourceRequest which contains the URL of the request, the method (i.e. POST) and the headers. There are answers out there which state that adding the headers by doing request.Headers.Add("Name", "Value") is a viable solution but this is wrong. The IWebResourceRequest is not used by the WebView's internal logic so modifying it is useless!
You can write your own HTTP client in ShouldInterceptRequest which includes your own headers to perform the requests and return a WebResourceResponse object. Again, this works for GET requests, but the problem with this is that even though we can intercept a POST request, we cannot determine the content in the request as the request content is not included in the IWebResourceRequest object. As a result, we cannot accurately perform the request manually. So, unless the content of the POST request is unimportant or can somehow be fetched, this method is not viable.
An additional note on this method: returning null tells the WebView to handle the request for us. In other words 'I don't want to intercept the request'. If the return is not null however, the WebView will display whatever is in the WebResourceResponse object.
I also tried overriding the PostUrl and LoadUrl methods in the WebView itself. These methods are not called by the internal logic, so unless you are calling them yourself, this does not work.
So what can be done? There are a few hacky solutions (see github.com/KeejOow/android-post-webview) to get around this problem, but they rely on javascript and are not suitable in all cases (I have read that they don't work with forms). If you want to use them in Xamarin, you're going to need to adapt the code for C# anyway, and there is no guarantee that it will solve your problem.
I'm writing this so no one else has to waste countless hours finding a solution that doesn't really exist.
If only the Android devs had decided to include the POST content in the IWebResourceRequest object...
And apologies for the length, if you've read to this point, you're probably as desperate as I was.
I am using Zuul to proxy a strange client that sends a body as part of a GET request. There is unfortunately no way I can change the client.
With curl such a request can be sent as:
curl -XGET 'localhost:8765/kibana/index.html' -d' {"key": "value"}'
And the data is really sent in the body. On zuul side, however, when I try to read the body it is empty. Here is my prototype zuul code:
#Configuration
#ComponentScan
#EnableAutoConfiguration
#Controller
#EnableZuulProxy
public class ZuulServerApplication {
#Bean
public ZuulFilter myFilter() {
return new ZuulFilter(){
#Override
public Object run() {
RequestContext ctx = RequestContext.getCurrentContext();
HttpServletRequest request=(HttpServletRequest)ctx.getRequest();
try {
InputStream is=request.getInputStream();
String content=IOUtils.toString(is);
System.out.println("Request content:"+content);
} catch (IOException e) {
// TODO Auto-generated catch block
e.printStackTrace();
}
return null;
}
#Override
public boolean shouldFilter() {
return true;
}
#Override
public int filterOrder() {
return 10;
}
#Override
public String filterType() {
return "pre";
}};
}
public static void main(String[] args) {
new SpringApplicationBuilder(ZuulServerApplication.class).web(true).run(args);
}
}
If I send a POST request, the this code prints the request body without problem. However, if I send the above GET request, the body is not printed. Anything I can do to actually get the body sent as part of a GET request?
It seems that some underlying machinery[0], e.g. some built-in Zuul filter with lesser filter order, replaces default "raw" HttpServletRequest with HttpServletRequestWrapper which, under standard circumstances (i.e. not GET method with body), is able to handle multiple acquisition of input stream. But in the case of GET method with body HttpServletRequestWrapper seems to not proxy input stream at all.
Thus solution could be to change filterOrder e.g. to -10.
Then it works for the filter since HttpServletRequest is used - the mentioned machinery did not get to its turn and thus didn't replace HttpServletRequest with HttpServletRequestWrapper yet. But potential issue with this solution is that the filter might exhaust input stream for something else, e.g. filter with higher filter order. But since GET with body is not a good practice anyway, it might be good enough solution after all :)
[0] I've debug into this longer time ago, but did not get to exact point - thus vague definition of "the machinery".
I'm trying to write a global error handler for ASP.NET web api that is able to log the request details of requests that cause unhandled exception in my api. I've registered the below GlobalExceptionHandler class in my OWIN startup class, but I'm unable to retrieve the content of any data posted in the body of requests.
public class GlobalExecptionHander : ExceptionHandler
{
public override void Handle(ExceptionHandlerContext context)
{
var body = context.Request.Content.ReadAsStringAsync().Result;
//body here is an empty string
context.Result = new UnhandledErrorActionResult
{
Request = context.Request,
};
}
}
In my startup class
config.Services.Replace(typeof(IExceptionHandler), new GlobalExecptionHander());
Since I just came across this exact problem, I was amazed to find this question without an answer! I hope you've managed to solve the problem after all this time. I'd still like to answer this question regardless.
The thing is that by the time your GlobalExceptionHandler handles the exception, something (like Newtonsoft Json or any other request handler) has already read the contentstream of the HTTP request. When the stream is read, you cannot read it again, unless there was some way to reset that stream to its initial position...
public override void Handle(ExceptionHandlerContext context)
{
string requestContent = "";
using(System.IO.Stream stream = context.Request.Content.ReadAsStreamAsync().Result)
{
// Set the stream back to position 0...
if (stream.CanSeek)
{
stream.Position = 0;
}
// ... and read the content once again!
requestContent = context.Request.Content.ReadAsStringAsync().Result;
}
/* ..Rest of code handling the Exception.. */
}
The reason requestContent is outside that using block, is because the stream gets disposed after the block closes. You could also get rid of using and call stream.Dispose() after you've done reading the content.