Hi I have a simple restlet get method which returns a static string. It looks like the following:
#Get
public String represent() {
return "mystring\r\n";
}
A low level c app is invoking this get by going into a read loop. It never receives a finish confirmation signaling it that there is no more data left to read and times out after 20 seconds. Is there code I need to send to alert the client app that no more data is coming? Or that the get is finished?
[Note: The code written below is partly based on the samples available on http://www.restlet.org ]
HTTP 1.0 and 1.1 have a header named Content-Length. Whatever is the numeric value of that header, is the length of the body of the HTTP-response. Going a step further in HTTP 1.1, there is another header name-value, Tranfer-Encoding: chunked which indicates that the response-body is divided into parts (chunks) and each part's length is mentioned in a line just before that part is delivered.(I am not including other values of Transfer-Encoding to keep this answer concised.)
If this is my restlet server:
package restletapp;
import org.restlet.Component;
import org.restlet.data.Protocol;
import org.restlet.resource.Get;
import org.restlet.resource.ServerResource;
public class RestletApp extends ServerResource {
public static void main(String[] args) throws Exception {
Component component = new Component();
component.getServers().add(Protocol.HTTP, 8182);
component.getDefaultHost().attach("/trace", RestletApp.class);
component.start();
}
#Get
public String toAtGet() {
return "Resource URI : " + getReference() + '\n'
+ "Root URI : " + getRootRef() + '\n'
+ "Routed part : " + getReference().getBaseRef() + '\n'
+ "Remaining part: " + getReference().getRemainingPart()
;
}
}
And this is my client (written using Sockets in Java. Just sends a minimal HTTP request, and prints each character of response on console.)
package restletapp;
import java.io.BufferedReader;
import java.io.IOException;
import java.io.InputStream;
import java.io.InputStreamReader;
import java.io.OutputStream;
import java.net.Socket;
import java.net.UnknownHostException;
public class Requester {
public static void main(String[] args) throws UnknownHostException, IOException {
Socket s=new Socket("localhost", 8182);
OutputStream os = s.getOutputStream();
os.write((
"GET /trace HTTP/1.1\r\n" //request
+ "host: localhost:8182\r\n" //request
+ "Connection-type: close\r\n\r\n" //request
).getBytes());
InputStream is = s.getInputStream();
for(int ch;(ch=is.read())!=-1;System.out.flush())
System.out.write(ch); //response, one char at a time.
is.close();
os.close();
s.close();
}
}
The client process never ends. But if I change my client program to this:
package restletapp;
import java.io.BufferedReader;
import java.io.IOException;
import java.io.InputStream;
import java.io.InputStreamReader;
import java.io.OutputStream;
import java.net.Socket;
import java.net.UnknownHostException;
public class Requester {
public static void main(String[] args) throws UnknownHostException, IOException {
Socket s=new Socket("localhost", 8182);
OutputStream os = s.getOutputStream();
os.write((
"GET /trace HTTP/1.1\r\n"
+ "host: localhost:8182\r\n"
+ "Connection-type: close\r\n\r\n"
).getBytes());
InputStream is = s.getInputStream();
BufferedReader br = new BufferedReader(new InputStreamReader(is));
int bytesRead=0;
int contentLength=0;
//response headers.
for(String line;!(line=br.readLine()).isEmpty();System.out.flush()){
System.out.println(line);
String[] tokens = line.split(":| ");
if(tokens[0].equalsIgnoreCase("content-length")){
contentLength=Integer.parseInt(tokens[2]);
}
}
//response separator, between headers and body.
System.out.println();
//response body.
while(bytesRead<contentLength){
System.out.write(br.read());
System.out.flush();
bytesRead++;
}
is.close();
os.close();
s.close();
}
}
In the second version of Requester, you can see that the connection is closed by the client when response body's content-length-number of characters are read.
This is what i get using curl:
command line $ curl -i "http://localhost:8182/trace"
HTTP/1.1 200 OK
Date: Fri, 14 Jun 2013 11:54:32 GMT
Server: Restlet-Framework/2.0.15
Vary: Accept-Charset, Accept-Encoding, Accept-Language, Accept
Content-Length: 148
Content-Type: text/plain; charset=UTF-8
Resource URI : http://localhost:8182/trace
Root URI : http://localhost:8182/trace
Routed part : http://localhost:8182/trace
Remaining part:
command line $
You can see, that curl exits after reading the content.
Related
I'm trying to implement filter for logging requests and responses in Spring MVC application.
I use the following code:
#Component
public class LoggingFilter extends OncePerRequestFilter {
private static final Logger LOGGER = LoggerFactory.getLogger(LoggingFilter.class);
#Override
protected void doFilterInternal(HttpServletRequest request, HttpServletResponse response, FilterChain filterChain) throws ServletException, IOException {
ContentCachingRequestWrapper requestWrapper = new ContentCachingRequestWrapper(request);
ContentCachingResponseWrapper responseWrapper = new ContentCachingResponseWrapper(response);
LOGGER.debug(REQUEST_MESSAGE_FORMAT, requestWrapper.getRequestURI(), requestWrapper.getMethod(), requestWrapper.getContentType(),
new ServletServerHttpRequest(requestWrapper).getHeaders(), IOUtils.toString(requestWrapper.getInputStream(), UTF_8));
filterChain.doFilter(requestWrapper, responseWrapper);
LOGGER.debug(RESPONSE_MESSAGE_FORMAT, responseWrapper.getStatus(), responseWrapper.getContentType(),
new ServletServerHttpResponse(responseWrapper).getHeaders(), IOUtils.toString(responseWrapper.getContentInputStream(), UTF_8));
}
}
So, I get my request and respone logged as expected. Here are the logs:
2016-10-08 19:10:11.212 DEBUG 11072 --- [qtp108982313-19] by.kolodyuk.logging.LoggingFilter
----------------------------
ID: 1
URI: /resources/1
Http-Method: GET
Content-Type: null
Headers: {User-Agent=[curl/7.41.0], Accept=[*/*], Host=[localhost:9015]}
Body:
--------------------------------------
2016-10-08 19:10:11.277 DEBUG 11072 --- [qtp108982313-19] by.kolodyuk.logging.LoggingFilter
----------------------------
ID: 1
Response-Code: 200
Content-Type: application/json;charset=UTF-8
Headers: {}
Body: {"id":"1"}
--------------------------------------
However, the empty response is returned. Here's the output from curl:
$ curl http://localhost:9015/resources/1 --verbose
* Trying ::1...
* Connected to localhost (::1) port 9015 (#0)
> GET /resources/1 HTTP/1.1
> User-Agent: curl/7.41.0
> Host: localhost:9015
> Accept: */*
>
< HTTP/1.1 200 OK
< Date: Sat, 08 Oct 2016 17:10:11 GMT
< Content-Type: application/json;charset=UTF-8
< Content-Length: 0
<
* Connection #0 to host localhost left intact
Any ideas?
Thanks
After couple of hours of struggling, I've finally found the solution.
In short, ContentCachingResponseWrapper.copyBodyToResponse() should be called in the end of the filter method.
ContentCachingResponseWrapper caches the response body by reading it from response output stream. So, the stream becomes empty. To write response back to the output stream ContentCachingResponseWrapper.copyBodyToResponse() should be used.
Finally solved the problem. Here is the perfect solution:
import com.fasterxml.jackson.databind.JsonNode;
import com.fasterxml.jackson.databind.ObjectMapper;
import org.apache.commons.io.IOUtils;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.http.*;
import org.springframework.stereotype.Component;
import org.springframework.web.filter.OncePerRequestFilter;
import org.springframework.web.util.ContentCachingRequestWrapper;
import org.springframework.web.util.ContentCachingResponseWrapper;
import javax.servlet.FilterChain;
import javax.servlet.ServletException;
import javax.servlet.http.HttpServletRequest;
import javax.servlet.http.HttpServletResponse;
import java.io.BufferedReader;
import java.io.IOException;
import java.io.InputStreamReader;
import java.net.URI;
import java.util.Enumeration;
import java.util.Map;
import static java.nio.charset.StandardCharsets.UTF_8;
import static net.logstash.logback.marker.Markers.appendFields;
#Component
public class LoggingFilter extends OncePerRequestFilter {
private static final Logger LOGGER = LoggerFactory.getLogger(LoggingFilter.class);
#Autowired
private ObjectMapper objectMapper;
#Override
protected void doFilterInternal(HttpServletRequest httpServletRequest, HttpServletResponse httpServletResponse, FilterChain filterChain) throws ServletException, IOException {
ContentCachingRequestWrapper requestWrapper = new ContentCachingRequestWrapper(httpServletRequest);
ContentCachingResponseWrapper responseWrapper = new ContentCachingResponseWrapper(httpServletResponse);
filterChain.doFilter(requestWrapper, responseWrapper);
String requestUrl = requestWrapper.getRequestURL().toString();
HttpHeaders requestHeaders = new HttpHeaders();
Enumeration headerNames = requestWrapper.getHeaderNames();
while (headerNames.hasMoreElements()) {
String headerName = (String) headerNames.nextElement();
requestHeaders.add(headerName, requestWrapper.getHeader(headerName));
}
HttpMethod httpMethod = HttpMethod.valueOf(requestWrapper.getMethod());
Map<String, String[]> requestParams = requestWrapper.getParameterMap();
String requestBody = IOUtils.toString(requestWrapper.getInputStream(),UTF_8);
JsonNode requestJson = objectMapper.readTree(requestBody);
RequestEntity<JsonNode> requestEntity = new RequestEntity<>(requestJson,requestHeaders, httpMethod, URI.create(requestUrl));
LOGGER.info(appendFields(requestEntity),"Logging Http Request");
HttpStatus responseStatus = HttpStatus.valueOf(responseWrapper.getStatusCode());
HttpHeaders responseHeaders = new HttpHeaders();
for (String headerName : responseWrapper.getHeaderNames()) {
responseHeaders.add(headerName, responseWrapper.getHeader(headerName));
}
String responseBody = IOUtils.toString(responseWrapper.getContentInputStream(), UTF_8);
JsonNode responseJson = objectMapper.readTree(responseBody);
ResponseEntity<JsonNode> responseEntity = new ResponseEntity<>(responseJson,responseHeaders,responseStatus);
LOGGER.info(appendFields(responseEntity),"Logging Http Response");
responseWrapper.copyBodyToResponse();
}
}
The pattern I like to use is to split this into 2 filters, one for extracting the raw body and another one to do the logging - feels a more SRP.
#Slf4j // lombok logging
#Component // spring loads filter into it's filter chain
#Order(1) // Best if this goes first (or early in filter chain)
public class CachingBodyFilter implements Filter {
#Override
public void doFilter(ServletRequest req, ServletResponse res, FilterChain chain) {
ContentCachingRequestWrapper reqWrapper = new ContentCachingRequestWrapper((HttpServletRequest) req);
ContentCachingResponseWrapper resWrapper = new ContentCachingResponseWrapper((HttpServletResponse) res);
try {
chain.doFilter(reqWrapper, resWrapper);
resWrapper.copyBodyToResponse(); // Necessary (see answer by StasKolodyuk above)
} catch (IOException | ServletException e) {
log.error("Error extracting body", e);
}
}
}
And then we create another filter to do the logging part.
#Slf4j
#Component
#Order(2) // This needs to come after `CachingBodyFilter`
public class PayloadLogFilter implements Filter {
public void doFilter(ServletRequest req, ServletResponse res, FilterChain chain) throws IOException, ServletException {
chain.doFilter(req, res);
if (req instanceof ContentCachingRequestWrapper) {
ContentCachingRequestWrapper reqWrapper = (ContentCachingRequestWrapper) req;
String payload = new String (reqWrapper.getContentAsByteArray(), "utf-8");
log.debug("Request [ {} ] has payload [ {} ]", reqWrapper.getRequestURI(), payload);
}
}
}
A nice advantage of splitting these up is that other classes (e.g. a Spring AOP interceptor or a Spring controller) can also access / use the HTTP body.
I am trying to test a Servlet with special character which is deployed in jetty 9.
I am posting a String with a single quote character to the Servlet. Though I have mentioned to use UTF-8 character encoding in the Servlet , the Servlet cannot print the single quote character. I don't know what's wrong in the following code :
import javax.servlet.ServletException;
import javax.servlet.http.HttpServlet;
import javax.servlet.http.HttpServletRequest;
import javax.servlet.http.HttpServletResponse;
import java.io.BufferedReader;
import java.io.IOException;
import java.io.InputStreamReader;
public class SpecialCharacterServlet extends HttpServlet {
protected void doPost(HttpServletRequest request, HttpServletResponse response) throws ServletException, IOException {
request.setCharacterEncoding("UTF-8");
BufferedReader in = new BufferedReader(
new InputStreamReader(request.getInputStream()));
String inputLine;
while ((inputLine = in.readLine()) != null) {
System.out.println(inputLine);
}
in.close();
}
protected void doGet(HttpServletRequest request, HttpServletResponse response) throws ServletException, IOException {
}
}
The main method
public static void main(String s[]) {
try {
HttpURLConnection conn = (HttpURLConnection) new URL("http://localhost:8080/test/SpecialCharacterServlet").openConnection();
conn.setDoInput(true);
conn.setDoOutput(true);
conn.setUseCaches(false);
conn.setRequestMethod("POST");
conn.setRequestProperty("Connection", "Keep-Alive");
conn.setRequestProperty("Content-Type", "multipart/form-data;boundary=*****");
String str = "Hello ‘World’";
DataOutputStream dos = new DataOutputStream(conn.getOutputStream());
dos.writeBytes(str);
dos.flush();
dos.close();
InputStream is = conn.getInputStream();
byte[] content = new byte[is.available()];
is.read(content);
}catch(Exception e) {
System.out.println(e);
}
}
The output does not show the single quoted character , the output is Hello World , it's some block in there .
You need to remove any action where character sets are implied but not defined.
Ensure that you're Main.class is UTF-8 encoded.
Ensure that javac is aware that Main.class is UTF-8 encoded. Most IDE's do this for you but it requires you to set the encoding in the file properties
In main() wrap the OutputStream with OutputStreamWriter with an explicit charset declaration. This ensures everything written through the OutputStreamWriter is in the correct charset:
osw = OutputStreamWriter(conn.getOutputStream(), "UTF-8");
osw.write(str);
In your servlet, tell your inputStream to decode from "UTF-8":
BufferedReader in = new BufferedReader(
new InputStreamReader(request.getInputStream(), "UTF-8"));
System.out is also subject to charset conversion. This is particularly troublesome on DOS consoles. Instead, write the result to a text file and check the results in a good text editor, such as Notepad++. Again, when writing to the text file, set the character set in the constructor of your writer.
I am working on Ubuntu 12.04. This is my simple code for implementing HTTP GET method using URLConnection.
import java.io.BufferedReader;
import java.io.DataOutputStream;
import java.io.InputStreamReader;
import java.net.HttpURLConnection;
import java.net.URL;
import javax.net.ssl.HttpsURLConnection;
public class HttpURLConnectionExample {
private final String USER_AGENT = "Mozilla/5.0";
public static void main(String[] args) throws Exception {
HttpURLConnectionExample http = new HttpURLConnectionExample();
System.out.println("Testing 1 - Send Http GET request");
http.sendGet();
}
// HTTP GET request
private void sendGet() throws Exception {
String url = "https://www.google.com/search?q=flower";
URL obj = new URL(url);
HttpURLConnection con = (HttpURLConnection) obj.openConnection();
// optional default is GET
con.setRequestMethod("GET");
//add request header
con.setRequestProperty("User-Agent", USER_AGENT);
int responseCode = con.getResponseCode();
System.out.println("\nSending 'GET' request to URL : " + url);
System.out.println("Response Code : " + responseCode);
BufferedReader in = new BufferedReader(
new InputStreamReader(con.getInputStream()));
String inputLine;
StringBuffer response = new StringBuffer();
while ((inputLine = in.readLine()) != null) {
response.append(inputLine);
}
in.close();
//print result
System.out.println(response.toString());
}
}
But when i compile and run this code from ubuntu terminal, the output of this code does not give the content of the page specified by the URL. Rather it gives the following output
Testing 1 - Send Http GET request
Sending 'GET' request to URL : http://www.google.com/search?q=flower
Response Code : 307
<!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN"><html><head><title>307 Temporary Redirect</title></head><body><h1>Temporary Redirect</h1><p>The document has moved here.</p><hr><address>Apache/2.2.22 (Fedora) Server at www.google.com Port 80</address></body></html>
This issue holds for any URL I specify in the code. Moreover, I tried to access web content using telnet client like
telnet www.google.com 80
GET /
and it gives the similar result not only for www.google.com but for every URL.
I am a student at IIT Bombay and may be it has something to do with https://ifwb.iitb.ac.in.
I also want to stick to java.net and not apache httpclient. So help me out of this.
It seems you're rejected by the server due incomplete request. It's a good idea to use any sniffer like Fiddler or Wireshark to "learn by example": compare your requests and requests from particular software like browsers.
Below is an excerpt from Wireshark dump, how IE10 sends GET request to interested URL. As you can see, there are various fields that describe capabilities and expectations of your client side so queried server can return the answer in most suitable and consumable form. Consult with Google/RFC to see the meaning of each parameter passed in:
GET /search?q=flower HTTP/1.1
Accept: text/html, application/xhtml+xml, /
Accept-Language: en-US
User-Agent: Mozilla/5.0 (compatible; MSIE 10.0; Windows NT 6.1; WOW64;
Trident/6.0)
Accept-Encoding: gzip, deflate
Host: www.google.com
DNT: 1
Connection: Keep-Alive
Cookie: [some private info here]
I am trying to write a http client which connects to a kerberos enabled tomcat(tested to be correct using browsers). It first gets the response code (which will be 401) and as according continue with its work.
The code is
import java.io.BufferedReader;
import java.io.InputStream;
import java.io.InputStreamReader;
import java.net.*;
public class SampleHTTP2 {
static final String kuser = "correctusername"; // your account name
static final String kpass = "correctpassword"; // your password for the account
static class MyAuthenticator extends Authenticator {
public PasswordAuthentication getPasswordAuthentication() {
//System.out.println("I am reaching here");
// I haven't checked getRequestingScheme() here, since for NTLM
// and Negotiate, the username and password are all the same.
System.err.println("Feeding username and password for "
+ getRequestingScheme());
return (new PasswordAuthentication(kuser, kpass.toCharArray()));
}
}
public static void main(String[] args) throws Exception {
URL url = new URL("http://mycompname:6008/examples/");
HttpURLConnection h1 = (HttpURLConnection) url.openConnection();
int rescode = h1.getResponseCode();
System.out.println(rescode);
System.setProperty("sun.security.krb5.debug", "true");
System.setProperty("java.security.auth.login.config", "C:\\login2.conf");
System.setProperty("javax.security.auth.useSubjectCredsOnly","false");
System.setProperty("java.security.krb5.conf", "C:\\krb5.ini");
if(rescode == 401){
Authenticator.setDefault(new MyAuthenticator());
URL url2 = new URL("http://mycompname/examples/");
URLConnection h2 = url2.openConnection();
InputStream ins2 = h2.getInputStream();
BufferedReader reader = new BufferedReader(new InputStreamReader(ins2));
String str;
while((str = reader.readLine()) != null)
System.out.println(str);
}
}
}
Now when i comment the line:-
int rescode = h1.getResponseCode();
and put if(true) instead of if(rescode ==401), it works.
I am not sure what is going wrong. getResponseCode() internally calls getinputStream and thus I have used a separate url connection. Even still it does not work
P.S - Server is perfectly set up and the Authenticator class is also correct.
Using Clojure 1.4 (Java 7) and the clj-http (0.6.0) library.
When doing a get request the Content-Length header is automatically included and set to zero. Some servers (lighttpd for instance) don't like this and respond with Bad Request. Is it possible to remove the said header or make the library not include it by default? I couldn't find anything relevant in the docs and googling gave me only this, which doesn't really help.
If I try:
(client/get "http://thepiratebay.se" {:debug true})
I get:
Request: nil
{:scheme :http,
:http-url "http://thepiratebay.se",
:request-method :get,
:query-string nil,
:uri "",
:server-name "thepiratebay.se",
:headers {"accept-encoding" "gzip, deflate"},
:debug true,
:body-type nil,
:server-port nil,
:body nil,
:user-info nil}
HttpRequest:
{:requestLine #<BasicRequestLine GET http://thepiratebay.se HTTP/1.1>,
:protocolVersion #<HttpVersion HTTP/1.1>,
:params
#<BasicHttpParams org.apache.http.params.BasicHttpParams#5b14a306>,
:method "GET",
:entity nil,
:class
clj_http.core.proxy$org.apache.http.client.methods.HttpEntityEnclosingRequestBase$0,
:allHeaders
[#<BasicHeader Connection: close>,
#<BasicHeader accept-encoding: gzip, deflate>],
:aborted false,
:URI #<URI http://thepiratebay.se>}
Which yields a 400 error. I tried to reproduce it in Java, using Apache HttpClient directly:
import org.apache.http.client.methods.HttpGet;
import org.apache.http.impl.client.DefaultHttpClient;
import org.apache.http.client.HttpClient;
import org.apache.http.HttpResponse;
import org.apache.http.Header;
public class Get {
public static void main(String args[]) throws Exception {
HttpClient httpclient = new DefaultHttpClient();
HttpGet httpget = new HttpGet(args[0]);
httpget.addHeader("Connection", "close");
httpget.addHeader("accept-encoding", "gzip, deflate");
Header[] headers = httpget.getAllHeaders();
for (Header h : headers) {
System.out.println(h.getName() + ", " + h.getValue());
}
System.out.println();
HttpResponse response = httpclient.execute(httpget);
System.out.println(response);
}
}
However, this works fine. My guess is that before calling HttpClient, clj-http is doing something that forces an empty body in the response, so HttpClient sets the header Content-Length to 0. The header is not set by clj-http if you look at the source. I would file this as an issue for clj-http.
https://github.com/dakrone/clj-http/issues