I want to parse with XmlSlurper a HTML document which I read using HTTPBuilder. Initialy I tried to do it this way:
def response = http.get(path: "index.php", contentType: TEXT)
def slurper = new XmlSlurper()
def xml = slurper.parse(response)
But it produces an exception:
java.io.IOException: Server returned HTTP response code: 503 for URL: http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd
I found a workaround to provide cached DTD files. I found a simple implementation of class which should help here:
class CachedDTD {
/**
* Return DTD 'systemId' as InputSource.
* #param publicId
* #param systemId
* #return InputSource for locally cached DTD.
*/
def static entityResolver = [
resolveEntity: { publicId, systemId ->
try {
String dtd = "dtd/" + systemId.split("/").last()
Logger.getRootLogger().debug "DTD path: ${dtd}"
new org.xml.sax.InputSource(CachedDTD.class.getResourceAsStream(dtd))
} catch (e) {
//e.printStackTrace()
Logger.getRootLogger().fatal "Fatal error", e
null
}
}
] as org.xml.sax.EntityResolver
}
My package tree looks as shown below:
I modified also a little code for parsing response, so it looks like this:
def response = http.get(path: "index.php", contentType: TEXT)
def slurper = new XmlSlurper()
slurper.setEntityResolver(org.yuri.CachedDTD.entityResolver)
def xml = slurper.parse(response)
But now I'm getting java.net.MalformedURLException. Logged DTD path from CachedDTD entityResolver is org/yuri/dtd/xhtml1-transitional.dtd and I can't get it working...
there is a HTML parse that you could use, in conjunction with XmlSlurper to address these problems
http://sourceforge.net/projects/nekohtml/
Sample useage here
http://groovy.codehaus.org/Testing+Web+Applications
I was able to solve my parsing issue by using another XmlSlurper constructor:
public XmlSlurper(boolean validating, boolean namespaceAware, boolean allowDocTypeDeclaration)
like this:
def parser = new XmlSlurper(false, false, true)
In my XML case, disabling the validation (1st parameter false) and enabling the DOCTYPE declaration (3rd parameter true) did the trick.
Note:
Related
Using Quarkus, can somebody give an example on how the server and client side code using a reactive API to download a file over http looks?
So far I tried to return a Flux of nio ByteBuffers but it seems not to be supported:
#RegisterRestClient(baseUri = "http://some-page.com")
interface SomeService {
// same interface for client and server
#GET
#Produces(MediaType.APPLICATION_OCTET_STREAM)
#Path("/somePath")
fun downloadFile(): reactor.core.publisher.Flux<java.nio.ByteBuffer>
}
Trying to return a Flux on the server-side results in the following exception:
ERROR: RESTEASY002005: Failed executing GET /somePath
org.jboss.resteasy.core.NoMessageBodyWriterFoundFailure: Could not find MessageBodyWriter for response object of type: kotlinx.coroutines.reactor.FlowAsFlux of media type: application/octet-stream
at org.jboss.resteasy.core.ServerResponseWriter.lambda$writeNomapResponse$3(ServerResponseWriter.java:124)
at org.jboss.resteasy.core.interception.jaxrs.ContainerResponseContextImpl.filter(ContainerResponseContextImpl.java:403)
at org.jboss.resteasy.core.ServerResponseWriter.executeFilters(ServerResponseWriter.java:251)
...
Here is an example how to start reactive file download with smallrye mutiny. Main function is getFile
#GET
#Path("/f/{fileName}")
#Produces(MediaType.APPLICATION_OCTET_STREAM)
public Uni<Response> getFile(#PathParam String fileName) {
File nf = new File(fileName);
log.info("file:" + nf.exists());
ResponseBuilder response = Response.ok((Object) nf);
response.header("Content-Disposition", "attachment;filename=" + nf);
Uni<Response> re = Uni.createFrom().item(response.build());
return re;
}
You can test in your local with mvn quarkus:dev and go to this url to see what files are there http://localhost:8080/hello/list/test and after that you can call this url to start download http://localhost:8080/hello/f/reactive-file-download-dev.jar
I did not check about Flux(which looks like more spring then quarkus), feel free to share your thoughts. I am just learning and answering/sharing.
As of this commit, Quarkus has out-of-the-box support for AsyncFile. So, we can stream down a file by returning an AsyncFile instance.
For example, in a JAX-RS resource controller:
// we need a Vertx instance for accessing filesystem
#Inject
Vertx vertx;
#GET
#Path("/file-data-1")
#Produces(MediaType.TEXT_PLAIN)
public Uni<Response> streamDataFromFile1()
{
final OpenOptions openOptions = (new OpenOptions()).setCreate(false).setWrite(false);
Uni<AsyncFile> uni1 = vertx.fileSystem()
.open("/srv/texts/hello.txt", openOptions);
return uni1.onItem()
.transform(asyncFile -> Response.ok(asyncFile)
.header("Content-Disposition", "attachment;filename=\"Hello.txt\"")
.build());
}
I want to implement an http4s server that receives the content from another service, processes it and return the response.
The original service uses redirects so I added the Follow redirect middleware. I also added the Logger middleware to check the logs produced.
The skeleton of the service is:
implicit val clientResource = BlazeClientBuilder[F](global).resource
val wikidataEntityUrl = "http://www.wikidata.org/entity/Q"
def routes(implicit timer: Timer[F]): HttpRoutes[F] = HttpRoutes.of[F] {
case GET -> Root / "e" / entity => {
val uri = uri"http://www.wikidata.org/entity/" / ("Q" + entity)
val req: Request[F] = Request(uri = uri)
clientResource.use { c => {
val req: Request[F] = Request(Method.GET, uri)
def cb(resp: Response[F]): F[Response[F]] = Ok(resp.bodyAsText)
val redirectClient = Logger(true,true,_ => false)(FollowRedirect[F](10, _ => true)(c))
redirectClient.fetch[Response[F]](req)(cb)
}}}}
When I try to access the service with curl as:
curl -v http://localhost:8080/e/33
The response contains the first part of the original content and finnishes with:
transfer closed with outstanding read data remaining
* Closing connection 0
Looking at the logs, they content the following line:
ERROR o.h.s.blaze.Http1ServerStage$$anon$1 - Error writing body
org.http4s.InvalidBodyException: Received premature EOF.
which suggests that there was an error receiving a premature EOF.
I found a possible answer in this issue: but the answers suggest to use deprecated methods like tohttpService.
I think I would need to rewrite the code using a streams, but I am not sure what's the more idiomatic way to do it. Some suggestions?
I received some help in the http4s gitter channel to use the toHttpApp method instead of the fetch method.
I was also suggested also to pass the client as a parameter.
The resulting code is:
case GET -> Root / "s" / entity => {
val uri = uri"http://www.wikidata.org/entity/" / ("Q" + entity)
val req: Request[F] = Request(Method.GET, uri)
val redirectClient = Logger(true,true,_ => false)(FollowRedirect[F](10, _ => true)(client))
redirectClient.toHttpApp.run(req)
}
and now it works as expected.
The toHttpApp method is intended for use in proxy servers.
I'm trying to use the Groovy HTTPBuilder library to delete some data from Firebase via a HTTP DELETE request. If I use curl, the following works
curl -X DELETE https://my.firebase.io/users/bob.json?auth=my-secret
Using the RESTClient class from HTTPBuilder works if I use it like this:
def client = new RESTClient('https://my.firebase.io/users/bob.json?auth=my-secret')
def response = client.delete(requestContentType: ContentType.ANY)
However, when I tried breaking down the URL into it's constituent parts, it doesn't work
def client = new RESTClient('https://my.firebase.io')
def response = client.delete(
requestContentType: ContentType.ANY,
path: '/users/bob.json',
query: [auth: 'my-secret']
)
I also tried using the HTTPBuilder class instead of RESTClient
def http = new HTTPBuilder('https://my.firebase.io')
// perform a POST request, expecting TEXT response
http.request(Method.DELETE, ContentType.ANY) {
uri.path = '/users/bob.json'
uri.query = [auth: 'my-secret']
// response handler for a success response code
response.success = { resp, reader ->
println "response status: ${resp.statusLine}"
}
}
But this also didn't work. Surely there's a more elegant approach than stuffing everything into a single string?
There's an example of using HttpURLClient in the tests to do a delete, which in its simplest form looks like:
def http = new HttpURLClient(url:'https://some/path/')
resp = http.request(method:DELETE, contentType:JSON, path: "destroy/somewhere.json")
def json = resp.data
assert json.id != null
assert resp.statusLine.statusCode == 200
Your example is very close to the test for the delete in a HTTPBuilder.
A few differences I see are:
Your path is absolute and not relative
Your http url path doesn't end with trailing slash
You're using content type ANY where test uses JSON. Does the target need the content type to be correct? (Probably not as you're not setting it in curl example unless it's doing some voodoo on your behalf)
Alternatively you could use apache's HttpDelete but requires more boiler plate. For a HTTP connection this is some code I've got that works. You'll have to fix it for HTTPS though.
def createClient() {
HttpParams params = new BasicHttpParams()
HttpProtocolParams.setVersion(params, HttpVersion.HTTP_1_1)
HttpProtocolParams.setContentCharset(params, "UTF-8")
params.setBooleanParameter(ClientPNames.HANDLE_REDIRECTS, true)
SchemeRegistry registry = new SchemeRegistry()
registry.register(new Scheme("http", PlainSocketFactory.getSocketFactory(), 80))
ClientConnectionManager ccm = new PoolingClientConnectionManager(registry)
HttpConnectionParams.setConnectionTimeout(params, 8000)
HttpConnectionParams.setSoTimeout(params, 5400000)
HttpClient client = new DefaultHttpClient(ccm, params)
return client
}
HttpClient client = createClient()
def url = new URL("http", host, Integer.parseInt(port), "/dyn/admin/nucleus$component/")
HttpDelete delete = new HttpDelete(url.toURI())
// if you have any basic auth, you can plug it in here
def auth="USER:PASS"
delete.setHeader("Authorization", "Basic ${auth.getBytes().encodeBase64().toString()}")
// convert a data map to NVPs
def data = [:]
List<NameValuePair> nvps = new ArrayList<NameValuePair>(data.size())
data.each { name, value ->
nvps.add(new BasicNameValuePair(name, value))
}
delete.setEntity(new UrlEncodedFormEntity(nvps))
HttpResponse response = client.execute(delete)
def status = response.statusLine.statusCode
def content = response.entity.content
I adopted the code above from a POST version, but the principle is the same.
I have a spring-boot 1.1.7 application that uses Thymeleaf for much of the UI, so the response from my controllers hasn't really been a concern. However, now I need to provide a XML response when a user submits a request via URL.
Here is a typical Request:
http://localhost:9001/remote/search?sdnName=Victoria&address=123 Maple Ave
Here is most of my gradle configuration:
project.ext {
springBootVersion = '1.1.7.RELEASE'
}
dependencies {
compile("org.springframework.boot:spring-boot-starter-web:$springBootVersion")
compile("org.springframework.boot:spring-boot-starter-thymeleaf")
compile("org.springframework.boot:spring-boot-starter-security")
compile("org.springframework.boot:spring-boot-starter-data-jpa:$springBootVersion")
compile("org.springframework.security:spring-security-web:4.0.0.M1")
compile("org.springframework.security:spring-security-config:4.0.0.M1")
compile('org.thymeleaf.extras:thymeleaf-extras-springsecurity3:2.1.1.RELEASE')
compile("org.springframework.boot:spring-boot-starter-actuator")
compile('com.fasterxml.jackson.dataformat:jackson-dataformat-xml:2.5.0')
}
And here is my controller:
#Controller
public class RemoteSearchController {
#Autowired
private SdnSearchService sdnSearchService;
#RequestMapping(value = "/remote/search", method = RequestMethod.GET, produces = MediaType.APPLICATION_XML_VALUE)
public List<Sdn> search(#ModelAttribute SdnSearch sdnSearch) {
List<Sdn> foundSdns = sdnSearchService.find( sdnSearch );
return foundSdns;
}
Here is my Object to be returned:
#Entity
public class Sdn {
#Id
private long entNum;
private String sdnName;
...
//getters & setters here
}
I am able to receive the request via REST client (such as CocoaREST) and handle it. But When I return the list of SDN i get the following exception, even though I do have Jackson & jackson-dataformat-xml on my classpath:
org.springframework.web.HttpMediaTypeNotAcceptableException: Could not find acceptable representation
at org.springframework.web.servlet.mvc.method.RequestMappingInfoHandlerMapping.handleNoMatch(RequestMappingInfoHandlerMapping.java:229)
at org.springframework.web.servlet.handler.AbstractHandlerMethodMapping.lookupHandlerMethod(AbstractHandlerMethodMapping.java:301)
at org.springframework.web.servlet.handler.AbstractHandlerMethodMapping.getHandlerInternal(AbstractHandlerMethodMapping.java:248)
at org.springframework.web.servlet.handler.AbstractHandlerMethodMapping.getHandlerInternal(AbstractHandlerMethodMapping.java:57)
at org.springframework.web.servlet.handler.AbstractHandlerMapping.getHandler(AbstractHandlerMapping.java:299)
My REST Client is including a Accept Header of "text/xml" (but in all honesty I would rather them not have to set this. Ideally any call to this Controller would always get XML, regardless of header being present).
Is there a way to handle this? I thought the Media Converters were included and just returned whatever the controller told them to?
SOLUTION:
See below for the answer I posted.
I had the exact same problem and I found the solution on Spring documentation website : here
In synthesis, I added the following dependency to the pom.xml of my project :
<dependency>
<groupId>com.fasterxml.jackson.dataformat</groupId>
<artifactId>jackson-dataformat-xml</artifactId>
</dependency>
Then I added the following code block to the class that the service had to return :
import javax.xml.bind.annotation.XmlRootElement;
#XmlRootElement
public class Greeting {...}
And it worked.
SOLUTION: I used a combination of both answers below (thank you very much!). I am posting here in case anyone else needs help.
My modified controller:
#Controller
public class RemoteSearchController {
#Autowired
private SdnSearchService sdnSearchService;
#RequestMapping(value = "/remote/search", method = RequestMethod.GET, produces = { "application/xml", "text/xml" }, consumes = MediaType.ALL_VALUE )
#ResponseBody
public SdnSearchResults search(#ModelAttribute SdnSearch sdnSearch) {
List<Sdn> foundSdns = sdnSearchService.find( sdnSearch );
SdnSearchResults results = new SdnSearchResults();
results.setSdns( foundSdns );
return results;
}
}
And on my client, I set the request headers:
Content-type: application/text
Accept: text/xml
I think ultimately the problem was that my client headers were not being set correctly, so I may not have had to make some of these changes. But I liked the idea of a SearchResults class containing a list of results:
#XmlRootElement
public class SdnSearchResults {
private List<Sdn> sdns;
...
}
It may be better to create a new class:
public class SdnSearchResult {
private List<Sdn> sdns;
...
}
Then, a slight change will be required to the existing classes as follows:
public interface SdnSearchService {
SdnSearchResult find(SdnSearch sdnSearch);
}
#Controller
public class UISearchController {
#Autowired
private SdnSearchService sdnSearchService;
#RequestMapping("/search")
public ModelAndView search(#ModelAttribute SdnSearch sdnSearch) {
return new ModelAndView("pages/search/results", "sdns", sdnSearchService.find(sdnSearch).getSdns());
}
}
Once this is done, the other controller must be coded as:
#Controller
public class RemoteSearchController {
#Autowired
private SdnSearchService sdnSearchService;
#RequestMapping("/remote/search")
#ResponseBody
public SdnSearchResult search(#RequestBody SdnSearch sdnSearch) {
return sdnSearchService.find(sdnSearch);
}
}
A quick explanation of the changes from your code:
#RequestBody will automatically deserialize the entire HTTP request body to an SdnSearch instance. External applications will typically submit the request data as HTTP body, so #RequestBody will ensure that the deserialization to Java object happens automatically.
#ResponseBody will automatically serialize the return value according to the external client's capabilities and the libraries available on the classpath. If Jackson is available on the classpath and the client has indicated that they can accept JSON, the return value will be automatically sent as JSON. If the JRE is 1.7 or higher (which means that JAXB is included with the JRE) and the client has indicated that they can accept XML, the return value will be automatically sent as XML.
List<Sdn> needs to be changed to SdnSearchResult to ensure that the application can exchange JSON, XML, RSS and ATOM formats with a single controller method, since XML (and XML based formats) require a root-tag on the output, which a List<Sdn> cannot be translated to.
Once these changes are done, fire up a REST client such as the Postman extension for Chrome and submit a request to /remote/search with the following information:
Request header Accepts set to application/json.
Request header Content-Type set to application/json.
Request body set to the JSON string { "sdnName" : "Victoria", "address" : "123 Maple Ave" }.
This will give you a JSON response.
You've marked the controller method as producing application/xml responses (produces = MediaType.APPLICATION_XML_VALUE). The request's accept header (Accept: text/xml) doesn't match so Spring determines that your search method cannot handle the request.
There are a few different ways to fix this on the server, depending on your exact requirements:
You could remove the produces attribute entirely
You could specify multiple media types: produces = { "application/xml", "text/xml" }
I am not sure about your version of Spring Boot (1.1.7.RELEASE) but I am on version 1.5.2.RELEASE and this xml conversion / serialization happens automatically without usage of any jackson dependencies as mentioned in few of the answers.
I guess that is happening because org.springframework.http.converter.xml.Jaxb2RootElementHttpMessageConverter is automatically configured since Spring Boot version 1.5.1.RELEASE & that converter uses default JAXB implementation of JRE ( so no explicit xml conversion dependency needed ) .
Second, Accept header set by clients in request decides which format the output is expected so a request mapping like below ( i.e. a single end point ) ,
#RequestMapping(method = RequestMethod.GET, value = "/remote/search", produces = {
MediaType.APPLICATION_JSON_VALUE, MediaType.APPLICATION_XML_VALUE, MediaType.TEXT_XML_VALUE })
can be used to produce an xml as well as a JSON response ( if Accept header is set as text/xml or application/xml & application/json respectively.
Note 1 : javax.xml.bind.annotation.XmlRootElement needs to be specified on root class if xml response is expected for a Java class. This is mandatory.
Note 2 : Jackson for json is already included in Spring Boot so that is not to be explicitly included for json outputs
Note 3 : Accept header - Output match off happens automatically by framework & developer doesn't have to code anything specific for that.
So in my opinion, if you only add XmlRootElement to your base class & upgrade your Spring Boot version, your server side is all set. Responsibility to set correct Accept header lies with the clients.
In addition to what Michael told in his answer, I added the following dependencies as well to pom.xml
<dependency>
<groupId>org.codehaus.woodstox</groupId>
<artifactId>woodstox-core-asl</artifactId>
<version>4.4.1</version>
</dependency>
For some reason, the jackson-dataformat-xml alone was not helping.
I also made sure that ResponseEntity is returned in the get call and removed the produces=MediaType from the RequestMapping annotation.
With these changes, I was able to get the correct data but I had to give the extension of mime type to the REST URL during get call. ie, specify explicitly like: http://localhost:8080/hello.xml or http://localhost:8080/hello.json in browser
In my case I wanted to return a formatted XML string and it was all combined into one line.
Adding produces = { "application/xml", "text/xml" } to the request mapping was enough to return the string as formatted XML (with indentation).
example:
#RequestMapping(method= RequestMethod.GET, value="/generate/{blabla}", produces = { "application/xml", "text/xml" })
public String getBlaBla(#PathVariable("param") String param) throws IOException {
}
Goodluck.
I am trying to import a CSV from a url using the feeds module, I set everything up but when I start the import I get the following error:
An AJAX HTTP error occurred. HTTP Result Code: 200 Debugging information follows. Path: XXXXX StatusText: OK ResponseText: Fatal error: Unsupported operand types in XXXXX/sites/all/modules/feeds/includes/FeedsConfigurable.inc on line 149
I have traced that error to this code, specifically the last line:
public function getConfig() {
$defaults = $this->configDefaults();
return $this->config + $defaults;
}
Any idea why I am getting this error?
I removed the default mappings because they were for an RSS feed, with GUID title etc....Could this have something to do with it? Are those required?
Also there is a bug that provokes an error if input user is set to Anonymous, as seen here, that is not my problem.
also, this is configdefaults():
* Return default configuration.
*
* #todo rename to getConfigDefaults().
*
* #return
* Array where keys are the variable names of the configuration elements and
* values are their default values.
*/
public function configDefaults() {
return array();
}
See if drupal.org/node/1213472 helps you