I want to implement an http4s server that receives the content from another service, processes it and return the response.
The original service uses redirects so I added the Follow redirect middleware. I also added the Logger middleware to check the logs produced.
The skeleton of the service is:
implicit val clientResource = BlazeClientBuilder[F](global).resource
val wikidataEntityUrl = "http://www.wikidata.org/entity/Q"
def routes(implicit timer: Timer[F]): HttpRoutes[F] = HttpRoutes.of[F] {
case GET -> Root / "e" / entity => {
val uri = uri"http://www.wikidata.org/entity/" / ("Q" + entity)
val req: Request[F] = Request(uri = uri)
clientResource.use { c => {
val req: Request[F] = Request(Method.GET, uri)
def cb(resp: Response[F]): F[Response[F]] = Ok(resp.bodyAsText)
val redirectClient = Logger(true,true,_ => false)(FollowRedirect[F](10, _ => true)(c))
redirectClient.fetch[Response[F]](req)(cb)
}}}}
When I try to access the service with curl as:
curl -v http://localhost:8080/e/33
The response contains the first part of the original content and finnishes with:
transfer closed with outstanding read data remaining
* Closing connection 0
Looking at the logs, they content the following line:
ERROR o.h.s.blaze.Http1ServerStage$$anon$1 - Error writing body
org.http4s.InvalidBodyException: Received premature EOF.
which suggests that there was an error receiving a premature EOF.
I found a possible answer in this issue: but the answers suggest to use deprecated methods like tohttpService.
I think I would need to rewrite the code using a streams, but I am not sure what's the more idiomatic way to do it. Some suggestions?
I received some help in the http4s gitter channel to use the toHttpApp method instead of the fetch method.
I was also suggested also to pass the client as a parameter.
The resulting code is:
case GET -> Root / "s" / entity => {
val uri = uri"http://www.wikidata.org/entity/" / ("Q" + entity)
val req: Request[F] = Request(Method.GET, uri)
val redirectClient = Logger(true,true,_ => false)(FollowRedirect[F](10, _ => true)(client))
redirectClient.toHttpApp.run(req)
}
and now it works as expected.
The toHttpApp method is intended for use in proxy servers.
Related
getting a strange error when trying simple gRPC implementation I.e. following the standard python example. Server seems to run OK, but get the error when I ping it with a client
grpc:
package pas;
// The PAS service definition
service PAS {
// analyze single file
rpc getPhotonRecords (PhotonRecordsRequest) returns (PhotonRecordsReply) {}
}
message PhotonRecordsRequest {
string fileName = 1;
}
message PhotonRecordsReply {
repeated uint32 PhotonRecords = 1;
}
client:
with grpc.insecure_channel("localhost:50051") as channel:
stub = pas_pb2_grpc.PASStub(channel)
msg = pas_pb2.PhotonRecordsRequest(fileName='testingFilename.flb')
response = stub.getPhotonRecords(msg)
server:
class PAS_GRPC(pas_pb2_grpc.PASServicer):
def getPhotonRecords(self, request: pas_pb2.PhotonRecordsRequest, context):
# check for required fields and error if not there or valid
# update any optional fields that the request has specified
PhotonRecordsReply = pas_pb2.PhotonRecordsReply()
PhotonRecordsReply.PhotonRecords.extend([1, 3, 7])
return pas_pb2.PhotonRecordsReply
client error:
<_InactiveRpcError of RPC that terminated with:
status = StatusCode.INTERNAL
details = "Failed to serialize response!"
server error:
TypeError: IsInitialized() missing 1 required positional argument: 'self'
Your server method getPhotonRecords returns the type:
return pas_pb2.PhotonRecordsReply
But it should return the variable you created:
return PhotonRecordsReply
You may want to use snake_case for variables to help differentiate from CamelCase class names, i.e.:
photon_records_reply = pas_pb2.PhotonRecordsReply()
...
I'm writing a "hello world" HTTP server with Hyper, however I am not able to find the Server and rt modules when trying to import them.
When invoking cargo run, then I see this error message:
26 | let server = hyper::Server::bind(&addr).serve(router);
| ^^^^^^ could not find `Server` in `hyper`
I must be missing something obvious about Rust and Hyper. What I am trying to do is writing something as dry/simple as possible with just the HTTP layer and some basic routes. I would like to include as little as possible 3rd party dependencies e.g avoiding Tokio which I think involves async behavior, but I am not sure about the context as I am new to Rust.
Looks like I must use futures, so I included this dependency and perhaps futures only work with the async reserved word (which I am not sure if it comes from Tokio or Rust itself).
What confuses me is that in the Hyper examples I do see imports like use hyper::{Body, Request, Response, Server};, so that Server thing must be there, somewhere.
These are the dependencies in Cargo.toml:
hyper = "0.14.12"
serde_json = "1.0.67"
futures = "0.3.17"
This is the code in main.rs:
use futures::future;
use hyper::service::service_fn;
use hyper::{Body, Method, Response, StatusCode};
use serde_json::json;
fn main() {
let router = || {
service_fn(|req| match (req.method(), req.uri().path()) {
(&Method::GET, "/foo") => {
let mut res = Response::new(
Body::from(json!({"message": "bar"}).to_string())
);
future::ok(res)
},
(_, _) => {
let mut res = Response::new(
Body::from(json!({"content": "route not found"}).to_string())
);
*res.status_mut() = StatusCode::NOT_FOUND;
future::ok(res)
}
})
};
let addr = "127.0.0.1:8080".parse::<std::net::SocketAddr>().unwrap();
let server = hyper::Server::bind(&addr).serve(router); // <== this line fails to compile
// hyper::rt::run(server.map_err(|e| {
// eprintln!("server error: {}", e);
// }));
}
How do I make the code above compile and run?
According to documentation, you are missing one module namespace in your call hyper::server::Server:
let server = hyper::server::Server::bind(&addr).serve(router)
In order to use server you need to activate the feature flag in cargo:
hyper = { version = "0.14.12", features = ["server"] }
I'm trying to use the Groovy HTTPBuilder library to delete some data from Firebase via a HTTP DELETE request. If I use curl, the following works
curl -X DELETE https://my.firebase.io/users/bob.json?auth=my-secret
Using the RESTClient class from HTTPBuilder works if I use it like this:
def client = new RESTClient('https://my.firebase.io/users/bob.json?auth=my-secret')
def response = client.delete(requestContentType: ContentType.ANY)
However, when I tried breaking down the URL into it's constituent parts, it doesn't work
def client = new RESTClient('https://my.firebase.io')
def response = client.delete(
requestContentType: ContentType.ANY,
path: '/users/bob.json',
query: [auth: 'my-secret']
)
I also tried using the HTTPBuilder class instead of RESTClient
def http = new HTTPBuilder('https://my.firebase.io')
// perform a POST request, expecting TEXT response
http.request(Method.DELETE, ContentType.ANY) {
uri.path = '/users/bob.json'
uri.query = [auth: 'my-secret']
// response handler for a success response code
response.success = { resp, reader ->
println "response status: ${resp.statusLine}"
}
}
But this also didn't work. Surely there's a more elegant approach than stuffing everything into a single string?
There's an example of using HttpURLClient in the tests to do a delete, which in its simplest form looks like:
def http = new HttpURLClient(url:'https://some/path/')
resp = http.request(method:DELETE, contentType:JSON, path: "destroy/somewhere.json")
def json = resp.data
assert json.id != null
assert resp.statusLine.statusCode == 200
Your example is very close to the test for the delete in a HTTPBuilder.
A few differences I see are:
Your path is absolute and not relative
Your http url path doesn't end with trailing slash
You're using content type ANY where test uses JSON. Does the target need the content type to be correct? (Probably not as you're not setting it in curl example unless it's doing some voodoo on your behalf)
Alternatively you could use apache's HttpDelete but requires more boiler plate. For a HTTP connection this is some code I've got that works. You'll have to fix it for HTTPS though.
def createClient() {
HttpParams params = new BasicHttpParams()
HttpProtocolParams.setVersion(params, HttpVersion.HTTP_1_1)
HttpProtocolParams.setContentCharset(params, "UTF-8")
params.setBooleanParameter(ClientPNames.HANDLE_REDIRECTS, true)
SchemeRegistry registry = new SchemeRegistry()
registry.register(new Scheme("http", PlainSocketFactory.getSocketFactory(), 80))
ClientConnectionManager ccm = new PoolingClientConnectionManager(registry)
HttpConnectionParams.setConnectionTimeout(params, 8000)
HttpConnectionParams.setSoTimeout(params, 5400000)
HttpClient client = new DefaultHttpClient(ccm, params)
return client
}
HttpClient client = createClient()
def url = new URL("http", host, Integer.parseInt(port), "/dyn/admin/nucleus$component/")
HttpDelete delete = new HttpDelete(url.toURI())
// if you have any basic auth, you can plug it in here
def auth="USER:PASS"
delete.setHeader("Authorization", "Basic ${auth.getBytes().encodeBase64().toString()}")
// convert a data map to NVPs
def data = [:]
List<NameValuePair> nvps = new ArrayList<NameValuePair>(data.size())
data.each { name, value ->
nvps.add(new BasicNameValuePair(name, value))
}
delete.setEntity(new UrlEncodedFormEntity(nvps))
HttpResponse response = client.execute(delete)
def status = response.statusLine.statusCode
def content = response.entity.content
I adopted the code above from a POST version, but the principle is the same.
I m using dispatch-nio library to make several Http request calls. But I cannot find an example how to handle callback responses through nio library.
And also I am unable to handle exceptions thrown by nio.Http call. Could anyone please post an example or an url where we can find some information of it.
So far I achieved:
val http = new nio.Http
withShutdown(http) {
val host = :/("http://example.org") /"api"
val withParams = host <<? Map(
"page" -> "1"
)
val future = http(withParams as_str)
while (!future.isSet) {
println("Waiting for results...")
Thread.sleep(15000)
}
future.apply()
}
}
How to set headers and onComplete? any example please
Thanks in advance.
An image gets corrupted while being retrieved (through HTTP) and then sent (through HTTP) to a database. Image's raw data is handled in String form.
The service sends a GET for an image file, receives response with the raw image data (response's body) and the Content-Type. Then, a PUT request is sent with the aforementioned request's body and Content-Type header. (The PUT request is constructed by providing the body in String) This PUT request is sent to a RESTful database (CouchDB), creating an attachment (for those unfamiliar with CouchDB an attachment acts like a static file).
Now I have the original image, which my service GETs and PUTs to a database, and this 'copy' of the original image, that I can now GET from the database. If I then `curl --head -v "[copy's url]" it has the Content-Type of the original image, but Content-Length has changed, went from 200kb to about 400kb. If I GET the 'copy' image with a browser, it is not rendered, whereas, the original renders fine. It is corrupted.
What might be the cause? My guess is that while handling the raw data as a string, my framework guesses the encoding wrong and corrupts it. I have not been able to confirm or deny this. How could I handle this raw data/request body in a safe manner, or how could I properly handle the encoding (if that proves to be the problem)?
Details: Play2 Framework's HTTP client, Scala. Below a test to reproduce:
"able to copy an image" in {
def waitFor[T](future:Future[T]):T = { // to bypass futures
Await.result(future, Duration(10000, "millis"))
}
val originalImageUrl = "http://laughingsquid.com/wp-content/uploads/grumpy-cat.jpg"
val couchdbUrl = "http://admin:admin#localhost:5984/testdb"
val getOriginal:ws.Response = waitFor(WS.url(originalImageUrl).get)
getOriginal.status mustEqual 200
val rawImage:String = getOriginal.body
val originalContentType = getOriginal.header("Content-Type").get
// need an empty doc to have something to attach the attachment to
val emptyDocUrl = couchdbUrl + "/empty_doc"
val putEmptyDoc:ws.Response = waitFor(WS.url(emptyDocUrl).put("{}"))
putEmptyDoc.status mustEqual 201
//uploading an attachment will require the doc's revision
val emptyDocRev = (putEmptyDoc.json \ "rev").as[String]
// create actual attachment/static file
val attachmentUrl = emptyDocUrl + "/0"
val putAttachment:ws.Response = waitFor(WS.url(attachmentUrl)
.withHeaders(("If-Match", emptyDocRev), ("Content-Type", originalContentType))
.put(rawImage))
putAttachment.status mustEqual 201
// retrieve attachment
val getAttachment:ws.Response = waitFor(WS.url(attachmentUrl).get)
getAttachment.status mustEqual 200
val attachmentContentType = getAttachment.header("Content-Type").get
originalContentType mustEqual attachmentContentType
val originalAndCopyMatch = getOriginal.body == getAttachment.body
originalAndCopyMatch aka "original matches copy" must beTrue // << false
}
Fails at the last 'must':
[error] x able to copy an image
[error] original matches copy is false (ApplicationSpec.scala:112)
The conversion to String is definitely going to cause problems. You need to work with the bytes as Daniel mentioned.
Looking at the source it looks like ws.Response is just a wrapper. If you get to the underlying class then there are some methods that may help you. On the Java side, someone made a commit on GitHub to expose more ways of getting the response data other than a String.
I'm not familiar with scala but something like this may work:
getOriginal.getAHCResponse.getResponseBodyAsBytes
// instead of getOriginal.body
WS.scala
https://github.com/playframework/playframework/blob/master/framework/src/play/src/main/scala/play/api/libs/ws/WS.scala
WS.java
Here you can see that Response has some new methods, getBodyAsStream() and asByteArray.
https://github.com/playframework/playframework/blob/master/framework/src/play-java/src/main/java/play/libs/WS.java