I use the http-conduit library version 2.0+ to fetch the contents from a HTTP webservice:
import Network.HTTP.Conduit
main = do content <- simpleHttp "http://stackoverflow.com"
print $ content
As stated in the docs, the default timeout is 5 seconds.
Note: This question was answered by me immediately and therefore intentionally does not show further research effort.
Similar to this previous question you can't do that with simpleHttp alone. You need to use a Manager together with httpLbs in order to be able to set the timeout.
Note that you don't need to set the timeout in the manager but you can set it for each request individually.
Here is a full example that behaves like your function above, but allows you to modify the timeout:
import Network.HTTP.Conduit
import Control.Monad (liftM)
import qualified Data.ByteString.Lazy.Char8 as LB
-- | A simpleHttp alternative that allows to specify the timeout
-- | Note that the timeout parameter is in microseconds!
downloadHttpTimeout :: Manager -> String -> Int -> IO LB.ByteString
downloadHttpTimeout manager url timeout = do req <- parseUrl url
let req' = req {responseTimeout = Just timeout}
liftM responseBody $ httpLbs req' manager
main = do manager <- newManager conduitManagerSettings
let timeout = 15000000 -- Microseconds --> 15 secs
content <- downloadHttpTimeout manager "http://stackoverflow.com" timeout
print $ content
I've found the following to be a version of Uli's downloadHttpTimeout that resembles simpleHTTP more closely:
simpleHTTPWithTimeout :: Int -> Request a -> IO (Response LB.ByteString)
simpleHTTPWithTimeout timeout req =
do mgr <- newManager tlsManagerSettings
let req = req { responseTimeout = Just timeout }
httpLbs req mgr
the only difference from simpleHTTP being a slightly different return type, so to extract e.g. the response body, one uses conduit's responseBody not Network.HTTP.getResponseBody.
Related
I want to implement an http4s server that receives the content from another service, processes it and return the response.
The original service uses redirects so I added the Follow redirect middleware. I also added the Logger middleware to check the logs produced.
The skeleton of the service is:
implicit val clientResource = BlazeClientBuilder[F](global).resource
val wikidataEntityUrl = "http://www.wikidata.org/entity/Q"
def routes(implicit timer: Timer[F]): HttpRoutes[F] = HttpRoutes.of[F] {
case GET -> Root / "e" / entity => {
val uri = uri"http://www.wikidata.org/entity/" / ("Q" + entity)
val req: Request[F] = Request(uri = uri)
clientResource.use { c => {
val req: Request[F] = Request(Method.GET, uri)
def cb(resp: Response[F]): F[Response[F]] = Ok(resp.bodyAsText)
val redirectClient = Logger(true,true,_ => false)(FollowRedirect[F](10, _ => true)(c))
redirectClient.fetch[Response[F]](req)(cb)
}}}}
When I try to access the service with curl as:
curl -v http://localhost:8080/e/33
The response contains the first part of the original content and finnishes with:
transfer closed with outstanding read data remaining
* Closing connection 0
Looking at the logs, they content the following line:
ERROR o.h.s.blaze.Http1ServerStage$$anon$1 - Error writing body
org.http4s.InvalidBodyException: Received premature EOF.
which suggests that there was an error receiving a premature EOF.
I found a possible answer in this issue: but the answers suggest to use deprecated methods like tohttpService.
I think I would need to rewrite the code using a streams, but I am not sure what's the more idiomatic way to do it. Some suggestions?
I received some help in the http4s gitter channel to use the toHttpApp method instead of the fetch method.
I was also suggested also to pass the client as a parameter.
The resulting code is:
case GET -> Root / "s" / entity => {
val uri = uri"http://www.wikidata.org/entity/" / ("Q" + entity)
val req: Request[F] = Request(Method.GET, uri)
val redirectClient = Logger(true,true,_ => false)(FollowRedirect[F](10, _ => true)(client))
redirectClient.toHttpApp.run(req)
}
and now it works as expected.
The toHttpApp method is intended for use in proxy servers.
I am looking for the path of least resistance for doing NTLM authentication in a Go HTTP request using the system credentials of the Windows user calling the application.
In C#/.NET, I would be able to achieve this through
WebRequest request = WebRequest.Create(url);
request.Credentials = CredentialCache.DefaultCredentials;
WebResponse response = request.GetResponse();
Stream receiveStream = response.GetResponseStream();
and in Python, the equivalent result can be obtained through
import win32com.client
h = win32com.client.Dispatch('WinHTTP.WinHTTPRequest.5.1')
h.SetAutoLogonPolicy(0)
h.Open('GET', url, False)
h.Send()
but I have not been able to find any resources on how to do the same thing in Go. I could of course use a library for NTLM authentication and manually provide a username/password, but the goal here is to avoid ever putting those in.
After digging into it a bit further, it looks like go-ole can be utilized to make use of WinHTTPRequest in the same way as the Python example in the question. Ignoring all error catching,
package main
import (
"fmt"
ole "github.com/go-ole/go-ole"
"github.com/go-ole/go-ole/oleutil"
)
func main() {
ole.CoInitialize(0)
defer ole.CoUninitialize()
unknown, _ := oleutil.CreateObject("WinHTTP.WinHTTPRequest.5.1")
request, _ := unknown.QueryInterface(ole.IID_IDispatch)
oleutil.CallMethod(request, "SetAutoLogonPolicy", 0)
oleutil.CallMethod(request, "Open", "GET", "http://example.com", false)
oleutil.CallMethod(request, "Send")
resp := oleutil.MustGetProperty(request, "ResponseText")
fmt.Println(resp.ToString())
}
I have a working getRequest via a Proxy :
main = do
rsp <- browse $ do
setProxy . fromJust $ parseProxy "128.199.232.117:3128"
request $ getRequest "https://www.youtube.com/watch?v=yj_wyw6Xrq4"
print $ rspBody <$> rsp
But it's htpps and so basically I get an Exception. But I foud out here that it can also work with htpps :
import Network.Connection (TLSSettings (..))
import Network.HTTP.Conduit
main :: IO ()
main = do
request <- parseUrl "https://github.com/"
let settings = mkManagerSettings (TLSSettingsSimple True False False) Nothing
manager <- newManager settings
res <- httpLbs request manager
print res
But I have no idea how to integrate this into my Proxy getRequest Code?
Could someone show me please? Thanks
Looks like you are using HTTP package it the first snippet and http-conduit in the second one.
Unfortunately HTTP doesn't support https, so your can't "integrate" the second snippet into the first one. But http-conduit supports proxies, so you can use addProxy function to set proxy host and port (not tested):
{-# LANGUAGE OverloadesStrings #-}
...
request <- do
req <- parseUrl "https://github.com/"
return $ addProxy "128.199.232.117" 3128 req
...
I'm trying to use the Groovy HTTPBuilder library to delete some data from Firebase via a HTTP DELETE request. If I use curl, the following works
curl -X DELETE https://my.firebase.io/users/bob.json?auth=my-secret
Using the RESTClient class from HTTPBuilder works if I use it like this:
def client = new RESTClient('https://my.firebase.io/users/bob.json?auth=my-secret')
def response = client.delete(requestContentType: ContentType.ANY)
However, when I tried breaking down the URL into it's constituent parts, it doesn't work
def client = new RESTClient('https://my.firebase.io')
def response = client.delete(
requestContentType: ContentType.ANY,
path: '/users/bob.json',
query: [auth: 'my-secret']
)
I also tried using the HTTPBuilder class instead of RESTClient
def http = new HTTPBuilder('https://my.firebase.io')
// perform a POST request, expecting TEXT response
http.request(Method.DELETE, ContentType.ANY) {
uri.path = '/users/bob.json'
uri.query = [auth: 'my-secret']
// response handler for a success response code
response.success = { resp, reader ->
println "response status: ${resp.statusLine}"
}
}
But this also didn't work. Surely there's a more elegant approach than stuffing everything into a single string?
There's an example of using HttpURLClient in the tests to do a delete, which in its simplest form looks like:
def http = new HttpURLClient(url:'https://some/path/')
resp = http.request(method:DELETE, contentType:JSON, path: "destroy/somewhere.json")
def json = resp.data
assert json.id != null
assert resp.statusLine.statusCode == 200
Your example is very close to the test for the delete in a HTTPBuilder.
A few differences I see are:
Your path is absolute and not relative
Your http url path doesn't end with trailing slash
You're using content type ANY where test uses JSON. Does the target need the content type to be correct? (Probably not as you're not setting it in curl example unless it's doing some voodoo on your behalf)
Alternatively you could use apache's HttpDelete but requires more boiler plate. For a HTTP connection this is some code I've got that works. You'll have to fix it for HTTPS though.
def createClient() {
HttpParams params = new BasicHttpParams()
HttpProtocolParams.setVersion(params, HttpVersion.HTTP_1_1)
HttpProtocolParams.setContentCharset(params, "UTF-8")
params.setBooleanParameter(ClientPNames.HANDLE_REDIRECTS, true)
SchemeRegistry registry = new SchemeRegistry()
registry.register(new Scheme("http", PlainSocketFactory.getSocketFactory(), 80))
ClientConnectionManager ccm = new PoolingClientConnectionManager(registry)
HttpConnectionParams.setConnectionTimeout(params, 8000)
HttpConnectionParams.setSoTimeout(params, 5400000)
HttpClient client = new DefaultHttpClient(ccm, params)
return client
}
HttpClient client = createClient()
def url = new URL("http", host, Integer.parseInt(port), "/dyn/admin/nucleus$component/")
HttpDelete delete = new HttpDelete(url.toURI())
// if you have any basic auth, you can plug it in here
def auth="USER:PASS"
delete.setHeader("Authorization", "Basic ${auth.getBytes().encodeBase64().toString()}")
// convert a data map to NVPs
def data = [:]
List<NameValuePair> nvps = new ArrayList<NameValuePair>(data.size())
data.each { name, value ->
nvps.add(new BasicNameValuePair(name, value))
}
delete.setEntity(new UrlEncodedFormEntity(nvps))
HttpResponse response = client.execute(delete)
def status = response.statusLine.statusCode
def content = response.entity.content
I adopted the code above from a POST version, but the principle is the same.
import Network.URI
import Network.HTTP
import Network.Browser
get :: URI -> IO String
get uri = do
let req = Request uri GET [] ""
resp <- browse $ do
setAllowRedirects True -- handle HTTP redirects
request req
return $ rspBody $ snd resp
main = do
case parseURI "http://cn.bing.com/search?q=hello" of
Nothing -> putStrLn "Invalid search"
Just uri -> do
body <- get uri
writeFile "output.txt" body
Here is the diff between haskell output and curl output
It's probably not a good idea to use String as the intermediate data type here, as it will cause character conversions both when reading the HTTP response, and when writing to the file. This can cause corruption if these conversions are nor consistent, as it would appear they are here.
Since you just want to copy the bytes directly, it's better to use a ByteString. I've chosen to use a lazy ByteString here, so that it does not have to be loaded into memory all at once, but can be streamed lazily into the file, just like with String.
import Network.URI
import Network.HTTP
import Network.Browser
import qualified Data.ByteString.Lazy as L
get :: URI -> IO L.ByteString
get uri = do
let req = Request uri GET [] L.empty
resp <- browse $ do
setAllowRedirects True -- handle HTTP redirects
request req
return $ rspBody $ snd resp
main = do
case parseURI "http://cn.bing.com/search?q=hello" of
Nothing -> putStrLn "Invalid search"
Just uri -> do
body <- get uri
L.writeFile "output.txt" body
Fortunately, the functions in Network.Browser are overloaded so that the change to lazy bytestrings only involves changing the request body to L.empty, replacing writeFile with L.writeFile, as well as changing the type signature of the function.