Http can only get httpbin.org - http

When running the following program in elm-reactor (or using elm-make for that matter), the only webpage I have found that it actually GETs is httpbin. Otherwise, I see a Http.NetworkError, even on reliable sites such as "http://google.com" or "http://stackoverflow.com". I am utterly befuddled as to why this might be, can anyone point out my error?
module Main (..) where
import Http
import Html exposing (..)
import Effects
import Html.Events as Events
import StartApp
import Task
-- MODEL
type alias Model =
{ output : String }
type Action
= Response String
| Request String
| HTTPError Http.Error
-- UPDATE
update : Action -> Model -> ( Model, Effects.Effects Action )
update act mod =
case act of
Response str ->
( { mod | output = str }, Effects.none )
Request srv ->
let
effects =
srv
|> Http.getString
|> Task.map Response
|> flip Task.onError (Task.succeed << HTTPError)
|> Effects.task
in
( { mod | output = "GET: " ++ srv }, effects )
HTTPError err ->
( { mod
| output =
"Error: "
++ case err of
Http.Timeout ->
"Timeout"
Http.UnexpectedPayload str ->
"Unexpected payload: " ++ str
Http.BadResponse code str ->
"Bad response: " ++ toString code ++ ": " ++ str
Http.NetworkError ->
"Network error"
}
, Effects.none
)
-- VIEW
view : Signal.Address Action -> Model -> Html.Html
view address mod =
div
[]
[ div
[]
[ input
[ Events.on "input" Events.targetValue (Request >> Signal.message address) ]
[]
]
, div [] [ text mod.output ]
]
-- MAIN
app : StartApp.App Model
app =
StartApp.start
{ init = ( { output = "No requests made" }, Effects.none )
, update = update
, view = view
, inputs = []
}
main =
app.html
port tasks : Signal (Task.Task Effects.Never ())
port tasks =
app.tasks

You are most likely running into cross-origin browser restrictions. Take a look at your browser javascript console as you make requests that fail. Chrome logs an error like this:
XMLHttpRequest cannot load https://www.stackoverflow.com/. No 'Access-Control-Allow-Origin' header is present on the requested resource. Origin 'http://elm-lang.org' is therefore not allowed access.
Your working httpbin example link includes the HTTP header Access-Control-Allow-Origin: *, which is the least restrictive setting.
You can read up more on these types of issues, and the way to get around them, by looking up information on CORS (which stands for Cross Origin Http Request).

Related

Elm Http request through proxy

I am trying to make an Http request which needs to go through a proxy. I saw an answer referring how to do it with create-elm-app, but I am not using this and was wondering if there is an easy method of doing this? I have tried with the Via header (left down below to see the use I am going for), but that probably has nothing to do with it.
Do I need to use some kind of behind the scene trickery or is there a native way of doing it?
fetchUiElements : Cmd Msg
fetchUiElements =
let
baseUrl = "https://example.com/"
responseDecoder =
Json.Decode.field "data" Json.Decode.string
httpBody =
Http.jsonBody <|
Json.Encode.object
[ ("", Json.Encode.string "")
, ("", Json.Encode.string "")
]
in
Http.request
{ method = "POST"
, headers = [ Http.header "Via" "http://proxy.com:8888"]
, url = baseUrl ++ "getUiElements"
, body = httpBody
, timeout = Nothing
, tracker = Nothing
, expect = Http.expectJson FetchUiElements responseDecoder
}
Edit: For further clarification I want the same behaviour as this curl
curl -x "http://proxy:8888" -v https://endpoint/getstuff

http4s Received premature EOF

I want to implement an http4s server that receives the content from another service, processes it and return the response.
The original service uses redirects so I added the Follow redirect middleware. I also added the Logger middleware to check the logs produced.
The skeleton of the service is:
implicit val clientResource = BlazeClientBuilder[F](global).resource
val wikidataEntityUrl = "http://www.wikidata.org/entity/Q"
def routes(implicit timer: Timer[F]): HttpRoutes[F] = HttpRoutes.of[F] {
case GET -> Root / "e" / entity => {
val uri = uri"http://www.wikidata.org/entity/" / ("Q" + entity)
val req: Request[F] = Request(uri = uri)
clientResource.use { c => {
val req: Request[F] = Request(Method.GET, uri)
def cb(resp: Response[F]): F[Response[F]] = Ok(resp.bodyAsText)
val redirectClient = Logger(true,true,_ => false)(FollowRedirect[F](10, _ => true)(c))
redirectClient.fetch[Response[F]](req)(cb)
}}}}
When I try to access the service with curl as:
curl -v http://localhost:8080/e/33
The response contains the first part of the original content and finnishes with:
transfer closed with outstanding read data remaining
* Closing connection 0
Looking at the logs, they content the following line:
ERROR o.h.s.blaze.Http1ServerStage$$anon$1 - Error writing body
org.http4s.InvalidBodyException: Received premature EOF.
which suggests that there was an error receiving a premature EOF.
I found a possible answer in this issue: but the answers suggest to use deprecated methods like tohttpService.
I think I would need to rewrite the code using a streams, but I am not sure what's the more idiomatic way to do it. Some suggestions?
I received some help in the http4s gitter channel to use the toHttpApp method instead of the fetch method.
I was also suggested also to pass the client as a parameter.
The resulting code is:
case GET -> Root / "s" / entity => {
val uri = uri"http://www.wikidata.org/entity/" / ("Q" + entity)
val req: Request[F] = Request(Method.GET, uri)
val redirectClient = Logger(true,true,_ => false)(FollowRedirect[F](10, _ => true)(client))
redirectClient.toHttpApp.run(req)
}
and now it works as expected.
The toHttpApp method is intended for use in proxy servers.

Elm 0.19: How to obtain request body when receiving BadStatus with elm/http 2.0.0

elm/http 1.0.0 defined Http.Error as
type Error
= BadUrl String
| Timeout
| NetworkError
| BadStatus (Response String)
| BadPayload String (Response String)
but 2.0.0 changed it to
type Error
= BadUrl String
| Timeout
| NetworkError
| BadStatus Int
| BadBody String
When receiving BadStatus I cannot obtain the body of the request, only the status code. In the docs Evan suggests a solution for this, but I don't understand how to make it work.
If we defined our own expectJson similar to
expectJson : (Result Http.Error a -> msg) -> D.Decoder a -> Expect msg
expectJson toMsg decoder =
expectStringResponse toMsg <|
\response ->
case response of
Http.BadStatus_ metadata body ->
Err (Http.BadStatus metadata.statusCode)
...
Then we have access to the metadata and body, but how do I use them? Should I define my own myBadStatus and return that instead?
Http.BadStatus_ metadata body ->
Err (myBadStatus metadata.statusCode body)
Would this work?
What I need is to convert the following code:
myErrorMessage : Http.Error -> String
myErrorMessage error =
case error of
Http.BadStatus response ->
case Decode.decodeString myErrorDecoder response.body of
Ok err ->
err.message
Err e ->
"Failed to parse JSON response."
...
Thank you.
Edit 22/4/2019: I updated this answer for version 2.0+ of http-extras which has some API changes. Thanks to Berend de Boer for pointing this out!
The answer below gives a solution using a package I wrote (as per request), but you don't have to use the package! I wrote an entire article on how to extract detailed information from an HTTP response, it includes multiple Ellie examples that don't require the package, as well as an example that uses the package.
As Francesco mentioned, I created a package for exactly this purpose, using a similar approach described in the question: https://package.elm-lang.org/packages/jzxhuang/http-extras/latest/.
Specifically, the module to use Http.Detailed. It defines an Error type that keeps the original body around on error:
type Error body
= BadUrl String
| Timeout
| NetworkError
| BadStatus Metadata body Int
| BadBody Metadata body String
Make a request like so:
type Msg
= MyAPIResponse (Result (Http.Detailed.Error String) ( Http.Metadata, String ))
sendRequest : Cmd Msg
sendRequest =
Http.get
{ url = "/myapi"
, expect = Http.Detailed.expectString MyAPIResponse
In your update, handle the result including decoding the body when it is BadStatus:
update msg model =
case msg of
MyAPIResponse httpResponse ->
case httpResponse of
Ok ( metadata, respBody ) ->
-- Do something with the metadata if you need! i.e. access a header
Err error ->
case error of
Http.Detailed.BadStatus metadata body statusCode ->
-- Try to decode the body the body here...
...
...
Thanks the Francisco for reaching out to me about this, hopefully this answer helps anyone who faces the same problem as OP.

Request without output Swift 3

I have that code
typealias escHandler = ( URLResponse?, Data? ) -> Void
func getRequest(url : URL, _ handler : #escaping escHandler){
let session = URLSession.shared
let task = session.dataTask(with: url){ (data,response,error) in
print("--")
handler(response,data)
}
task.resume()
}
I wanna get output from this call, but, for some reason the program just finish without output.
let x = "someURL"
let url = URL(string: x)!
getRequest(url: url){ response,data in
print("Handling")
}
That's my problem, can anyone help me?
The code is correct but if by any chance you are using a "http" URL and not a "https" url then you have to make the following change to the info.plist file
Add "App Transport Security Settings" -> Add "Allow Arbitrary Loads" and set it to "YES".
Refer the screenshot below

Increasing request timeout for Network.HTTP.Conduit

I use the http-conduit library version 2.0+ to fetch the contents from a HTTP webservice:
import Network.HTTP.Conduit
main = do content <- simpleHttp "http://stackoverflow.com"
print $ content
As stated in the docs, the default timeout is 5 seconds.
Note: This question was answered by me immediately and therefore intentionally does not show further research effort.
Similar to this previous question you can't do that with simpleHttp alone. You need to use a Manager together with httpLbs in order to be able to set the timeout.
Note that you don't need to set the timeout in the manager but you can set it for each request individually.
Here is a full example that behaves like your function above, but allows you to modify the timeout:
import Network.HTTP.Conduit
import Control.Monad (liftM)
import qualified Data.ByteString.Lazy.Char8 as LB
-- | A simpleHttp alternative that allows to specify the timeout
-- | Note that the timeout parameter is in microseconds!
downloadHttpTimeout :: Manager -> String -> Int -> IO LB.ByteString
downloadHttpTimeout manager url timeout = do req <- parseUrl url
let req' = req {responseTimeout = Just timeout}
liftM responseBody $ httpLbs req' manager
main = do manager <- newManager conduitManagerSettings
let timeout = 15000000 -- Microseconds --> 15 secs
content <- downloadHttpTimeout manager "http://stackoverflow.com" timeout
print $ content
I've found the following to be a version of Uli's downloadHttpTimeout that resembles simpleHTTP more closely:
simpleHTTPWithTimeout :: Int -> Request a -> IO (Response LB.ByteString)
simpleHTTPWithTimeout timeout req =
do mgr <- newManager tlsManagerSettings
let req = req { responseTimeout = Just timeout }
httpLbs req mgr
the only difference from simpleHTTP being a slightly different return type, so to extract e.g. the response body, one uses conduit's responseBody not Network.HTTP.getResponseBody.

Resources