Encoding problem with GET requests in Haskell - http

I'm trying to get some Json data from a Jira server using Haskell. I'm counting this as "me having problems with Haskell" rather than encodings or Jira because my problem is when doing this in Haskell.
The problem occurs when the URL (or query) has plus signs. After building my request for theproject+order+by+created, Haskell prints it as:
Request {
host = "myjiraserver.com"
port = 443
secure = True
requestHeaders = [("Content-Type","application/json"),("Authorization","<REDACTED>")]
path = "/jira/rest/api/2/search"
queryString = "?jql=project%3Dtheproject%2Border%2Bby%2Bcreated"
method = "GET"
proxy = Nothing
rawBody = False
redirectCount = 10
responseTimeout = ResponseTimeoutDefault
requestVersion = HTTP/1.1
}
But the request fails with this response:
- 'Error in the JQL Query: The character ''+'' is a reserved JQL character. You must
enclose it in a string or use the escape ''\u002b'' instead. (line 1, character
21)'
So it seems like Jira didn't like Haskell's %2B. Do you have any suggestions on what I can do to fix this, or any resources that might be helpful? The same request sans the +order+by+created part is successful.
The code (patched together from these examples):
{-# LANGUAGE OverloadedStrings #-}
import Data.Aeson
import qualified Data.ByteString.Char8 as S8
import qualified Data.Yaml as Yaml
import Network.HTTP.Simple
import System.Environment (getArgs)
-- auth' is echo -e "username:passwd" | base64
foo urlBase proj' auth' = do
let proj = S8.pack (proj' ++ "+order+by+created")
auth = S8.pack auth'
request'' <- parseRequest urlBase
let request'
= setRequestMethod "GET"
$ setRequestPath "/jira/rest/api/2/search"
$ setRequestHeader "Content-Type" ["application/json"]
$ request''
request
= setRequestQueryString [("jql", Just (S8.append "project=" proj))]
$ setRequestHeader "Authorization" [S8.append "Basic " auth]
$ request'
return request
main :: IO ()
main = do
args <- getArgs
case args of
(urlBase:proj:auth:_) -> do
request <- foo urlBase proj auth
putStrLn $ show request
response <- httpJSON request
S8.putStrLn $ Yaml.encode (getResponseBody response :: Value) -- apparently this is required
putStrLn ""
_ -> putStrLn "usage..."
(If you know a simpler way to do the above then I'd take such suggestions as well, I'm just trying to do something analogous to this Python:
import requests
import sys
if len(sys.argv) >= 4:
urlBase = sys.argv[1]
proj = sys.argv[2]
auth = sys.argv[3]
urlBase += "/jira/rest/api/2/search?jql=project="
proj += "+order+by+created"
h = {}
h["content-type"] = "application/json"
h["authorization"] = "Basic " + auth
r = requests.get(urlBase + proj, headers=h)
print(r.json())
)

project+order+by+created is the URL-encoded string for the actual request project order by created (with spaces instead of +). The function setRequestQueryString expects a raw request (with spaces, not URL-encoded), and URL-encodes it.
The Python script you give for comparison essentially does the URL-encoding by hand.
So the fix is to put the raw request in proj:
foo urlBase proj' auth' = do
let proj = S8.pack (proj' ++ " order by created") -- spaces instead of +
...

Related

How to make a request to an IPv6 address using the http-client package in haskell?

I've been trying to make a request to an IPv6 address using the parseRequest function from Network.HTTP.Client (https://hackage.haskell.org/package/http-client-0.7.10/docs/Network-HTTP-Client.html) package as follows:
request <- parseRequest "http://[2001:0db8:85a3:0000:0000:8a2e:0370:7334]"
Instead of parsing it as an address/addrInfo, it is parsed as a hostname and throws the error: does not exist (Name or service not known). As a next step, I tried pointing a domain to the same IPv6 address and then using the domain name in parseRequest, then it successfully resolves that into the IPv6 address and makes the request. Is there some other way I can directly use the IPv6 address to make the request using the http-client package?
PS: I also tried without square brackets around the IP address, in this case the error is Invalid URL:
request <- parseRequest "http://2001:0db8:85a3:0000:0000:8a2e:0370:7334"
More context:
For an IPv4 address, the getAddrInfo function generates the address as:
AddrInfo {addrFlags = [AI_NUMERICHOST], addrFamily = AF_INET, addrSocketType = Stream, addrProtocol = 6, addrAddress = 139.59.90.1:80, addrCanonName = Nothing}
whereas for IPv6 address(inside the square brackets format):
AddrInfo {addrFlags = [AI_ADDRCONFIG], addrFamily = AF_UNSPEC, addrSocketType = Stream, addrProtocol = 6, addrAddress = 0.0.0.0:0, addrCanonName = Nothing}
and the error prints as:
(ConnectionFailure Network.Socket.getAddrInfo (called with preferred socket type/protocol: AddrInfo {addrFlags = [AI_ADDRCONFIG], addrFamily = AF_UNSPEC, addrSocketType = Stream, addrProtocol = 6, addrAddress = 0.0.0.0:0, addrCanonName = Nothing}, host name: Just "[2001:0db8:85a3:0000:0000:8a2e:0370:7334]", service name: Just "80"): does not exist (Name or service not known))
When a literal IPv6 address is used in a URL, it should be surrounded by square brackets (as per RFC 2732) so the colons in the literal address aren't misinterpreted as some kind of port designation.
When a literal IPv6 address is resolved using the C library function getaddrinfo (or the equivalent Haskell function getAddrInfo), these functions are not required to handle these extra square brackets, and at least on Linux they don't.
Therefore, it's the responsibility of the HTTP client library to remove the square brackets from the hostname extracted from the URL before resolving the literal IPv6 address using getaddrinfo, and the http-client package doesn't do this, at least as of version 0.7.10. So, this is a bug, and I can see you've appropriately filed a bug report.
Unfortunately, I don't see an easy way to work around the issue. You can manipulate the Request after parsing to remove the square brackets from the host field, like so:
{-# LANGUAGE OverloadedStrings #-}
import Data.ByteString (ByteString)
import qualified Data.ByteString as BS
import Network.HTTP.Client
import Network.HTTP.Types.Status (statusCode)
main :: IO ()
main = do
manager <- newManager defaultManagerSettings
request <- parseRequest "http://[::1]"
let request' = request { host = removeBrackets (host request) }
response <- httpLbs request' manager
print response
removeBrackets :: ByteString -> ByteString
removeBrackets bs =
case BS.stripPrefix "[" bs >>= BS.stripSuffix "]" of
Just bs' -> bs'
Nothing -> bs
The problem with this is that it also removes the square brackets from the value in the Host header, so the HTTP request will contain the header:
Host: ::1
instead of the correct
Host: [::1]
which may or may not cause problems, depending on the web server at the other end.
You could try using a patched http-client package. The following patch against version 0.7.10 seems to work, but I didn't test it very extensively:
diff --git a/Network/HTTP/Client/Connection.hs b/Network/HTTP/Client/Connection.hs
index 0e329cd..719822e 100644
--- a/Network/HTTP/Client/Connection.hs
+++ b/Network/HTTP/Client/Connection.hs
## -15,6 +15,7 ## module Network.HTTP.Client.Connection
import Data.ByteString (ByteString, empty)
import Data.IORef
+import Data.List (stripPrefix, isSuffixOf)
import Control.Monad
import Network.HTTP.Client.Types
import Network.Socket (Socket, HostAddress)
## -158,8 +159,12 ## withSocket :: (Socket -> IO ())
withSocket tweakSocket hostAddress' host' port' f = do
let hints = NS.defaultHints { NS.addrSocketType = NS.Stream }
addrs <- case hostAddress' of
- Nothing ->
- NS.getAddrInfo (Just hints) (Just host') (Just $ show port')
+ Nothing -> do
+ let port'' = Just $ show port'
+ case ip6Literal host' of
+ Just lit -> NS.getAddrInfo (Just hints { NS.addrFlags = [NS.AI_NUMERICHOST] })
+ (Just lit) port''
+ Nothing -> NS.getAddrInfo (Just hints) (Just host') port''
Just ha ->
return
[NS.AddrInfo
## -173,6 +178,11 ## withSocket tweakSocket hostAddress' host' port' f = do
E.bracketOnError (firstSuccessful addrs $ openSocket tweakSocket) NS.close f
+ where
+ ip6Literal h = case stripPrefix "[" h of
+ Just rest | "]" `isSuffixOf` rest -> Just (init rest)
+ _ -> Nothing
+
openSocket tweakSocket addr =
E.bracketOnError
(NS.socket (NS.addrFamily addr) (NS.addrSocketType addr)

How to download JIRA attachment files with Python

I want to download attachment files of an issue in JIRA Python.
I use jira python lib ,you can use pip install JIRA
# -- coding: UTF-8 --
from jira import JIRA
import requests
url = 'https://jira.1234.com'
jira = JIRA(server=url, basic_auth=('admin', 'password'))
attachment=jira.attachment(12345) #12345 is attachment_key
image = attachment.get()
with open("Image.png", 'wb') as f:
f.write(image)
JIRA exposes its REST services and through that and some python you can download any attachment.
It worked for me like this (you'll need to adjust the variables):
#!/usr/bin/python
# miguel ortiz
# Requests module: http://docs.python-requests.org/en/latest/
# Documentation: <url>
#----------------------------------------------------------------Modules
import sys
import csv, json
import requests
#----------------------------------------------------------------Variables
myTicket= sys.argv[1] # Your ticket: ABC-123
user = 'miguel' # JIRA user
pasw = 'password' # JIRA password
jiraURL = 'https://yourinstance.jira.com/rest/api/latest/issue/'
fileName = 'my_attached_file' # In this case we'll be looking for a specific file in the attachments
attachment_final_url="" # To validate if there are or not attachments
def main() :
print '\n\n [ You are checking ticket: ' + myTicket+ ' ]\n'
# Request Json from JIRA API
r = requests.get(jiraURL+myTicket, auth=(user, pasw),timeout=5)
# status of the request
rstatus = r.status_code
# If the status isn't 200 we leave
if not rstatus == 200 :
print 'Error accesing JIRA:' + str(rstatus)
exit()
else:
data = r.json()
if not data['fields']['attachment'] :
status_attachment = 'ERROR: Nothing attached, attach a file named: ' + fileName
attachment_final_url=""
else:
for i in data['fields']['attachment'] :
if i['filename'] == fileName :
attachment_final_url = i['content']
status_attachment_name = 'OK: The desired attachment exists: ' + fileName
attachment_name = False
attachment_amount = False
attachment_files = False
break
else :
attachment_files = False
status_attachment_name = + 'ERROR: None of the files has the desired name '
attachment_final_url=""
attachment_name = True
attachment_amount = True
continue
if attachment_final_url != "" :
r = requests.get(attachment_final_url, auth=(user, pasw), stream=True)
with open(fileName, "wb") as f:
f.write(r.content.decode('iso-8859-1').encode('utf8'))
f.close()
else:
print status_attachment
if __name__ == "__main__" :
main()
If you do not understand the code I've detailed it better in my blog.
EDIT: Be careful, in JIRA you can add many files with the same name.

Download pdf file from wikipedia

Wikipedia provides a link (left side on Print/export) on every article to download the article as pdf. I wrote a small Haskell script which first gets the Wikipedia link and output the rendering link. When I am giving the rendering url as input, I am getting empty tags but the same url in browser provides download link.
Could someone please tell me how to solve this problem? Formated code on ideone.
import Network.HTTP
import Text.HTML.TagSoup
import Data.Maybe
parseHelp :: Tag String -> Maybe String
parseHelp ( TagOpen _ y ) = if any ( \( a , b ) -> b == "Download a PDF version of this wiki page" ) y
then Just $ "http://en.wikipedia.org" ++ snd ( y !! 0 )
else Nothing
parse :: [ Tag String ] -> Maybe String
parse [] = Nothing
parse ( x : xs )
| isTagOpen x = case parseHelp x of
Just s -> Just s
Nothing -> parse xs
| otherwise = parse xs
main = do
x <- getLine
tags_1 <- fmap parseTags $ getResponseBody =<< simpleHTTP ( getRequest x ) --open url
let lst = head . sections ( ~== "<div class=portal id=p-coll-print_export>" ) $ tags_1
url = fromJust . parse $ lst --rendering url
putStrLn url
tags_2 <- fmap parseTags $ getResponseBody =<< simpleHTTP ( getRequest url )
print tags_2
If you try requesting the URL through some external tool like wget, you will see that Wikipedia does not serve up the result page directly. It actually returns a 302 Moved Temporarily redirect.
When entering this URL in a browser, it will be fine, as the browser will follow the redirect automatically. simpleHTTP, however, will not. simpleHTTP is, as the name suggests, rather simple. It does not handle things like cookies, SSL or redirects.
You'll want to use the Network.Browser module instead. It offers much more control over how the requests are done. In particular, the setAllowRedirects function will make it automatically follow redirects.
Here's a quick and dirty function for downloading an URL into a String with support for redirects:
import Network.Browser
grabUrl :: String -> IO String
grabUrl url = fmap (rspBody . snd) . browse $ do
-- Disable logging output
setErrHandler $ const (return ())
setOutHandler $ const (return ())
setAllowRedirects True
request $ getRequest url

Haskell Network.Browser HTTPS Connection

Is there a way to make https calls with the Network.Browser package.
I'm not seeing it in the documentation on Hackage.
If there isn't a way to do it with browse is there another way to fetch https pages?
My current test code is
import Network.HTTP
import Network.URI (parseURI)
import Network.HTTP.Proxy
import Data.Maybe (fromJust)
import Control.Applicative ((<$>))
import Network.Browser
retrieveUrl :: String -> IO String
retrieveUrl url = do
rsp <- browse $ request (Request (fromJust uri) POST [] "Body")
return $ snd (rspBody <$> rsp)
where uri = parseURI url
I've been running nc -l -p 8000 and watching the output.
I see that it doesn't encrypt it when I do retrieveUrl https://localhost:8000
Also when I try a real https site I get:
Network.Browser.request: Error raised ErrorClosed
*** Exception: user error (Network.Browser.request: Error raised ErrorClosed)
Edit: Network.Curl solution (For doing a SOAP call)
import Network.Curl (curlGetString)
import Network.Curl.Opts
soapHeader s = CurlHttpHeaders ["Content-Type: text/xml", "SOAPAction: " ++ s]
proxy = CurlProxy "proxy.foo.org"
envelope = "myRequestEnvelope.xml"
headers = readFile envelope >>= (\x -> return [ soapHeader "myAction"
, proxy
, CurlPost True
, CurlPostFields [x]])
main = headers >>= curlGetString "https://service.endpoint"
An alternative and perhaps more "haskelly" solution as Travis Brown put it with http-conduit:
To just fetch https pages:
import Network.HTTP.Conduit
import qualified Data.ByteString.Lazy as L
main = simpleHttp "https://www.noisebridge.net/wiki/Noisebridge" >>= L.putStr
The below shows how to pass urlencode parameters.
{-# LANGUAGE OverloadedStrings #-}
import Network.HTTP.Conduit
import qualified Data.ByteString.Lazy as L
main = do
initReq <- parseUrl "https://www.googleapis.com/urlshortener/v1/url"
let req' = initReq { secure = True } -- Turn on https
let req = (flip urlEncodedBody) req' $
[ ("longUrl", "http://www.google.com/")
-- ,
]
response <- withManager $ httpLbs req
L.putStr $ responseBody response
You can also set the method, content-type, and request body manually. The api is the same as in http-enumerator a good example is: https://stackoverflow.com/a/5614946
I've wondered about this in the past and have always ended up just using the libcurl bindings. It would be nice to have a more Haskelly solution, but Network.Curl is very convenient.
If all you want to do is fetch a page, Network.HTTP.Wget is the most simple way. Exhibit a:
import Network.HTTP.Wget
main = putStrLn =<< wget "https://www.google.com" [] []

HTTP POST contents in Haskell

I'm trying to post some data to a server in Haskell and the server side is coming up empty.
I'm using the Network.HTTP library for the request.
module Main (main) where
import Network.URI (URI (..), parseURI, uriScheme, uriPath, uriQuery, uriFragment)
import Network.HTTP
import Network.TCP as TCP
main = do
conn <- TCP.openStream "localhost" 80
rawResponse <- sendHTTP conn updateTest
body <- getResponseBody rawResponse
if body == rqBody updateTest
then print "test passed"
else print (body ++ " != " ++ (rqBody updateTest))
updateURI = case parseURI "http://localhost/test.php" of
Just u -> u
updateTest = Request { rqURI = updateURI :: URI
, rqMethod = POST :: RequestMethod
, rqHeaders = [ Header HdrContentType "text/plain; charset=utf-8"
] :: [Header]
, rqBody = "Test string"
}
This test is returning the empty string as the response body from the server, when I think it should be echoing the "Test string" post.
I would ideally like to replicate the functionality of:
curl http://localhost/test.php -d 'Test string' -H 'Content-type:text/plain; charset=utf-8'
and am validating results with serverside test.php:
<?php
print (#file_get_contents('php://input'));
Am I doing this wrong or should I just be trying another library?
You need to specify a Content-Length HTTP header, whose value must be the length of the raw posted data:
updateTest = Request { rqURI = updateURI
, rqMethod = POST
, rqHeaders = [ mkHeader HdrContentType "application/x-www-form-urlencoded"
, mkHeader HdrContentLength "8"
]
, rqBody = "raw data"
}
And with http-conduit:
{-# LANGUAGE OverloadedStrings #-}
import Network.HTTP.Conduit
import qualified Data.ByteString.Lazy as L
main = do
initReq <- parseUrl "http://localhost/test.php"
let req = (flip urlEncodedBody) initReq $
[ ("", "Test string")
-- ,
]
response <- withManager $ httpLbs req
L.putStr $ responseBody response
The "Test string", in the above example, is urlEncoded before being posted.
You can also set the method, content-type, and request body manually. The api is the same as in http-enumerator a good example is:
https://stackoverflow.com/a/5614946

Resources