HTTP POST contents in Haskell - http

I'm trying to post some data to a server in Haskell and the server side is coming up empty.
I'm using the Network.HTTP library for the request.
module Main (main) where
import Network.URI (URI (..), parseURI, uriScheme, uriPath, uriQuery, uriFragment)
import Network.HTTP
import Network.TCP as TCP
main = do
conn <- TCP.openStream "localhost" 80
rawResponse <- sendHTTP conn updateTest
body <- getResponseBody rawResponse
if body == rqBody updateTest
then print "test passed"
else print (body ++ " != " ++ (rqBody updateTest))
updateURI = case parseURI "http://localhost/test.php" of
Just u -> u
updateTest = Request { rqURI = updateURI :: URI
, rqMethod = POST :: RequestMethod
, rqHeaders = [ Header HdrContentType "text/plain; charset=utf-8"
] :: [Header]
, rqBody = "Test string"
}
This test is returning the empty string as the response body from the server, when I think it should be echoing the "Test string" post.
I would ideally like to replicate the functionality of:
curl http://localhost/test.php -d 'Test string' -H 'Content-type:text/plain; charset=utf-8'
and am validating results with serverside test.php:
<?php
print (#file_get_contents('php://input'));
Am I doing this wrong or should I just be trying another library?

You need to specify a Content-Length HTTP header, whose value must be the length of the raw posted data:
updateTest = Request { rqURI = updateURI
, rqMethod = POST
, rqHeaders = [ mkHeader HdrContentType "application/x-www-form-urlencoded"
, mkHeader HdrContentLength "8"
]
, rqBody = "raw data"
}

And with http-conduit:
{-# LANGUAGE OverloadedStrings #-}
import Network.HTTP.Conduit
import qualified Data.ByteString.Lazy as L
main = do
initReq <- parseUrl "http://localhost/test.php"
let req = (flip urlEncodedBody) initReq $
[ ("", "Test string")
-- ,
]
response <- withManager $ httpLbs req
L.putStr $ responseBody response
The "Test string", in the above example, is urlEncoded before being posted.
You can also set the method, content-type, and request body manually. The api is the same as in http-enumerator a good example is:
https://stackoverflow.com/a/5614946

Related

Encoding problem with GET requests in Haskell

I'm trying to get some Json data from a Jira server using Haskell. I'm counting this as "me having problems with Haskell" rather than encodings or Jira because my problem is when doing this in Haskell.
The problem occurs when the URL (or query) has plus signs. After building my request for theproject+order+by+created, Haskell prints it as:
Request {
host = "myjiraserver.com"
port = 443
secure = True
requestHeaders = [("Content-Type","application/json"),("Authorization","<REDACTED>")]
path = "/jira/rest/api/2/search"
queryString = "?jql=project%3Dtheproject%2Border%2Bby%2Bcreated"
method = "GET"
proxy = Nothing
rawBody = False
redirectCount = 10
responseTimeout = ResponseTimeoutDefault
requestVersion = HTTP/1.1
}
But the request fails with this response:
- 'Error in the JQL Query: The character ''+'' is a reserved JQL character. You must
enclose it in a string or use the escape ''\u002b'' instead. (line 1, character
21)'
So it seems like Jira didn't like Haskell's %2B. Do you have any suggestions on what I can do to fix this, or any resources that might be helpful? The same request sans the +order+by+created part is successful.
The code (patched together from these examples):
{-# LANGUAGE OverloadedStrings #-}
import Data.Aeson
import qualified Data.ByteString.Char8 as S8
import qualified Data.Yaml as Yaml
import Network.HTTP.Simple
import System.Environment (getArgs)
-- auth' is echo -e "username:passwd" | base64
foo urlBase proj' auth' = do
let proj = S8.pack (proj' ++ "+order+by+created")
auth = S8.pack auth'
request'' <- parseRequest urlBase
let request'
= setRequestMethod "GET"
$ setRequestPath "/jira/rest/api/2/search"
$ setRequestHeader "Content-Type" ["application/json"]
$ request''
request
= setRequestQueryString [("jql", Just (S8.append "project=" proj))]
$ setRequestHeader "Authorization" [S8.append "Basic " auth]
$ request'
return request
main :: IO ()
main = do
args <- getArgs
case args of
(urlBase:proj:auth:_) -> do
request <- foo urlBase proj auth
putStrLn $ show request
response <- httpJSON request
S8.putStrLn $ Yaml.encode (getResponseBody response :: Value) -- apparently this is required
putStrLn ""
_ -> putStrLn "usage..."
(If you know a simpler way to do the above then I'd take such suggestions as well, I'm just trying to do something analogous to this Python:
import requests
import sys
if len(sys.argv) >= 4:
urlBase = sys.argv[1]
proj = sys.argv[2]
auth = sys.argv[3]
urlBase += "/jira/rest/api/2/search?jql=project="
proj += "+order+by+created"
h = {}
h["content-type"] = "application/json"
h["authorization"] = "Basic " + auth
r = requests.get(urlBase + proj, headers=h)
print(r.json())
)
project+order+by+created is the URL-encoded string for the actual request project order by created (with spaces instead of +). The function setRequestQueryString expects a raw request (with spaces, not URL-encoded), and URL-encodes it.
The Python script you give for comparison essentially does the URL-encoding by hand.
So the fix is to put the raw request in proj:
foo urlBase proj' auth' = do
let proj = S8.pack (proj' ++ " order by created") -- spaces instead of +
...

How to download JIRA attachment files with Python

I want to download attachment files of an issue in JIRA Python.
I use jira python lib ,you can use pip install JIRA
# -- coding: UTF-8 --
from jira import JIRA
import requests
url = 'https://jira.1234.com'
jira = JIRA(server=url, basic_auth=('admin', 'password'))
attachment=jira.attachment(12345) #12345 is attachment_key
image = attachment.get()
with open("Image.png", 'wb') as f:
f.write(image)
JIRA exposes its REST services and through that and some python you can download any attachment.
It worked for me like this (you'll need to adjust the variables):
#!/usr/bin/python
# miguel ortiz
# Requests module: http://docs.python-requests.org/en/latest/
# Documentation: <url>
#----------------------------------------------------------------Modules
import sys
import csv, json
import requests
#----------------------------------------------------------------Variables
myTicket= sys.argv[1] # Your ticket: ABC-123
user = 'miguel' # JIRA user
pasw = 'password' # JIRA password
jiraURL = 'https://yourinstance.jira.com/rest/api/latest/issue/'
fileName = 'my_attached_file' # In this case we'll be looking for a specific file in the attachments
attachment_final_url="" # To validate if there are or not attachments
def main() :
print '\n\n [ You are checking ticket: ' + myTicket+ ' ]\n'
# Request Json from JIRA API
r = requests.get(jiraURL+myTicket, auth=(user, pasw),timeout=5)
# status of the request
rstatus = r.status_code
# If the status isn't 200 we leave
if not rstatus == 200 :
print 'Error accesing JIRA:' + str(rstatus)
exit()
else:
data = r.json()
if not data['fields']['attachment'] :
status_attachment = 'ERROR: Nothing attached, attach a file named: ' + fileName
attachment_final_url=""
else:
for i in data['fields']['attachment'] :
if i['filename'] == fileName :
attachment_final_url = i['content']
status_attachment_name = 'OK: The desired attachment exists: ' + fileName
attachment_name = False
attachment_amount = False
attachment_files = False
break
else :
attachment_files = False
status_attachment_name = + 'ERROR: None of the files has the desired name '
attachment_final_url=""
attachment_name = True
attachment_amount = True
continue
if attachment_final_url != "" :
r = requests.get(attachment_final_url, auth=(user, pasw), stream=True)
with open(fileName, "wb") as f:
f.write(r.content.decode('iso-8859-1').encode('utf8'))
f.close()
else:
print status_attachment
if __name__ == "__main__" :
main()
If you do not understand the code I've detailed it better in my blog.
EDIT: Be careful, in JIRA you can add many files with the same name.

How to POST raw file using Tornado HTTPRequest

I want to send some data in POST request using Tornado (AsyncHTTPClient)
rec_body = {'source': self.request.body, 'top': str(self.config["top"]), 'model': self.config["model"]}
where self.request.body is a raw binary file (image).
I try doing this:
http_client = AsyncHTTPClient()
rec_body = {'source': self.request.body, 'top': str(self.config["top"]), 'model': self.config["model"]}
request = HTTPRequest( url = os.path.join(self.config["dest_addr"], self.config["sub_sect"]) , method='POST', body =rec_body)
result = http_client.fetch( request, callback=self.handle_request)
but got this errors
File "/usr/local/lib/python2.7/dist-packages/tornado/httpclient.py", line 424, in __init__
self.body = body
File "/usr/local/lib/python2.7/dist-packages/tornado/httpclient.py", line 468, in body
self._body = utf8(value)
File "/usr/local/lib/python2.7/dist-packages/tornado/escape.py", line 203, in utf8
"Expected bytes, unicode, or None; got %r" % type(value)
TypeError: Expected bytes, unicode, or None; got <type 'dict'>
ERROR:tornado.access:500 POST /upload (192.168.72.84) 13.14ms
What I doing wrong?
I tried curl (naively), requests module all works fine, ut not asynchronously. For tornado`s AsyncHTTPClient there is good recipe from flickr.
Deals with multipart POST requests.
The code is adapted from the recipe found at :
http://code.activestate.com/recipes/146306/
No author name was given.
Author : Alexis Mignon (c)
email : alexis.mignon#gmail.Com
Date : 06/08/2011
Here is the code:
import mimetypes
from tornado.gen import coroutine, Return
from tornado.httpclient import HTTPRequest
from tornado_flickrapi.httpclient import fetch
#coroutine
def posturl(url, fields, files):
try:
response = yield post_multipart(url, fields, files)
except Exception as e:
raise e
raise Return(response)
#coroutine
def post_multipart(url, fields, files):
"""
Post fields and files to an http host as multipart/form-data.
fields is a sequence of (name, value) elements for regular form fields.
files is a sequence of (name, filename, value) elements for data to be
uploaded as files.
Return the server's response page.
"""
content_type, body = encode_multipart_formdata(fields, files)
headers = {"Content-Type": content_type, 'content-length': str(len(body))}
request = HTTPRequest(url, "POST", headers=headers, body=body, validate_cert=False)
try:
response = yield fetch(request)
except Exception as e:
raise e
raise Return(response)
def encode_multipart_formdata(fields, files):
"""
fields is a sequence of (name, value) elements for regular form fields.
files is a sequence of (name, filename, value) elements for data to be
uploaded as files.
Return (content_type, body) ready for httplib.HTTP instance
"""
BOUNDARY = '----------ThIs_Is_tHe_bouNdaRY_$'
CRLF = '\r\n'
L = []
for (key, value) in fields:
L.append('--' + BOUNDARY)
L.append('Content-Disposition: form-data; name="%s"' % key)
L.append('')
L.append(value)
for (key, filename, value) in files:
filename = filename.encode("utf8")
L.append('--' + BOUNDARY)
L.append(
'Content-Disposition: form-data; name="%s"; filename="%s"' % (
key, filename
)
)
L.append('Content-Type: %s' % get_content_type(filename))
L.append('')
L.append(value)
L.append('--' + BOUNDARY + '--')
L.append('')
body = CRLF.join(L)
content_type = 'multipart/form-data; boundary=%s' % BOUNDARY
return content_type, body
def get_content_type(filename):
return mimetypes.guess_type(filename)[0] or 'application/octet-stream'

Download pdf file from wikipedia

Wikipedia provides a link (left side on Print/export) on every article to download the article as pdf. I wrote a small Haskell script which first gets the Wikipedia link and output the rendering link. When I am giving the rendering url as input, I am getting empty tags but the same url in browser provides download link.
Could someone please tell me how to solve this problem? Formated code on ideone.
import Network.HTTP
import Text.HTML.TagSoup
import Data.Maybe
parseHelp :: Tag String -> Maybe String
parseHelp ( TagOpen _ y ) = if any ( \( a , b ) -> b == "Download a PDF version of this wiki page" ) y
then Just $ "http://en.wikipedia.org" ++ snd ( y !! 0 )
else Nothing
parse :: [ Tag String ] -> Maybe String
parse [] = Nothing
parse ( x : xs )
| isTagOpen x = case parseHelp x of
Just s -> Just s
Nothing -> parse xs
| otherwise = parse xs
main = do
x <- getLine
tags_1 <- fmap parseTags $ getResponseBody =<< simpleHTTP ( getRequest x ) --open url
let lst = head . sections ( ~== "<div class=portal id=p-coll-print_export>" ) $ tags_1
url = fromJust . parse $ lst --rendering url
putStrLn url
tags_2 <- fmap parseTags $ getResponseBody =<< simpleHTTP ( getRequest url )
print tags_2
If you try requesting the URL through some external tool like wget, you will see that Wikipedia does not serve up the result page directly. It actually returns a 302 Moved Temporarily redirect.
When entering this URL in a browser, it will be fine, as the browser will follow the redirect automatically. simpleHTTP, however, will not. simpleHTTP is, as the name suggests, rather simple. It does not handle things like cookies, SSL or redirects.
You'll want to use the Network.Browser module instead. It offers much more control over how the requests are done. In particular, the setAllowRedirects function will make it automatically follow redirects.
Here's a quick and dirty function for downloading an URL into a String with support for redirects:
import Network.Browser
grabUrl :: String -> IO String
grabUrl url = fmap (rspBody . snd) . browse $ do
-- Disable logging output
setErrHandler $ const (return ())
setOutHandler $ const (return ())
setAllowRedirects True
request $ getRequest url

Haskell Network.Browser HTTPS Connection

Is there a way to make https calls with the Network.Browser package.
I'm not seeing it in the documentation on Hackage.
If there isn't a way to do it with browse is there another way to fetch https pages?
My current test code is
import Network.HTTP
import Network.URI (parseURI)
import Network.HTTP.Proxy
import Data.Maybe (fromJust)
import Control.Applicative ((<$>))
import Network.Browser
retrieveUrl :: String -> IO String
retrieveUrl url = do
rsp <- browse $ request (Request (fromJust uri) POST [] "Body")
return $ snd (rspBody <$> rsp)
where uri = parseURI url
I've been running nc -l -p 8000 and watching the output.
I see that it doesn't encrypt it when I do retrieveUrl https://localhost:8000
Also when I try a real https site I get:
Network.Browser.request: Error raised ErrorClosed
*** Exception: user error (Network.Browser.request: Error raised ErrorClosed)
Edit: Network.Curl solution (For doing a SOAP call)
import Network.Curl (curlGetString)
import Network.Curl.Opts
soapHeader s = CurlHttpHeaders ["Content-Type: text/xml", "SOAPAction: " ++ s]
proxy = CurlProxy "proxy.foo.org"
envelope = "myRequestEnvelope.xml"
headers = readFile envelope >>= (\x -> return [ soapHeader "myAction"
, proxy
, CurlPost True
, CurlPostFields [x]])
main = headers >>= curlGetString "https://service.endpoint"
An alternative and perhaps more "haskelly" solution as Travis Brown put it with http-conduit:
To just fetch https pages:
import Network.HTTP.Conduit
import qualified Data.ByteString.Lazy as L
main = simpleHttp "https://www.noisebridge.net/wiki/Noisebridge" >>= L.putStr
The below shows how to pass urlencode parameters.
{-# LANGUAGE OverloadedStrings #-}
import Network.HTTP.Conduit
import qualified Data.ByteString.Lazy as L
main = do
initReq <- parseUrl "https://www.googleapis.com/urlshortener/v1/url"
let req' = initReq { secure = True } -- Turn on https
let req = (flip urlEncodedBody) req' $
[ ("longUrl", "http://www.google.com/")
-- ,
]
response <- withManager $ httpLbs req
L.putStr $ responseBody response
You can also set the method, content-type, and request body manually. The api is the same as in http-enumerator a good example is: https://stackoverflow.com/a/5614946
I've wondered about this in the past and have always ended up just using the libcurl bindings. It would be nice to have a more Haskelly solution, but Network.Curl is very convenient.
If all you want to do is fetch a page, Network.HTTP.Wget is the most simple way. Exhibit a:
import Network.HTTP.Wget
main = putStrLn =<< wget "https://www.google.com" [] []

Resources