Access the Betfair Exchange API using Julia
I've been using Julia for about 2mths now, and have recently been trying to use Julia to access the Betfair API.
Note about this service are here.
https://docs.developer.betfair.com/display/1smk3cen4v3lu3yomq5qye0ni/Getting+Started
Whilst I can get the Python example working (& I have an appKey & sessionToken though not shown), I've not been able to successfully translate this Python into Julia.
In example below I get a StatusError 400 response (which is the closest I've gotten). Other attempts indicated Bound issues probably from the Python example using {} and ' which Ive attempted to then translate.
I've looked at other Stackflow questions, but found they don't have the complexity associated with this example.
Wondering if anyone has any thoughts.
Thanks in advance
using HTTP
url="https://api.betfair.com/exchange/betting/json-rpc/v1"
header = "\"X-Application\" : \"appKey\", \"X-Authentication\" : \"sessionToken\" ,\"content-type\" : \"application/json\" "
jsonrpc_req="\"jsonrpc\": \"2.0\", \"method\": \"SportsAPING/v1.0/listEventTypes\", \"params\": {\"filter\":{ }}, \"id\": 1"
response = HTTP.post(url, data=[jsonrpc_req], headers=[header])
println(response.text)
Expected Results.
In Python, I get a summary of Betfair Sports and Market's.
{"jsonrpc":"2.0","result":[{"eventType":{"id":"1","name":"Soccer"},"marketCount":10668},{"eventType":{"id":"2","name":"Tennis"},"marketCount":4590},{"eventType":{"id":"3","name":"Golf"},"marketCount":43},{"eventType":{"id":"4","name":"Cricket"},"marketCount":394},{"eventType":{"id":"5","name":"Rugby Union"},"marketCount":37},{"eventType":{"id":"1477","name":"Rugby League"},"marketCount":24},{"eventType":{"id":"6","name":"Boxing"},"marketCount":27},{"eventType"
...etc...
Currently get
HTTP.ExceptionRequest.StatusError(400, HTTP.Messages.Response:
400 Bad Request.
While the interaction with a particular REST service is a problem-specific issue here are the general guidelines.
Firstly, you need to properly format headers - HTTP.jl manual reads: "headers can be any collection where [string(k) => string(v) for (k,v) in headers] yields Vector{Pair}."
Since we do not have Betfair API key let's have a look on a more generic example using https://postman-echo.com/ which is a free simple API testing that simply returns as JSON whatever it gets as the input.
using HTTP
using JSON
headers = (("X-Application","appKey"),("X-Authentication","sessionToken"),
("content-type","application/json"))
url="https://postman-echo.com/post"
req = Dict("jsonrpc" => "2.0", "params" => Dict("filet" => Dict()))
response = HTTP.post(url, headers, JSON.json(req))
response_text = String(response.body)
json_obj = JSON.parse()
Now let us parse the output from postman-echo.com:
julia> display(JSON.parse(response_text))
Dict{String,Any} with 7 entries:
"headers" => Dict{String,Any}("x-forwarded-port"=>"443","host"=>"postman-echo.com","x-application"=>"appKey","content-type"… "json" => Dict{String,Any}("params"=>Dict{String,Any}("filet"=>Dict{String,Any}()),"jsonrpc"=>"2.0")
"files" => Dict{String,Any}()
"args" => Dict{String,Any}()
"data" => Dict{String,Any}("params"=>Dict{String,Any}("filet"=>Dict{String,Any}()),"jsonrpc"=>"2.0")
"url" => "https://postman-echo.com/post"
"form" => Dict{String,Any}()
You can easily adopt the above code to any RESTful JSON API.
Thanks Przemyslaw Szufel for your response. After a few more days of frustration, I managed to get the first part of the API working using the Excel/VBA sample here: https://github.com/betfair/API-NG-Excel-Toolkit (my translations of the Python examples did not work).
Your comment helped in terms of understanding how to structure multiple headers, and using Dict( =>) rather than string for the manipulations I attempted above.
using HTTP
using JSON
const ListEventTypesMethod = "listEventTypes"
const AppKey = "appKey"
const Session = "sessionToken"
function SendRequest(Url, AppKey, Session, Data)
headers = (("X-Application", AppKey),
("content-type", "application/json"),
("Accept", "application/json"),
("X-Authentication", Session))
HTTP.get(Url,headers,JSON.json(Data))
end
function ParseJsonRpcResponseToCollection(Response)
ParseJsonRpcResponseToCollection = JSON.parse(Response)
end
function GetJsonRpcUrl()
GetJsonRpcUrl = "https://api.betfair.com/exchange/betting/json-rpc/v1/"
end
function MakeJsonRpcRequestString(Method, RequestString)
#MakeJsonRpcRequestString = "{""jsonrpc"": ""2.0"", ""method"": ""SportsAPING/v1.0/" & Method & """, ""params"": " & RequestString & ", ""id"": 1}"
MakeJsonRpcRequestString = Dict("jsonrpc" => "2.0", "method" =>"SportsAPING/v1.0/"*Method, "params" => RequestString, "id"=> 1 )
end
function GetListEventTypesRequestString()
#GetListEventTypesRequestString = "{""filter"":{}}"
GetListEventTypesRequestString=Dict("filter" =>Dict())
end
Request = MakeJsonRpcRequestString(ListEventTypesMethod, GetListEventTypesRequestString())
ListEventTypesResponse = SendRequest(GetJsonRpcUrl(), AppKey, Session, Request)
Response
HTTP/1.1 200 OK
Date: Sun, 17 Feb 2019 17:28:08 GMT
Server: Cougar - 4.4.2 (Cougar 2 mode)
Cache-Control: no-cache
Content-Type: application/json
Vary: Accept-Encoding, User-Agent
Content-Length: 1850
{"jsonrpc":"2.0","result":[{"eventType":{"id":"1","name":"Soccer"},"marketCount":6553},{"eventType":{"id":"2","name":"Tennis"},"marketCount":5511},{"eventType":{"id":"3","name":"Golf"},"marketCount":34}
etc...
Hope this helps others as well.
Related
I try to post a form to server and here is the code:
ar request = new http.MultipartRequest("POST", _uri);
request.fields['user_acc'] = _userAcc;
// this issue should be solve
request.fields['user_nick_name'] = '中文名字';
request.fields['user_password'] = _password;
But the server side in the user_nick_name field always got null, note that is always, but I change it into English the server can receive that. I test on postman, the server can got Chinese correctly, so it's MultipartRequest issue on this problem.
My question is: Why the Dart or Flutter team so careless on this so important basic library? They even not consider about this simply issue. I opened a issue on github but no-one response, I think the team is done. So I ask the develop communit here, how to solve this problem anyway?
[UPDATE]
As kindly people suggested, I update my golang server now, if anyone else got this problem, you may wonna answer and suggestions too.
func HandleUserRegister(context *gin.Context) {
userAcc := context.PostForm("user_acc")
userAvatar := context.PostForm("user_avatar")
userNickName := context.PostForm("user_nick_name")
userPassword := context.PostForm("user_password")
userPhone := context.PostForm("user_phone")
userEmail := context.PostForm("user_email")
userGender := context.PostForm("user_gender")
userSign := context.PostForm("user_sign")
userType := context.PostForm("user_type")
userTypeInt, _ := strconv.Atoi(userType)
log.Infof("userAcc: %s, userNickName: %s, userPassword: %s", userAcc, userNickName, userPassword)}
This is based on gin, and this function is the api solver. If anyone wanna help, please help me figure it out.
OK! I update the question now, because it's really weird!. I did those test:
Post multiform via Flutter to Django server, it receives Chinese filed correctly;
Post multiform data via Postman, the golang(gin) server gots Chinese correctly;
Post multiform data via Flutter to golang(gin) server gots Chinese field null;
For more detail, I log the headers from my server for both postman(normal) and flutter (abnormal):
Postman:
request header: map[Content-Type:[multipart/form-data; boundary=--------------------------022341683711652813100488] Postman-Token:[855646d7-5bea-4b8f-b8df-81366226cd49] User-Agent:[PostmanRuntime/7.1.1] Content-Length:[422] Connection:[keep-alive] Cache-Control:[no-cache] Accept:[*/*] Accept-Encoding:[gzip, deflate]]
Flutter:
request header: map[User-Agent:[Dart/2.0 (dart:io)] Content-Type:[multipart/form-data; boundary=dart-http-boundary-.XUeYeqXpg4Yfyh8QhH1T5JB4zi_f3WxX9t7Taxhw91EFqhyki4] Accept-Encoding:[gzip] Content-Length:[574]]
Does anyone can notice the difference and let me know how to change the it make server can receive the Chinese Characters?
#DannyTuppeny is correct. This is a server problem.
When asked to include a non-ASCII field into a multi-part request, the Dart library correctly wraps this with a binary content-transfer-encoding.
String _headerForField(String name, String value) {
var header =
'content-disposition: form-data; name="${_browserEncode(name)}"';
if (!isPlainAscii(value)) {
header = '$header\r\n'
'content-type: text/plain; charset=utf-8\r\n'
'content-transfer-encoding: binary';
}
return '$header\r\n\r\n';
}
(Postman does not and simply sends the utf8 encoded string without any headers.)
Dart/ASCII looks like this:
--dart-http-boundary-HjDS88CmQicdgd8VaHSwPqJK8iR4H6rTG3LovSZy-QXGpU7pAB0
content-disposition: form-data; name="test"
stackover
--dart-http-boundary-HjDS88CmQicdgd8VaHSwPqJK8iR4H6rTG3LovSZy-QXGpU7pAB0
Dart/non-ASCII looks like this:
First boundary: --dart-http-boundary-58NU6u6_Fo22xjH8H7yPCtKuoKgB+A8+RTJ82iIK1gs3nnGMLlp\r\n
Encapsulated multipart part: (text/plain)
content-disposition: form-data; name="test"\r\n
content-type: text/plain; charset=utf-8\r\n
content-transfer-encoding: binary\r\n\r\n
Line-based text data: text/plain
\344\270\255\346\226\207\345\220\215\345\255\227
Boundary: \r\n--dart-http-boundary-58NU6u6_Fo22xjH8H7yPCtKuoKgB+A8+RTJ82iIK1gs3nnGMLlp\r\n
So the problem is that the server is unable to unwrap the value from the encapsulation.
EDIT
Here's the Postman trace I captured yesterday. It's multi-form, but fails to add the content-type-encoding header despite the field being non-ASCII.
MIME Multipart Media Encapsulation, Type: multipart/form-data, Boundary: "--------------------------595246000077585285134204"
[Type: multipart/form-data]
First boundary: ----------------------------595246000077585285134204\r\n
Encapsulated multipart part:
Content-Disposition: form-data; name="name"\r\n\r\n
Data (12 bytes)
0000 e4 b8 ad e6 96 87 e5 90 8d e5 ad 97 ............
Data: e4b8ade69687e5908de5ad97
[Length: 12]
Last boundary: \r\n----------------------------595246000077585285134204--\r\n
I tested by posting to httpbin and the response suggests that the characters were posted correctly:
"user_nick_name":"\u4e2d\u6587\u540d\u5b57"
I tried with both the Stable v1 SDK and a v2 SDK from Flutter. Is it possible the issue is on the server? Have you tried using something like Fiddler to capture what's actually being sent?
Edit: My guess is that your server side code is not correctly reading the data as MultipartForm data (eg. you should be using ParseMultipartForm and reading from MultipartForm).
The problem, it appears, is in formdata.go part of multipart. Go assumes that any multipart part with an Content-Type header is a file (not a field). However, knowing this you can change your server code as follows:
func main() {
r := gin.Default()
r.POST("/sotest", func(c *gin.Context) {
formValue := c.PostForm("form_value")
if formValue == "" {
formFile, _ := c.FormFile("form_value")
file, _ := formFile.Open()
b1 := make([]byte, formFile.Size)
file.Read(b1)
formValue = string(b1)
}
c.JSON(200, gin.H{
"status": "posted",
"formValue": formValue,
})
})
r.Run() // listen and serve on 0.0.0.0:8080
}
When you detect that PostForm returns the empty string, you know that Go has treated the field as a file, in which case you can Open and Read the 'file' and decode it as the utf-8 string that we know it is. Obviously, you could encapsulate the "try as PostForm and if that's empty, try as FormFile" test into a function.
If you don't want to have to test for empty string at the server, you could change your Dart end code to always utf-8 encode even non-ascii strings with
request.files.add(
new http.MultipartFile.fromBytes(
'some_form_value_name',
utf8.encode('the string value'),
contentType: new MediaType('text', 'plain', {'charset': 'utf-8'}),
),
);
and read them at the server with the Open/Read/string method.
I have now solved this. Thanks to Richard and Danny for their help.
1. Reason for this
No matter what happens but this really not only one-side problem, we can not say it's Flutter or Go wrong. But the combination, Flutter + Go server just may be got this issue. The behind reason I still not quit sure, but it must some head not right set (postman can do it right);
2. Solution
We don't only need know why but also how to solve it. Here is what I do:
Do not use the official http package. Using dio, which is a extension Dart package. link: https://pub.dartlang.org/packages/dio
It's more clean and easy to use, so my code becomes to:
FormData _formData = new FormData.from({
"user_acc": _userAcc,
"user_nick_name": _userNickName,
'user_password': _password,
});
Dio dio = new Dio();
Response response = await dio.post(usersUrl, data: _formData);
print(response.data);
I can not post the none-English words now:
INFO[0668] userAcc: ww, userNickName: 小鹿叮叮婴儿湿巾手口专用80抽湿纸巾婴儿湿巾婴儿100抽带盖批发【原价】34.90元【券后】9.9元【省】25元【复制此信息打开手机淘宝即可查看并下单】¥Tnsx0E77pFs¥【必买理由】新品预售80抽*3仙女联盟,更多优惠fd.loliloli.pro , userPassword: ww
INFO[0671] user exist.
Using python requests session I can connect to JIRA and retrieve issue information ...
session = requests.Session()
headers = {"Authorization": "Basic %s" % bas64_val}
session.post(jira_rest_url, headers=headers)
jira = session.get(jira_srch_issue_url + select_fields)
# select_fields = the fields I want from the issue
Now I'm trying to post a payload via the JIRA API, using a fixed issue url e.g. "https://my_jira_server.com:1234/rest/api/latest/issue/KEY-9876"
Which should be a case of the following, given: https://developer.atlassian.com/jiradev/jira-apis/about-the-jira-rest-apis/jira-rest-api-tutorials/jira-rest-api-example-edit-issues
payload = { "update": {
"fixVersions": [ {"set": "release-2.139.0"} ]
}}
posted = session.post(jira_task_url, data=payload)
# returns <Response [405]>
# jira_task_url = https://my_jira_server.com:1234/rest/api/latest/issue/KEY-9876
But this doesn't appear to work! Looking into the http 405 response, suggests that my payload is not properly formatted! Which notably, is the not easiest thing to diagnose.
What am I doing wrong here? Any help on this would be much appreciated.
Please note, I am not looking to use the python jira module, I am using requests.session to manage several sessions for different systems i.e. JIRA, TeamCity, etc..
Found the solution! I had two problems:
1) The actual syntax structure should have been:
fix_version = { "update": { "fixVersions": [ {"set" : [{ "name" : "release-2.139.0" }]}]
2) To ensure the payload is actually presented as JSON, use json.dumps() which takes an object and produces a string (see here) AND set 'content-type' to 'application/json':
payload = json.dumps(fix_version)
app_json = { 'content-type': 'application/json' }
session.put(https://.../rest/api/latest/issue/KEY-9876, headers=app_json, data=payload)
Rather than trying to define the JSON manually!
I'm new to both: R and OAUTH. I've learned a little using coursera examples on github API where OAUTH request gave plaintext response but now I'm trying to do something that is practicall for me and access EVE-Online CREST OAUTH API but instead of what I got when I tried github API (im using "httr" libary):
Response [https://api.github.com/users/jtleek/repos]
Date: 2014-12-14 08:57
Status: 200
Content-type: application/json; charset=utf-8
Size: 154 kB
[
{
"id": 12441219,
"name": "ballgown",
"full_name": "jtleek/ballgown",
"owner": {
"login": "jtleek",
"id": 1571674,
"avatar_url": "https://avatars.githubusercontent.com/u/1571674?v=3",
"gravatar_id": "",
...
I got this BINARY BODY response:
Response [https://crest-tq.eveonline.com/market/10000002/orders/buy/?type=https://crest-tq.eveonline.com/types/185/]
Date: 2014-12-14 08:05
Status: 200
Content-type: application/vnd.ccp.eve.MarketOrderCollection-v1+json; charset=utf-8
Size: 7.61 kB
<BINARY BODY>
And frankly I have no idea what to do with it. I'm preety sure its gzip (I used chrome extension postman to access the same information and header says its encoded with gzip) but I dont know how to uncompress it, maybe there is standard way of dealing with binary/gzip response but my google foo have failed me.
Here is exact code I'm running:
library(httr)
myapp <- oauth_app("my app name redacted", "my id redacted", "my secret redacted")
eve_token <- oauth2.0_token(oauth_endpoint(authorize = "https://login-tq.eveonline.com/oauth/authorize/",access = "https://login-tq.eveonline.com/oauth/token/"), myapp, scope = "publicData")
token <- config(token = eve_token)
req <- GET("https://crest-tq.eveonline.com/market/10000002/orders/buy/?type=https://crest-tq.eveonline.com/types/185/", token)
EDIT:
YES!!! :)
managed to figure it out :)
result <- content(req, type = "application/json; charset=utf-8")
while the reqular content(req) produced just raw binary data, the above translated it to json :)
Like I wrote above, what I needed to do was pass more information about content type and encoding used to content function like this:
result <- content(req, type = "application/json; charset=utf-8")
gzip part as its turned out was handled automagically, but the issue was strage content-type used by EVE API. when i explicitly passed desired content type R was able to read data as json without problem
Intuit offers these instructions for uploading attachments (which become Attachable objects that can be associated with one or more transactions).
I believe I'm using python's requests module (via rauth's OAuth1Session module—see below for how I'm creating the session object) to generate these requests. Here's the code leading up to the request:
print request_type
print url
print headers
print request_body
r = session.request(request_type, url, header_auth,
self.company_id, headers = headers,
data = request_body, **req_kwargs)
result = r.json()
print json.dumps(result, indent=4)
and the output of these things:
POST
https://quickbooks.api.intuit.com/v3/company/0123456789/upload
{'Accept': 'application/json'}
Content-Disposition: form-data; name="Invoice 003"; filename="Invoice 003.pdf"
Content-Type: application/pdf
<#INCLUDE */MyDir/Invoice 003.pdf*#>
{
"Fault": {
"type": "SystemFault",
"Error": [
{
"Message": "An application error has occurred while processing your request",
"code": "10000",
"Detail": "System Failure Error: Cannot consume content type"
}
]
},
"time": "[timestamp]"
}
I have confirmed (by uploading an attachment through the QBO web UI and then querying the Attachable object through the API) that application/pdf is included in the list of acceptable file types.
At sigmavirus24's suggestion, I tried removing the Content-Type line from the headers, but I got the same result.
Here's how I'm creating the session object (which, again, is working fine for other QBO v3 API requests of every type you see in Intuit's API Explorer):
from rauth import OAuth1Session
def create_session(self):
if self.consumer_secret and self.consumer_key and self.access_token_secret and self.access_token:
session = OAuth1Session(self.consumer_key,
self.consumer_secret,
self.access_token,
self.access_token_secret,
)
self.session = session
else:
raise Exception("Need four creds for Quickbooks.create_session.")
return self.session
What might I be missing here?
EDIT: current area of exploration is here; I just formed the header you see (that has the "INCLUDE" string there) directly. Perhaps I should be using rauth to attach the file...
Without being able to see what code you're using with requests, I'm going to take a shot in the dark and tell you to remove setting your own Content-Type. You probably don't want that. It looks like you want multipart/form-data and requests will set that on its own if you stop fighting it.
It looks like you're missing the boundaries that QuickBooks is expecting (based on what you linked).
---------------------------acebdf13572468
Content-Disposition: form-data; name="file_content_01"; filename="IMG_0771.jpg"
Content-Type: image/jpeg
<#INCLUDE *Y:\Documents\IMG_0771.jpg*#>
---------------------------acebdf13572468--
The first and last line above seem to be what you're missing.
Is it possible to submit a Freebase mqlread request via POST in Python? I have tried to search for documentation but everything refers to GET. Thanks.
It is possible.
You will need issue a POST and add a specific header: X-HTTP-Method-Override: GET (basically tells the server to emulate a GET with the POST's content). Specifically for me I used the Content-Encoding: application/x-www-form-urlencode.
Here's the relevant part of my code (coffeescript) if it helps:
mqlread = (query, queryEnvelope, cb) ->
## build URL
url = urlparser.format
protocol: 'https'
host: 'www.googleapis.com'
pathname: 'freebase/v1/mqlread'
## build POST body
queryEnvelope ?= {}
queryEnvelope.key = config.GOOGLE_API_SERVER_KEY
queryEnvelope.query = JSON.stringify query
options =
url: url
method: 'POST'
headers:
'X-HTTP-Method-Override': 'GET'
'User-Agent': config.wikipediaScraperUserAgent
timeout: 3000
form: queryEnvelope
## invoke API
request options, (err, response, body) ->
if err then return cb err
if response.statusCode != 200
try
json = JSON.parse(body)
errmsg = json?.error?.message or "(unknown JSON)"
catch e
errmsg = body?[..50]
return cb "#{response.statusCode} #{errmsg}"
r = JSON.parse response.body
decodeStringsInResponse r
cb null, r
I don't think POST is supported for MQLread, but you could use the HTTP Batch facility.
Here's an example in Python:
https://github.com/tfmorris/freebase-python-samples/blob/master/client-library/mqlread-batch.py