I want to do something very similar to what's shown in the docs for FSharp.Data:
The URL I'm requesting from though (TFS) requires client authentication. Is there any way I can provide this by propagating my Windows creds? I notice JsonProvider has a few other compile-time parameters, but none seem to be in support of this.
You don't have to provide a live URL as a type parameter to JsonProvider; you can also provide the filename of a sample file that reflects the structure you expect to see. With that feature, you can do the following steps:
First, log in to the service and save a JSON file that reflects the API you're going to use.
Next, do something like the following:
type TfsData = JsonProvider<"/path/to/sample/file.json">
let url = "https://example.com/login/etc"
// Use standard .Net API to log in with your Windows credentials
// Save the results in a variable `jsonResults`
let parsedResults = TfsData.Parse(jsonResults)
printfn "%A" parsedResults.Foo // At this point, Intellisense should work
This is all very generic, of course, since I don't know precisely what you need to do to log in to your service; presumably you already know how to do that. The key is to retrieve the JSON yourself, then use the .Parse() method of your provided type to parse it.
Related
I wrote a small service in go (although I don't think this would be a language specific issue), that caches some results by saving it to a file, and writing a URL query parameter into the filename with "prefix" + param + ".json" using ioutil.WriteFile. The service runs on Ubuntu.
Is it possible to do something malicious, by passing an unexpected string via the query?
Relevant attacks that come to mind are called path injection. For example what if the query parameter is something like ../../etc/passwd (okthis would probably not work as the user running this service would have no permissions, but you get the point). For example it could be possible to overwrite your service code itself.
You should sanitize the parameter before adding it to the filename. The best would be a strict whitelist of letters and numbers that are allowed, anything else should ve removed from the parameter. That way injection would not be possible.
You can also check whether the path you are writing to is actually under an explicitly allowed directory.
I will make a test in python, here is the struct of the project
app1/main.py
while True:
a = input() # passing query
with open("{}.json".format(a), "w") as f:
f.write("Hello world")
now i am a hacker, and i want to change "yourfile.json"
so i passed this
and than, the content of yourfile.json become: Hello world
As a end-point user of an API, how can I list all parameters available to pass the query? In my case (stats about Age of Empires 2 matches), the website describing the API has a list with some of them but it seems there are more available.
To provide more context, I'm extracting the following information:
GET("https://aoe2.net/api/matches?game=aoe2de&count=1000&since=1632744000&map_type=12")
but for some reason the last condition, map_type=12 does nothing (output is the same as without it). I'm after the list of parameters available, so I can extract what I want.
PD: this post is closely related but does not focus on API. Perhaps this makes a difference, as the second answer there seems to suggest.
It is not possible to find out all available (undocumented) query parameters for a query, unless the API explicitly provides such a method or you can find out how the API server processes the query.
For instance, if the API server code is open source, you could find out from the code how the query is processed. Provided that you find the code also.
The answers in the post you linked are similarly valid for an API site as well as for one that provides content for a web browser (a web server can be both).
Under the hood, there is not necessarily any difference between an API server or a server that provides web content (html) in terms of how queries are handled.
As for the parameters seemingly without an effect, it seems that the API in question does not validate the query parameters, i.e., you can put arbitrary parameters in the query and the server will simply ignore parameters that it is not specifically programmed to use.
The documentation on their website is all any of us have to go by https://aoe2.net/#api
You can't just add your own parameters to the URL and expect it to return a value back as they have to have coded it to work that way.
Your best bet is to just extract as much data as you can by increasing the count parameter, then loop through the JSON response and extract the map_type from there.
JavaScript example:
<script>
json=[{"match_id":"1953364","lobby_id":null,"game_type":0},
{"match_id":"1961217","lobby_id":null,"game_type":0},
{"match_id":"1962068","lobby_id":null,"game_type":1},
{"match_id":"1962821","lobby_id":null,"game_type":0},
{"match_id":"1963814","lobby_id":null,"game_type":0},
{"match_id":"1963807","lobby_id":null,"game_type":0},
{"match_id":"1963908","lobby_id":null,"game_type":0},
{"match_id":"1963716","lobby_id":null,"game_type":0},
{"match_id":"1964491","lobby_id":null,"game_type":0},
{"match_id":"1964535","lobby_id":null,"game_type":12},];
for(var i = 0; i < json.length; i++) {
var obj = json[i];
if(obj.game_type==12){
//do something with game_type 12 json object
console.log(obj);
}
}
</script>
I know that in most MVC frameworks, for example, both query string params and form params will be made available to the processing code, and usually merged into one set of params (often with POST taking precedence). However, is it a valid thing to do according to the HTTP specification? Say you were to POST to:
http://1.2.3.4/MyApplication/Books?bookCode=1234
... and submit some update like a change to the book name whose book code is 1234, you'd be wanting the processing code to take both the bookCode query string param into account, and the POSTed form params with the updated book information. Is this valid, and is it a good idea?
Is it valid according HTTP specifications ?
Yes.
Here is the general syntax of URL as defined in those specs
http_URL = "http:" "//" host [ ":" port ] [ abs_path [ "?" query ]]
There is no additional constraints on the form of the http_URL. In particular, the http method (i.e. POST,GET,PUT,HEAD,...) used don't add any restriction on the http URL format.
When using the GET method : the server can consider that the request body is empty.
When using the POST method : the server must handle the request body.
Is it a good idea ?
It depends what you need to do. I suggest you this link explaining the ideas behind GET and POST.
I can think that in some situation it can be handy to always have some parameters like the user language in the query part of the url.
I know that in most MVC frameworks, for example, both query string params and form params will be made available to the processing code, and usually merged into one set of params (often with POST taking precedence).
Any competent framework should support this.
Is this valid
Yes. The POST method in HTTP does not impose any restrictions on the URI used.
is it a good idea?
Obviously not, if the framework you are going to use is still clue-challenged. Otherwise, it depends on what you want to accomplish. The major use case (redirection of a data subset to a new POST target) has been irretrievably broken by browser implementations (all mechanically following the broken lead of Mosaic/Netscape), so the considerations here are mostly theoretical.
I have a Java component which scans through a set of folders (input/processing/output) and returns the list of files in JSON format.
The REST URL for the same is:
GET http://<baseurl>/files/<foldername>
Now, I need to perform certain actions on each of the files, like validate, process, delete, etc. I'm not sure of the best way to design the REST URLs for these actions.
Since its a direct file manipulation, I don't have any unique identifier for the files, except their paths. So I'm not sure if the following is a good URL:
POST http://<baseurl>/file/validate?path=<filepath>
Edit: I would have ideally liked to use something like /file/fileId/validate. But the only unique id for files is its path, and I don't think I can use that as part of the URL itself.
And finally, I'm not sure which HTTP verb to use for such custom actions like validate.
Thanks in advance!
Regards,
Anand
When you implement a route like http:///file/validate?path you encode the action in your resource that's not a desired effect when modelling a resource service.
You could do the following for read operations
GET http://api.example.com/files will return all files as URL reference such as
http://api.example.com/files/path/to/first
http://api.example.com/files/path/to/second
...
GET http://api.example.com/files/path/to/first will return validation results for the file (I'm using JSON for readability)
{
name : first,
valid : true
}
That was the simple read only part. Now to the write operations:
DELETE http://api.example.com/files/path/to/first will of course delete the file
Modelling the file processing is the hard part. But you could model that as top level resource. So that:
POST http://api.example.com/FileOperation?operation=somethingweird will create a virtual file processing resource and execute the operation given by the URL parameter 'operation'. Modelling these file operations as resources gives you the possibility to perform the operations asynchronous and return a result that gives additional information about the process of the operation and so on.
You can take a look at Amazon S3 REST API for additional examples and inspiration on how to model resources. I can highly recommend to read RESTful Web Services
Now, I need to perform certain actions on each of the files, like validate, process, delete, etc. I'm not sure of the best way to design the REST URLs for these actions. Since its a direct file manipulation, I don't have any unique identified for the files, except their paths. So I'm not sure if the following is a good URL: POST http:///file/validate?path=
It's not. /file/validate doesn't describe a resource, it describes an action. That means it is functional, not RESTful.
Edit: I would have ideally liked to use something like /file/fileId/validate. But the only unique id for files is its path, and I don't think I can use that as part of the URL itself.
Oh yes you can! And you should do exactly that. Except for that final validate part; that is not a resource in any way, and so should not be part of the path. Instead, clients should POST a message to the file resource asking it to validate itself. Luckily, POST allows you to send a message to the file as well as receive one back; it's ideal for this sort of thing (unless there's an existing verb to use instead, whether in standard HTTP or one of the extensions such as WebDAV).
And finally, I'm not sure which HTTP verb to use for such custom actions like validate.
POST, with the action to perform determined by the content of the message that was POSTed to the resource. Custom “do something non-standard” actions are always mapped to POST when they can't be mapped to GET, PUT or DELETE. (Alas, a clever POST is not hugely discoverable and so causes problems for the HATEOAS principle, but that's still better than violating basic REST principles.)
REST requires a uniform interface, which in HTTP means limiting yourself to GET, PUT, POST, DELETE, HEAD, etc.
One way you can check on each file's validity in a RESTful way is to think of the validity check not as an action to perform on the file, but as a resource in its own right:
GET /file/{file-id}/validity
This could return a simple True/False, or perhaps a list of the specific constraint violations. The file-id could be a file name, an integer file number, a URL-encoded path, or perhaps an unencoded path like:
GET /file/bob/dir1/dir2/somefile/validity
Another approach would be to ask for a list of the invalid files:
GET /file/invalid
And still another would be to prevent invalid files from being added to your service in the first place, ie, when your service processes a PUT request with bad data:
PUT /file/{file-id}
it rejects it with an HTTP 400 (Bad Request). The body of the 400 response could contain information on the specific error.
Update: To delete a file you would of course use the standard HTTP REST verb:
DELETE /file/{file-id}
To 'process' a file, does this create a new file (resource) from one that was uploaded? For example Flickr creates several different image files from each one you upload, each with a different size. In this case you could PUT an input file and then trigger the processing by GET-ing the corresponding output file:
PUT /file/input/{file-id}
GET /file/output/{file-id}
If the processing isn't near-instantaneous, you could generate the output files asynchronously: every time a new input file is PUT into the web service, the web service starts up an asynchronous activity that eventually results in the output file being created.
I am working on an iPad app that is fed data via web service returning JSON. Watching some iTunes U episodes, it looks like sending back Plist would save me a ton of time and speed up my app quite a bit on the parsing side of things.
Does anyone know of a .net library that converts objects into this Plist to return instead?
EDIT (this is my very limited understanding of this topic):
An Plist is a Property List that iOS can use to easily encode and/or parse data. It is very similar to JSON except parsing takes a fraction of the time and can be done in 1 line of code. If your server uses WebObjects then encoding can also be done in 1 line of code, I am using IIS so I need a solution for this if one exists before I write my own.
You can see the videos here:
http://developer.apple.com/videos/wwdc/2010/
In particular watch Session 117 - Building a Server-Driven User Experience
You may checkout this project. Sample usage:
object value = ...
string plist = Plist.PlistDocument.CreateDocument(value);
The only requirement is to decorate your object with [Serializable] attribute.
If you're using WebObjects, the appserver from apple, there's a java mirror class of NSPropertyListSerialization that does all of this for you; you can pass it NSArray's, NSDictionaries, etc and it will just work. Not sure if that's what you're talking about; confused as to the WebObjects in your question. HTH's.