I want html source code with asp.net HttpWebRequest but there is a problem.
I can get it. No problem that :
https://www.nesine.com/iddaa/default.aspx
But I do not get it :
http://www.iddaa.com/program/futbol.html
in my opinion problem url routing.
I use the code
public static string GetSourceCode(string url)
{
HttpWebRequest req = (HttpWebRequest)WebRequest.Create(url);
HttpWebResponse resp = (HttpWebResponse)req.GetResponse();
StreamReader sRead = new StreamReader(resp.GetResponseStream(), Encoding.UTF8);
return sRead.ReadToEnd();
}
help me please.
Thanks.
The http://www.iddaa.com/program/futbol.html url is heavily using javascript and AJAX in order to load its dynamic content. It's a SPA application. The HttpWebRequest class in .NET doesn't execute any javascript. It is simply loading the contents returned by the server at the specified location.
By inspecting the network traffic you will notice that the information you are interested in is located on the following location: http://www.iddaa.com/program/data?sportId=1&date=&sortType=&marketType=1
So you can go ahead and scrape this url:
using System;
using System.IO;
using System.Net;
static class Program
{
static void Main()
{
var request = WebRequest.Create("http://www.iddaa.com/program/data?sportId=1&date=&sortType=&marketType=1");
using (var response = request.GetResponse())
using (var stream = response.GetResponseStream())
using (var reader = new StreamReader(stream))
{
string responseHtml = reader.ReadToEnd();
Console.WriteLine(responseHtml);
}
}
}
Obviously, as always with scraping, if the website changes the location of this dynamic url your application will stop working. That's why it's better to use an API if the website provides one. Contact the authors of the website for more information.
Related
I'm just a beginner on the .NET world and I've created a web api (.NET 4.5.2) and I'm using the annotation [Authorize] above my controllers like shown below:
[Authorize]
public class PhasesController : ApiController
{
private TestReportEntities db = new TestReportEntities();
// GET: api/Phases
public IQueryable<Phase> GetPhase()
{
return db.Phase;
}
}
I've already created my DB and I'm using the default tables that the web.api uses to manage the access, as you can see on this image:
My tables
I've already done a method to request to my web api, in another project/solution, it's working fine when I remove the annotation [Authorize] from my web api controllers.
this is an example about how I'm requesting my api:
public int GetCurrentIdPhase(int idProject)
{
int phaseId = -1;
WebRequest request = WebRequest.Create(string.Concat(URL, string.Format("api/phases/?idProject={0}", idProject)));
using (var resp = (HttpWebResponse)request.GetResponse())
{
using (var reader = new StreamReader(resp.GetResponseStream()))
{
string objText = reader.ReadToEnd();
var phase = JsonConvert.DeserializeObject<List<Phase>>(objText);
phaseId = phase[0].id;
}
}
if (phaseId != -1)
{
return phaseId;
}
else
{
throw new Exception("Phase not found");
}
}
At the end of the day my questions are:
How can I request a token to my api (POST - www.myApi/token) using the example above?
How can I use the token, once I've got it, on every request to my API?
if you can help me I would really appreciate it.
Thanks.
I've created a method to get the Token from my Web API, this is the method:
var request = (HttpWebRequest)WebRequest.Create(string.Concat(URL, "token"));
var postData = "grant_type=password";
postData += string.Format("&userName={0}", user);
postData += string.Format("&password={0}", pass);
var data = Encoding.ASCII.GetBytes(postData);
request.Method = "POST";
request.ContentType = "application/x-www-form-urlencoded";
request.ContentLength = data.Length;
using (var stream = request.GetRequestStream())
{
stream.Write(data, 0, data.Length);
}
var response = (HttpWebResponse)request.GetResponse();
string objText = new StreamReader(response.GetResponseStream()).ReadToEnd();
var requestedToken = (JObject)JsonConvert.DeserializeObject(objText);
token = string.Concat(token, requestedToken["access_token"].Value<string>());
And to request something to my API all I need to do is just add the token on the header of all requests like shown on the line below:
request.Headers.Add(HttpRequestHeader.Authorization, getToke());
Hope it can help someone else who is beginning to work with .NET web API like me.
Regards.
Im assuming the "GetCurrentIdPhase" call is from an unrelated app with unrealted auth - if any auth.
The difficulty here is in using Authorize and the traidtional browser authentication flow. Here's an example of changing the pipeline a bit to use a different auth form for using console/desktop apps. You don't say where you are calling GetCurrentIdPhase from so I'll have to assume either a separate app. If its a web app and you are authenticated using the same tables, then you will have to share the token between them using for ex. the url blackice provided above.
If the app is a desktop/console/etc (not another app that the user had to auth against the same tables) then you can try this approach to change how auth is done to make it easier to access.
MVC WebAPI authentication from Windows Forms
I'm working on a web service using ASP.NET MVC's new WebAPI that will serve up binary files, mostly .cab and .exe files.
The following controller method seems to work, meaning that it returns a file, but it's setting the content type to application/json:
public HttpResponseMessage<Stream> Post(string version, string environment, string filetype)
{
var path = #"C:\Temp\test.exe";
var stream = new FileStream(path, FileMode.Open);
return new HttpResponseMessage<Stream>(stream, new MediaTypeHeaderValue("application/octet-stream"));
}
Is there a better way to do this?
Try using a simple HttpResponseMessage with its Content property set to a StreamContent:
// using System.IO;
// using System.Net.Http;
// using System.Net.Http.Headers;
public HttpResponseMessage Post(string version, string environment,
string filetype)
{
var path = #"C:\Temp\test.exe";
HttpResponseMessage result = new HttpResponseMessage(HttpStatusCode.OK);
var stream = new FileStream(path, FileMode.Open, FileAccess.Read);
result.Content = new StreamContent(stream);
result.Content.Headers.ContentType =
new MediaTypeHeaderValue("application/octet-stream");
return result;
}
A few things to note about the stream used:
You must not call stream.Dispose(), since Web API still needs to be able to access it when it processes the controller method's result to send data back to the client. Therefore, do not use a using (var stream = …) block. Web API will dispose the stream for you.
Make sure that the stream has its current position set to 0 (i.e. the beginning of the stream's data). In the above example, this is a given since you've only just opened the file. However, in other scenarios (such as when you first write some binary data to a MemoryStream), make sure to stream.Seek(0, SeekOrigin.Begin); or set stream.Position = 0;
With file streams, explicitly specifying FileAccess.Read permission can help prevent access rights issues on web servers; IIS application pool accounts are often given only read / list / execute access rights to the wwwroot.
For Web API 2, you can implement IHttpActionResult. Here's mine:
using System;
using System.IO;
using System.Net;
using System.Net.Http;
using System.Net.Http.Headers;
using System.Threading;
using System.Threading.Tasks;
using System.Web;
using System.Web.Http;
class FileResult : IHttpActionResult
{
private readonly string _filePath;
private readonly string _contentType;
public FileResult(string filePath, string contentType = null)
{
if (filePath == null) throw new ArgumentNullException("filePath");
_filePath = filePath;
_contentType = contentType;
}
public Task<HttpResponseMessage> ExecuteAsync(CancellationToken cancellationToken)
{
var response = new HttpResponseMessage(HttpStatusCode.OK)
{
Content = new StreamContent(File.OpenRead(_filePath))
};
var contentType = _contentType ?? MimeMapping.GetMimeMapping(Path.GetExtension(_filePath));
response.Content.Headers.ContentType = new MediaTypeHeaderValue(contentType);
return Task.FromResult(response);
}
}
Then something like this in your controller:
[Route("Images/{*imagePath}")]
public IHttpActionResult GetImage(string imagePath)
{
var serverPath = Path.Combine(_rootPath, imagePath);
var fileInfo = new FileInfo(serverPath);
return !fileInfo.Exists
? (IHttpActionResult) NotFound()
: new FileResult(fileInfo.FullName);
}
And here's one way you can tell IIS to ignore requests with an extension so that the request will make it to the controller:
<!-- web.config -->
<system.webServer>
<modules runAllManagedModulesForAllRequests="true"/>
For those using .NET Core:
You can make use of the IActionResult interface in an API controller method, like so.
[HttpGet("GetReportData/{year}")]
public async Task<IActionResult> GetReportData(int year)
{
// Render Excel document in memory and return as Byte[]
Byte[] file = await this._reportDao.RenderReportAsExcel(year);
return File(file, "application/vnd.openxmlformats", "fileName.xlsx");
}
This example is simplified, but should get the point across. In .NET Core this process is so much simpler than in previous versions of .NET - i.e. no setting response type, content, headers, etc.
Also, of course the MIME type for the file and the extension will depend on individual needs.
Reference: SO Post Answer by #NKosi
While the suggested solution works fine, there is another way to return a byte array from the controller, with response stream properly formatted :
In the request, set header "Accept: application/octet-stream".
Server-side, add a media type formatter to support this mime type.
Unfortunately, WebApi does not include any formatter for "application/octet-stream". There is an implementation here on GitHub: BinaryMediaTypeFormatter (there are minor adaptations to make it work for webapi 2, method signatures changed).
You can add this formatter into your global config :
HttpConfiguration config;
// ...
config.Formatters.Add(new BinaryMediaTypeFormatter(false));
WebApi should now use BinaryMediaTypeFormatter if the request specifies the correct Accept header.
I prefer this solution because an action controller returning byte[] is more comfortable to test. Though, the other solution allows you more control if you want to return another content-type than "application/octet-stream" (for example "image/gif").
For anyone having the problem of the API being called more than once while downloading a fairly large file using the method in the accepted answer, please set response buffering to true
System.Web.HttpContext.Current.Response.Buffer = true;
This makes sure that the entire binary content is buffered on the server side before it is sent to the client. Otherwise you will see multiple request being sent to the controller and if you do not handle it properly, the file will become corrupt.
The overload that you're using sets the enumeration of serialization formatters. You need to specify the content type explicitly like:
httpResponseMessage.Content.Headers.ContentType = new MediaTypeHeaderValue("application/octet-stream");
You could try
httpResponseMessage.Content.Headers.Add("Content-Type", "application/octet-stream");
You can try the following code snippet
httpResponseMessage.Content.Headers.Add("Content-Type", "application/octet-stream");
Hope it will work for you.
I have a webservice which i need to call in .net application. The link looks like this.
http://www.contoso.com/student?id=12345
This will work only when its called like this. For rest of the this i dont have access. ie if i call it on a browser without the querystring it will not work. but with querystring it will return an XML data.
Now, when i call this in the .net application its not working?
How can I call this in a .NET application?
The Normal Webservice Importing methods are not working since it needs a querystring with value and we dont have access to the links which doesnt have the querystring.
How are you currently trying to download it?
A very simple way to do this is to use the HttpWebRequest and HttpWebResponse classes;
public XmlDocument GetStudentXml(int studentId)
{
XmlDocument targetXml = new XmlDocument();
HttpWebRequest webRequest = (HttpWebRequest)WebRequest.Create(String.Format("http://www.contoso.com/student?id={0}", studentId));
webRequest.Credentials = System.Net.CredentialCache.DefaultCredentials;
webRequest.Accept = "text/xml";
HttpWebResponse webResponse = (HttpWebResponse)webRequest.GetResponse();
using (Stream responseStream = webResponse.GetResponseStream())
{
XmlTextReader reader = new XmlTextReader(responseStream);
targetXml.Load(reader);
reader.Close();
}
webResponse.Close();
return targetXml;
}
This method simply creates a HttpWebRequest, initializes it with the URL (via String.Format so as to append the student id), some windows credentials and the expected content type.
It then calls the remote address via the GetResponse method. The response is then loaded into a stream, and an XmlTextReader is used to load the Xml data from the response stream into the XmlDocument, which is then returned to the caller.
You can also use WebClient and XDocument to achieve the same thing:
string url = String.Format("http://www.contoso.com/student?id={0}", studentId);
string remoteXml;
using (var webClient = new WebClient())
{
remoteXml = webClient.DownloadString(url);
}
XDocument doc = XDocument.Parse(remoteXml);
I have a website that I have registered as a facebook app - I now have an app ID.
My website is ASP.net C#. When the user clicks a button I'd like it to post a pre-defined message to their wall. I'm expecting Facebook to present a login dialog to the user - they login and grant publish permission to for my website app.
Does anyone have any sample code that would do this? I think I need to use the graph API but all the examples I've seen use PHP - which I know nothing about. I'm looking for an example that would use Java Script (of which I know almost nothing) or C# (beautiful!).
* Update *
I have managed to get the access_token. Now I make a call through the Facebook C# API to post to the wall. I get the error message:
(#803) Some of the aliases you requested do not exist: profile_id
I've stepped through the api code and found that it is trying to post to the following address: {https://graph.facebook.com/PROFILE_ID/feed}, the post data is: message=Sample+message+from+c%23+sdk&access_token=199209316768200|2.1avFTZuDGR4HJ7jPFeaO3Q__.3600.1302897600.1-100000242760733|R4DkNDf4JCb6B2F64n5TSQwBqvM
I'm pretty sure my token should be valid. Prior to requesting access token I requested publish_stream on the app authorization request as follows:
Response.Redirect ("https://www.facebook.com/dialog/oauth?client_id=" + myAppId + "&redirect_uri=" + myURL + "&scope=publish_stream");
The sdk code that actually makes the request is as follows:
private string MakeRequest(Uri url, HttpVerb httpVerb,
Dictionary<string, string> args)
{
if (args != null && args.Keys.Count > 0 && httpVerb == HttpVerb.GET)
{
url = new Uri(url.ToString() + EncodeDictionary(args, true));
}
HttpWebRequest request = WebRequest.Create(url) as HttpWebRequest;
request.Method = httpVerb.ToString();
if (httpVerb == HttpVerb.POST)
{
string postData = EncodeDictionary(args, false);
ASCIIEncoding encoding = new ASCIIEncoding();
byte[] postDataBytes = encoding.GetBytes(postData);
request.ContentType = "application/x-www-form-urlencoded";
request.ContentLength = postDataBytes.Length;
Stream requestStream = request.GetRequestStream();
requestStream.Write(postDataBytes, 0, postDataBytes.Length);
requestStream.Close();
}
try
{
using (HttpWebResponse response
= request.GetResponse() as HttpWebResponse)
{
StreamReader reader
= new StreamReader(response.GetResponseStream());
return reader.ReadToEnd();
}
}
Can anyone see what I'm doing wrong?
Many thanks,
Rob.
First of all, you need to take care of Authentication. You need to create an Application, and use OAuth to get hold of the access token. It's all described in the Authentication guide.
To post something to the user's wall, take a look at the Graph API under Publishing.
As a start, you could use Facebook's C# SDK
You could look to use a .NET library like http://facebooknet.codeplex.com/ to do this. There are a couple out there, I just remembered this one...
HTH.
I created a video showing how to do this using OG: http://www.markhagan.me/Samples/Grant-Access-And-Post-As-Facebook-User-ASPNet
In case you don't have time to watch the video, here is the full code:
using System;
using System.Collections.Generic;
using System.IO;
using System.Linq;
using System.Net;
using System.Web;
using System.Web.UI;
using System.Web.UI.WebControls;
using Facebook;
namespace FBO
{
public partial class facebooksync : System.Web.UI.Page
{
protected void Page_Load(object sender, EventArgs e)
{
CheckAuthorization();
}
private void CheckAuthorization()
{
string app_id = "374961455917802";
string app_secret = "9153b340ee604f7917fd57c7ab08b3fa";
string scope = "publish_stream,manage_pages";
if (Request["code"] == null)
{
Response.Redirect(string.Format(
"https://graph.facebook.com/oauth/authorize?client_id={0}&redirect_uri={1}&scope={2}",
app_id, Request.Url.AbsoluteUri, scope));
}
else
{
Dictionary<string, string> tokens = new Dictionary<string, string>();
string url = string.Format("https://graph.facebook.com/oauth/access_token?client_id={0}&redirect_uri={1}&scope={2}&code={3}&client_secret={4}",
app_id, Request.Url.AbsoluteUri, scope, Request["code"].ToString(), app_secret);
HttpWebRequest request = WebRequest.Create(url) as HttpWebRequest;
using (HttpWebResponse response = request.GetResponse() as HttpWebResponse)
{
StreamReader reader = new StreamReader(response.GetResponseStream());
string vals = reader.ReadToEnd();
foreach (string token in vals.Split('&'))
{
//meh.aspx?token1=steve&token2=jake&...
tokens.Add(token.Substring(0, token.IndexOf("=")),
token.Substring(token.IndexOf("=") + 1, token.Length - token.IndexOf("=") - 1));
}
}
string access_token = tokens["access_token"];
var client = new FacebookClient(access_token);
client.Post("/me/feed", new { message = "markhagan.me video tutorial" });
}
}
}
}
I am using this http://facebooksdk.codeplex.com/ . I am using the latest stable release, easy to use. To comment, just post with /OBJECT_ID/comments for more, refer http://developers.facebook.com/docs/reference/api/#publishing and http://developers.facebook.com/docs/reference/api/post/
How about trying this API tha I recently developed to make integrating with Facebook easier.
Here is a code sample for you there's more documentation on the site.
Authenticating Users
Imports Branches.FBAPI
...
Dim SI As New SessionInfo("[application_id]","applicaiton_secret")
'Redirects user to facebooks
SI.AuthenticateUser("http://[my url]", New SessionInfo.PermissionsEnum(){SessionInfo.PermissionsEnum.email, SessionInfo.PermissionsEnum.read_stream}))
'Called when the user is returned to your page
Dim FSR = FS.ReadFacebooAuthResponse
Response.Write(FSR.Access_Token)
Response.Write(FSR.UserID)
Making Posts
Imports Branches.FBAPI
...
Dim SI As New SessionInfo("[access_token]"))
Dim Posts = New Functions.Posts(SI)
Dim P As New Post
P.name = "name of post"
P.message = "message"
P.link = "www.cnn.com"
P.caption = "my caption"
Posts.PublishCreate("[object ID to post to]", P)
Dim PostID = P.id
Getting stuff from the graph.
Dim SI As New SessionInfo("[access_token]"))
Dim Req New Functions.Requests(SI)
Dim User = Req.GetUserInfo("[optional user ID]")
Response.Write(U.name)
I don't want to do anything fancy on Twitter except post to it via my site once a day. I have searched around a bit and there are all sorts of super-complex ways to do every little thing that Twitter does, but there seems to be little documentation on how to do the simplest thing, which is make a post!
Does anyone know how to do this? Or can you at least point me in the right direction? I don't need full wrappers or anything (http://apiwiki.twitter.com/Libraries#C/NET), just one simple function that will post to Twitter.
Thanks!
This is the easiest implementation ever. Up and running in under 2 minutes: Twitterizer
Its fairly simple; you just need to post an xml file to a web page using webrequest.create. This example is close (assumes you have the xml for the message in another place and just pass it into twitterxml variable as a string. The url might not be the right one; found it on this [page][1] which defines the interface
WebRequest req = null;
WebResponse rsp = null;
try
{
string twitterXML = "xml as string";
string uri = "http://twitter.com/statuses/update.format";
req = WebRequest.Create(uri);
//req.Proxy = WebProxy.GetDefaultProxy(); // Enable if using proxy
req.Method = "POST"; // Post method
req.ContentType = "text/xml"; // content type
// Wrap the request stream with a text-based writer
StreamWriter writer = new StreamWriter(req.GetRequestStream());
// Write the XML text into the stream
writer.WriteLine(twitterXML);
writer.Close();
// Send the data to the webserver
rsp = req.GetResponse();
}
[1]: http://apiwiki.twitter.com/Twitter-REST-API-Method%3A-statuses update
There are a couple different ways of doing this, they vary depending on the tools you want to use and have access to. Option 1 will work right out of the box, but the coding can be complicated. Option 3 you will have to download tools for, but once there installed and loaded you should be able to consume the twitter api very quickly.
Use WebRequest.Create to create/send messages to remote endpoints
Use WCF, create a mirror endpoint and access the twitter api using client only endpoint.
Use the WCF REST Starter Kit Preview 2, which has a new class called the HttpClient. I would have to recommend this technique if you can. Here is a great video Consuming a REST Twitter Feed in under 3 minutes.
Here is a sample of using the WCF REST Starter Kit's HttpClient:
public void CreateFriendship(string friend)
{
using (var client = new HttpClient())
{
var url = string.Format("http://www.twitter.com/friendships/create/{0}.xml?follow=true", friend);
client.Post(url)
.CheckForTwitterError()
.EnsureStatusIs(HttpStatusCode.OK);
}
}
Add a comment if you'd like more info about a particular method.
Update:
For Option #1 see this question: Remote HTTP Post with C#
There are a few ways of doing this, you can check out http://restfor.me/twitter and it will give you the code from RESTful documentation.
Essentially making any authenticated call you can follow this logic:
///
/// Executes an HTTP POST command and retrives the information.
/// This function will automatically include a "source" parameter if the "Source" property is set.
///
/// The URL to perform the POST operation
/// The username to use with the request
/// The password to use with the request
/// The data to post
/// The response of the request, or null if we got 404 or nothing.
protected string ExecutePostCommand(string url, string userName, string password, string data) {
WebRequest request = WebRequest.Create(url);
request.ContentType = "application/x-www-form-urlencoded";
request.Method = "POST";
if (!string.IsNullOrEmpty(userName) && !string.IsNullOrEmpty(password)) {
request.Credentials = new NetworkCredential(userName, password);
byte[] bytes = Encoding.UTF8.GetBytes(data);
request.ContentLength = bytes.Length;
using (Stream requestStream = request.GetRequestStream()) {
requestStream.Write(bytes, 0, bytes.Length);
using (WebResponse response = request.GetResponse()) {
using (StreamReader reader = new StreamReader(response.GetResponseStream())) {
return reader.ReadToEnd();
}
}
}
}
return null;
}