I would like to utilize oData without HTTP.
Scenario: "Load Balancer" oData-capable service receives requests from
clients, puts them into a queue (in serialized form), then "background" workers pick up
messages from the queue and process the request (get data from data
storage and provide response)
And it seems that such functionality either it is not exposed in MS oData libs, or it is so simple and obvious (though not for me) that nobody cares to highlight it in the docs.
I see it something like (pseudo-code)
var model = GetEdmModel();
var processor = GetProcessor(); // something like ODataController in AspNet.oData - contains functions and whatever
var request = GetRequestFromQueueAndParse();
var uriPath = request.Path; // like "/Books"
var queryString = request.QueryString; // like "$filter=price lt 50"
var method = request.Method; // GET or POST or ...
var body = request.Body;
// *** here is what I am looking for ***
var response = SomeMagicODataHelper.ProcessQuery(model, processor, method, uriPath, queryString, body);
ProvideResponseBackToBalancer(response);
Is there something alike provided by (preferable) standard MS oData library, or as a third-party library?
Related
Currently, I have dozens of .NET services hosted on various machines that show up as Resources on my AppInsights Application Map, which also shows their dependencies with respect to each other, based on the HTTP requests they make.
However, the relationships between services that communicate through NServiceBus (RabbitMQ) are not shown. Now, I am able to show the messages that are either sent or handled by a service via calls to TelemetryClient.TrackXXX(), but not connect Resources on the map using this information.
I have even gone so far as to attach the parent operation ID from the NSB message sender to the message itself, and assign it to the telemetry object in the receiver, but there is still no line drawn between the services in the Application Map.
To reiterate, this is what I'm getting in the Application Map:
(NSB Message Sender) --> (Message sent/handled)
And this is what I want:
(NSB Sender) --> (Receiver)
The services in question are .NET Core 3.1.
I cannot provide the code, as this is for my work, but any help would be greatly appreciated. I've searched everywhere, and even sources that seemed like they would help, didn't.
(not signed in, posting from work)
Alright, I finally got it. My approach to correlate AppInsights resources using their NSB communication is to mimic HTTP telemetry correlation.
Below is an extension method I wrote for AppInsights' TelemetryClient. I made a subclass named RbmqMessage:NServiceBus.IMessage, given my applications use RBMQ, and gave it the following properties for the sake of correlation (all set in the service that sends the message) :
parentId: equal to DependencyTelemetry.Id
opId: value is the same in the sender's DependencyTelemetry and the receiver's RequestTelemetry. Equal to telemetry.context.operation.id
startTime: DateTime.Now was good enough for my purposes
The code in the service that sends the NSB message:
public static RbmqMessage TrackRbmq(this TelemetryClient client, RbmqMessage message)
{
var msg = message;
// I ran into some issues with Reflection
var classNameIdx = message.ToString().LastIndexOf('.') + 1;
var messageClassName = message.ToString().Substring(classNameIdx);
var telemetry = new DependencyTelemetry
{
Type = "RabbitMQ",
Data = "SEND "+messageClassName,
Name = "SEND "+messageClassName,
Timestamp = DateTime.Now,
Target = "RECEIVE "+messageClassName //matches name in the service receiving this message
};
client.TrackDependency(telemetry);
msg.parentId = telemetry.Id;
msg.opId = telemetry.Context.Operation.Id; //this wont have a value until TrackDependency is called
msg.startTime = telemetry.Timestamp;
return msg;
}
The code where you send the NSB message:
var msg = new MyMessage(); //make your existing messages inherit RbmqMessage
var correlatedMessage = _telemetryClient.TrackRbmq(msg);
MessageSession.Publish(correlatedMessage); //or however the NSB message goes out in your application
The extension method in the NServiceBus message-receiving service:
public static void TrackRbmq(this TelemetryClient client, RbmqMessage message)
{
var classnameIdx = message.ToString().LastIndexOf('.')+1;
var telemetry = new RequestTelemetry
{
Timestamp = DateTime.Now,
Name = "RECEIVE "+message.ToString().Substring(classNameIdx)
};
telemetry.Context.Operation.ParentId = message.parentId;
telemetry.Context.Operation.Id = message.opId;
telemetry.Duration = message.startTime - telemetry.Timestamp;
client.TrackRequest(telemetry);
}
And finally, just track and send the message:
var msg = new MyMessage();
_telemetryClient.TrackRbmq(msg);
MessagePipeline.Send(msg); //or however its sent in your app
I hope this saves someone the trouble I went through.
We are sending the purchase events from the server with code like this:
using (var httpClient = new RestClient())
{
httpClient.SendAsync(new HttpRequestMessage
{
RequestUri = new Uri(url),
Method = HttpMethod.Get
});
}
But around 15-20% of the events never gets registered in GA.
Google always seem to respond with a GIF and status code 200, so it is hard to tell which events are not processed successfully.
In the beginning we were using the javascript API to send the event, but when we switched to server side, we copied the request it was creating and tried to replicate it with HttpClient.
The request send looks like the following:
https://www.google-analytics.com/collect?v=1&_v=j47&a=817546713&t=event&ni=0&_s=1&
dl=#scheme + host + pathAndQuery#&dp=#path#&dt=#path#&ul=#browser language#&de=#browser encoding#&sd=#bit#&sr=#screen resolution#&vp=#viewable browser area#&cid=#Id taken from the _ga cookie#&je=0&fl=24.0%20r0&ec=Ecommerce&ea=purchase &_u=SCEAAAALI20%25~&jid=&tid=#TrackingId#>m=#TagManagerId#&ti=#OrderId#&ta=&
tr=#TotalPrice#&tt=#TotalTax#&ts=#ShippingPrice#&tcc=#VoucherCode# &pa=purchase&cu=#CurrencyCode#&pr1nm=#ProducteName#&pr1id=#ProductId#&pr1pr=#ProductPrice#&pr1br=#Brand#&pr1ca=&pr1va=#Variant#&pr1qt=#Quantity#&z=#Randomly generated unique id#
Any ideas about what is wrong or how to debug it is welcome
You shouldn't do that on the backend. The correct way is to do that on the frontend
The easiest and the correct way is to send data to your dataLayer and then in GTM send an event to GA.
P.S. In your C# code I can see the problem that you are not awaiting async method. If your method is not async, then you can use it like that:
var temp = httpClient.SendAsync(new HttpRequestMessage
{
RequestUri = new Uri(url),
Method = HttpMethod.Get
}).Result;
We have a callout to another API within a javascript policy:
var calloutresponse, status, headers = {
'key':context.getVariable("request.header.key")
};
var myRequest = new Request(url, "POST", headers, data);
var exchange = httpClient.send(myRequest);
exchange.waitForComplete();
var calloutResponse = exchange.getResponse();
context.setVariable("calloutstatus",calloutResponse.status.code);
context.setVariable("calloutresponse",calloutResponse.content);
There are times when this callout takes an exceptionally long time and we would like to be able to set a timeout limit for it (like one would a target endpoint) and have the calloutResponse.status.code be a 503.
Is there a value which can be set for either the httpClient or Request to set this? I have looked through the Apigee documentation as well as here and can't find anything.
In the xml policy that calls the script, there's a timeLimit property that defaults to 200ms. You can change this at will.
how can i directly call my own server side function using XMLHttpRequest.
suppose i have one static webmethod in my aspx file then how can i call it by XMLHttpRequest. what header info i need to pass to asp.net engine and as a result asp.net engine can invoke my method and return the response back in the out going stream.
this way i need to call my server side method
<script type="text/javascript">
var request;
// A
// Here is the new function that is called when the user submits the form.
// This example uses POST.
function submitCallback() {
var inputValue = document.getElementById("SearchInput").value;
var sendStr = "name=" + inputValue;
request = new XMLHttpRequest();
// B
// Specify the POST method and send it.
request.open("POST", "Default.aspx/Getdata");
request.onreadystatechange = readyCallback;
request.setRequestHeader("Content-Type", "application/x-www-form-urlencoded");
request.setRequestHeader("IsLookup", "true");
request.send(sendStr);
}
</script>
please guide me....thanks
I believe you are probably referring to ASP.NET Page Methods when you say One static webmethod in my aspx file. ASP.NET page methods (or web services for consumption in JS) uses JSON serialization for i/p and o/p. So you need to set JSON as the content type for the request and actually send the JSON string in the body i.e.
...
var sendStr = "{name:" + inputValue + "}";
...
request.setRequestHeader("Content-Type", "application/json");
...
request.send(sendStr);
Further, I will suggest you to use jquery or ASP.NET generated proxies (by ScriptManager) instead of directly coding against XmlHttpRequest. See this article for how to use jquery for calling Page methods.
I think, the easiest way will be usage of jQuery for ajax requests: http://api.jquery.com/category/ajax/
Our Situation:
Our team needs to retrieve log information from a 3rd party website (Specifically, this log
information is call logs -- our client rents an 866 number. When calls come in, they assist
people and need to make notes accordingly in our application that will correspond with the
current call). Our client has a web account with the 3rd party that allows them to view the
current call logs (date/time, phone number, amount of time on each call, etc).
I contacted the developer of their website and inquired about API or any other means of syncing
our database with their constantly updating database. They currently DO NOT support API. I
informed them of my situation and they are perfectly fine with any way we can retrieve the
information (bot/crawler). *The 3rd party said that they are working on API but could not give
us a general timeline as to when it will be up... and as with every client, they need to start
production ASAP.
I completely understand that if the 3rd party were to change their HTML layout, it may cause a
slight headache for us (sorting the data from the webpage). That being said, this is a temporary
solution to a long term issue. Once they implement their API, we will switch them over to it.
So my question is this:
What is the best way to log into the 3rd party website (see image: http://i903.photobucket.com/albums/ac239/jreedinc/customtf.jpg)
and retrieve certain HTML pages? We have reviewed source codes of webcrawlers, but none of them
have the capability of storing cookies and posting information back to the website (with log in information). We would prefer to do this in ASP.NET.
Is there another way to accomplish logging on to the website, then retrieving said information?
The classes you'll need to use are in the System.Net namespace. Below is some quick and dirty proof of concept code. To login in to a site that uses form login + cookies for security and then scrape the HTML output of a page.
In order to parse the HTML results you'll need to use an additional tool.
Possible HTML parsing tools.
SgmlReader, can convert HTML to XML. You then use .NET's XML features to extract data from the XML.
http://code.msdn.microsoft.com/SgmlReader
HTML Agility Pack, allows XPath queries against HTML documents.
http://htmlagilitypack.codeplex.com/
using System;
using System.Collections.Generic;
using System.Linq;
using System.Text;
class WebWorker {
/// <summary>
/// Cookies for use by web worker
/// </summary>
private System.Collections.Generic.List `<System.Net.Cookie` > cookies = new List < System.Net.Cookie > ();
public string GetWebPageContent(string url) {
System.Net.HttpWebRequest request = (System.Net.HttpWebRequest) System.Net.WebRequest.Create(url);
System.Net.CookieContainer cookieContainer = new System.Net.CookieContainer();
request.CookieContainer = cookieContainer;
request.Method = "GET";
//add cookies to maintain session state
foreach(System.Net.Cookie c in this.cookies) {
cookieContainer.Add(c);
}
System.Net.HttpWebResponse response = request.GetResponse() as System.Net.HttpWebResponse;
System.IO.Stream responseStream = response.GetResponseStream();
System.IO.StreamReader sReader = new System.IO.StreamReader(responseStream);
System.Diagnostics.Debug.WriteLine("Content:\n" + sReader.ReadToEnd());
return sReader.ReadToEnd();
}
public string Login(string url, string userIdFormFieldName, string userIdValue, string passwordFormFieldName, string passwordValue) {
System.Net.HttpWebRequest request = (System.Net.HttpWebRequest) System.Net.WebRequest.Create(url);
System.Net.CookieContainer cookieContainer = new System.Net.CookieContainer();
request.CookieContainer = cookieContainer;
request.Method = "POST";
request.ContentType = "application/x-www-form-urlencoded";
string postData = System.Web.HttpUtility.UrlEncode(userIdFormFieldName) + "=" + System.Web.HttpUtility.UrlEncode(userIdValue) +
"&" + System.Web.HttpUtility.UrlEncode(passwordFormFieldName) + "=" + System.Web.HttpUtility.UrlEncode(passwordValue);
request.ContentLength = postData.Length;
request.AllowAutoRedirect = false; //allowing redirect seems to loose cookies
byte[] postDataBytes = System.Text.Encoding.UTF8.GetBytes(postData);
System.IO.Stream requestStream = request.GetRequestStream();
requestStream.Write(postDataBytes, 0, postDataBytes.Length);
System.Net.HttpWebResponse response = request.GetResponse() as System.Net.HttpWebResponse;
// System.Diagnostics.Debug.Write(WriteLine(new StreamReader(response.GetResponseStream()).ReadToEnd());
System.IO.Stream responseStream = response.GetResponseStream();
System.IO.StreamReader sReader = new System.IO.StreamReader(responseStream);
System.Diagnostics.Debug.WriteLine("Content:\n" + sReader.ReadToEnd());
this.cookies.Clear();
if (response.Cookies.Count > 0) {
for (int i = 0; i < response.Cookies.Count; i++) {
this.cookies.Add(response.Cookies[i]);
}
}
return "OK";
}
} //end class
//sample to use class
WebWorker worker = new WebWorker();
worker.Login("http://localhost/test/default.aspx", "uid", "bob", "pwd", "secret");
worker.GetWebPageContent("http://localhost/test/default.aspx");
I used a tool recently called WebQL (its a web scraper tool that lets the developer use SQL like syntax to scrape information from web pages.
WebQL on Wikipedia
This is actually a relatively simple operation. What you need to do is get the page that the screenshot posts back to (something like login.php, etc) and then construct a webrequest to that page with the login data you have. You will most likely get back a cookiecontainer that will have your login cookie to use on all subsequent requests.
You can look at this MSDN article for the basics of how to do it, but their write-up is kind of confusing. Look at the community comments at the end for an example of how to post back page variables (like the username and password). You will need to make sure you pass the cookiecontainer around on subsequent requests.
Unfortunately .NET does not natively have something like WWW::Mechanize, but the Webclient does have an "upload value" which might make it easier. You will still have to manually parse the page to figure out what fields you need to pass.