I have to write an android client, in which i should use HttpComponents to connect to a specific Server on Port 8080.
For now, all i've found, was the Examplecode from the Apache-site, which is nearly perfect for what i need, except the Port it connects to:
if (isSet)
{
throw new IOException("Hostname or Port are not set!");
}
HttpClient httpclient = new DefaultHttpClient();
HttpGet httpget = new HttpGet(serverURL + ":" + serverPort + "/maps");
HttpResponse response = httpclient.execute(httpget);
HttpEntity entity = response.getEntity();
if (entity != null)
{
InputStream instream = entity.getContent();
int l;
byte[] tmp = new byte[2048];
while ((l = instream.read(tmp)) != -1)
{
}
}
Is there any way to change the Port?
Any help would be appreciated.
I know a little about HttpComponents, but I find an example code from Apache-site, which may help you. You can try to modify the code like below for your problem.
HttpHost target = new HttpHost("issues.apache.org", 443, "https");
HttpGet request = new HttpGet("/");
CloseableHttpResponse response = httpclient.execute(target, request);
Related
I am trying to implement the certificate authentication in .net core API(Server/target) and this API will be invoked in to another API(Client) .Here is the piece of code of client api which makes request to server/target api.But I'm facing an error on the server/target api .I'm running these two services from local and both certificates have already installed
Client side controller logic
[HttpGet]
public async Task<List<WeatherForecast>> Get()
{
List<WeatherForecast> weatherForecastList = new List<WeatherForecast>();
X509Certificate2 clientCert = Authentication.GetClientCertificate();
if (clientCert == null)
{
HttpActionContext actionContext = null;
actionContext.Response = new HttpResponseMessage(System.Net.HttpStatusCode.Forbidden)
{
ReasonPhrase = "Client Certificate Required"
};
}
HttpClientHandler requestHandler = new HttpClientHandler();
requestHandler.ClientCertificates.Add(clientCert);
requestHandler.ServerCertificateCustomValidationCallback += (sender, cert, chain, sslPolicyErrors) => true;
HttpClient client = new HttpClient(requestHandler)
{
BaseAddress = new Uri("https://localhost:11111/ServerAPI")
};
client.DefaultRequestHeaders
.Accept
.Add(new MediaTypeWithQualityHeaderValue("application/xml"));//ACCEPT head
using (var httpClient = new HttpClient())
{
//httpClient.DefaultRequestHeaders.Accept.Clear();
var request = new HttpRequestMessage()
{
RequestUri = new Uri("https://localhost:44386/ServerAPI"),
Method = HttpMethod.Get,
};
request.Headers.Add("X-ARR-ClientCert", clientCert.GetRawCertDataString());
httpClient.DefaultRequestHeaders.Accept.Add(new MediaTypeWithQualityHeaderValue("application/json"));//ACCEPT head
//using (var response = await httpClient.GetAsync("https://localhost:11111/ServerAPI"))
using (var response = await httpClient.SendAsync(request))
{
if (response.StatusCode == System.Net.HttpStatusCode.OK)
{
string apiResposne = await response.Content.ReadAsStringAsync();
weatherForecastList = JsonConvert.DeserializeObject<List<WeatherForecast>>(apiResposne);
}
}
}
return weatherForecastList;
}
authentication class
public static X509Certificate2 GetClientCertificate()
{
X509Store userCaStore = new X509Store(StoreName.TrustedPeople, StoreLocation.CurrentUser);
try
{
string str_API_Cert_Thumbprint = "aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa";
userCaStore.Open(OpenFlags.ReadOnly);
X509Certificate2Collection certificatesInStore = userCaStore.Certificates;
X509Certificate2Collection findResult = certificatesInStore.Find(X509FindType.FindByThumbprint, str_API_Cert_Thumbprint, false);
X509Certificate2 clientCertificate = null;
if (findResult.Count == 1)
{
clientCertificate = findResult[0];
if(System.DateTime.Today >= System.Convert.ToDateTime(clientCertificate.GetExpirationDateString()))
{
throw new Exception("Certificate has already been expired.");
}
else if (System.Convert.ToDateTime(clientCertificate.GetExpirationDateString()).AddDays(-30) <= System.DateTime.Today)
{
throw new Exception("Certificate is about to expire in 30 days.");
}
}
else
{
throw new Exception("Unable to locate the correct client certificate.");
}
return clientCertificate;
}
catch (Exception ex)
{
throw;
}
finally
{
userCaStore.Close();
}
}
Server/target api code
[HttpGet]
public IEnumerable<WeatherForecast> Getcertdata()
{
IHeaderDictionary headers = base.Request.Headers;
X509Certificate2 clientCertificate = null;
string certHeaderString = headers["X-ARR-ClientCert"];
if (!string.IsNullOrEmpty(certHeaderString))
{
//byte[] bytes = Encoding.ASCII.GetBytes(certHeaderString);
//byte[] bytes = Convert.FromBase64String(certHeaderString);
//clientCertificate = new X509Certificate2(bytes);
clientCertificate = new X509Certificate2(WebUtility.UrlDecode(certHeaderString));
var serverCertificate = new X509Certificate2(Path.Combine("abc.pfx"), "pwd");
if (clientCertificate.Thumbprint == serverCertificate.Thumbprint)
{
//Valida Cert
}
}
var rng = new Random();
return Enumerable.Range(1, 5).Select(index => new WeatherForecast
{
Date = DateTime.Now.AddDays(index),
TemperatureC = rng.Next(-20, 55),
Summary = Summaries[rng.Next(Summaries.Length)]
}).ToArray();
//return new List<WeatherForecast>();
}
You have much more problems here, the code is significantly flawed and insecure in various ways. Let's explain each issue:
HttpClient in using clause in client side controller logic
Although you expect to wrap anything that implements IDisposable in using statement. However, it is not really the case with HttpClient. Connections are not closed immediately. And with every request to client controller action, a new connection is established to remote endpoint, while previous connections sit in TIME_WAIT state. Under certain constant load, your HttpClient will exhaust TCP port pool (which is limited) and any new attempt to create a new connection will throw an exception. Here are more details on this problem: You're using HttpClient wrong and it is destabilizing your software
Microsoft recommendation is to re-use existing connections. One way to do this is to Use IHttpClientFactory to implement resilient HTTP requests. Microsoft article talks a bit about this problem:
Though this class implements IDisposable, declaring and instantiating
it within a using statement is not preferred because when the
HttpClient object gets disposed of, the underlying socket is not
immediately released, which can lead to a socket exhaustion problem.
BTW, you have created a client variable, but do not use it in any way.
Ignore certificate validation problems
The line:
requestHandler.ServerCertificateCustomValidationCallback += (sender, cert, chain, sslPolicyErrors) => true;
make you vulnerable to MITM attack.
you are doing client certificate authentication wrong
The line:
request.Headers.Add("X-ARR-ClientCert", clientCert.GetRawCertDataString());
It is not the proper way how to do client cert authentication. What you literally doing is passing certificate's public part to server. That's all. You do not prove private key possession which is required to authenticate you. The proper way to do so is:
requestHandler.ClientCertificates.Add(clientCert);
This will force client and server to perform proper client authentication and check if you possess the private key for certificate you pass (it is done in TLS handshake automatically). If you have ASP.NET on server side, then you read it this way (in controller action):
X509Certificate2 clientCert = Request.HttpContext.Connection.ClientCertificate
if (clientCert == null) {
return Unauthorized();
}
// perform client cert validation according server-side rules.
Non-standard cert store
In authentication class you open StoreName.TrustedPeople store, while normally it should be StoreName.My. TrustedPeople isn't designed to store certs with private key. It isn't a functional problem, but it is bad practice.
unnecessary try/catch clause in authentication class
If you purposely throw exceptions in method, do not use try/catch. In your case you simply rethrow exception, thus you are doing a double work. And this:
throw new Exception("Certificate is about to expire in 30 days.");
is behind me. Throwing exception on technically valid certificate? Really?
server side code
As said, all this:
IHeaderDictionary headers = base.Request.Headers;
X509Certificate2 clientCertificate = null;
string certHeaderString = headers["X-ARR-ClientCert"];
if (!string.IsNullOrEmpty(certHeaderString))
{
//byte[] bytes = Encoding.ASCII.GetBytes(certHeaderString);
//byte[] bytes = Convert.FromBase64String(certHeaderString);
//clientCertificate = new X509Certificate2(bytes);
clientCertificate = new X509Certificate2(WebUtility.UrlDecode(certHeaderString));
var serverCertificate = new X509Certificate2(Path.Combine("abc.pfx"), "pwd");
if (clientCertificate.Thumbprint == serverCertificate.Thumbprint)
{
//Valida Cert
}
}
must be replaced with:
X509Certificate2 clientCert = Request.HttpContext.Connection.ClientCertificate
if (clientCert == null) {
return Unauthorized();
}
// perform client cert validation according server-side rules.
BTW:
var serverCertificate = new X509Certificate2(Path.Combine("abc.pfx"), "pwd");
if (clientCertificate.Thumbprint == serverCertificate.Thumbprint)
{
//Valida Cert
}
This is another disaster in your code. You are loading the server certificate from PFX just to compare their thumbprints? So, you suppose that client will have a copy of server certificate? Client and server certificates must not be the same. Next thing is you are generating a lot of copies of server certificate's private key files. More private key files you generate, the slower the process is and you just generate a lot of garbage. More details on this you can find in my blog post: Handling X509KeyStorageFlags in applications
I am trying to specify the local endpoint for some HTTP requests that I need to send.
So, I have multiple IPs on my device (Wifi and Cellular) and would like to chose which one of those I want to use when sending HTTP request to get/post data from the server.
I learned that it is possible with HttpClient with .NET core 5.0 (that isn't supported with Xamarin)
HttpClient specify interface to bind to / make requests from
SocketsHttpHandler socketsHttpHandler = new SocketsHttpHandler();
socketsHttpHandler.ConnectCallback = async (context, token) =>
{
Socket s = new Socket(AddressFamily.InterNetwork, SocketType.Stream, ProtocolType.Tcp);
s.Bind(new IPEndPoint(IPAddress.Loopback, 0));
await s.ConnectAsync(context.DnsEndPoint, token);
s.NoDelay = true;
return new NetworkStream(s, ownsSocket: true);
};
using (HttpClient Client = new HttpClient(socketsHttpHandler))
{
Client.BaseAddress = new Uri("PutIPHere//");
HttpResponseMessage Response = await Client.GetAsync("");
// use Response as needed...
}
Another way is with reflection (but only with .NET framework), see L.B Answer
HttpClientHandler SetServicePointOptions(HttpClientHandler handler)
{
var field = handler.GetType().GetField("_startRequest", BindingFlags.NonPublic | BindingFlags.Instance); // Fieldname has a _ due to being private
var startRequest = (Action<object>)field.GetValue(handler);
Action<object> newStartRequest = obj =>
{
var webReqField = obj.GetType().GetField("webRequest", BindingFlags.NonPublic | BindingFlags.Instance);
var webRequest = webReqField.GetValue(obj) as HttpWebRequest;
webRequest.ServicePoint.BindIPEndPointDelegate = new BindIPEndPoint(BindIPEndPointCallback);
startRequest(obj); //call original action
};
field.SetValue(handler, newStartRequest); //replace original 'startRequest' with the one above
return handler;
}
private static IPEndPoint BindIPEndPointCallback(ServicePoint servicePoint, IPEndPoint remoteEndPoint, int retryCount)
{
return new IPEndPoint(IPAddress.Parse("PutIPHere"), 0);
}
Is there any way to specify the local endpoint for HTTP requests (using Xamarin)?
I would like to do it from the PLC, but also native to android and iOS should be ok.
Thanks in advance
I am a newbie into async programming and am trying to use the httpclient to fire bulk URL requests for the page content.
Here is my attempt:
private async void ProcessUrlAsyncWithHttp(HttpClient httpClient, string purl)
{
Stopwatch sw = new Stopwatch();
sw.Start();
HttpResponseMessage response = null;
try
{
Interlocked.Increment(ref _activeRequestsCount);
var request = new HttpRequestMessage()
{
RequestUri = new Uri(purl),
Method = HttpMethod.Get,
};
request.Headers.TryAddWithoutValidation("User-Agent", "MozillaMozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/39.0.2171.95 Safari/537.36");
request.Headers.TryAddWithoutValidation("Accept", "text/html,*.*");
request.Headers.TryAddWithoutValidation("Connection", "Keep-Alive");
request.Headers.TryAddWithoutValidation("Accept-Encoding", "gzip, deflate, sdch");
request.Headers.TryAddWithoutValidation("Accept-Language", "en-US,en;q=0.8");
response = await httpClient.SendAsync(request).ConfigureAwait(false);
string html = await response.Content.ReadAsStringAsync().ConfigureAwait(false);
response.Dispose();
if (IsCaptcha(html)) throw new Exception("Captcha was returned");
request.Dispose();
Interlocked.Increment(ref _successfulCalls);
}
catch (HttpRequestException hex)
{
Console.WriteLine("http:" + hex.Message);
Interlocked.Increment(ref _failedCalls);
}
catch (Exception ex)
{
Console.WriteLine(ex.GetType().AssemblyQualifiedName + " " + ex.Message);
Interlocked.Increment(ref _failedCalls);
}
finally
{
Interlocked.Decrement(ref _activeRequestsCount);
Interlocked.Decrement(ref _itemsLeft);
if (response != null) response.Dispose();
if (httpClient != null) httpClient.Dispose();
sw.Stop();
DateTime currentTime = DateTime.UtcNow;
TimeSpan elapsedTillNow = (currentTime - _overallStartTime).Duration();
Console.WriteLine("Left:" + _itemsLeft + ", Current execution:" + sw.ElapsedMilliseconds + " (ms), Average execution:" + Math.Round((elapsedTillNow.TotalMilliseconds / (_totalItems - _itemsLeft)), 0) + " (ms)");
lock(_syncLock)
{
if (_itemsLeft == 0)
{
_overallEndTime = DateTime.UtcNow;
this.DisplayTestResults();
}
}
}
}
As you can see I am passing an httpclient to the function and it gets destroyed everytime the URL is downloaded. I know this is an overkill and ideally we should be reusing the httpclient. But since I cant use a single httpclient with different proxies for each URL (the handler needs to be passed to the constructor of httpclient and cannot be changed, hence a fresh proxy cant be given without recreating the httpclient object), I needed to use this approach.
At the caller side, I have a pretty basic code:
public async void TestAsyncWithHttp()
{
ServicePointManager.DefaultConnectionLimit = 10;
//ServicePointManager.UseNagleAlgorithm = false;
List<string> urlList = SetUpURLList();
urlList = urlList.GetRange(1, 50);
_itemsLeft = urlList.Count();
_totalItems = _itemsLeft;
List<string> proxies = new List<string>();
proxies.Add("124.161.94.8:80");
proxies.Add("183.207.228.8:80");
proxies.Add("202.29.97.5:3128");
proxies.Add("210.75.14.158:80");
proxies.Add("203.100.80.81:8080");
proxies.Add("218.207.172.236:80");
proxies.Add("218.59.144.120:81");
proxies.Add("218.59.144.95:80");
proxies.Add("218.28.35.234:8080");
proxies.Add("222.88.236.236:83");
Random rnd = new Random();
foreach (string url in urlList)
{
int ind = rnd.Next(0, proxies.Count-1);
var httpClientHandler = new HttpClientHandler
{
Proxy = new WebProxy(proxies.ElementAt(ind), false),
UseProxy = true
};
HttpClient httpClient = new HttpClient(httpClientHandler);
//HttpClient httpClient = new HttpClient();
httpClient.Timeout = TimeSpan.FromMinutes(2);
ProcessUrlAsyncWithHttp(httpClient, url);
}
}
Question is:
1) Why the TCP ports gets closed for each request. I wanted to open the max connections number of ports and reuse them across calls. e.g in the example above I can have 10 concurrent connections. Hence I wanted this to open 10 TCP ports and the rest of the 40 requests could then use these 10 ports in tandem. This is a normal behaviour expected in httpwebrequest. I have a working code for using httpwebrequest that depicts this behavior of reusing ports. Can post the code of that on demand for anyone who might want to have a look. So its kind of weird that httpclient does not mimic this behaviour although it is based on httpwebrequest.
2) How do we assign autoredirect to false for such calls?
3) I intend to use this function for multiple calls - say around 50K. Anything wrong in the way the code is written that might need a correction
4) Lets assume that I somehow manage to use a single httpclient object instead of one object per request. What is the way to ensure I read cookies for all these individual requests and also alter them if necessary all the while remembering that I have a single httpclient class for the whole set of URL requests?
Tks
Kallol
In my experience (I once had a similar problem with TCP port congestion, because of ports always getting closed, when I was hitting a server with around 6000 connections a minute) it suffices to reuse the HttpClientHandler objects, which actually manage the connection pooling, and always recreate the HttpClient objects for each request (using the constructor with HttpClientManager parameter).
Hope this helps.
Matthias
have you tried putting the HttpClient code in class and create 10 classes, each with a HttpClient?
I am trying to Bind my Outbound IP for my webRequest
HttpWebRequest reqhttp = (HttpWebRequest)req;
reqhttp.ServicePoint.BindIPEndPointDelegate = new System.Net.BindIPEndPoint(BindIPEndPointCallback);
reqhttp.Credentials = null;
reqhttp.AuthenticationLevel = AuthenticationLevel.None;
reqhttp.Method = "POST";
reqhttp.ContentLength = send.Length;
reqhttp.ContentType = "text/xml";
Stream dataStream = reqhttp.GetRequestStream();
dataStream.Write(send, 0, send.Length);
dataStream.Close();
public delegate IPEndPoint BindIPEndPoint(ServicePoint servicePoint, IPEndPoint remoteEndPoint, int retryCount);
private IPEndPoint BindIPEndPointCallback(ServicePoint servicePoint, IPEndPoint remoteEndPoint, int retryCount)
{
return new IPEndPoint(IPAddress.Parse("111.111.11.11"), 0); //bind to a specific ip address on your server
}
for some reason when i do this, it is throwing an error
if fails to execute this line
Stream dataStream = reqhttp.GetRequestStream();
An existing connection was forcibly closed by the remote host
i don't understand what is wrong in here.
Can any one help to understand whats wrong in this code and fix the issue.
The GetRequestStream() method will first trigger BindIPEndPointDelegat, then it will try to connect to the remote server. If you bind to a local end point that does not exists, or the remote server is unavailable, you will get an exception.
try something like this
HttpWebRequest request = (HttpWebRequest)WebRequest.Create(myIP);
request.Proxy = myProxy;
ServicePoint sp = ServicePointManager.FindServicePoint(new Uri(myIP), myProxy);
sp.BindIpEndPointDelegate = new BindIpEndPoint(BindIpEndPointCallback);
HttpWebResponse = (HttpWebResponse)request.GetResponse();
I have WCF REST web service hosted by IIS, it works on HTTPS, I generate Certificate on IIS and assign Https to a port
I generate cer through IE browser. I create a test application and regardless Add a client certificate or not or even add a wrong certificate the connection take place and a I get correct response. I am wondering how the message was decrypted if there is no certificate sent.
Either the destination is not secured or I misunderstand the whole thing.
also
The error I have from the callback "CheckValidationResult()" is either
CertCN_NO_MATCH = 0x800B010F
or
"Unknown Certificate Problem" , the certificateProblem (parameter of CheckValidationResult) is 0 for this case
What is CertCN_NO_MATCH eror, what is CN?
See code below.
ServicePointManager.CertificatePolicy = new CertPolicy();
HttpWebRequest request = (HttpWebRequest)HttpWebRequest.Create(String.Format("https://{0}/uri", ip));
//request.ClientCertificates.Add(new X509Certificate("D:\\ThePubKey.cer"));
request.ContentType = "application/x-www-form-urlencoded";
request.Method = "POST";
using (StreamWriter stream = new StreamWriter(request.GetRequestStream()))
{
stream.Write("RequestType=CheckStatus&ReportType=Fulfillment&ReportID=5");
}
using (StreamReader stream = new StreamReader(request.GetResponse().GetResponseStream()))
{
Response.ContentType = "text/xml";
Response.Output.Write(stream.ReadToEnd());
Response.End();
}
class CertPolicy : ICertificatePolicy
{
public enum CertificateProblem : uint
{
CertEXPIRED = 0x800B0101,
CertVALIDITYPERIODNESTING = 0x800B0102,
CertROLE = 0x800B0103,
CertPATHLENCONST = 0x800B0104,
CertCRITICAL = 0x800B0105,
CertPURPOSE = 0x800B0106,
CertISSUERCHAINING = 0x800B0107,
CertMALFORMED = 0x800B0108,
CertUNTRUSTEDROOT = 0x800B0109,
CertCHAINING = 0x800B010A,
CertREVOKED = 0x800B010C,
CertUNTRUSTEDTESTROOT = 0x800B010D,
CertREVOCATION_FAILURE = 0x800B010E,
CertCN_NO_MATCH = 0x800B010F,
CertWRONG_USAGE = 0x800B0110,
CertUNTRUSTEDCA = 0x800B0112
}
public bool CheckValidationResult(ServicePoint srvPoint, X509Certificate certificate, WebRequest request, int certificateProblem)
{
// You can do your own certificate checking.
// You can obtain the error values from WinError.h.
// Return true so that any certificate will work with this sample.
String error = "";
using (StringWriter writer = new StringWriter())
{
writer.WriteLine("Certificate Problem with accessing " + request.RequestUri);
writer.Write("Problem code 0x{0:X8},", (int)certificateProblem);
writer.WriteLine(GetProblemMessage((CertificateProblem)certificateProblem));
error = writer.ToString();
}
return true;
}
private String GetProblemMessage(CertificateProblem Problem)
{
String ProblemMessage = "";
CertificateProblem problemList = new CertificateProblem();
String ProblemCodeName = Enum.GetName(problemList.GetType(), Problem);
if (ProblemCodeName != null)
ProblemMessage = ProblemMessage + "-Certificateproblem:" +
ProblemCodeName;
else
ProblemMessage = "Unknown Certificate Problem";
return ProblemMessage;
}
}
I've just replied to this similar question (in Java).
CN is the "Common Name". It ought to be the hostname of the server to which you're connecting (unless it's in the subject alternative name). I guess from your code sample that you're using the IP address directly. In this case, the CN should be that IP address (it tends to be better to use a hostname rather than an IP address). See RFC 2818 (sec 3.1) for the specifications.
Note that the CN or subject alternative name is from the point of view of the client, so if you connect to https://some.example.com/, then the name in the cert should be some.example.com, if you connect to https://localhost/, then the name in the cert should be localhost, even if some.example.com and localhost may be the same server effectively.
(I guess that by default, IIS might generate a certificate for the external name, but you'd have to look at the certificate to know; this should be visible in the certificate properties somewhere.)