Keep TCP port open using httpclient in C# - asynchronous

Question

I am a newbie into async programming and am trying to use the httpclient to fire bulk URL requests for the page content.
Here is my attempt:
private async void ProcessUrlAsyncWithHttp(HttpClient httpClient, string purl)
{
Stopwatch sw = new Stopwatch();
sw.Start();
HttpResponseMessage response = null;
try
{
Interlocked.Increment(ref _activeRequestsCount);
var request = new HttpRequestMessage()
{
RequestUri = new Uri(purl),
Method = HttpMethod.Get,
};
request.Headers.TryAddWithoutValidation("User-Agent", "MozillaMozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/39.0.2171.95 Safari/537.36");
request.Headers.TryAddWithoutValidation("Accept", "text/html,*.*");
request.Headers.TryAddWithoutValidation("Connection", "Keep-Alive");
request.Headers.TryAddWithoutValidation("Accept-Encoding", "gzip, deflate, sdch");
request.Headers.TryAddWithoutValidation("Accept-Language", "en-US,en;q=0.8");
response = await httpClient.SendAsync(request).ConfigureAwait(false);
string html = await response.Content.ReadAsStringAsync().ConfigureAwait(false);
response.Dispose();
if (IsCaptcha(html)) throw new Exception("Captcha was returned");
request.Dispose();
Interlocked.Increment(ref _successfulCalls);
}
catch (HttpRequestException hex)
{
Console.WriteLine("http:" + hex.Message);
Interlocked.Increment(ref _failedCalls);
}
catch (Exception ex)
{
Console.WriteLine(ex.GetType().AssemblyQualifiedName + " " + ex.Message);
Interlocked.Increment(ref _failedCalls);
}
finally
{
Interlocked.Decrement(ref _activeRequestsCount);
Interlocked.Decrement(ref _itemsLeft);
if (response != null) response.Dispose();
if (httpClient != null) httpClient.Dispose();
sw.Stop();
DateTime currentTime = DateTime.UtcNow;
TimeSpan elapsedTillNow = (currentTime - _overallStartTime).Duration();
Console.WriteLine("Left:" + _itemsLeft + ", Current execution:" + sw.ElapsedMilliseconds + " (ms), Average execution:" + Math.Round((elapsedTillNow.TotalMilliseconds / (_totalItems - _itemsLeft)), 0) + " (ms)");
lock(_syncLock)
{
if (_itemsLeft == 0)
{
_overallEndTime = DateTime.UtcNow;
this.DisplayTestResults();
}
}
}
}
As you can see I am passing an httpclient to the function and it gets destroyed everytime the URL is downloaded. I know this is an overkill and ideally we should be reusing the httpclient. But since I cant use a single httpclient with different proxies for each URL (the handler needs to be passed to the constructor of httpclient and cannot be changed, hence a fresh proxy cant be given without recreating the httpclient object), I needed to use this approach.
At the caller side, I have a pretty basic code:
public async void TestAsyncWithHttp()
{
ServicePointManager.DefaultConnectionLimit = 10;
//ServicePointManager.UseNagleAlgorithm = false;
List<string> urlList = SetUpURLList();
urlList = urlList.GetRange(1, 50);
_itemsLeft = urlList.Count();
_totalItems = _itemsLeft;
List<string> proxies = new List<string>();
proxies.Add("124.161.94.8:80");
proxies.Add("183.207.228.8:80");
proxies.Add("202.29.97.5:3128");
proxies.Add("210.75.14.158:80");
proxies.Add("203.100.80.81:8080");
proxies.Add("218.207.172.236:80");
proxies.Add("218.59.144.120:81");
proxies.Add("218.59.144.95:80");
proxies.Add("218.28.35.234:8080");
proxies.Add("222.88.236.236:83");
Random rnd = new Random();
foreach (string url in urlList)
{
int ind = rnd.Next(0, proxies.Count-1);
var httpClientHandler = new HttpClientHandler
{
Proxy = new WebProxy(proxies.ElementAt(ind), false),
UseProxy = true
};
HttpClient httpClient = new HttpClient(httpClientHandler);
//HttpClient httpClient = new HttpClient();
httpClient.Timeout = TimeSpan.FromMinutes(2);
ProcessUrlAsyncWithHttp(httpClient, url);
}
}
Question is:
1) Why the TCP ports gets closed for each request. I wanted to open the max connections number of ports and reuse them across calls. e.g in the example above I can have 10 concurrent connections. Hence I wanted this to open 10 TCP ports and the rest of the 40 requests could then use these 10 ports in tandem. This is a normal behaviour expected in httpwebrequest. I have a working code for using httpwebrequest that depicts this behavior of reusing ports. Can post the code of that on demand for anyone who might want to have a look. So its kind of weird that httpclient does not mimic this behaviour although it is based on httpwebrequest.
2) How do we assign autoredirect to false for such calls?
3) I intend to use this function for multiple calls - say around 50K. Anything wrong in the way the code is written that might need a correction
4) Lets assume that I somehow manage to use a single httpclient object instead of one object per request. What is the way to ensure I read cookies for all these individual requests and also alter them if necessary all the while remembering that I have a single httpclient class for the whole set of URL requests?
Tks
Kallol

Answer 1

In my experience (I once had a similar problem with TCP port congestion, because of ports always getting closed, when I was hitting a server with around 6000 connections a minute) it suffices to reuse the HttpClientHandler objects, which actually manage the connection pooling, and always recreate the HttpClient objects for each request (using the constructor with HttpClientManager parameter).
Hope this helps.
Matthias

Answer 2

have you tried putting the HttpClient code in class and create 10 classes, each with a HttpClient?