what is the best approach to make an http request - http

Question

what is the best method to do this: i have a txt file filled with web addresses, i have to check all of them using idHTTP component,only a simple check from a web server, downloading the html and finding a match, i want it to to be fast, there is different types of threads and i am not sure what is the best to use, a TParallel for or Task threads or regular threads?
i tried before TParallel for and i got stuck at AV, also i've tried Task threads but its not fast, the http request becomes slower by time, i tried also the regular threads and i didnt know how to use it because its complicated to use.
note: Please do NOT downvote i just need advice from the experts. Thank you

Answer 1

First advice: do not use Indy. Use THTTPClient (unit System.Net.HttpClient) -- native for Delphi XE?+
I am still using old TThreads. I could make suggestion only with TThread.
Workflow:
Main thread -- reading your TXT file line by line.
After line was readed, you create NEW thread, which are downloading information from WWW.
Sample of application:
unit ufmMain;
interface
uses
Winapi.Windows, Winapi.Messages,
System.SysUtils, System.Variants,
{ TThread }
System.Classes,
Vcl.Graphics, Vcl.Controls, Vcl.Forms, Vcl.Dialogs, Vcl.StdCtrls;
type
TForm1 = class(TForm)
Button1: TButton;
procedure Button1Click(Sender: TObject);
private
{ Private declarations }
public
{ Public declarations }
end;
TLoad = class(TThread)
protected
FURL,
FOutputFileName: String;
procedure Execute; override;
public
constructor Create(const AURL, AOutputFileName: String); overload;
end;
HTTP = class
public
class procedure Get(const AURL: String; out AOutputStream: TMemoryStream);
end;
var
Form1: TForm1;
implementation
{$R *.dfm}
uses
{ THTTPClient }
System.Net.HttpClient;
procedure TForm1.Button1Click(Sender: TObject);
var
LLoad: TLoad;
LFile: TextFile;
LCycle: Integer;
LUrl: String;
begin
LCycle := 0;
AssignFile(LFile, 'urls.txt');
try
Reset(LFile);
while not Eof(LFile) do
begin
{ Using for generate file name. All file names must be unique }
Inc(LCycle);
{ Read next URL }
ReadLn(LFile, LUrl);
{ Create new thread }
LLoad := TLoad.Create(LUrl, 'Output file No ' + LCycle.ToString + '.htm');
LLoad.FreeOnTerminate := True;
LLoad.Start;
end;
finally
CloseFile(LFile);
end;
end;
{ TLoad }
constructor TLoad.Create(const AURL, AOutputFileName: String);
begin
inherited Create(True);
FURL := AURL;
FOutputFileName := AOutputFileName;
end;
procedure TLoad.Execute;
var
LResponse: TMemoryStream;
begin
inherited;
LResponse := TStringStream.Create;
try
HTTP.Get(FURL, LResponse);
{ Save result to file }
LResponse.SaveToFile(GetCurrentDir + PathDelim + FOutputFileName);
finally
LResponse.Free;
end;
end;
{ HTTP }
class procedure HTTP.Get(const AURL: String; out AOutputStream: TMemoryStream);
var
LStream: TStream;
LHTTPClient: THTTPClient;
begin
LHTTPClient := THTTPClient.Create;
try
LStream := LHTTPClient.Get(AURL).ContentStream;
AOutputStream.CopyFrom(LStream, LStream.Size);
finally
LHTTPClient.Free;
end;
end;
end.
Why I against Indy:
1) THTTPClient do not required additional DLL for works with SSL protocol
2) THTTPClient is modern from Delphi XE8
3) My subjective opinion: THTTPClient works much more smoothly (with less issues) then Indy library. I used Indy for last 10 years, but now all my supported project moved to THTTPClient.

Answer 2

You can use TTask and Indy (TIdHTTP). Example:
function GetUrl(const aUrl: string): ITask;
begin
Result := TTask.Run(
procedure
var
FOutput: string;
FHTTP: TIdHTTP;
begin
FHTTP:=TIdHTTP.Create(nil);
try
try
FOutput:=FHTTP.Get(aUrl);
except
// handle errors
end;
finally
FHTTP.Free;
end;
TThread.Synchronize(nil,
procedure
begin
ProcessOutput(FOutput); // send your output/result to main thread
end);
end );
end;
procedure TForm1.Button1Click(Sender: TObject);
var
i: Integer;
list: TStringList;
begin
list:=TStringList.Create;
try
list.LoadFromFile('yourfile.txt');
// get all your urls
// you should control how many threads run at the same time
for i := 0 to list.Count-1 do
GetUrl(list[i]);
finally
list.Free;
end;
end;