Using classic ASP, how to fetch or screenscrape meta tags of html page? - asp-classic

Question

Using the following code, I can reach the site, grab the data, but I cannot obtain the meta title tag. Surprisingly, I searched for methods to obtain meta tags while screenscraping with classic ASP and only found a couple of example, neither of which I could get to work.
Any assistance?
rss_url = "https://www.nationalgeographic.com/science/2019/06/opal-fossils-reveal-new-species-dinosaur-australia-fostoria/"
Set objHTTP = CreateObject("Microsoft.XMLHTTP")
objHTTP.setRequestHeader "User-Agent", "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/74.0.3729.169 Safari/537.36"
objHTTP.Open "GET", rss_url, False
objHTTP.Send
if objHTTP.Status = 200 Then sdata = BinaryToString(objHTTP.ResponseBody)
Set objHTTP = Nothing
Set regEx = New RegExp
regEx.Pattern = "<meta.*property=""og:image"".*content=""(.*)"".*\/>"
regEx.IgnoreCase = True
Set matches = regEx.Execute(sdata)
if matches.Count > 0 then
KeywordAl = matches(0).SubMatches(0)
response.write "Image = " & KeywordAl&"<hr>"
end if
I included the BinaryToString Function just to be complete:
Function BinaryToString(byVal Binary)
'--- Converts the binary content to text using ADODB Stream
'--- Set the return value in case of error
BinaryToString = ""
'--- Creates ADODB Stream
Dim BinaryStream
Set BinaryStream = CreateObject("ADODB.Stream")
'--- Specify stream type.
BinaryStream.Type = 1 '--- adTypeBinary
'--- Open the stream And write text/string data To the object
BinaryStream.Open
BinaryStream.Write Binary
'--- Change stream type to text
BinaryStream.Position = 0
BinaryStream.Type = 2 '--- adTypeText
'--- Specify charset for the source text (unicode) data.
BinaryStream.CharSet = "UTF-8"
'--- Return converted text from the object
BinaryToString = BinaryStream.ReadText
End Function

Answer 1

Try this:
Function GetTextFromUrl(url)
Dim oXMLHTTP
Dim strStatusTest
Set oXMLHTTP = CreateObject("MSXML2.ServerXMLHTTP.3.0")
oXMLHTTP.Open "GET", url, False
oXMLHTTP.Send
If oXMLHTTP.Status = 200 Then
GetTextFromUrl = oXMLHTTP.responseText
End If
End Function
Dim sResult : sResult = GetTextFromUrl("https://www.nationalgeographic.com/science/2019/06/opal-fossils-reveal-new-species-dinosaur-australia-fostoria/")
Set regEx = New RegExp
regEx.Pattern = "<meta.*property=""og:image"".*content=""(.*)"".*\/>"
regEx.IgnoreCase = True
Set matches = regEx.Execute(sResult)
if matches.Count > 0 then
KeywordAl = matches(0).SubMatches(0)
response.write "Image = " & KeywordAl&"<hr>"
end if
For me this outputs for that page:
Image = https://www.nationalgeographic.com/content/dam/science/2019/05/22/gemstone-dino/og-fostoria_final.ngsversion.1559624211907.adapt.1900.1.jpg
edit: Added some debugging info here. Try this snippet and see what it says about your TLS version - it is possible this site refuses connections below a certain TLS level.
Set objHttp = Server.CreateObject("WinHTTP.WinHTTPRequest.5.1")
objHttp.open "GET", "https://howsmyssl.com/a/check", False
objHttp.Send
Response.Write objHttp.responseText
Set objHttp = Nothing
Response.End