I am able to start R 4.2.0 with RGui just fine, when I use RStudio however, it fails.
R Session Startup Failure Report
RStudio Version
RStudio 2021.09.0+351, "Ghost Orchid" (077589bc, 2021-09-20) for Windows
Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) QtWebEngine/5.12.8 Chrome/69.0.3497.128 Safari/537.36
Error message
[No error available]
Process Output
The R session exited with code -1073740791.
Error output:
[No errors emitted]
Standard output:
[No output emitted]
Logs
C:/Users/bha485/AppData/Local/RStudio/log/rsession-bha485.log
2022-06-10T14:21:27.071373Z [rsession-bha485] WARNING findProgramOnPath returns wrong result: C:\PROGRA~1\MIKTEX~1.9\miktex\bin\x64\pdflatex.exe != C:/Program Files/MiKTeX 2.9/miktex/bin/x64/pdflatex.exe; LOGGED FROM: class rstudio::core::FilePath __cdecl rstudio::session::module_context::findProgram(const class std::basic_string<char,struct std::char_traits<char>,class std::allocator<char> > &) src/cpp/session/SessionModuleContext.cpp:1206
2022-06-10T14:21:29.197389Z [rsession-bha485] ERROR CLIENT EXCEPTION (rsession-bha485): (TypeError) : Cannot read property 'M' of null;|||org/rstudio/studio/client/workbench/views/source/editors/text/AceEditor.java#4487::setScrollSpeed|||org/rstudio/studio/client/workbench/views/source/editors/text/AceEditorMonitor.java#46::monitor|||org/rstudio/studio/client/workbench/views/source/editors/text/AceEditorMonitor.java#70::execute|||com/google/gwt/core/client/impl/SchedulerImpl.java#140::execute|||com/google/gwt/core/client/impl/Impl.java#306::apply|||com/google/gwt/core/client/impl/Impl.java#345::entry0|||rstudio-0.js#-1::eval|||com/google/gwt/cell/client/AbstractEditableCell.java#41::viewDataMap|||Client-ID: 33e600bb-c1b1-46bf-b562-ab5cba070b0e|||User-Agent: Mozilla/5.0 (Windows NT 10.0 Win64 x64) AppleWebKit/537.36 (KHTML, like Gecko) QtWebEngine/5.12.8 Chrome/69.0.3497.128 Safari/537.36
2022-06-10T14:21:29.197389Z [rsession-bha485] ERROR CLIENT EXCEPTION (rsession-bha485): (TypeError) : Cannot read property 'M' of null;|||org/rstudio/studio/client/workbench/views/source/editors/text/AceEditor.java#4487::setScrollSpeed|||org/rstudio/studio/client/workbench/views/source/editors/text/AceEditorMonitor.java#46::monitor|||org/rstudio/studio/client/workbench/views/source/editors/text/AceEditorMonitor.java#70::execute|||com/google/gwt/core/client/impl/SchedulerImpl.java#140::execute|||com/google/gwt/core/client/impl/Impl.java#306::apply|||com/google/gwt/core/client/impl/Impl.java#345::entry0|||rstudio-0.js#-1::eval|||com/google/gwt/cell/client/AbstractEditableCell.java#41::viewDataMap|||Client-ID: 33e600bb-c1b1-46bf-b562-ab5cba070b0e|||User-Agent: Mozilla/5.0 (Windows NT 10.0 Win64 x64) AppleWebKit/537.36 (KHTML, like Gecko) QtWebEngine/5.12.8 Chrome/69.0.3497.128 Safari/537.36
How do I change the user-agent in a headless Chrome created by Symfony's Panther createChromeClient()?
When I create a Chrome client with
$client = \Symfony\Component\Panther\Client::createChromeClient();
I see in the access_log a user-agent of
"Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) HeadlessChrome/77.0.3865.90 Safari/537.36"
I searched for solutions, and think I have to change the user-agent string via the arguments of the chrome, but can't find the right way, because the answers on the web aren't for PHP or Panther.
Cheers!
I found it:
$client = \Symfony\Component\Panther\Client::createChromeClient(null, [
'--user-agent=Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/79.0.3945.88 Safari/537.36',
'--window-size=1200,1100',
'--headless',
'--disable-gpu',
]);
This question gave me the idea.
When I add a content module to a page I get a 404 error.
In the logbook I see the following entry
TabId:
PortalAlias:staging.2-le-marche.com/nl-nl
OriginalUrl:/nl-nl/desktopmodules/2sxc/api/view/Module/GetSelectableTemplates
Referer:http://staging.2-le-marche.com/desktopmodules/tosic_sexycontent/dist/dnn/ui.html?sxcver=8.5.1.26679
Url:http://staging.2-le-marche.com/nl-nl/desktopmodules/2sxc/api/view/Module/GetSelectableTemplates
UserAgent:Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/52.0.2743.116 Safari/537.36
My guess is a miss-configuration of DNN and portal domains. That kind of thing can work for quite a while (pages do work) but the js-framework of dnn will then report a (slightly) different url than is actually in use, and end up pointing to a url which doesn't work.
Consider the case,
I want to crawl websites frequently, but my IP address got blocked after some day/limit.
So, how can change my IP address dynamically or any other ideas?
An approach using Scrapy will make use of two components, RandomProxy and RotateUserAgentMiddleware.
Modify DOWNLOADER_MIDDLEWARES as follows. You will have to insert the new components in the settings.py:
DOWNLOADER_MIDDLEWARES = {
'scrapy.contrib.downloadermiddleware.retry.RetryMiddleware': 90,
'tutorial.randomproxy.RandomProxy': 100,
'scrapy.contrib.downloadermiddleware.httpproxy.HttpProxyMiddleware': 110,
'scrapy.contrib.downloadermiddleware.useragent.UserAgentMiddleware' : None,
'tutorial.spiders.rotate_useragent.RotateUserAgentMiddleware' :400,
}
Random Proxy
You can use scrapy-proxies. This component will process Scrapy requests using a random proxy from a list to avoid IP ban and improve crawling speed.
You can build up your proxy list from a quick internet search. Copy links in the list.txt file according to requested url format.
Rotation of user agent
For each scrapy request a random user agent will be used from a list you define in advance:
class RotateUserAgentMiddleware(UserAgentMiddleware):
def __init__(self, user_agent=''):
self.user_agent = user_agent
def process_request(self, request, spider):
ua = random.choice(self.user_agent_list)
if ua:
request.headers.setdefault('User-Agent', ua)
# Add desired logging message here.
spider.log(
u'User-Agent: {} {}'.format(request.headers.get('User-Agent'), request),
level=log.DEBUG
)
# the default user_agent_list composes chrome,I E,firefox,Mozilla,opera,netscape
# for more user agent strings,you can find it in http://www.useragentstring.com/pages/useragentstring.php
user_agent_list = [
"Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.1 (KHTML, like Gecko) Chrome/22.0.1207.1 Safari/537.1",
"Mozilla/5.0 (X11; CrOS i686 2268.111.0) AppleWebKit/536.11 (KHTML, like Gecko) Chrome/20.0.1132.57 Safari/536.11",
"Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/536.6 (KHTML, like Gecko) Chrome/20.0.1092.0 Safari/536.6",
"Mozilla/5.0 (Windows NT 6.2) AppleWebKit/536.6 (KHTML, like Gecko) Chrome/20.0.1090.0 Safari/536.6",
"Mozilla/5.0 (Windows NT 6.2; WOW64) AppleWebKit/537.1 (KHTML, like Gecko) Chrome/19.77.34.5 Safari/537.1",
"Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/536.5 (KHTML, like Gecko) Chrome/19.0.1084.9 Safari/536.5",
"Mozilla/5.0 (Windows NT 6.0) AppleWebKit/536.5 (KHTML, like Gecko) Chrome/19.0.1084.36 Safari/536.5",
"Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/536.3 (KHTML, like Gecko) Chrome/19.0.1063.0 Safari/536.3",
"Mozilla/5.0 (Windows NT 5.1) AppleWebKit/536.3 (KHTML, like Gecko) Chrome/19.0.1063.0 Safari/536.3",
"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_8_0) AppleWebKit/536.3 (KHTML, like Gecko) Chrome/19.0.1063.0 Safari/536.3",
"Mozilla/5.0 (Windows NT 6.2) AppleWebKit/536.3 (KHTML, like Gecko) Chrome/19.0.1062.0 Safari/536.3",
"Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/536.3 (KHTML, like Gecko) Chrome/19.0.1062.0 Safari/536.3",
"Mozilla/5.0 (Windows NT 6.2) AppleWebKit/536.3 (KHTML, like Gecko) Chrome/19.0.1061.1 Safari/536.3",
"Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/536.3 (KHTML, like Gecko) Chrome/19.0.1061.1 Safari/536.3",
"Mozilla/5.0 (Windows NT 6.1) AppleWebKit/536.3 (KHTML, like Gecko) Chrome/19.0.1061.1 Safari/536.3",
"Mozilla/5.0 (Windows NT 6.2) AppleWebKit/536.3 (KHTML, like Gecko) Chrome/19.0.1061.0 Safari/536.3",
"Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/535.24 (KHTML, like Gecko) Chrome/19.0.1055.1 Safari/535.24",
"Mozilla/5.0 (Windows NT 6.2; WOW64) AppleWebKit/535.24 (KHTML, like Gecko) Chrome/19.0.1055.1 Safari/535.24"
]
More details here.
You can try using proxy servers to prevent being blocked. There are services providing working proxies. The best I tried is https://gimmeproxy.com - they frequently check proxies for various parameters.
In order to get proxy from them, you need just to make the following request:
https://gimmeproxy.com/api/getProxy
They will provide JSON response with all proxy data which you can use later as needed:
{
"supportsHttps": true,
"protocol": "socks5",
"ip": "179.162.22.82",
"port": "36915",
"get": true,
"post": true,
"cookies": true,
"referer": true,
"user-agent": true,
"anonymityLevel": 1,
"websites": {
"example": true,
"google": false,
"amazon": true
},
"country": "BR",
"tsChecked": 1517952910,
"curl": "socks5://179.162.22.82:36915",
"ipPort": "179.162.22.82:36915",
"type": "socks5",
"speed": 37.78,
"otherProtocols": {}
}
You can use it like this with Curl:
curl -x socks5://179.162.22.82:36915 http://example.com
If you are using R, you could do the web crawling through TOR. I think TOR resets its IP-adress every 10 minutes(?) automatically. I think there is a way forcing TOR to change the IP in shorter intervals, but that didn't work for me. Instead you could set up multiple instances of TOR and then switch between the independent instances (here you can find a good explaination of how to set up multiple instances of TOR: https://tor.stackexchange.com/questions/2006/how-to-run-multiple-tor-browsers-with-different-ips)
After that you could do something like the following in R (use the ports of your independent TOR browsers and a list of useragents. Every time you call the 'getURL'-function cycle through your list of ports/useragents)
library(RCurl)
port <- c(a list of your ports)
proxy <- paste("socks5h://127.0.0.1:",port,sep="")
ua <- c(a list of your useragents)
opt <- list(proxy=sample(proxy,1),
useragent=sample(ua,1),
followlocation=TRUE,
referer="",
timeout=timeout,
verbose=verbose,
ssl.verifypeer=ssl)
webpage <- getURL(url=url,.opts=opt)
Some VPN applications allow you to automatically change your IP address to a new random IP address at a set interval such as: every 2 minutes. Both HMA! Pro VPN and VPN4ALL software support this feature.
Word of warning about VPNs, check their Terms and Conditions carefully because scraping using them goes against their user policy ( One such example would be Astrill). I tried a scraping tool and got my account locked
If you have public IPs. Add them on your interface and if you are using Linux use Iptables for switching those public IPs.
Iptables sample rules for two IPs
iptables -t nat -A POSTROUTING -m statistic --mode random --probability 0.5 -j SNAT --to-source 192.168.0.2
iptables -t nat -A POSTROUTING -m statistic --mode random --probability 0.5 -j SNAT --to-source 192.168.0.3
If you have 4 IPs then probablity will become 0.25.
You can also create your own proxy with simple steps.
These rules will allow the proxy server to switch its outgoing IPS.
Here are some spider user agent strings I've seen recently. They all seem to include a URL prefixed with +:
Mozilla/5.0 (compatible; Baiduspider/2.0; +http://www.baidu.com/search/spider.html)
Mozilla/5.0 (compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm)
Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)
Mozilla/5.0 (compatible; meanpathbot/1.0; +http://www.meanpath.com/meanpathbot.html)
Mozilla/5.0 (compatible; YandexBot/3.0; +http://yandex.com/bots)
Mozilla/5.0 (iPhone; CPU iPhone OS 6_0 like Mac OS X) AppleWebKit/536.26 (KHTML, like Gecko) Version/6.0 Mobile/10A5376e Safari/8536.25 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)
Is that just a convention that most spiders follow? Or is it specified somewhere. I couldn't find it.
It's just a convention that some spiders follow. There is no constraint on what people can put in a user agent header.
Take a look at this list of user agents that contain "GoogleBot". You'll notice that many of these don't contain "+http".