I don't have Wordpress on my server, someone are trying requesting "/wp08/wp-includes/dtcla.php" (these is no this file in my server) all day, I have banned the ip of requester, but he still requesting, I want to know what is "dtcla.php", How should I solved it?
om" AGENT: "Mozilla/5.0 (Windows; U; MSIE 7.0; Windows NT 5.2) Java/1.5.0_08" FORWARDED: "-"
IP: 127.0.0.1 - USER: - [17/Feb/2023:10:05:53 +0800] "GET / HTTP/1.1" 301 SENT: 169 REFERER: "-" AGENT: "curl/7.29.0" FORWARDED: "-"
IP: 123.51.239.35 - USER: - [17/Feb/2023:10:05:55 +0800] "GET /wp08/wp-includes/dtcla.php HTTP/1.1" 301 SENT: 169 REFERER: "http://www.google.com" AGENT: "Mozilla/5.0 (Windows; U; MSIE 7.0; Windows NT 5.2) Java/1.5.0_08" FORWARDED: "-"
IP: 123.51.239.35 - USER: - [17/Feb/2023:10:05:55 +0800] "GET /wp08/wp-includes/dtcla.php HTTP/1.1" 301 SENT: 169 REFERER: "http://www.google.com" AGENT: "Mozilla/5.0 (Windows; U; MSIE 7.0; Windows NT 5.2) Java/1.5.0_08" FORWARDED: "-"
IP: 123.51.239.35 - USER: - [17/Feb/2023:10:05:55 +0800] "GET /wp08/wp-includes/dtcla.php HTTP/1.1" 403 SENT: 555 REFERER: "http://www.google.com" AGENT: "Mozilla/5.0 (Windows; U; MSIE 7.0; Windows NT 5.2) Java/1.5.0_08" FORWARDED: "-"
IP: 123.51.239.35 - USER: - [17/Feb/2023:10:05:57 +0800] "GET /wp08/wp-includes/dtcla.php HTTP/1.1" 301 SENT: 169 REFERER: "http://www.google.com" AGENT: "Mozilla/5.0 (Windows; U; MSIE 7.0; Windows NT 5.2) Java/1.5.0_08" FORWARDED: "-"
IP: 123.51.239.35 - USER: - [17/Feb/2023:10:05:58 +0800] "GET /wp08/wp-includes/dtcla.php HTTP/1.1" 301 SENT: 169 REFERER: "http://www.google.com" AGENT: "Mozilla/5.0 (Windows; U; MSIE 7.0; Windows NT 5.2) Java/1.5.0_08" FORWARDED: "-"
IP: 123.51.239.35 - USER: - [17/Feb/2023:10:05:58 +0800] "GET /wp08/wp-includes/dtcla.php HTTP/1.1" 403 SENT: 555 REFERER: "http://www.google.com" AGENT: "Mozilla/5.0 (Windows; U; MSIE 7.0; Windows NT 5.2) Java/1.5.0_08" FORWARDED: "-"
IP: 127.0.0.1 - USER: - [17/Feb/2023:10:05:58 +0800] "GET / HTTP/1.1" 301 SENT: 169 REFERER: "-" AGENT: "curl/7.29.0" FORWARDED: "-"
IP: 123.51.239.35 - USER: - [17/Feb/2023:10:06:00 +0800] "GET /wp08/wp-includes/dtcla.php HTTP/1.1" 301 SENT: 169 REFERER: "http://www.google.com" AGENT: "Mozilla/5.0 (Windows; U; MSIE 7.0; Windows NT 5.2) Java/1.5.0_08" FORWARDED: "-"
IP: 123.51.239.35 - USER: - [17/Feb/2023:10:06:00 +0800] "GET /wp08/wp-includes/dtcla.php HTTP/1.1" 301 SENT: 169 REFERER: "http://www.google.com" AGENT: "Mozilla/5.0 (Windows; U; MSIE 7.0; Windows NT 5.2) Java/1.5.0_08" FORWARDED: "-"
IP: 123.51.239.35 - USER: - [17/Feb/2023:10:06:02 +0800] "GET /wp08/wp-includes/dtcla.php HTTP/1.1" 403 SENT: 555 REFERER: "http://www.google.com" AGENT: "Mozilla/5.0 (Windows; U; MSIE 7.0; Windows NT 5.2) Java/1.5.0_08" FORWARDED: "-"
IP: 127.0.0.1 - USER: - [17/Feb/2023:10:06:03 +0800] "GET / HTTP/1.1" 301 SENT: 169 REFERER: "-" AGENT: "curl/7.29.0" FORWARDED: "-"
IP: 123.51.239.35 - USER: - [17/Feb/2023:10:06:04 +0800] "GET /wp08/wp-includes/dtcla.php HTTP/1.1" 301 SENT: 169 REFERER: "http://www.google.com" AGENT: "Mozilla/5.0 (Windows; U; MSIE 7.0; Windows NT 5.2) Java/1.5.0_08" FORWARDED: "-"
IP: 123.51.239.35 - USER: - [17/Feb/2023:10:06:04 +0800] "GET /wp08/wp-includes/dtcla.php HTTP/1.1" 301 SENT: 169 REFERER: "http://www.google.com" AGENT: "Mozilla/5.0 (Windows; U; MSIE 7.0; Windows NT 5.2) Java/1.5.0_08" FORWARDED: "-"
IP: 123.51.239.35 - USER: - [17/Feb/2023:10:06:04 +0800] "GET /wp08/wp-includes/dtcla.php HTTP/1.1" 403 SENT: 555 REFERER: "http://www.google.com" AGENT: "Mozilla/5.0 (Windows; U; MSIE 7.0; Windows NT 5.2) Java/1.5.0_08" FORWARDED: "-"
IP: 123.51.239.35 - USER: - [17/Feb/2023:10:06:07 +0800] "GET /wp08/wp-includes/dtcla.php HTTP/1.1" 301 SENT: 169 REFERER: "http://www.google.com" AGENT: "Mozilla/5.0 (Windows; U; MSIE 7.0; Windows NT 5.2) Java/1.5.0_08" FORWARDED: "-"
IP: 123.51.239.35 - USER: - [17/Feb/2023:10:06:08 +0800] "GET /wp08/wp-includes/dtcla.php HTTP/1.1" 301 SENT: 169 REFERER: "http://www.google.com" AGENT: "Mozilla/5.0 (Windows; U; MSIE 7.0; Windows NT 5.2) Java/1.5.0_08" FORWARDED: "-"
IP: 123.51.239.35 - USER: - [17/Feb/2023:10:06:08 +0800] "GET /wp08/wp-includes/dtcla.php HTTP/1.1" 403 SENT: 555 REFERER: "http://www.google.com" AGENT: "Mozilla/5.0 (Windows; U; MSIE 7.0; Windows NT 5.2) Java/1.5.0_08" FORWARDED: "-"
IP: 127.0.0.1 - USER: - [17/Feb/2023:10:06:08 +0800] "GET / HTTP/1.1" 301 SENT: 169 REFERER: "-" AGENT: "curl/7.29.0" FORWARDED: "-"
IP: 123.51.239.35 - USER: - [17/Feb/2023:10:06:12 +0800] "GET /wp08/wp-includes/dtcla.php HTTP/1.1" 301 SENT: 169 REFERER: "http://www.google.com" AGENT: "Mozilla/5.0 (Windows; U; MSIE 7.0; Windows NT 5.2) Java/1.5.0_08" FORWARDED: "-"
IP: 123.51.239.35 - USER: - [17/Feb/2023:10:06:12 +0800] "GET /wp08/wp-includes/dtcla.php HTTP/1.1" 301 SENT: 169 REFERER: "http://www.google.com" AGENT: "Mozilla/5.0 (Windows; U; MSIE 7.0; Windows NT 5.2) Java/1.5.0_08" FORWARDED: "-"
IP: 123.51.239.35 - USER: - [17/Feb/2023:10:06:12 +0800] "GET /wp08/wp-includes/dtcla.php HTTP/1.1" 403 SENT: 555 REFERER: "http://www.google.com" AGENT: "Mozilla/5.0 (Windows; U; MSIE 7.0; Windows NT 5.2) Java/1.5.0_08" FORWARDED: "-"
IP: 127.0.0.1 - USER: - [17/Feb/2023:10:06:13 +0800] "GET / HTTP/1.1" 301 SENT: 169 REFERER: "-" AGENT: "curl/7.29.0" FORWARDED: "-"
IP: 123.51.239.35 - USER: - [17/Feb/2023:10:06:15 +0800] "GET /wp08/wp-includes/dtcla.php HTTP/1.1" 301 SENT: 169 REFERER: "http://www.google.com" AGENT: "Mozilla/5.0 (Windows; U; MSIE 7.0; Windows NT 5.2) Java/1.5.0_08" FORWARDED: "-"
IP: 123.51.239.35 - USER: - [17/Feb/2023:10:06:15 +0800] "GET /wp08/wp-includes/dtcla.php HTTP/1.1" 301 SENT: 169 REFERER: "http://www.google.com" AGENT: "Mozilla/5.0 (Windows; U; MSIE 7.0; Windows NT 5.2) Java/1.5.0_08" FORWARDED: "-"
IP: 123.51.239.35 - USER: - [17/Feb/2023:10:06:15 +0800] "GET /wp08/wp-includes/dtcla.php HTTP/1.1" 403 SENT: 555 REFERER: "http://www.google.com" AGENT: "Mozilla/5.0 (Windows; U; MSIE 7.0; Windows NT 5.2) Java/1.5.0_08" FORWARDED: "-"
IP: 123.51.239.35 - USER: - [17/Feb/2023:10:06:17 +0800] "GET /wp08/wp-includes/dtcla.php HTTP/1.1" 301 SENT: 169 REFERER: "http://www.google.com" AGENT: "Mozilla/5.0 (Windows; U; MSIE 7.0; Windows NT 5.2) Java/1.5.0_08" FORWARDED: "-"
I banned the ip and response 403
I am trying to run my Django app (V 1.8.6) in a docker container using docker-compose.
This is my setting.py:
MEDIA_ROOT = "/srv/media/"
MEDIA_URL = "/media/"
STATIC_URL = "/static/"
STATIC_ROOT = os.path.join(BASE_DIR, "static")
This is my docker compose:
version: "3"
services:
#
# Nettle API server
#
nettle:
image: mcvitty_img
command: python manage.py runserver 0.0.0.0:8000 --settings=sites.production.settings
ports:
- "80:8000"
volumes:
bluebell: {}
postgres-data: {}
rabbitmq-data: {}
I am successfully running the collect static in my Dockerfile:
RUN python manage.py migrate --settings=sites.production.settings --fake-initial
RUN python manage.py sitetree_resync_apps --settings=sites.production.settings
RUN python manage.py collectstatic --settings=sites.production.settings --noinput
But sadly this is the result:
This are the logs I get:
nettle_1 | [29/Sep/2020 15:08:19] "GET /static/css/all.min.css HTTP/1.1" 404 4463
nettle_1 | [29/Sep/2020 15:08:19] "GET /static/autocomplete_light/widget.js HTTP/1.1" 404 4463
nettle_1 | [29/Sep/2020 15:08:19] "GET /static/js/all.min.js HTTP/1.1" 404 4463
nettle_1 | [29/Sep/2020 15:08:19] "GET /static/autocomplete_light/autocomplete.js HTTP/1.1" 404 4463
nettle_1 | [29/Sep/2020 15:08:19] "GET /static/autocomplete_light/style.css HTTP/1.1" 404 4463
nettle_1 | [29/Sep/2020 15:08:19] "GET /static/autocomplete_light/remote.js HTTP/1.1" 404 4463
nettle_1 | [29/Sep/2020 15:08:19] "GET /static/autocomplete_light/text_widget.js HTTP/1.1" 404 4463
nettle_1 | [29/Sep/2020 15:08:19] "GET /static/autocomplete_light/autocomplete.js HTTP/1.1" 404 4463
nettle_1 | [29/Sep/2020 15:08:19] "GET /static/autocomplete_light/addanother.js HTTP/1.1" 404 4463
nettle_1 | [29/Sep/2020 15:08:19] "GET /static/autocomplete_light/widget.js HTTP/1.1" 404 4463
nettle_1 | [29/Sep/2020 15:08:19] "GET /static/js/datatableview.js HTTP/1.1" 404 4463
nettle_1 | [29/Sep/2020 15:08:19] "GET /static/images/logo.png HTTP/1.1" 404 4463
nettle_1 | [29/Sep/2020 15:08:19] "GET /static/images/frontimage1.jpg HTTP/1.1" 404 4463
nettle_1 | [29/Sep/2020 15:08:19] "GET /static/autocomplete_light/addanother.js HTTP/1.1" 404 4463
nettle_1 | [29/Sep/2020 15:08:19] "GET /static/images/frontimage2.jpg HTTP/1.1" 404 4463
nettle_1 | [29/Sep/2020 15:08:19] "GET /static/autocomplete_light/text_widget.js HTTP/1.1" 404 4463
nettle_1 | [29/Sep/2020 15:08:19] "GET /static/images/frontimage3.png HTTP/1.1" 404 4463
nettle_1 | [29/Sep/2020 15:08:19] "GET /static/autocomplete_light/remote.js HTTP/1.1" 404 4463
nettle_1 | [29/Sep/2020 15:08:19] "GET /static/images/frontimage4.jpg HTTP/1.1" 404 4463
nettle_1 | [29/Sep/2020 15:08:19] "GET /static/autocomplete_light/style.css HTTP/1.1" 404 4463
nettle_1 | [29/Sep/2020 15:08:19] "GET /static/images/frontimage6.jpg HTTP/1.1" 404 4463
nettle_1 | [29/Sep/2020 15:08:19] "GET /static/images/frontimage8.png HTTP/1.1" 404 4463
nettle_1 | [29/Sep/2020 15:08:19] "GET /static/css/all.min.css HTTP/1.1" 404 4463
nettle_1 | [29/Sep/2020 15:08:19] "GET /static/images/frontimage7.jpg HTTP/1.1" 404 4463
nettle_1 | [29/Sep/2020 15:08:19] "GET /static/images/frontimage5.png HTTP/1.1" 404 4463
nettle_1 | [29/Sep/2020 15:08:19] "GET /static/js/datatableview.js HTTP/1.1" 404 4463
If I hop into the container I can see the static folder has been created:
I'm trying to ban annoying bot by user-agent. I put this into server section of nginx config:
server {
listen 80 default_server;
....
if ($http_user_agent ~* (AhrefsBot)) {
return 444;
}
checking by curl:
[root#vm85559 site_avaliable]# curl -I -H 'User-agent: Mozilla/5.0 (compatible; AhrefsBot/5.2; +http://ahrefs.com/robot/)' localhost/
curl: (52) Empty reply from server
so i check /var/log/nginx/access.log and i see some connections get 444, but another connections get 200!
51.255.65.78 - - [25/Jun/2017:15:47:36 +0300 - -] "GET /product/kovriki-avtomobilnie/volkswagen/?PAGEN_1=10 HTTP/1.1" 444 0 "-" "Mozilla/5.0 (compatible; AhrefsBot/5.2; +http://ahrefs.com/robot/)" 1498394856.155
217.182.132.60 - - [25/Jun/2017:15:47:50 +0300 - 2.301] "GET /product/bryzgoviki/toyota/ HTTP/1.1" 200 14500 "-" "Mozilla/5.0 (compatible; AhrefsBot/5.2; +http://ahrefs.com/robot/)" 1498394870.955
How is it possible?
Ok, got it!
I've add $server_name and $server_addr to nginx log format, and saw that cunning bot connects by ip without server_name:
51.255.65.40 - _ *myip* - [25/Jun/2017:16:22:27 +0300 - 2.449] "GET /product/soyuz_96_2/mitsubishi/l200/ HTTP/1.1" 200 9974 "-" "Mozilla/5.0 (compatible; AhrefsBot/5.2; +http://ahrefs.com/robot/)" 1498396947.308
so i added this and bot can't connect anymore
server {
listen *myip*:80;
server_name _;
return 403;
}
I am facing an issue while accepting site invitation while email, when I am trying to accept by clicking the link from email, getting the error, but user is added to site.
It is working fine in case if I have a browser session open for share. when I close browser and trying to accept the link its errored. When I got error I see the accept link request 2 times in logs. First request success and 2nd one failure.
"Processing invite acceptance failed Unfortunately, your invite acceptance could not be registered. Either you have already accepted or rejected the invite, or the inviter cancelled your invitation."
Fiddler session summary from client side:
Result Protocol Host URL Body Caching Content-Type Process Comments Custom
1 200 HTTPS www.fiddler2.com /UpdateCheck.aspx?isBeta=False 659 private text/plain; charset=utf-8
2 200 HTTP Tunnel to safebrowsing.google.com:443 2,018 chrome:13808 [#1]
3 200 HTTP share.xxx.com /share/page/accept-invite?inviteId=activiti$2287867&inviteeUserName=user1&siteShortName=test-invitation-site&inviteTicket=30457b99-a49f-4159-a304-8730920bc932 17,708 no-cache text/html;charset=utf-8 chrome:13808 [#2]
4 401 HTTP share.xxx.com /share/page/site-index?site=test-invitation-site 164 text/html;charset=ISO-8859-1 chrome:13808 [#3]
5 302 HTTP share.xxx.com /share/page/site-index?site=test-invitation-site 0 no-cache text/html;charset=utf-8 chrome:13808 [#4]
6 401 HTTP share.xxx.com /share/proxy/alfresco/api/people/guest/preferences 975 no-cache; Expires: Thu, 01 Jan 1970 00:00:00 GMT text/html;charset=utf-8 chrome:13808 [#5]
7 200 HTTP share.xxx.com /share/page/accept-invite?inviteId=activiti$2287867&inviteeUserName=user1&siteShortName=test-invitation-site&inviteTicket=30457b99-a49f-4159-a304-8730920bc932 20,767 no-cache text/html;charset=utf-8 chrome:13808 [#6]
here is the access log:
10.64.11.250 - A11949874768A1CAD071FD9CD9DF5BB9.alfresco02 - [28/Feb/2017:16:14:53 -0500] "GET /share/page/accept-invite?inviteId=activiti$2233022&inviteeUserName=myuser&siteShortName=test-invitation-error&inviteTicket=7ce8693f-e105-4eee-b53a-7e605cb28fba HTTP/1.1" 200 17683 200
127.0.0.1 - - - [28/Feb/2017:16:14:53 -0500] "PUT /alfresco/s/api/invite/activiti$2233022/7ce8693f-e105-4eee-b53a-7e605cb28fba/accept?inviteeUserName=myuser HTTP/1.1" 200 79 187
127.0.0.1 - 5FF1ACC4670F3607F8104779F3BB8E0A.alfresco02 - [28/Feb/2017:16:14:54 -0500] "GET /alfresco/wcs/touch HTTP/1.1" 200 - 3
10.64.11.249 - 345B5286F8F72790BBF8484D125344F5.alfresco02 - [28/Feb/2017:16:14:54 -0500] "GET /share/keepAlive.jsp HTTP/1.1" 200 317 1
127.0.0.1 - 70C5283382FC5038FAEF2603CCAE82B8.alfresco02 - [28/Feb/2017:16:14:54 -0500] "GET /alfresco/wcs/touch HTTP/1.1" 401 162 1
10.64.11.250 - A11949874768A1CAD071FD9CD9DF5BB9.alfresco02 - [28/Feb/2017:16:14:54 -0500] "GET /share/page/site-index?site=test-invitation-error HTTP/1.1" 401 164 3
127.0.0.1 - D1754CF6E7351D045BD6B15D6764890B.alfresco02 - [28/Feb/2017:16:14:54 -0500] "GET /alfresco/wcs/touch HTTP/1.1" 401 162 0
127.0.0.1 - D1754CF6E7351D045BD6B15D6764890B.alfresco02 - [28/Feb/2017:16:14:54 -0500] "GET /alfresco/wcs/touch HTTP/1.1" 200 - 31
10.64.11.251 - BC8EDCF38E2520AD658140F65C3B1E57.alfresco02 - [28/Feb/2017:16:14:54 -0500] "GET /share/keepAlive.jsp HTTP/1.1" 200 317 1
127.0.0.1 - 5FF1ACC4670F3607F8104779F3BB8E0A.alfresco02 - [28/Feb/2017:16:14:54 -0500] "GET /alfresco/wcs/common/downloadFiles?nodeRef=workspace://SpacesStore/439802eb-bc72-479e-b5d5-ba88f717600c&a=true HTTP/1.1" 200 2588898 496
127.0.0.1 - D1754CF6E7351D045BD6B15D6764890B.alfresco02 - [28/Feb/2017:16:14:54 -0500] "GET /alfresco/wcs/touch HTTP/1.1" 200 - 4
127.0.0.1 - D1754CF6E7351D045BD6B15D6764890B.alfresco02 - [28/Feb/2017:16:14:55 -0500] "GET /alfresco/wcs/api/people/myuser?groups=true HTTP/1.1" 200 4809 656
127.0.0.1 - D1754CF6E7351D045BD6B15D6764890B.alfresco02 - [28/Feb/2017:16:14:55 -0500] "GET /alfresco/wcs/webframework/content/metadata?user=myuser HTTP/1.1" 200 5082 46
10.64.11.250 - A11949874768A1CAD071FD9CD9DF5BB9.alfresco02 - [28/Feb/2017:16:14:55 -0500] "GET /share/page/site-index?site=test-invitation-error HTTP/1.1" 302 - 757
127.0.0.1 - D1754CF6E7351D045BD6B15D6764890B.alfresco02 - [28/Feb/2017:16:14:55 -0500] "GET /alfresco/wcs/remoteadm/has/alfresco/site-data/pages/user/myuser/dashboard.xml?s=sitestore HTTP/1.1" 200 4 7
127.0.0.1 - D1754CF6E7351D045BD6B15D6764890B.alfresco02 - [28/Feb/2017:16:14:55 -0500] "GET /alfresco/wcs/api/people/myuser?groups=true HTTP/1.1" 200 4809 595
127.0.0.1 - D1754CF6E7351D045BD6B15D6764890B.alfresco02 - [28/Feb/2017:16:14:55 -0500] "GET /alfresco/wcs/touch HTTP/1.1" 200 - 4
127.0.0.1 - D1754CF6E7351D045BD6B15D6764890B.alfresco02 - [28/Feb/2017:16:14:55 -0500] "GET /alfresco/wcs/remoteadm/has/alfresco/site-data/pages/site/test-invitation-error/dashboard.xml?s=sitestore HTTP/1.1" 200 4 12
10.64.11.250 - A11949874768A1CAD071FD9CD9DF5BB9.alfresco02 - [28/Feb/2017:16:14:55 -0500] "GET /share/proxy/alfresco/api/people/guest/preferences HTTP/1.1" 401 975 616
127.0.0.1 - D1754CF6E7351D045BD6B15D6764890B.alfresco02 - [28/Feb/2017:16:14:55 -0500] "GET /alfresco/wcs/api/people/guest/preferences HTTP/1.1" 401 514 15
127.0.0.1 - B6FE21069F0F5E7F1282FD2596560B63.alfresco02 - [28/Feb/2017:16:14:55 -0500] "GET /alfresco/wcs/touch HTTP/1.1" 200 - 1
127.0.0.1 - D1754CF6E7351D045BD6B15D6764890B.alfresco02 - [28/Feb/2017:16:14:55 -0500] "GET /alfresco/wcs/remoteadm/get/alfresco/site-data/pages/site/test-invitation-error/dashboard.xml?s=sitestore HTTP/1.1" 200 657 32
127.0.0.1 - D1754CF6E7351D045BD6B15D6764890B.alfresco02 - [28/Feb/2017:16:14:55 -0500] "GET /alfresco/wcs/api/people/myuser/preferences HTTP/1.1" 200 402 17
127.0.0.1 - D1754CF6E7351D045BD6B15D6764890B.alfresco02 - [28/Feb/2017:16:14:55 -0500] "GET /alfresco/wcs/api/sites/test-invitation-error HTTP/1.1" 200 514 13
127.0.0.1 - D1754CF6E7351D045BD6B15D6764890B.alfresco02 - [28/Feb/2017:16:14:55 -0500] "GET /alfresco/wcs/api/admin/usage HTTP/1.1" 200 288 14
127.0.0.1 - D1754CF6E7351D045BD6B15D6764890B.alfresco02 - [28/Feb/2017:16:14:55 -0500] "GET /alfresco/wcs/remoteadm/has/alfresco/site-data/components/page.full-width-dashlet.site~test-invitation-error~dashboard.xml?s=sitestore HTTP/1.1" 200 4 15
127.0.0.1 - D1754CF6E7351D045BD6B15D6764890B.alfresco02 - [28/Feb/2017:16:14:55 -0500] "GET /alfresco/wcs/remoteadm/get/alfresco/site-data/components/page.full-width-dashlet.site~test-invitation-error~dashboard.xml?s=sitestore HTTP/1.1" 200 423 24
127.0.0.1 - D1754CF6E7351D045BD6B15D6764890B.alfresco02 - [28/Feb/2017:16:14:55 -0500] "GET /alfresco/wcs/remoteadm/has/alfresco/site-data/components/page.component-1-1.site~test-invitation-error~dashboard.xml?s=sitestore HTTP/1.1" 200 4 8
127.0.0.1 - - - [28/Feb/2017:16:14:55 -0500] "PUT /alfresco/s/api/invite/activiti$2233022/7ce8693f-e105-4eee-b53a-7e605cb28fba/accept?inviteeUserName=myuser HTTP/1.1" 409 5993 47
10.64.11.250 - A11949874768A1CAD071FD9CD9DF5BB9.alfresco02 - [28/Feb/2017:16:14:55 -0500] "GET /share/page/accept-invite?inviteId=activiti$2233022&inviteeUserName=myuser&siteShortName=test-invitation-error&inviteTicket=7ce8693f-e105-4eee-b53a-7e605cb28fba HTTP/1.1" 200 19476 95
Here is alfresco.log
16:14:55,356 ERROR [org.springframework.extensions.webscripts.AbstractRuntime] Exception from executeScript - redirecting to status template error: 012873015 org.alfresco.service.cmr.invitation.InvitationExceptionUserError: 0128576624 "Invitation, activiti$2233022 has already been accepted, cancelled or rejected"
org.springframework.extensions.webscripts.WebScriptException: 012873015 org.alfresco.service.cmr.invitation.InvitationExceptionUserError: 0128576624 "Invitation, activiti$2233022 has already been accepted, cancelled or rejected"
at org.alfresco.repo.web.scripts.invite.InviteResponse.execute(InviteResponse.java:146)
at org.alfresco.repo.web.scripts.invite.InviteResponse.access$000(InviteResponse.java:45)
at org.alfresco.repo.web.scripts.invite.InviteResponse$1.doWork(InviteResponse.java:105)
at org.alfresco.repo.web.scripts.invite.InviteResponse$1.doWork(InviteResponse.java:94)
at org.alfresco.repo.tenant.TenantUtil.runAsWork(TenantUtil.java:119)
at org.alfresco.repo.tenant.TenantUtil.runAsTenant(TenantUtil.java:88)
at org.alfresco.repo.tenant.TenantUtil$1.doWork(TenantUtil.java:62)
at org.alfresco.repo.security.authentication.AuthenticationUtil.runAs(AuthenticationUtil.java:548)
at org.alfresco.repo.tenant.TenantUtil.runAsUserTenant(TenantUtil.java:58)
at org.alfresco.repo.tenant.TenantUtil.runAsSystemTenant(TenantUtil.java:112)
at org.alfresco.repo.web.scripts.invite.InviteResponse.executeImpl(InviteResponse.java:93)
at org.springframework.extensions.webscripts.DeclarativeWebScript.executeImpl(DeclarativeWebScript.java:235)
at org.springframework.extensions.webscripts.DeclarativeWebScript.__AW_execute(DeclarativeWebScript.java:64)
at org.springframework.extensions.webscripts.DeclarativeWebScript.execute(DeclarativeWebScript.java)
at org.alfresco.repo.web.scripts.RepositoryContainer$3.__AW_execute(RepositoryContainer.java:489)
at org.alfresco.repo.web.scripts.RepositoryContainer$3.execute(RepositoryContainer.java)
at org.alfresco.repo.transaction.RetryingTransactionHelper.__AW_doInTransaction(RetryingTransactionHelper.java:457)
at org.alfresco.repo.transaction.RetryingTransactionHelper.doInTransaction(RetryingTransactionHelper.java)
at org.alfresco.repo.web.scripts.RepositoryContainer.transactionedExecute(RepositoryContainer.java:551)
at org.alfresco.repo.web.scripts.RepositoryContainer.transactionedExecuteAs(RepositoryContainer.java:619)
at org.alfresco.repo.web.scripts.RepositoryContainer.executeScriptInternal(RepositoryContainer.java:326)
at org.alfresco.repo.web.scripts.RepositoryContainer.executeScript(RepositoryContainer.java:280)
at org.springframework.extensions.webscripts.AbstractRuntime.executeScript(AbstractRuntime.java:378)
at org.springframework.extensions.webscripts.AbstractRuntime.__AW_executeScript(AbstractRuntime.java:209)
at org.springframework.extensions.webscripts.AbstractRuntime.executeScript(AbstractRuntime.java)
at org.springframework.extensions.webscripts.servlet.WebScriptServlet.service(WebScriptServlet.java:132)
at javax.servlet.http.HttpServlet.__AW_service(HttpServlet.java:727)
at javax.servlet.http.HttpServlet.service(HttpServlet.java)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:303)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:52)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
at org.alfresco.web.app.servlet.GlobalLocalizationFilter.doFilter(GlobalLocalizationFilter.java:61)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:220)
at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:122)
at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:504)
at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:170)
at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:103)
at org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:950)
at org.apache.catalina.valves.RemoteIpValve.invoke(RemoteIpValve.java:683)
at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:116)
at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:421)
at org.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:1074)
at org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:611)
at org.apache.tomcat.util.net.JIoEndpoint$SocketProcessor.run(JIoEndpoint.java:316)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
at java.lang.Thread.run(Thread.java:745)
I'm coding on a simple HTTP crawler but I have an issue running the code at the bottom. I'm requesting 50 URLs and get the content of 20+ back. I've generated few files with 150kB size each to test the crawler. So I think the 20+ responses are limited by the bandwidth? BUT: how to tell the Erlang snippet not to quit until the last file is not fetched? The test data server is online, so plz try the code out and any hints are welcome :)
-module(crawler).
-define(BASE_URL, "http://46.4.117.69/").
-export([start/0, send_reqs/0, do_send_req/1]).
start() ->
ibrowse:start(),
proc_lib:spawn(?MODULE, send_reqs, []).
to_url(Id) ->
?BASE_URL ++ integer_to_list(Id).
fetch_ids() ->
lists:seq(1, 50).
send_reqs() ->
spawn_workers(fetch_ids()).
spawn_workers(Ids) ->
lists:foreach(fun do_spawn/1, Ids).
do_spawn(Id) ->
proc_lib:spawn_link(?MODULE, do_send_req, [Id]).
do_send_req(Id) ->
io:format("Requesting ID ~p ... ~n", [Id]),
Result = (catch ibrowse:send_req(to_url(Id), [], get, [], [], 10000)),
case Result of
{ok, Status, _H, B} ->
io:format("OK -- ID: ~2..0w -- Status: ~p -- Content length: ~p~n", [Id, Status, length(B)]);
Err ->
io:format("ERROR -- ID: ~p -- Error: ~p~n", [Id, Err])
end.
That's the output:
Requesting ID 1 ...
Requesting ID 2 ...
Requesting ID 3 ...
Requesting ID 4 ...
Requesting ID 5 ...
Requesting ID 6 ...
Requesting ID 7 ...
Requesting ID 8 ...
Requesting ID 9 ...
Requesting ID 10 ...
Requesting ID 11 ...
Requesting ID 12 ...
Requesting ID 13 ...
Requesting ID 14 ...
Requesting ID 15 ...
Requesting ID 16 ...
Requesting ID 17 ...
Requesting ID 18 ...
Requesting ID 19 ...
Requesting ID 20 ...
Requesting ID 21 ...
Requesting ID 22 ...
Requesting ID 23 ...
Requesting ID 24 ...
Requesting ID 25 ...
Requesting ID 26 ...
Requesting ID 27 ...
Requesting ID 28 ...
Requesting ID 29 ...
Requesting ID 30 ...
Requesting ID 31 ...
Requesting ID 32 ...
Requesting ID 33 ...
Requesting ID 34 ...
Requesting ID 35 ...
Requesting ID 36 ...
Requesting ID 37 ...
Requesting ID 38 ...
Requesting ID 39 ...
Requesting ID 40 ...
Requesting ID 41 ...
Requesting ID 42 ...
Requesting ID 43 ...
Requesting ID 44 ...
Requesting ID 45 ...
Requesting ID 46 ...
Requesting ID 47 ...
Requesting ID 48 ...
Requesting ID 49 ...
Requesting ID 50 ...
OK -- ID: 49 -- Status: "200" -- Content length: 150000
OK -- ID: 47 -- Status: "200" -- Content length: 150000
OK -- ID: 50 -- Status: "200" -- Content length: 150000
OK -- ID: 17 -- Status: "200" -- Content length: 150000
OK -- ID: 48 -- Status: "200" -- Content length: 150000
OK -- ID: 45 -- Status: "200" -- Content length: 150000
OK -- ID: 46 -- Status: "200" -- Content length: 150000
OK -- ID: 10 -- Status: "200" -- Content length: 150000
OK -- ID: 09 -- Status: "200" -- Content length: 150000
OK -- ID: 19 -- Status: "200" -- Content length: 150000
OK -- ID: 13 -- Status: "200" -- Content length: 150000
OK -- ID: 21 -- Status: "200" -- Content length: 150000
OK -- ID: 16 -- Status: "200" -- Content length: 150000
OK -- ID: 27 -- Status: "200" -- Content length: 150000
OK -- ID: 03 -- Status: "200" -- Content length: 150000
OK -- ID: 23 -- Status: "200" -- Content length: 150000
OK -- ID: 29 -- Status: "200" -- Content length: 150000
OK -- ID: 14 -- Status: "200" -- Content length: 150000
OK -- ID: 18 -- Status: "200" -- Content length: 150000
OK -- ID: 01 -- Status: "200" -- Content length: 150000
OK -- ID: 30 -- Status: "200" -- Content length: 150000
OK -- ID: 40 -- Status: "200" -- Content length: 150000
OK -- ID: 05 -- Status: "200" -- Content length: 150000
Update:
thanks stemm for the hint with the wait_workers. I've combined your and mine code but same behaviour :(
-module(crawler).
-define(BASE_URL, "http://46.4.117.69/").
-export([start/0, send_reqs/0, do_send_req/2]).
start() ->
ibrowse:start(),
proc_lib:spawn(?MODULE, send_reqs, []).
to_url(Id) ->
?BASE_URL ++ integer_to_list(Id).
fetch_ids() ->
lists:seq(1, 50).
send_reqs() ->
spawn_workers(fetch_ids()).
spawn_workers(Ids) ->
%% collect reference to each worker
Refs = [ do_spawn(Id) || Id <- Ids ],
%% wait for response from each worker
wait_workers(Refs).
wait_workers(Refs) ->
lists:foreach(fun receive_by_ref/1, Refs).
receive_by_ref(Ref) ->
%% receive message only from worker with specific reference
receive
{Ref, done} ->
done
end.
do_spawn(Id) ->
Ref = make_ref(),
proc_lib:spawn_link(?MODULE, do_send_req, [Id, {self(), Ref}]),
Ref.
do_send_req(Id, {Pid, Ref}) ->
io:format("Requesting ID ~p ... ~n", [Id]),
Result = (catch ibrowse:send_req(to_url(Id), [], get, [], [], 10000)),
case Result of
{ok, Status, _H, B} ->
io:format("OK -- ID: ~2..0w -- Status: ~p -- Content length: ~p~n", [Id, Status, length(B)]),
%% send message that work is done
Pid ! {Ref, done};
Err ->
io:format("ERROR -- ID: ~p -- Error: ~p~n", [Id, Err]),
%% repeat request if there was error while fetching a page,
do_send_req(Id, {Pid, Ref})
%% or - if you don't want to repeat request, put there:
%% Pid ! {Ref, done}
end.
Running the crawler forks fine for a handful of files, but then the code even doesnt fetch the entire files (file size each 150000 bytes) - he crawler fetches some files partially, see the following web server log :(
82.114.62.14 - - [13/Sep/2012:15:17:00 +0200] "GET /10 HTTP/1.1" 200 150000 "-" "-"
82.114.62.14 - - [13/Sep/2012:15:17:00 +0200] "GET /1 HTTP/1.1" 200 150000 "-" "-"
82.114.62.14 - - [13/Sep/2012:15:17:00 +0200] "GET /3 HTTP/1.1" 200 150000 "-" "-"
82.114.62.14 - - [13/Sep/2012:15:17:00 +0200] "GET /8 HTTP/1.1" 200 150000 "-" "-"
82.114.62.14 - - [13/Sep/2012:15:17:00 +0200] "GET /39 HTTP/1.1" 200 150000 "-" "-"
82.114.62.14 - - [13/Sep/2012:15:17:00 +0200] "GET /7 HTTP/1.1" 200 150000 "-" "-"
82.114.62.14 - - [13/Sep/2012:15:17:00 +0200] "GET /6 HTTP/1.1" 200 150000 "-" "-"
82.114.62.14 - - [13/Sep/2012:15:17:00 +0200] "GET /2 HTTP/1.1" 200 150000 "-" "-"
82.114.62.14 - - [13/Sep/2012:15:17:00 +0200] "GET /5 HTTP/1.1" 200 150000 "-" "-"
82.114.62.14 - - [13/Sep/2012:15:17:00 +0200] "GET /50 HTTP/1.1" 200 150000 "-" "-"
82.114.62.14 - - [13/Sep/2012:15:17:00 +0200] "GET /9 HTTP/1.1" 200 150000 "-" "-"
82.114.62.14 - - [13/Sep/2012:15:17:00 +0200] "GET /44 HTTP/1.1" 200 150000 "-" "-"
82.114.62.14 - - [13/Sep/2012:15:17:00 +0200] "GET /38 HTTP/1.1" 200 150000 "-" "-"
82.114.62.14 - - [13/Sep/2012:15:17:00 +0200] "GET /47 HTTP/1.1" 200 150000 "-" "-"
82.114.62.14 - - [13/Sep/2012:15:17:00 +0200] "GET /49 HTTP/1.1" 200 150000 "-" "-"
82.114.62.14 - - [13/Sep/2012:15:17:00 +0200] "GET /43 HTTP/1.1" 200 150000 "-" "-"
82.114.62.14 - - [13/Sep/2012:15:17:00 +0200] "GET /37 HTTP/1.1" 200 150000 "-" "-"
82.114.62.14 - - [13/Sep/2012:15:17:00 +0200] "GET /46 HTTP/1.1" 200 150000 "-" "-"
82.114.62.14 - - [13/Sep/2012:15:17:00 +0200] "GET /48 HTTP/1.1" 200 150000 "-" "-"
82.114.62.14 - - [13/Sep/2012:15:17:00 +0200] "GET /36 HTTP/1.1" 200 150000 "-" "-"
82.114.62.14 - - [13/Sep/2012:15:17:01 +0200] "GET /42 HTTP/1.1" 200 150000 "-" "-"
82.114.62.14 - - [13/Sep/2012:15:17:01 +0200] "GET /41 HTTP/1.1" 200 150000 "-" "-"
82.114.62.14 - - [13/Sep/2012:15:17:01 +0200] "GET /45 HTTP/1.1" 200 150000 "-" "-"
82.114.62.14 - - [13/Sep/2012:15:17:01 +0200] "GET /17 HTTP/1.1" 200 150000 "-" "-"
82.114.62.14 - - [13/Sep/2012:15:17:01 +0200] "GET /35 HTTP/1.1" 200 150000 "-" "-"
82.114.62.14 - - [13/Sep/2012:15:17:01 +0200] "GET /16 HTTP/1.1" 200 150000 "-" "-"
82.114.62.14 - - [13/Sep/2012:15:17:01 +0200] "GET /15 HTTP/1.1" 200 17020 "-" "-"
82.114.62.14 - - [13/Sep/2012:15:17:01 +0200] "GET /21 HTTP/1.1" 200 120360 "-" "-"
82.114.62.14 - - [13/Sep/2012:15:17:01 +0200] "GET /40 HTTP/1.1" 200 117600 "-" "-"
82.114.62.14 - - [13/Sep/2012:15:17:01 +0200] "GET /34 HTTP/1.1" 200 60660 "-" "-"
Any hints are welcome. I have no clue what's going wrong there :(
So, if I've understand you correctly - you don't want to return control from function spawn_workers until each of worker is not stopped (and fetched a page)? If that - you may change your code in such way:
spawn_workers(Ids) ->
%% collect reference to each worker
Refs = [ do_spawn(Id) || Id <- Ids ],
%% wait for response from each worker
wait_workers(Refs).
wait_workers(Refs) ->
lists:foreach(fun receive_by_ref/1, Refs).
receive_by_ref(Ref) ->
%% receive message only from worker with specific reference
receive
{Ref, done} ->
done
end.
do_spawn(Id) ->
Ref = make_ref(),
proc_lib:spawn_link(?MODULE, do_send_req, [Id, {self(), Ref}]),
Ref.
do_send_req(Id, {Pid, Ref}) ->
io:format("Requesting ID ~p ... ~n", [Id]),
Result = (catch ibrowse:send_req(to_url(Id), [], get, [], [], 10000)),
case Result of
{ok, Status, _H, B} ->
io:format("OK -- ID: ~2..0w -- Status: ~p -- Content length: ~p~n", [Id, Status, length(B)]),
%% send message that work is done
Pid ! {Ref, done};
Err ->
io:format("ERROR -- ID: ~p -- Error: ~p~n", [Id, Err]),
%% repeat request if there was error while fetching a page,
do_send_req(Id, {Pid, Ref})
%% or - if you don't want to repeat request, put there:
%% Pid ! {Ref, done}
end.
Edit:
I've noticed that your entry point (function start) returns control without waiting for all workers are end their tasks (because of you calling there spawn). If you want to wait there too - just do the similar trick:
start() ->
ibrowse:start(),
Ref = make_ref(),
proc_lib:spawn(?MODULE, send_reqs, [self(), Ref]),
receive_by_ref(Ref).
send_reqs(Pid, Ref) ->
spawn_workers(fetch_ids()),
Pid ! {Ref, done}.
You can use a combination of supervisors and the queue module: spawn N fetching children, each child fetches 1 item of the queue and processes it. When done notify parent process to continue with next item in queue. That way you can put a cap on number of concurrent requests.
If you spawn 500 reqs at the time ibrowse might be confused. Do you get any errors in the console?
See ibrowse:get_config_value/1 and ibrowse:set_config_value/2