So I am trying to set up a Jenkins deploy pipeline. Almost everything seems to be working fine, except for the last part of the job, which is to run wrangler publish (to publish the site to Cloudflare Workers).
I have tried running it twice now, and both times, it will fail during the job, and I will receive an error code "502" when I try to access my jenkins server. The only thing that works is a full reboot of the server.
I have tried checking logs, but nothing much shows up. In jenkins.error.log I see this:
2020/09/23 21:12:00 [error] 1098#1098: *498 connect() failed (111: Connection refused) while connecting to upstream, client: 162.158.94.165, server: jenkins.mydomain.com, request: "POST /job/my-project-staging/15/logText/progressiveHtml HTTP/1.1", upstream: "http://127.0.0.1:8080/job/my-project-staging/15/logText/progressiveHtml", host: "jenkins.mydomain.com", referrer: "https://jenkins.mydomain.com/job/my-project-staging/15/console"
2020/09/23 21:12:02 [error] 1098#1098: *500 connect() failed (111: Connection refused) while connecting to upstream, client: 162.158.93.72, server: jenkins.mydomain.com, request: "HEAD /job/my-project-staging/15/statusIcon HTTP/1.1", upstream: "http://127.0.0.1:8080/job/my-project-staging/15/statusIcon", host: "jenkins.mydomain.com"
2020/09/23 21:12:03 [error] 1098#1098: *502 connect() failed (111: Connection refused) while connecting to upstream, client: 172.69.34.207, server: jenkins.mydomain.com, request: "GET /job/my-project-staging/15/statusIcon HTTP/1.1", upstream: "http://127.0.0.1:8080/job/my-project-staging/15/statusIcon", host: "jenkins.mydomain.com"
2020/09/23 21:12:03 [error] 1098#1098: *504 connect() failed (111: Connection refused) while connecting to upstream, client: 162.158.91.152, server: jenkins.mydomain.com, request: "HEAD /job/my-project-staging/15/console HTTP/1.1", upstream: "http://127.0.0.1:8080/job/my-project-staging/15/console", host: "jenkins.mydomain.com"
2020/09/23 21:12:06 [error] 1098#1098: *506 connect() failed (111: Connection refused) while connecting to upstream, client: 162.158.91.146, server: jenkins.mydomain.com, request: "GET /job/my-project-staging/15/console HTTP/1.1", upstream: "http://127.0.0.1:8080/job/my-project-staging/15/console", host: "jenkins.mydomain.com"
2020/09/23 22:15:09 [error] 1098#1098: *1773 connect() failed (111: Connection refused) while connecting to upstream, client: 74.120.14.35, server: jenkins.mydomain.com, request: "GET / HTTP/1.1", upstream: "http://127.0.0.1:8080/", host: "123.123.123.123:443"
2020/09/23 22:15:10 [error] 1098#1098: *1775 connect() failed (111: Connection refused) while connecting to upstream, client: 74.120.14.35, server: jenkins.mydomain.com, request: "GET / HTTP/1.1", upstream: "http://127.0.0.1:8080/", host: "123.123.123.123"
2020/09/23 23:08:18 [error] 1098#1098: *2378 connect() failed (111: Connection refused) while connecting to upstream, client: 193.118.53.210, server: jenkins.mydomain.com, request: "GET /solr/ HTTP/1.1", upstream: "http://127.0.0.1:8080/solr/", host: "123.123.123.123"
2020/09/24 01:36:08 [error] 1098#1098: *7943 connect() failed (111: Connection refused) while connecting to upstream, client: 51.158.24.203, server: jenkins.mydomain.com, request: "GET / HTTP/1.1", upstream: "http://127.0.0.1:8080/", host: "123.123.123.123"
Any ideas?
I have tried restarting nginx, that doesn't prevent the 502 error. However, when I run sudo /etc/init.d/jenkins restart the jenkins server will reboot and be online again.
When I go into the failed job, I see this at the very end, before the server crashed:
memory allocation of 240904 bytes failedCreating placeholder flownodes because failed loading originals.
java.io.IOException: Tried to load head FlowNodes for execution Owner[jwr-nuxt-staging/15:jwr-nuxt-staging #15] but FlowNode was not found in storage for head id:FlowNodeId 1:26
at org.jenkinsci.plugins.workflow.cps.CpsFlowExecution.initializeStorage(CpsFlowExecution.java:689)
at org.jenkinsci.plugins.workflow.cps.CpsFlowExecution.onLoad(CpsFlowExecution.java:726)
at org.jenkinsci.plugins.workflow.job.WorkflowRun.getExecution(WorkflowRun.java:691)
at org.jenkinsci.plugins.workflow.job.WorkflowRun.onLoad(WorkflowRun.java:550)
at hudson.model.RunMap.retrieve(RunMap.java:225)
at hudson.model.RunMap.retrieve(RunMap.java:57)
at jenkins.model.lazy.AbstractLazyLoadRunMap.load(AbstractLazyLoadRunMap.java:501)
at jenkins.model.lazy.AbstractLazyLoadRunMap.load(AbstractLazyLoadRunMap.java:483)
at jenkins.model.lazy.AbstractLazyLoadRunMap.getByNumber(AbstractLazyLoadRunMap.java:381)
at hudson.model.RunMap.getById(RunMap.java:205)
at org.jenkinsci.plugins.workflow.job.WorkflowRun$Owner.run(WorkflowRun.java:940)
at org.jenkinsci.plugins.workflow.job.WorkflowRun$Owner.get(WorkflowRun.java:951)
at org.jenkinsci.plugins.workflow.flow.FlowExecutionList$1.computeNext(FlowExecutionList.java:65)
at org.jenkinsci.plugins.workflow.flow.FlowExecutionList$1.computeNext(FlowExecutionList.java:57)
at com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:143)
at com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:138)
at org.jenkinsci.plugins.workflow.flow.FlowExecutionList$ItemListenerImpl.onLoaded(FlowExecutionList.java:178)
at jenkins.model.Jenkins.<init>(Jenkins.java:1017)
at hudson.model.Hudson.<init>(Hudson.java:85)
at hudson.model.Hudson.<init>(Hudson.java:81)
at hudson.WebAppMain$3.run(WebAppMain.java:282)
Finished: FAILURE
Thinking this could be a memory related issue I went into /etc/default/jenkins and updated JAVA_ARGS to JAVA_ARGS="-Xmx4g -XX:MaxPermSize=512m -Djava.awt.headless=true" - but that didn't change anything. I have verified I can run wrangler publish via terminal on the server, without issues.
So aparently this was indeed a memory issue. My machine had 8Gb but turns out Jenkins use quite a lot, especially together with wrangler publish. So I added some 4GB swap memory and that seemed to resolve the issue.
You may follow this article on how to add swap memory on ubuntu: https://www.digitalocean.com/community/tutorials/how-to-add-swap-space-on-ubuntu-18-04
Related
I have a single solution with 2 projects - https://github.com/vickyrathee/dotnet-test-app
My Website: dotnet-test-website (domain.com)
My API: dotnet-test-api (api.domain.com)
How can I configure my projects to be deployed on single Ubuntu/Amazon linux server?
I already tried using the Urls option on appsettings.json to set port 5001 for API project and 5002 for website project as mentioned here
https://github.com/vickyrathee/dotnet-test-app/blob/master/dotnet-test-app/appsettings.json
{
"Urls" : "http://localhost:5001"
}
But as per the EB logs, it seems to trying port 5000 instead what I defined in appsettings.json
----------------------------------------
/var/log/nginx/error.log
----------------------------------------
2021/02/14 09:11:24 [error] 3583#0: *1 connect() failed (111: Connection refused) while connecting to upstream, client: 172.31.10.146, server: , request: "GET / HTTP/1.1", upstream: "http://127.0.0.1:5000/", host: "172.31.26.188"
2021/02/14 09:11:39 [error] 3583#0: *3 connect() failed (111: Connection refused) while connecting to upstream, client: 172.31.10.146, server: , request: "GET / HTTP/1.1", upstream: "http://127.0.0.1:5000/", host: "172.31.26.188"
2021/02/14 09:11:54 [error] 3583#0: *5 connect() failed (111: Connection refused) while connecting to upstream, client: 172.31.10.146, server: , request: "GET / HTTP/1.1", upstream: "http://127.0.0.1:5000/", host: "172.31.26.188"
Additionally, how to publish it correctly in separate folders, So that there are no conflict between them. EB allow single zip file upload with appsettings.json on root, while I have two with some other files as well in common and can't just put them on single directory to zip. Any help will be appreciated.
I run tomcat and my nginx.conf like this
If in http://127.0.0.1:8080/edu/b.html or http://192.168.0.102:8080/edu/b.html it's ok
If in http://192.168.0.102:9001/edu/b.html it can't work
my computer ip now is 192.168.0.102
error message
[error] 17810#17810: *12 connect() failed (111: Connection refused) while connecting to upstream, client: 192 .168.0.102, server: 192.168.0.102, request: "GET /edu/b.html HTTP/1.1", upstream: "http://127.0.0.1:8000/edu/b.html", host: "192. 168.0.102:9001"
Where is error? What should I do?
It is trying to connect to http://127.0.0.1:8000/edu/b.html(upstream) instead of http://192.168.0.102:8080/edu/b.html. You need to change the upstream port in your nginx.conf
I got this error when I want to open my website:
2015/01/27 07:04:38 [error] 1727#0: *1 connect() failed (111: Connection refused) while connecting to upstream, client: 62.217.151.69, server: localhost, request: "GET / HTTP/1.0", upstream: "fastcgi://127.0.0.1:9000", host: "www.alapar.az"
How can I resolve it?
Ensure that php-fpm or whatever you use as fastcgi backend is running and accepting connections at 127.0.0.1:9000
I've recently set up a WordPress blog on a VPS with Ubuntu 12.04 LTS, HHVM, Nginx (with the PageSpeed module included), and MariaDB. Everything runs really fast, but in the last few days I've noticed that HHVM stops working suddenly and from that moment the error.log shows first this message:
2014/09/18 12:27:56 [error] 24442#0: *121276 recv() failed (104: Connection reset by peer) while reading response header from upstream, client: 150.146.136.21, server: xxxxx.com, request: "POST /wp-login.php?PageSpeed=noscript HTTP/1.0", upstream: "fastcgi://127.0.0.1:9000", host: "xxxxx.com"
and then several messages with URLs corresponding to articles people are trying to read, but always with a 111 error:
2014/09/18 14:03:28 [error] 24442#0: *133594 connect() failed (111: Connection refused) while connecting to upstream, client: 66.249.78.121, server: xxxxx.com, request: "GET /2008/11/27/thanksgiving-whatever/ HTTP/1.1", upstream: "fastcgi://127.0.0.1:9000", host: "www.xxxx.com"
I'd say it has something to do with the PageSpeed module, but why is HHVM affected and not Nginx? DO you have some hint?
Thank
Hi I am getting the following error and have been searching and searching for an answer for my situation. I realize googling this issue has many solutions but ive tried them all and cant figure out why mine isnt working below is my nginx.conf and a log of the error please let me know if more information in needed
Log
2014/08/18 20:03:36 [error] 27960#0: *1 connect() failed (111: Connection refused) while connecting to upstream, client: x.x.x.x, server: , request: "GET / HTTP/1.1", upstream: "uwsgi://192.168.0.13:8081", host: "mysite.com"
2014/08/18 20:05:01 [error] 27960#0: *4 connect() failed (111: Connection refused) while connecting to upstream, client: 127.0.0.1, server: , request: "GET / HTTP/1.1", upstream: "uwsgi://192.168.0.13:8081", host: "mysite.com"
2014/08/18 20:08:19 [error] 28371#0: *1 connect() failed (111: Connection refused) while connecting to upstream, client: x.x.x.x, server: , request: "GET / HTTP/1.1", upstream: "uwsgi://127.0.0.1:8001", host: "mysite.com"
2014/08/18 20:08:21 [error] 28371#0: *1 connect() failed (111: Connection refused) while connecting to upstream, client: x.x.x.x, server: , request: "GET / HTTP/1.1", upstream: "uwsgi://127.0.0.1:8001", host: "mysite.com"
.conf
upstream django {
server 127.0.0.1:8001;
}
server {
location / {
include /etc/nginx/uwsgi_params;
uwsgi_pass django;
}
}
So it seems that whatever port I set in my uwsgi.ini file "socket = :" has to match the port I put as the server port in
upstream django {
server 127.0.0.1:8001;
}
Dont know if this is a true solution or whats actually going on behidn the scenes but it seemed to fix the issue