We have two dotnet applications that publish and subscribe to RabbitMQ respectively. Our developers tested it locally and they work fine on their workstation (launch through Visual Studio). However, when we build them into container and run them, RabbitMQ logs will show the following error messages:
2022-11-01 09:29:15.998692+00:00 [info] <0.1560.868> accepting AMQP connection <0.1560.868>
2022-11-01 09:31:22.599983+00:00 [error] <0.2112.868> closing AMQP connection <0.2112.868>
2022-11-01 09:31:22.599983+00:00 [error] <0.2112.868> {handshake_timeout,frame_header}
And below is the error message on our dotnet container:
RabbitMQ.Client.Exceptions.BrokerUnreachableException: None of the specified endpoints were reachable
---> RabbitMQ.Client.Exceptions.PossibleAuthenticationFailureException: Possibly caused by authentication failure
---> RabbitMQ.Client.Exceptions.OperationInterruptedException: The AMQP operation was interrupted: AMQP close-reason, initiated by Library, code=541, text='Unexpected Exception', classId=0, methodId=0, cause=System.IO.IOException: Unable to read data from the transport connection: Connection reset by peer.
---> System.Net.Sockets.SocketException (104): Connection reset by peer
at System.Net.Sockets.NetworkStream.Read(Byte[] buffer, Int32 offset, Int32 count)
--- End of inner exception stack trace ---
at System.Net.Sockets.NetworkStream.Read(Byte[] buffer, Int32 offset, Int32 count)
at System.IO.BufferedStream.ReadByteSlow()
at System.IO.BufferedStream.ReadByte()
at RabbitMQ.Client.Impl.InboundFrame.ReadFrom(Stream reader, Byte[] frameHeaderBuffer, ArrayPool`1 pool, UInt32 maxMessageSize)
at RabbitMQ.Client.Impl.SocketFrameHandler.ReadFrame()
at RabbitMQ.Client.Framing.Impl.Connection.MainLoopIteration()
at RabbitMQ.Client.Framing.Impl.Connection.MainLoop()
at RabbitMQ.Client.Impl.SimpleBlockingRpcContinuation.GetReply(TimeSpan timeout)
at RabbitMQ.Client.Impl.ModelBase.ConnectionStartOk(IDictionary`2 clientProperties, String mechanism, Byte[] response, String locale)
at RabbitMQ.Client.Framing.Impl.Connection.StartAndTune()
--- End of inner exception stack trace ---
at RabbitMQ.Client.Framing.Impl.Connection.StartAndTune()
at RabbitMQ.Client.Framing.Impl.Connection.Open(Boolean insist)
at RabbitMQ.Client.Framing.Impl.Connection..ctor(IConnectionFactory factory, Boolean insist, IFrameHandler frameHandler, String clientProvidedName)
at RabbitMQ.Client.Framing.Impl.Connection..ctor(IConnectionFactory factory, Boolean insist, IFrameHandler frameHandler, ArrayPool`1 memoryPool, String clientProvidedName)
at RabbitMQ.Client.Framing.Impl.AutorecoveringConnection.Init(IFrameHandler fh)
at RabbitMQ.Client.Framing.Impl.AutorecoveringConnection.Init(IEndpointResolver endpoints)
at RabbitMQ.Client.ConnectionFactory.CreateConnection(IEndpointResolver endpointResolver, String clientProvidedName)
--- End of inner exception stack trace ---
I have spent two days googling and troubleshooting but I am not a developer myself and couldn't find anything similar to our issue. Any help and guidance would be appreciated.
RabbitMQ version: 3.10.5
Erlang version: 24.3.4
.NET version: 6.0.10
RabbitMQ Client Library version: 6.2.1
Related
I recently launched a Blazor Server application on a Debian 11 server. The application is running through a Linux Docker container. I have used a DNS to connect my domain name to my server, and am trying to get an SSL certificate applied to my domain for my Blazor app.
By following Certbot guides, I have done the following on my Debian 11 server where my Blazor container is hosted to port 80:
apt-get install cerbot
certbot certonly --standalone
And I got the following message:
Congratulations!
Your certificate and chain have been saved at: /etc/letsencrypt/live/example.com/fullchain.pem
Your key file has been saved at: /etc/letsencrypt/live/example.com/privkey.pem
My site is still not using https. I cannot find any other instructions to apply the SSL from this point. Do I need to add code to my Blazor Server app to point to the SSL files? Do I need to mount the SSL files to somewhere in my Docker container? Thanks.
Update: I have been running my Docker container with the following command. I've tested the mounting of my keys folder and can see it is correct, but I still have issues when launching the container. It appears to be unable to find the keys even though they are mounted to the container correctly:
docker run -p 80:80 -p 443:443 -v /etc/letsencrypt/live/example.com:/https/ -e ASPNETCORE_URLS="https://+;http://+" -e ASPNETCORE_HTTPS_PORT=443 -e ASPNETCORE_Kestrel__Certificates__Default__Path=/https/fullchain.pem -e ASPNETCORE_Kestrel__Certificates__Default__KeyPath=/https/privkey.pem --restart always --name blazorserver blazorserver:1.01
I get these terminal errors, resulting in Program.cs failing to run app.Run();
warn: Microsoft.AspNetCore.DataProtection.Repositories.FileSystemXmlRepository[60]
Storing keys in a directory '/root/.aspnet/DataProtection-Keys' that may not be persisted outside of the container. Protected data will be unavailable when container is destroyed.
warn: Microsoft.AspNetCore.DataProtection.KeyManagement.XmlKeyManager[35]
No XML encryptor configured. Key {c9ce4f00-c893-456c-b894-6421bf21ffd8} may be persisted to storage in unencrypted form.
Unhandled exception. System.IO.FileNotFoundException: Could not find file '/https/fullchain.pem'.
File name: '/https/fullchain.pem'
at Interop.ThrowExceptionForIoErrno(ErrorInfo errorInfo, String path, Boolean isDirError)
at Microsoft.Win32.SafeHandles.SafeFileHandle.Open(String path, OpenFlags flags, Int32 mode, Func`4 createOpenException)
at Microsoft.Win32.SafeHandles.SafeFileHandle.Open(String fullPath, FileMode mode, FileAccess access, FileShare share, FileOptions options, Int64 preallocationSize, UnixFileMode openPermissions, Int64& fileLength, UnixFileMode& filePermissions, Func`4 createOpenException)
at System.IO.Strategies.OSFileStreamStrategy..ctor(String path, FileMode mode, FileAccess access, FileShare share, FileOptions options, Int64 preallocationSize, Nullable`1 unixCreateMode)
at System.IO.StreamReader.ValidateArgsAndOpenPath(String path, Encoding encoding, Int32 bufferSize)
at System.IO.File.ReadAllText(String path, Encoding encoding)
at System.Security.Cryptography.X509Certificates.X509Certificate2Collection.ImportFromPemFile(String certPemFilePath)
at Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Certificates.CertificateConfigLoader.LoadCertificate(CertificateConfig certInfo, String endpointName)
at Microsoft.AspNetCore.Server.Kestrel.KestrelConfigurationLoader.LoadDefaultCert()
at Microsoft.AspNetCore.Server.Kestrel.KestrelConfigurationLoader.Reload()
at Microsoft.AspNetCore.Server.Kestrel.KestrelConfigurationLoader.Load()
at Microsoft.AspNetCore.Server.Kestrel.Core.KestrelServerImpl.BindAsync(CancellationToken cancellationToken)
at Microsoft.AspNetCore.Server.Kestrel.Core.KestrelServerImpl.StartAsync[TContext](IHttpApplication`1 application, CancellationToken cancellationToken)
at Microsoft.AspNetCore.Hosting.GenericWebHostService.StartAsync(CancellationToken cancellationToken)
at Microsoft.Extensions.Hosting.Internal.Host.StartAsync(CancellationToken cancellationToken)
at Microsoft.Extensions.Hosting.HostingAbstractionsHostExtensions.RunAsync(IHost host, CancellationToken token)
at Microsoft.Extensions.Hosting.HostingAbstractionsHostExtensions.RunAsync(IHost host, CancellationToken token)
at Microsoft.Extensions.Hosting.HostingAbstractionsHostExtensions.Run(IHost host)
at Program.<Main>$(String[] args) in /src/Program.cs:line 36
So, the issue was that the /etc/letsencrypt/live/example.com/fullchain.pem file needs access to other files in the /etc/letsencrypt hierarchy. After mounting the entire /etc/letsencrypt directory to /https, I was able to access the SSL files with the full path, e.g. /https/live/example.com/fullchain.pem for the __Path variable.
Hopefully somebody finds my troubleshooting useful, as I could not find much information out there for Let's Encrypt SSL setup for Kestrel and/or Blazor.
I am running an ASP.Net Application in C#. I was trying to consuming web service. It works in previous server. However, when I migrated to code to Azure VM. It have error consuming the webservice as below. What is possible reason causing this issue?
Error occurs in the
'Sustainability.Service.Controllers.UsageController.GetAirportsDistance' method.
System.AggregateException: One or more errors occurred. ---> System.Net.Http.HttpRequestException: An error occurred while sending the request. ---> System.Net.WebException: The underlying connection was closed: An unexpected error occurred on a send. ---> System.IO.IOException: Unable to read data from the transport connection: An existing connection was forcibly closed by the remote host. ---> System.Net.Sockets.SocketException: An existing connection was forcibly closed by the remote host
at System.Net.Sockets.Socket.EndReceive(IAsyncResult asyncResult)
at System.Net.Sockets.NetworkStream.EndRead(IAsyncResult asyncResult)
--- End of inner exception stack trace ---
at System.Net.TlsStream.EndWrite(IAsyncResult asyncResult)
at System.Net.PooledStream.EndWrite(IAsyncResult asyncResult)
at System.Net.ConnectStream.WriteHeadersCallback(IAsyncResult ar)
--- End of inner exception stack trace ---
at System.Net.HttpWebRequest.EndGetResponse(IAsyncResult asyncResult)
at System.Net.Http.HttpClientHandler.GetResponseCallback(IAsyncResult ar)
--- End of inner exception stack trace ---
--- End of inner exception stack trace ---
at System.Threading.Tasks.Task.ThrowIfExceptional(Boolean includeTaskCanceledExceptions)
at System.Threading.Tasks.Task`1.GetResultCore(Boolean waitCompletionNotification)
at System.Threading.Tasks.Task`1.get_Result()
at Common.WebUtil.GetString(String baseAddress, String requestUri, HttpResponseMessage& response, NameValueCollection headers)
at Sustainability.BusinessObject.UsageManager.GetAirportsDistance(String sourceIATACode, String destinationIATACode) in C:\Users\HKHonCYP\Desktop\IESG_VSTS_2\Sustainability.BusinessObject\UsageManager.cs:line 254
at Sustainability.Service.Controllers.UsageController.GetAirportsDistance(String sourceIATACode, String destinationIATACode)
---> (Inner Exception #0) System.Net.Http.HttpRequestException: An error occurred while sending the request. ---> System.Net.WebException: The underlying connection was closed: An unexpected error occurred on a send. ---> System.IO.IOException: Unable to read data from the transport connection: An existing connection was forcibly closed by the remote host. ---> System.Net.Sockets.SocketException: An existing connection was forcibly closed by the remote host
at System.Net.Sockets.Socket.EndReceive(IAsyncResult asyncResult)
at System.Net.Sockets.NetworkStream.EndRead(IAsyncResult asyncResult)
--- End of inner exception stack trace ---
at System.Net.TlsStream.EndWrite(IAsyncResult asyncResult)
at System.Net.PooledStream.EndWrite(IAsyncResult asyncResult)
at System.Net.ConnectStream.WriteHeadersCallback(IAsyncResult ar)
--- End of inner exception stack trace ---
at System.Net.HttpWebRequest.EndGetResponse(IAsyncResult asyncResult)
at System.Net.Http.HttpClientHandler.GetResponseCallback(IAsyncResult ar)
--- End of inner exception stack trace ---<---
We are using Microsoft.AspNet.SignalR Version=2.1.1.0 running on Windows Server 2016, IIS 10.
We see the following errors in the server logs -
SignalR.Transports.WebSocketTransport Error: 0 : OnError(1531b342-8587-4082-a027-b57505e29a8d, System.Net.WebSockets.WebSocketException (0x80070026): Reached the end of the file
at System.Web.WebSockets.WebSocketPipe.<>c__DisplayClass9_0.b__0(Int32 hrError, Int32 cbIO, Boolean fUtf8Encoded, Boolean fFinalFragment, Boolean fClose)
What could be the cause of above error?
Been trying unsuccessfully to connect to EC-2 on AWS instance through Filezilla. Here is a screenshot of the settings I have on Filezilla Site manager
Here is the error log from Filezilla with the debug log set to 4
Trace: CControlSocket::DoClose(64)
Trace: CControlSocket::DoClose(64)
Trace: CControlSocket::DoClose(64)
Trace: CFileZillaEnginePrivate::ResetOperation(0)
Status: Connecting to ec2-54-251-155-167.ap-southeast-1.compute.amazonaws.com...
Trace: Going to execute /Users/zaidhumayun/Documents/FileZilla.app/Contents/MacOS/fzsftp
Response: fzSftp started, protocol_version=6
Trace: CSftpControlSocket::ConnectParseResponse(fzSftp started, protocol_version=6)
Trace: CSftpControlSocket::SendNextCommand()
Trace: CSftpControlSocket::ConnectSend()
Command: keyfile "/Users/zaidhumayun/Desktop/FirstInstance.pem"
Trace: CSftpControlSocket::ConnectParseResponse()
Trace: CSftpControlSocket::SendNextCommand()
Trace: CSftpControlSocket::ConnectSend()
Command: open "ubuntu#ec2-54-251-155-167.ap-southeast-1.compute.amazonaws.com" 22
Trace: Server version: SSH-2.0-OpenSSH_6.6.1p1 Ubuntu-2ubuntu2.8
Trace: We believe remote version has SSH-2 channel request bug
Trace: Using SSH protocol version 2
Trace: Doing ECDH key exchange with curve Curve25519 and hash SHA-256
Trace: Server also has ecdsa-sha2-nistp256/ssh-dss/ssh-rsa host keys, but we don't know any of them
Trace: Host key fingerprint is:
Trace: ssh-ed25519 256 7e:1c:94:69:78:c9:13:a3:38:0d:07:54:e9:28:5b:3b
Command: Trust new Hostkey: Once
Trace: Initialised AES-256 GCM client->server encryption
Trace: Initialised AES256 GCM client->server MAC algorithm (in ETM mode) (required by cipher)
Trace: Initialised AES-256 GCM server->client encryption
Trace: Initialised AES256 GCM server->client MAC algorithm (in ETM mode) (required by cipher)
Trace: Pageant is running. Requesting keys.
Trace: Pageant has 0 SSH-2 keys
Trace: Successfully loaded 1 key pair from file
Trace: Offered public key from "/Users/zaidhumayun/Desktop/FirstInstance.pem"
Trace: Server refused our key
Trace: Disconnected: No supported authentication methods available (server sent: publickey)
I have no idea why its not accepting my key pair instance. What do I need to do to fix this? Is there some way to re-download the key pair instance? Because according to the amazon documentation, you're allowed to download it once at instance launch.
According to this page, username should be 'bitnami', not 'ubuntu'
CCNET is throwing this exeption on the web dashboard. What is causing this?
System.Net.Sockets.SocketException: No connection could be made because the target machine actively refused it 127.0.0.1:21234 Server stack trace: at System.Net.Sockets.Socket.DoConnect(EndPoint endPointSnapshot, SocketAddress socketAddress) at System.Net.Sockets.Socket.Connect(EndPoint remoteEP) at System.Runtime.Remoting.Channels.RemoteConnection.CreateNewSocket(EndPoint ipEndPoint) at System.Runtime.Remoting.Channels.RemoteConnection.CreateNewSocket() at System.Runtime.Remoting.Channels.RemoteConnection.GetSocket() at System.Runtime.Remoting.Channels.SocketCache.GetSocket(String machinePortAndSid, Boolean openNew) at System.Runtime.Remoting.Channels.Tcp.TcpClientTransportSink.SendRequestWithRetry(IMessage msg, ITransportHeaders requestHeaders, Stream requestStream) at System.Runtime.Remoting.Channels.Tcp.TcpClientTransportSink.ProcessMessage(IMessage msg, ITransportHeaders requestHeaders, Stream requestStream, ITransportHeaders& responseHeaders, Stream& responseStream) at System.Runtime.Remoting.Channels.BinaryClientFormatterSink.SyncProcessMessage(IMessage msg) Exception rethrown at [0]: at System.Runtime.Remoting.Proxies.RealProxy.HandleReturnMessage(IMessage reqMsg, IMessage retMsg) at System.Runtime.Remoting.Proxies.RealProxy.PrivateInvoke(MessageData& msgData, Int32 type) at ThoughtWorks.CruiseControl.Remote.ICruiseManager.GetServerLog() at ThoughtWorks.CruiseControl.WebDashboard.ServerConnection.ServerAggregatingCruiseManagerWrapper.GetServerLog(IServerSpecifier serverSpecifier) at ThoughtWorks.CruiseControl.WebDashboard.Plugins.ServerReport.ServerLogServerPlugin.Execute(ICruiseRequest request) at ThoughtWorks.CruiseControl.WebDashboard.MVC.Cruise.ServerCheckingProxyAction.Execute(ICruiseRequest cruiseRequest) at ThoughtWorks.CruiseControl.WebDashboard.MVC.Cruise.CruiseActionProxyAction.Execute(IRequest request) at ThoughtWorks.CruiseControl.WebDashboard.MVC.Cruise.ExceptionCatchingActionProxy.Execute(IRequest request)
have you checked the ccservice is started? you can use telnet to check this, and, for testing, we can use ccnet.exe which is an console app for simplified
This issue looks like another process locks the same port that CCNet is using. You can use PortMon to track which process is locking that specific port. You can also configure CCNet in the config file to use a different port to avoid collision. The configuration should look as follow
<system.runtime.remoting>
<application>
<channels>
<!-- change port here -->
<channel ref="tcp" port="21234">
<serverProviders>
<formatter ref="binary" typeFilterLevel="Full"/>
</serverProviders>
</channel>
</channels>
</application>
</system.runtime.remoting>