I am trying to connect to an Apache Kafka server, through SSL connection and using an ASP.Net console application. This is my very first attempt at connecting to Kafka.
I have set up the producer config (shown below) but the people on Kafka side are Java shop and they use JKS certificate that apparently contains certificate and key. This is of no use to me. Is there a way to convert this to a .Net-friendly set of certificates?
var config = new ConsumerConfig
{
BootstrapServers = "kafka_broker_IP:9093",
SecurityProtocol = SecurityProtocol.Ssl,
SslCaLocation = "path/to/ca.crt",
SslCertificateLocation = "path/to/client.crt",
SslKeyLocation = "path/to/client.key"
};
Can I get ca.crt, client.cert and client.key from one single .jks file?
Related
I'm running a self contained dotnet app on a windows 2019 to execute a simple httpclient get on a remote machine using mTLS with a client certificate.
I am loading the client certificate in the application by passing a p12 keystore filepath, which has the certificate keypair and chain.
Executing this get on powershell with invoke-restmethod on the windows 2019 server works, which means the certificate loads properly and the server certificate is validated by accessing the certificate store.
Also, running the application locally works! so that means that both client and server certificates and chains are valid and my dotnet framework can access the local windows store.
Here is the simple call that is causing the issue:
var certificate = new X509Certificate2(filePath, password);
Console.WriteLine($"Certificate found in keystore: {certificate.FriendlyName}. {certificate.Thumbprint}. {certificate.Subject}.");
var handler = new HttpClientHandler();
var httpClient = new HttpClient(handler)
handler.ClientCertificates.Add(certificate);
var result = httpClient.GetAsync("https://urltoserverwithvalidmTLS").GetAwaiter().GetResult();
The exception i get is:
Exception: System.Net.Http.HttpRequestException: The SSL connection could not be established, see inner exception
---> System.Security.Authentication.AuthenticationException: Authentication failed, see inner exception.
---> System.ComponentModel.Win32Exception (0x80090304): The Local Security Authority cannot be contacted
--- End of inner exception stack trace ---
at System.Net.Security.SslStream.StartSendAuthResetSignal(ProtocolToken message, AsyncProtocolRequest asyncRequest, ExceptionDispatchInfo exception)
App is published with:
--configuration Release -r win-x64 --self-contained true
and csproj targets netcoreapp3.1 framework
At this point i have no clue why this is not working. Any help would be appreciated.
This was related to a combination of two issues related to ssl protocols. Dotnet httpclient defaults to an invalid TLS protocol on windows 2019.
Setting the SslProtocols to Tls12 resolved this.
The other issue is that windows 2019 does not work with ephemeral key when creating tls connections.
Setting the connection flag to X509KeyStorageFlags.PersistKeySet resolved this.
Here is a sample http client that works on windows 2019 with .net6:
var certificate = new X509Certificate2(filePath, password, X509KeyStorageFlags.PersistKeySet);
var handler = new HttpClientHandler();
handler.SslProtocols = SslProtocols.Tls12;
handler.ClientCertificates.Add(certificate);
var httpClient = new HttpClient(handler);
var result = await httpClient.GetAsync(https://pathToYourSecureUrl);
I hope, that you can help me here.
I'm trying to make a MQTT client (azure function in an app service environment) that extracts data from MQTT broker (hivemq), but I'm kind of lost when it comes to certificates (I'm not an expert).
Anyways, I received 3 files (look below) from our partner (that's them with the MQTT broker), but the question is. How should I use these PEM files in the application??
Client certificate == mqtt-client-cert.pem
Client key == mqtt-client-key_nopass.pem
CA certificate == server.pem
Here is the application and here is an example where I tested it locally (with a crt certificate) against an test broker (test.mosquitto.org). It worked perfectly, but now I just need to the same thing, just with 3 PEM files instead.
I also suspect that I need to do something on Azure (either on function app or app service environment level) in order to use these certificates?
// Create a new MQTT client.
var factory = new MqttFactory();
var mqttClient = factory.CreateMqttClient();
// Load certificate
X509Certificate caCertificate = new X509Certificate(#"..\mosquitto.org.crt");
// Create TLS based parameters.
var tlsParameters = new MqttClientOptionsBuilderTlsParameters
{
UseTls = true,
Certificates = new List<X509Certificate> { caCertificate },
SslProtocol = System.Security.Authentication.SslProtocols.Tls12
};
// Create TCP based options using the builder.
var connectOptions = new MqttClientOptionsBuilder()
.WithTcpServer("test.mosquitto.org", 8883)
.WithProtocolVersion(MQTTnet.Formatter.MqttProtocolVersion.V311)
.WithTls(tlsParameters)
.Build();
var conResult = await mqttClient.ConnectAsync(connectOptions);
mqttClient.UseConnectedHandler(e =>
{
Console.Write("Connected successfully with MQTT Brokers.");
});
mqttClient.UseDisconnectedHandler(e =>
{
Console.Write("Disconnected from MQTT Brokers.");
});
.crt files normally contain PEM encoded keys/certs so in this case the file extension doesn't make any difference.
You should be able to replace the mosquitto.org.crt with the server.pem.
The other 2 files are for what is known as mutual TLS authentication. For most TLS connections (e.g. when making a HTTPS request for a web page) only one side of the connection needs a certificate/private key. This is the server. The client uses a collections of CA certificates to validate that the service is what it claims to be. (This is what you are doing using the mosquitto.org.crt file).
In other cases we want to authenticate both ends of the connection (the client wants to know what the server is and the server wants to know who the client is). To do this the client need to also present a certificate to the server, this is what the other 2 files are for.
The MQTTNet documentation includes an example of setting up a connection using client certificates here but that uses a .pfx (a pfx is just another name for a PKCS12 container, if needed you can convert the .pem files into a .pfx/.p12 file using openssl e.g. openssl pkcs12 -export -out mqtt-client.p12 -inkey mqtt-client-key_nopass.pem -in mqtt-client-cert.pem -CAfile server.pem)
List<X509Certificate> certs = new List<X509Certificate>
{
new X509Certificate2("myCert.pfx")
};
var options = new MqttClientOptionBuilder()
.WithTcpServer(broker, port)
.WithTls(new MqttClientOptionsBuilderTlsParameters
{
UseTls = true,
Certificates = certs
})
.Build();
I'm exhausted after looking for an answer for 3 days. I don't know if my suggested flow is wrong or my Google skills have really deteriorated.
My API needs to create a valid certificate from a CSR it received, by signing it with a private key that exists ONLY inside an HSM-like service (Azure KeyVault), which unfortunately doesn't offer Certificate Authority functions BUT does offer signing data with a key that exists there. My CA certificate's private key is stored in the HSM. I'm using ECDSA.
My suggested flow:
Client generates Key Pair + CSR and sends CSR to API
API creates a certificate from the CSR
API asks HSM to sign the CSR data and receives back a signature
API appends the signature to the certificate and returns a signed (and including CA in chain) certificate to the Client
I'm using C# .NET Core and would like to keep it cross-platform (as it runs in Linux containers), so I have to keep it as native as possible or using Bouncy Castle (which I'm still not sure if runs in Linux .NET Core).
I really appreciate your help!
I had faced a similar issue and found a solution. You'll have to use the PKCS11Interop.X509Store library.
The solution uses dotnet core native System.Security.Cryptography.X509Certificates.CertificateRequest::Create method
for generating a certificate.
As per the docs:
Pkcs11Interop is managed library written in C# that brings the
full power of PKCS#11 API to the .NET environment
Pkcs11Interop.X509Store is managed library built on top of
Pkcs11Interop. It's main goal is to provide easy to use PKCS#11 based
read-only X.509 certificate store that can be easily integrated with
standard .NET ecosystem.
Till v0.3.0, implementation for issuing a certificate (i.e signing a CSR) is not available.
With minor modifications in the PKCS11Interop library, I was able to sign the CSR.
Mentioned in Issue #30, the code is now added in the PKCS11Interop.X509Store library version 0.4.0.
The below code is taken from test cases for BasicEcdsaCertificateRequestTest. Test cases for RSA CertificateRequest are also there.
// Load PKCS#11 based store
using (var pkcs11Store = new Pkcs11X509Store(SoftHsm2Manager.LibraryPath, SoftHsm2Manager.PinProvider))
{
// Find signing certificate (CA certificate)
Pkcs11X509Certificate pkcs11CertOfCertificateAuthority = Helpers.GetCertificate(pkcs11Store, SoftHsm2Manager.Token1Label, SoftHsm2Manager.Token1TestUserEcdsaLabel);
// Generate new key pair for end entity
ECDsa ecKeyPairOfEndEntity = ECDsa.Create(ECCurve.NamedCurves.nistP256);
// Define certificate request
CertificateRequest certificateRequest = new CertificateRequest(
new X500DistinguishedName("C=SK,L=Bratislava,CN=BasicEcdsaCertificateRequestTest"),
ecKeyPairOfEndEntity,
HashAlgorithmName.SHA256);
// Define certificate extensions
certificateRequest.CertificateExtensions.Add(new X509BasicConstraintsExtension(false, false, 0, true));
certificateRequest.CertificateExtensions.Add(new X509SubjectKeyIdentifierExtension(certificateRequest.PublicKey, false));
certificateRequest.CertificateExtensions.Add(new X509KeyUsageExtension(X509KeyUsageFlags.DigitalSignature, false));
// Issue X.509 certificate for end entity
X509Certificate2 certificateOfEndEntity = certificateRequest.Create(
pkcs11CertOfCertificateAuthority.Info.ParsedCertificate.SubjectName,
X509SignatureGenerator.CreateForECDsa(pkcs11CertOfCertificateAuthority.GetECDsaPrivateKey()),
DateTimeOffset.UtcNow,
DateTimeOffset.UtcNow.AddDays(365),
new BigInteger(1).ToByteArray());
// Verify signature on X.509 certificate for end entity
Assert.IsTrue(CaCertSignedEndEntityCert(pkcs11CertOfCertificateAuthority.Info.ParsedCertificate.RawData, certificateOfEndEntity.RawData));
// Asociate end entity certificate with its private key
certificateOfEndEntity = certificateOfEndEntity.CopyWithPrivateKey(ecKeyPairOfEndEntity);
// Export end entity certificate to PKCS#12 file
string basePath = Helpers.GetBasePath();
string pkcs12FilePath = Path.Combine(basePath, "BasicEcdsaCertificateRequestTest.p12");
File.WriteAllBytes(pkcs12FilePath, certificateOfEndEntity.Export(X509ContentType.Pkcs12, "password"));
}
Hope this helps.
Is there any way to change a http server to https using the library http4s? (https://http4s.org/)
I found myself facing this same issue but I managed to solve it, here's the thing:
You need to look for the moment when you build your server, presumably with BlazeServerBuilder.
BlazeServerBuilder has the method "withSslContext(sslContext: SSLContext)" to enable SSL. Thus, all you need to do is create a SSLContext object and pass it to the server builder.
Remember that you will probably have to store your SSL certificate in a keystore using Java's keytool utility before using it.
SSL context and SSL certificate
How to create an SSL context with an SSL certificate is another question, but here is an interesting post that covers the process of getting a free certificate from Let's Encrypt, storing it in a keystore and using it from a Java application: Using Let's Encrypt certificates in Java applications - Ken Coenen — Ordina JWorks Tech Blog
And here's the code I used for creating a SSLContext in Scala:
val keyStorePassword: String = your_keystore_password
val keyManagerPassword: String = your_certificate_password
val keyStorePath: String = your_keystore_location
val keyStore = KeyStore.getInstance(KeyStore.getDefaultType)
val in = new FileInputStream(keyStorePath)
keyStore.load(in, keyStorePassword.toCharArray)
val keyManagerFactory = KeyManagerFactory.getInstance(KeyManagerFactory.getDefaultAlgorithm)
keyManagerFactory.init(keyStore, keyStorePassword.toCharArray)
val trustManagerFactory = TrustManagerFactory.getInstance(TrustManagerFactory.getDefaultAlgorithm)
trustManagerFactory.init(keyStore)
val sslContext = SSLContext.getInstance("TLS")
sslContext.init(keyManagerFactory.getKeyManagers, trustManagerFactory.getTrustManagers, new SecureRandom())
sslContext
I have some issues with Akka http configuration on the client side. I am trying to connect to a server which doesn't provide:
- a public signed certificate
- a certificate corresponding to the hostname
I don't have the hand on this nginx so I cannot change the server side configuration. I can only change the client side.
After lots of investigation on configuring SSL, I have found that I need to configure SSL options in application.conf at two different levels :
akka.ssl-config.ssl.loose.acceptAnyCertificate=true
akka.ssl-config.loose.disableHostnameVerification = true
and
ssl-config.loose.acceptAnyCertificate=true
ssl-config.loose.disableHostnameVerification = true
I have checked the configuration is fine with
log-config-on-start = "on"
The problem is that I still get error at the akka debug level (not very clear)
[ingestionApiClient-akka.actor.default-dispatcher-13] [akka://ingestionApiClient/user/StreamSupervisor-0/flow-216-1-unknown-operation] closing output
Looking at wireshark I have found that's a problem of certificate validation
TLSv1 Record Layer: Alert (Level: Fatal, Description: Certificate Unknown)
I suppose the JVM configuration is overiding all I have done so I also tried to follow this method to modify JVM SSL config :
Java SSL: how to disable hostname verification
No problem with configuring the SSLContext and passing it to akka http because I can set the default HttpsContext with
val sc = SSLContext.getInstance("TLS")
*...configuration...*
val customContext =HttpsContext(sc, sslParameters = Some(params))
Http().setDefaultClientHttpsContext(customHttpsContext)
But I cannot find anyway to configure the default hostname verifier. The Http class doesn't have any method like Http().setDefaultHostnameVerifier
This how I connect to the server
val dataIngestFlow = Http().outgoingConnectionTls(config.httpEndpointHost,config.httpEndpointPort)
How can I achieve this ? Thanks a lot for your help
I don't know which version of akka and akka-http you use but have you tried to set the configuration field akka.ssl-config.hostnameVerifierClass to your specific implementation of the HostNameVerifier interface?
The simplest verifier which accepts everything looks like this:
public static class AcceptAllHostNameVerifier implements HostnameVerifier {
#Override
public boolean verify(String s, SSLSession sslSession) {
return true;
}
}
I also got stuck in similar issue and was getting similar errors. with following code I was able to get through:
val trustStoreConfig = TrustStoreConfig(None, Some("/etc/Project/keystore/my.cer")).withStoreType("PEM")
val trustManagerConfig = TrustManagerConfig().withTrustStoreConfigs(List(trustStoreConfig))
val badSslConfig = AkkaSSLConfig().mapSettings(s => s.withLoose(s.loose
.withAcceptAnyCertificate(true)
.withDisableHostnameVerification(true)
).withTrustManagerConfig(trustManagerConfig))
val badCtx = Http().createClientHttpsContext(badSslConfig)
Http().superPool[RequestTracker](badCtx)(httpMat)