How can I change runtime deterministic subject token partition number in golang?
For example,in account.conf I have:
mapping:{ "events.*" "events.{{wildcard(1)}}.{{partition(5,1)}}" }
How can I change 5 to another number in golang without restarting the whole server?
Related
I'm confused on how KCL works. First of all these are my understanding now.
1 KCL application uses one application name, creates one dynamodb table.
1 KCL application has one worker with x number of record-processor working parallel on x number of shards in a stream.
The dynamodb table keeps track of owner, checkpoints and etc of each shards.
If i create multiple, let's say 3, KCL application with different application name, then they are basically different application reading from the same stream, isolate from each other by having separate dynamodb tables. All 3 of them will read all x number of shards in the stream and keep track of the checkpoints separately.
Based on a few docs that i read, for example: https://docs.aws.amazon.com/streams/latest/dev/kinesis-record-processor-scaling.html
I would assume if i create another KCL application with the same application name, there would be 2 KCL application working on the same stream, with shards being load balanced to 2 workers in the 2 apps.
So, technically i can create 8 KCL app(let says there are 8 shards in the stream) in 8 ec2 instances, and each of them will process exactly one shard without clash, since each of them checkpoint in its own row in the dynamodb table.
I thought that is the case, but this post suggest otherwise: Multiple different consumers of same Kinesis stream
Else how can i achieve this
All workers associated with this application name are assumed to be working together on the same stream. These workers may be distributed on multiple instances. If you run an additional instance of the same application code, but with a different application name, the KCL treats the second instance as an entirely separate application that is also operating on the same stream.
as mentioned here https://docs.aws.amazon.com/streams/latest/dev/kinesis-record-processor-implementation-app-java.html#kinesis-record-processor-initialization-java
Reference:
https://www.amazonaws.cn/en/kinesis/data-streams/faqs/#recordprocessor
https://docs.aws.amazon.com/streams/latest/dev/kinesis-record-processor-scaling.html
https://docs.aws.amazon.com/streams/latest/dev/kinesis-record-processor-implementation-app-java.html#kinesis-record-processor-initialization-java
KCL library needs ConfigsBuilder where you pass streamName, applicationName, kinesisAsyncClient etc. Here, if you specify an application name associated with stream name, then
DynamoDB table with the application name and uses the table to
maintain state information
So if you have multiple streams, then you create multiple software.amazon.kinesis.common.ConfigsBuilder with individual streamNames and its associated applicationNames.
Pass individual configBuilder properties to software.amazon.kinesis.coordinator.Scheduler
This way you will have a dynamodb for every single streams. And your multi instance app can consume each stream event only once.
Ok so I have an application that I inherited that we do not know the root token and we do not have any recovery keys or unseal keys. The problem is, we cannot authenticate into Vault at all and we also cannot have the instance go down.
I do have access to the datastore it uses (DynamoDB) and the encrypting keys. My assumption is that it would be possible in theory to manually add an entry and set a password directly on the underlying datastore instance so that we can have a root account again.
I know this one is weird but we cannot re-initialize the database.
Any thoughts on how this could be done?
You can try one of the below -
The initial root token generated at vault operator init time -- this token has no expiration
By using another root token; a root token with an expiration cannot create a root token that never expires
By using vault operator generate-root (example) with the permission of a quorum of unseal key holders
Root tokens are useful in development but should be extremely carefully guarded in production. In fact, the Vault team recommends that root tokens are only used for just enough initial setup (usually, setting up auth methods and policies necessary to allow administrators to acquire more limited tokens) or in emergencies, and are revoked immediately after they are no longer needed. If a new root token is needed, the operator generate-root command and associated API endpoint can be used to generate one on-the-fly.
You can read more here - https://www.vaultproject.io/docs/concepts/tokens
No matter how bad the breakup was with the previous administrator, call him and ask for the shards. Now. It's an emergency.
To create a root token, you need a quorum of shards. A shard is a large number that could be in base64. For example, this is what the same shard looks in both formats:
9PTUFNoCFapAvxQ2L72Iox/hmpjyHGC5PpkDj9itaMo=
f4f4d414da0215aa40bf14362fbd88a31fe19a98f21c60b93e99038fd8ad68ca
You can mix and match formats, but each shard can be only entered once.
Run the command vault status to know how many different shards you need to find. The default Threshold is 3:
$ vault status
Key Value
--- -----
Recovery Seal Type shamir
Initialized true
Sealed false
Total Recovery Shares 5
Threshold 3
If you do get your hands on some shards, enter the command vault operator generate-root and enter them at the prompt. Don't cancel the ongoing root token generation, if someone entered a shard some time in the past, Vault has it (even if you don't). vault operator generate-root -status will tell you if Vault already has some shards. Here is an example where the first shard of three was entered:
$ vault operator generate-root -status
Nonce 9f435314-ce20-4716-cea7-a083de224e4e
Started true
Progress 1/3
Complete false
OTP Length 26
If you can't find the shards, you are in trouble. You will have to find a password and read all the secrets one by one (can be scripted), ideally every version of them. You say you can't log in, so you might have to ask your user to do it.
Keep in mind that some backends (like the PKI) can't be exported manually, not even by root.
I'm trying to incorporate end to end encryption using Apache Pulsar. So far the examples I've seen look similar to http://pulsar.apache.org/docs/en/security-encryption/
I saw that the encryption key (public / private key) is fetched every 4 hours in the key rotation section. It seems to indicate that you can only have a single key to encrypt the messages you send.
I’d like to be able to use different encryption keys for different topics/tenants. Does pulsar support different encryption keys for different topics?
Basically my client will pull a different key depending on which tenant the data belongs to and encrypt any messages pertaining to that specific key using pulsar's end to end encryption.
How would this look?
Is it as simple as creating a new producer any time I want to use a different key for encryption?
Producer producer = pulsarClient.newProducer()
.topic("persistent://my-tenant/my-ns/my-topic")
.addEncryptionKey("myTenant1Key")
.cryptoKeyReader(new RawFileKeyReader("tenant1_pubkey.pem", "tenant1_privkey.pem"))
.create();
So far I've used the pulsar client to send encrypted messages using a local private/public key pair. Can I just switch out the key pair and instantiate a new producer anytime I want a message to be encrypted differently? I've tried asking around and can't find the right answer.
Absolutely, you are creating separate producers for different topic and can use different keys for each producer.
I'm developing a sgx application using sgx_dh API. The workflow can be summarized as follows (two roles here are dh_initiator, dh_i in short, and dh_responder, i.e., dh_r.):
step0: Initiate session in dh_i and dh_r side;
step1: dh_r generates msg1 contains g_a and sends msg1 to dh_i;
step2: dh_i processes msg1 and generates msg2 containing g_b, and sends msg2 to dh_r;
step3: dh_r processes msg2 and genertaes msg3 including affirm of the sharing key, and then sends msg3 to dh_i;
step4: dh_i processes msg3. DH key exchange has done.
In step3, the dh_r will output a sgx_dh_session_enclave_identity_t type variable, and so as to dh_i in step4 that outputs a variable in the same type. According to the SGX Developer Reference, the two variables are used in dh session establishment.
But I cannot figure out where they will BE USED since they are generated when the key exchange and enclave authentication has done.
Any suggestion or hint is appreciated, and if any misunderstanding here, please correct me kindly. Thanks so much~
These two variables are provided to explicit the values you should verify to ensure you're communicating with the correct enclaves (you may check how they are used in the LocalAttestation sample: https://github.com/intel/linux-sgx/blob/62b116c502b09b125db9acc965694d3ecff8e698/SampleCode/LocalAttestation/EnclaveInitiator/EnclaveMessageExchange.cpp#L133-L144).
They are not here to be passed to other functions of the dh exchange protocol provided by the sdk, you should process them "manually" and define your own acceptation policy before relying on the secure channel established.
EDIT
I'm still hoping for some advice on this, i tried to clarify my intentions...
When i came upon device pairing in my mobile communication framework i studied a lot of papers on this topic and and also got some input from previous questions here. But, i didn't find a ready to implement protocol solution - so i invented a derivate and as i'm no crypto geek i'm not sure about the security caveats of the final solution:
The main questions are
Is SHA256 sufficient as a commit function?
Is the addition of the shared secret as an authentication info in the commit string safe?
What is the overall security of the 1024 bit group DH
I assume at most 2^-24 bit probability of succesful MITM attack (because of 24 bit challenge). Is this plausible?
What may be the most promising attack (besides ripping the device out off my numb, cold hands)
This is the algorithm sketch
For first time pairing, a solution proposed in "Key agreement in peer-to-peer wireless networks" (DH-SC) is implemented. I based it on a commitment derived from:
A fix "UUID" for the communicating entity/role (128 bit, sent at protocol start, before commitment)
The public DH key (192 bit private key, based on the 1024 bit Oakley group)
A 24 bit random challenge
Commit is computed using SHA256
c = sha256( UUID || DH pub || Chall)
Both parties exchange this commitment, open and transfer the plain content of the above values.
Alice Bob
ca = commit()
-------^ ca
cb = commit()
cb ^-----------
open
---^ DH pub a, chall a
open
DH pub b, chall b ^---
The 24 bit random is displayed to the user for manual authentication
DH session key (128 bytes, see above) is computed
When the user opts for persistent pairing, the session key is stored with the remote UUID as a shared secret
Next time devices connect, commit is computed by additionally hashing the previous DH session key before the random challenge. For sure it is not transfered when opening.
c = sha256( UUID || DH pub || DH sess || Chall)
Now the user is not bothered authenticating when the local party can derive the same commitment using his own, stored previous DH session key. After succesful connection the new DH session key becomes the new shared secret.
As this does not exactly fit the protocols i found so far (and as such their security proofs), i'd be very interested to get an opinion from some more crypto enabled guys here. BTW. i did read about the "EKE" protocol, but i'm not sure what the extra security level is.
"Is SHA256 sufficient as a commit function?"
The use of SHA256 should be just fine. The only issue I have heard of is that it has a hash extension vulnerability. If you produce multiple hashes using the same data don't simply concat more data to the end of the data you already hashed. In your post have have the two hashes "sha256( UUID || DH pub || Chall)" and "sha256( UUID || DH pub || DH sess || Chall)". If that second one was "sha256( UUID || DH pub || Chall || DH sess)" then there would be a relation between the hash values if UUID, DH pub, and Chall were all the same values as before. You should either take care to avoid the hash extension issue or include a salt value into the data to be hashed, either by communicating the salt across the link or having differing vales for each code path.
On a side note: is it really necessary to transmit a Chall when you already have a previous session key saved and don't need to ask the user to manually confirm the challenge code?
"Is the addition of the shared secret as an authentication info in the commit string safe?"
I'm guessing you mean to ask "Is it safe to include secret information in a hash that is to be made public?" If the secret is something that is really hard to guess and would take a really long time to perform a bruteforce attack against, then yes it is safe. If the secret is something easy to guess or has only a few possible values, then no, it's not safe unless you, at the same time, include some hard to guess secret to force a potential eavesdropper to have to guess all such secrets simultaneously. For a large, effectively random number like a DH shared secret then it should be just fine.
"What is the overall security of the 1024 bit group DH"
I'm not sure if DH group 1024 is what you want to use. A key exchange that is considered to be close to being as effective as the SHA256 hash algorithm you're using would be 521 bit ECDH. The cryptographic strength of ECDH is considered to be 1/2, so if you want 256 bit security, you want 521 bit ECDH. Unfortunately, I'm not certain about the security of the many individual 521 bit ECDH groups that have been published.
"I assume at most 2^-24 bit probability of succesful MITM attack (because of 24 bit challenge). Is this plausible?"
There is more than one way perform a MITM attack. Eve will use every resource at her disposal to perform her attacks and if you are not careful she will exploit something you didn't think of. This is why peer review is necessary in cryptography.
Simply, if you make your own cryptographic solution, then it's weak.
for the little story, the VISA guys have to start again 4 times before it's became strong enough.
I'm not a security expert, but it was what my crypto teacher told us everytime.
I've come up with this possible attack, based on my understanding of the protocol, inspired by Lowe's Attack to Needham-Shroeder Public Key Protocol:
Alice wants to reconnect. Calculates its committment ca and sends to Bob. The message is captured by Mallory.
Mallory answers. She does not know the shared secret, so she invents one. Calculates cb and sends to Alice.
At this step, Alice cannot verify the shared secret yet. So she sends DHpubA and ChallA.
Mallory ignores the messages from Alice and disappears.
Now Mallory has a valid DHpubA, ChallA and the corresponding (valid) ca.
Mallory sends ca to Bob.
Bob answers with cb.
Mallory sends a valid set of DhpubA, ChallA
Bob sends his DhpubB and ChallB
Since Bob can validate Mallory's messages, she is authenticated as Alice. Obviously Mallory does not know DHprivA, se she cannot calculate the current session key, but nevertheless you have a security flaw since Bob thinks he's talking to Alice.
General advice: avoid implementing you own cryptographic solution and don't trust security reviews from anyone else than an established security firm.
If you feel that your security requirements are not satisfied by standard mainstream crypto, try stating your requirements and asking whether is there a security procotol that matches them.
That sounds OK. Not sure what you meant by "fix[ed] UUID"?
Could a rogue app access the UUID and session keys: are they stored system-wide or in a service? There is some text in the SDK that suggests that any keystore always waits for user confirmation before returning a key.