The problem / use case:
In order for Salt to apply the correct states to minions, Salt needs to be aware of the role a given minion is intended to fill.
We define this through tags on Azure VMs and custom attributes on vSphere VMs.
In order to minimize auth concerns, we want the salt master to be responsible for retrieving this data and providing it to minions.
Does Salt provide these capabilities? What are they?
ext_pillar is how a master connects to azura and vsphere to collect data about minions and then send that data to minions. since this is pillar data it is also target-able in almost everything as pillar data. the one exception being pillar.
https://docs.saltproject.io/en/latest/topics/development/modules/external_pillars.html
https://docs.saltproject.io/en/latest/ref/pillar/all/salt.pillar.azureblob.html#module-salt.pillar.azureblob
https://docs.saltproject.io/en/latest/ref/pillar/all/salt.pillar.vmware_pillar.html#module-salt.pillar.vmware_pillar
Related
I am trying to encrypt my username and password on cloud foundry. Currently I am storing these values as a CUPS (VCAP_SERVICES).
SPRING_DATASOURCE_URL: jdbc:oracle:thin:#//spring.guru.csi0i9rgj9ws.us-east-1.rds.amazonaws.com:1521/ORC
SPRING_DATASOURCE_USERNAME: UserAdmin
SPRING_DATASOURCE_PASSWORD: p4ssw0rd
SPRING_DATASOURCE_initialize: false
I want to encrypt it so it would show some type of token/encryption or UUID instead of my actual username and password. How would I be able to encrypt these values so that, when I look at my VCAP_SERVICES these values would not be exposed?
Example from Cloud Foundry Provided Service
VCAP_SERVICES=
{
cleardb: [
{
name: "cleardb-1",
label: "cleardb",
plan: "spark",
credentials: {
SPRING_DATASOURCE_URL: "jdbc:oracle:thin:#//spring.guru.csi0i9rgj9ws.us-east-1.rds.amazonaws.com:1521/ORC",
SPRING_DATASOURCE_USERNAME: "UserAdmin",
SPRING_DATASOURCE_PASSWORD: "p4ssw0rd",
SPRING_DATASOURCE_initialize: "false"
}
}
]
As you can see VCAP_SERVICES above is exposed how can I encrypt it so that username and password is encrypted like example below
Desired output
Username: hVB5j5GgdiP78xCSV9sNv4FeqQJducBxXlB81090ozYB
Password: hVB523fff78xCSV9sNv4FeqQ341090324234fdfdsrrf
Depending on what do you want to archive you can use spring vault mentioned by you or external vault instance with hashicorp service broker https://github.com/hashicorp/cf-vault-service-broker to retrieve/store credentials within your application in a secure way.
As a side note - the Mongodb service credentials on the screenshot are not encrypted but randomly generated by the service broker.
Most importantly - you shouldn't store/provide service credentials in your application manifest, but obtain this credentials (for binded cloudfoundry services) by parsing environment variable VCAP_SERVICES.
https://docs.cloudfoundry.org/devguide/deploy-apps/environment-variable.html#VCAP-SERVICES
External services should be presented to cloud foundry apps via CUPS https://docs.cloudfoundry.org/devguide/services/user-provided.html
Since you appear to be using Spring already, you might want to look at Spring Cloud Config.
https://cloud.spring.io/spring-cloud-config/
For larger projects it makes it easy to externalize and manage your configuration. A common setup would be to store the configuration in Git (but there are other backends, including Vault), Spring Cloud Config then runs as a server and provides the configuration to your running applications (clients). Your application (client) doesn't need to do much beyond include the Spring Cloud Config dependencies and a couple lines of config. Settings are obtained automatically and integrated through the Environment and PropertySource abstractions, which makes for a very clean integration where you don't need to do a lot of work.
For smaller projects, it might be more overhead than you'd like. For starts, you have to run the Spring Cloud Server. If you only have one or two small apps, the resources to run the SCC server might be more than your app or apps total. Complexity would be another concern. Using SCC definitely adds some complexity and possible failure scenarios. You'd need to understand what's happening well enough to troubleshoot when there is a problem.
What you might want to consider instead for smaller projects is simply using user provided services in CF. These make for a central place to store your config settings (doesn't have to just be databases, could be keys and other things too). You can then bind these to your apps to expose the configuration settings to that app.
There is some security in that Cloud Controller manages who can access your services. I believe the information that Cloud Controller stores is also encrypted at rest, which is a plus. That said, information is exposed to your applications via environment variables (i.e. VCAP_SERVICES) so anything in the container that can see the environment variables will be able to read your settings.
Using user provided services is one step up on environment variables. Not really from a security stand point, but more from a management stand point. You create the user provided service once, and can then bind it to any number of apps. With env variables, you'd need to set those for every app. It's more tedious, and it's prone to typos. You can also put the service name into your manifest.yml file so it automatically binds to the app and still check that into source control. If you were putting env variables with sensitive info into your manifest.yml, you wouldn't want to check it into source control and you'd have to be a lot more careful with that file.
ext_pillar feature allows salt master to append dynamic data to each minion's pillar dictionary. This is achieved by salt master evaluating an eponymous python function on behalf of each minion during the pillar refresh stage.
When a large number of minions is present in the stack, the ext_pillar function would be evaluated multiple times, on behalf of each minion. In many cases this is highly undesirable, due to performance or other resource limits.
Thus, the question arises: is there a way to evaluate the ext_pillar function once per pillar_refresh command and then reuse the produced dictionary for all selected minions?
There isn't a way to do that right now. There has been an optimization done to many of the external pillar modules to cache the connection to the database, so that at least the connection is not created and torn down for each minion. But that doesn't have the optimization you're talking about regarding making one large query for all the minions at once.
Would you mind opening an issue here: https://github.com/saltstack/salt/issues/new
That will help make sure this gets discussed and considered. I see a lot of value in it.
Can I access persisted data of running orchestration Instances from the BizTalk database?
My BizTalk application deals with long-running processes and hundreds of orchestration instances can be running at a time.
I would like to access data persisted by these orchestration instances and display it on my application's UI. The data would give an insight about how many instances are running and at which state each of them is.
EDIT :
Let me try to be a little more specific.
My BizTalk application gets ticket requests (messages) from a source and after checking some business rules they are to be assigned to different departments of the company. The tickets can hop between inbox of different departments as each department completes its processing.
Now, the BizTalk orchestration instances are maintaining all the information that which department owns a particular ticket at a given time. I would want to read this orchestration information and generate inbox for each of the department at runtime. I can definitely do this by pushing this information to a separate database and populate the UI from there BUT as all this useful information is already available in the form of orchestration instances I would like to utilize it and avoid any syncing issues.
Does it make any sense?
The answer to your specific question is NO.
BAM exists for this purpose exactly.
Yes it is doable. Your question is little confusing. You can't get the data which is persisted for your orchestration instance, however You can get number of running or dehydrated instances using various options like WMI, ExplorerOM library. As a starting point you can look at some samples provided as part of BizTalk installation under SDK\Samples\Admin folder. Also you should be looking at MSBTS_ServiceInstance WMI class to get the service instances. You can also look at a sample http://msdn.microsoft.com/en-us/library/aa561219.aspx here. You can also use powershell to perform the same operation
I am using SignalR with my ASP.NET application. What my application needs is to pressist the groups data that is updated from various servers. According to SignalR documentation it's my responsibility to do this. It means that I need to use an external server/service that will collect the data from one or more servers and I can query that data from a single place.
I first thought that MemCached is the best candidate, because it's fast and the data that I need to put there is volatile. The problem is that I need to store collections, for example: collection A with user Ids, so I can have Collection A with 2000 user ids and Collection B with 40,000 ids. The problem is that I need to update this collection and remove and insert id very quickly. I afraid that because the commands will be initiated from several servers, and the fact that I might need to read the entire collection and update it on a either web servers, the data won't be consistent. Web Server A might update the data, but Server B will read the data before Server A finished updating it. There is a concurrency conflict.
I'm searching for the best way to implement this kind of strategy in my ASP.NET 4.5 application. I think that this might be a choice to use a in-memory database or that to insure no data integrity.
I want to ask you what is the best solution for my problem.
Here's an example for my problen:
MemCached Server - stores the collections (e.g. Collection A, B, C, D), each collection stores User Id's, which can be thousands of Ids and even much more.
Web Servers - My Amazon EC2 web servers with SignalR installed. Can be behind load balancer. Those servers need to gain access to the memcached server and get a complete collection items by the Collection name (e.g. "Collection_23"). They need to be able to remove items (User Id's) and add Items. All this should be fast as possible.
I hope that I explained myself right. Thanks.
Alternatively, you can use Redis, like Memcached everything is served from in-memory. Redis has many other capabilities beyond a simple key-value datastore; for your specific case you might use Redis transactions, which ensures data consistency.
In a comment in another post it shows a link to redis provider. The link is broken, it seems that it is now integrated in the main SignalR project: https://github.com/SignalR/SignalR/tree/master/src/Microsoft.AspNet.SignalR.Redis
You have the redis nuget here:
http://www.nuget.org/packages/Microsoft.AspNet.SignalR.Redis
and documentation here:
http://www.asp.net/signalr/overview/signalr-20/performance-and-scaling/scaleout-with-redis
We are thinking to make some architectural changes in our application, which might affect the technologies we'll be using as a result of those changes.
The change that I'm referring in this post is like this:
We've found out that some parts of our application have common data and common services, so we extracted those into a GlobalServices service, with its own master data db.
Now, this service will probably have its own cache, so that it won't have to retrieve data from the db on each call.
So, when one client makes a call to that service that updates data, other clients might be interested in that change, or not. Now that depends on whether we decide to keep a cache on the clients too.
Meaning that if the clients will have their own local cache, they will have to be notified somehow (and first register for notifications). If not, they will always get the data from the GlobalServices service.
I need your educated advice here guys:
1) Is it a good idea to keep a local cache on the clients to begin with?
2) If we do decide to keep a local cache on the clients, would you use
SqlCacheDependency to notify the clients, or would you use WCF for
notifications (each might have its cons and pros)
Thanks a lot folks,
Avi
I like the sound of your SqlCacheDependency, but I will answer this from a different perspective as I have worked with a team on a similar scenario. We created a master database and used triggers to create XML representations of data that was being changed in the master, and stored it in a TransactionQueue table, with a bit of meta data about what changed, when and who changed it. The client databases would periodically check the queue for items it was interested in, and would process the XML and update it's own tables as necessary.
We also did the same in reverse for the client to update the master. We set up triggers and a TransactionQueue table on the client databases to send data back to the master. This in turn would update all of the other client databases when they next poll.
The nice thing about this is that it is fairly agnostic on client platform, and client data structure, so we were able to use the method on a range of legacy and third party systems. The other great point here is that you can take any of the databases out of the loop (including the master - e.g. connection failure) and the others will still work fine. This worked well for us as our master database was behind our corporate firewall, and the simpler web databases were sitting with our ISP.
There are obviously cons to this approach, like race hazard, so we were careful with the order of transaction processing, error handling, de-duping etc. We also built a management GUI to provide a human interaction layer before important data was changed in the master.
Good luck! Tim