We process and refresh our AAS models in ADFv2 using API calls.
We want to be able vertically scale up (change the tier) just before we refresh the model and then scale back down once it's done.
I can't find an API call to do this or any kind of command to execute within a pipeline in ADF.
It's simple for Azure SQL database
ALTER DATABASE <db_name> MODIFY (SERVICE_OBJECTIVE = 'service-tier')
For reasons out of my control, using runbooks is not an option.
There must be alternatives.
You can leverage the REST API
Set-AzureRmAnalysisServicesServer -ResourceGroupName $ResourceGroupName -Name $AnalysisServerName -Sku "S1"
for altering the tier.
Note: you cannot change between basic to standard via manual or API approach.
You can only alter around within the same scope of tier like S1,S2.S4 etc
You can leverage ADF web activity to do the same
Related
I have an amplify and I get how to add an api function, use a lambda layer etc. What I don't see is how to create the database on startup -- it appears from the documentation that this is done from a CloudFormation stack, but I still can't see how to ensure that the database is set up on startup of the app (or build the tables if not) using something like SQLAlchemy.
What's the intended flow here?
I am using REST API (https://learn.microsoft.com/en-us/azure/kusto/api/rest/request) to interact with the database in ADX.
I want to create more databases in the same cluster. How should I do it using Java?
I am not using the Java SDK. I have relied on the REST APIs so far.
I think I cannot create a new database using the REST API, so looking for alternative.
It would have been really helpful if there was a command like ".create table tablename" just for the database.
Clusters and databases can be managed using the "Control Plane", aka ARM APIs. These APIs have libraries in different languanges (as well as REST).
For instance, for the java library use this link, for C# use this link
Example for how to create a database in C# library (Java should be very similar):
var database = managementClient.Databases.CreateOrUpdate(resourceGroup, clusterName, databaseName, new Database(location, softDeletePeriod: softDeletePeriod, hotCachePeriod: hotCachePeriod));
Read more here
I think you'll need to use the Azure ARM REST API since the database is treated as a resource. From that point you can interact with it through the ADX APIs.
In cloudera is there a way to update list of configurations at a time using CM-API or CURL?
Currently I am updating one by one one using below CM API.
services_api_instance.update_service_config()
How can we update all configurations stored in json/config file at a time.
The CM API endpoint you're looking for is PUT /cm/deployment. From the CM API documentation:
Apply the supplied deployment description to the system. This will create the clusters, services, hosts and other objects specified in the argument. This call does not allow for any merge conflicts. If an entity already exists in the system, this call will fail. You can request, however, that all entities in the system are deleted before instantiating the new ones.
This basically allows you to configure all your services with one call rather than doing them one at a time.
If you are using services that require a database (Hive, Hue, Oozie ...) then make sure you set them up before you call the API. It expects all the parameters you pass in to work so external dependencies must be resolved first.
How to improve datapower monitoring ? I want to improve our monitoring techniques say for example, want to check that all objects (FSH /MQFSHs, SSl proxy, crypto profile etc) are up and incase if it goes down , should be notified by email or something. checking number of files in file management ondisk folders.Basically validate the adapter after deployment (we use soapUi to test adapter functionality, however something else to improve or added validation).please suggest any ideas that can be implemented as a process improvement on Datapower
For example you can get the status off all your domains using this soma call. You can test this using soap UI. You can get the list of various soma calls using the datapower mgmt wsdl (available in datapower store directory).
<!-- get all the domains -->
<xsl:variable name="domainsList">
<dp:url-open target="{$XML-MGMT-URL}" response="responsecode">
<env:Envelope xmlns:env="http://schemas.xmlsoap.org/soap/envelope/">
<env:Body>
<dp:request xmlns:dp="http://www.datapower.com/schemas/management">
<dp:get-status class="DomainStatus"/>
</dp:request>
</env:Body>
</env:Envelope>
</dp:url-open>
</xsl:variable>
Try using SOMA commands of XML management interface to check the object status.
I am not sure if this is the best approach but this is how i have implemented it. You can always create a testing service in DataPower with/without interactive java application to perform all the soap test you are performing using soapUI. You can perform SOMA/AMP calls to check the status of objects, ping external services, etc. You can schedule these test on a regular interval or manual.
Depending how you set it up, you can either generate an email with status of each object/service you are testing or create a html dashboard that records the current status of everything.
I am creating cloud formation script, which will have ELB. In Auto Scaling launch configuration, I want to add encrypted EBS volume. Couldn't find an encrypted property withing blockdevicemapping. I need to encrypt volume. How can I attach an encrypted EBS volume to an EC2 instance through auto scaling launch configuration?
There is no such property for some strange reason when using launch configurations, however it is there when using blockdevicemappings with simple EC2 instances. See
launchconfig-blockdev vs ec2-blockdev
So you'll either have to use simple instances instead of autoscaling groups, or you can try this workaround:
SnapshotIds are accepted for launchconf blockdev too, and as stated here "Snapshots that are taken from encrypted volumes are automatically encrypted. Volumes that are created from encrypted snapshots are also automatically encrypted."
Create a snapshot from an encrypted empty EBS volume and use it in the CloudFormation template. If your template should work in multiple regions then of course you'll have to create the snapshot in every region and use a Mapping in the template.
As Marton says, there is no such property (unfortunately it often takes a while for CloudFormation to catch up with the main APIs).
Normally each encrypted volume you create will have a different key. However, when using the workaround mentioned (of using an encrypted snapshot) the resulting encrypted volumes will inherit the encryption key from the snapshot and all be the same.
From a cryptography point of view this is a bad idea as you potentially have multiple, different volumes and snapshots with the same key. If an attacker has access to all of these then he can potentially use differences to infer information about the key more easily.
An alternative is to write a script that creates and attaches a new encrypted volume at the boot time of a instance. This is fairly easy to do. You'll need to give the instance permissions to create and attach volumes and either have installed the AWS CLI tool or a library for your preferred scripting language. One you have that you can, from the instance that is booting, create a volume and attach it.
You can find a starting point for such a script here: https://github.com/guardian/machine-images/blob/master/packer/resources/features/ebs/add-encrypted.sh
There is an AutoScaling EBS Block Device type which provides the "Encrypted" option:
http://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aws-properties-as-launchconfig-blockdev-template.html
Hope this helps!
AWS recently announced Default Encryption for New EBS Volumes. You can enable this per region via
EC2 Console > Settings > Always encrypt new EBS volumes
https://aws.amazon.com/blogs/aws/new-opt-in-to-default-encryption-for-new-ebs-volumes/