This document describes how to create an Azure Container Registry task using an ARM template.
One thing that isn't clear from the document and the reference is how to create an ACR timer task with an ARM template. Effectively doing this same as the az acr task timer add command.
Is it possible to add a timer to an ACR task with an ARM template?
Of course, Yes. You can find the timer trigger in the template of the ACR task, and here is an example for the timer trigger:
"trigger": {
"timerTriggers": [
{
"schedule": "0 21 * * *",
"status": "Enabled",
"name": "t1"
}
]
Related
I am sending logs to an azure eventhub with Serilog (using WriteTo.AzureEventHub(eventHubClient)), after that I am running a filebeat process with the azure module enabled, so I send these logs to elasticsearch to be able to explore them with Kibana.
The problem I have is that all the information goes to the field "message", I would need to separate the information of my logs in different fields to be able to do good queries.
The way I found was create an ingest pipeline in Kibana and through a grok processor I separate the fields inside the "meessage" and generate multiple fields. In the filebeat.yml I set the pipeline name, but nothing happen, it seems the pipeline is not working.
output.elasticsearch:
# Array of hosts to connect to.
hosts: ["localhost:9200"]
pipeline: "filebeat-otc"
Does anybody knows what I am missing? THANKS in advance.
EDITION. I will add an example of my pipeline and my data. In the simulation is working properly:
POST _ingest/pipeline/_simulate
{
"pipeline": {
"processors": [
{
"grok": {
"field": "message",
"patterns": [
"%{TIME:timestamp}\\s%{LOGLEVEL}\\s{[a-zA-Z]*:%{UUID:CorrelationID},[a-zA-Z]*:%{TEXT:OperationTittle},[a-zA-Z]*:%{TEXT:OriginSystemName},[a-zA-Z]*:%{TEXT:TargetSystemName},[a-zA-Z]*:%{TEXT:OperationProcess},[a-zA-Z]*:%{TEXT:LogMessage},[a-zA-Z]*:%{TEXT:ErrorMessage}}"
],
"pattern_definitions": {
"LOGLEVEL" : "\\[[^\\]]*\\]",
"TEXT" : "[a-zA-Z0-9- ]*"
}
}
}
]
},
"docs": [
{
"_source": {
"message": "15:13:59 [INF] {CorrelationId:83355884-a351-4c8b-af8d-b77c48462f36,OperationTittle:Operation1,OriginSystemName:Fexa,TargetSystemName:Usina,OperationProcess:Testing Log Data,LogMessage:Esto es una buena prueba,ErrorMessage:null}"
}
},
{
"_source": {
"message": "20:13:48 [INF] {CorrelationId:8451ee54-efca-40be-91c8-8c8e18e33f58,OperationTittle:null,OriginSystemName:Fexa,TargetSystemName:Donna,OperationProcess:Testing Log Data,LogMessage:null,ErrorMessage:null}"
}
}
]
}
It seems when you use a module it will create and use an ingest pipeline in elasticsearch, and the pipeline option in the output is ignored.
So my solution was modify the index.final_pipeline. For this, in Kibana I went to Stack Management / Index Management there I found my index, there I went to Edit Settings and set "index.final_pipeline": "the-name-of-my-pipeline".
I hope this helps to anybody.
This was thanks to leandrojmp
The amplify docks here says that we can configure a lambda function as a dynamodb trigger by running **amplify add function** and selecting the "Lambda Trigger" option, but when I run the "amplify add api" (selected Python as runtime language) I am not getting the lambda trigger option, I'm only getting the "Serverless function" and "lambda layer" options.
Please help me to resolve this issue to access the feature.
docs snapshot - showing 4 options
my CLI snapshot - showing only 2 options
I know it works for nodejs runtime lambda, but I want this option for Python Lambda as well.
Just followed these steps with amplify CLI version 4.50.2.
To create a lambda function that is triggered by changes to a DynamoDB table, you can use the following command line actions, which are walked-through inside of the CLI after entering the below command:
amplify add function
Select which capability you want to add:
❯ Lambda function (serverless function)
Provide an AWS Lambda function name:
<YourFunctionsName>
Choose the runtime that you want to use:
> NodeJS # IMPORTANT: Must be NodeJS as of now, you can change this later by manually editing ...-cloudformation-template.json file inside function directory
Choose the function template you want to use
> Lambda Trigger
What event source do you want to associate with the lambda trigger
> Amazon DynamoDB Stream
Choose a DynamoDB event source option
>Use API category graphql #model backend DynamoDB table(s) in the current Amplify project
Choose the graphql #model(s)
<Select any models (using spacebar) you want to trigger the function after editing>
Do you want to trigger advanced settings
Y # IMPORTANT: If you are using a dynamodb event source based on a table defined by graphql schema, you will need to give this function read access to the api resource that contains the graphql schema that defines the table that drives the event
Do you want to access other resources in this project from your Lambda function?
y # See above, select your api that contains the data model and make sure that the function has at least read access.
After this, the other options (layer, call scheduling) are up to you.
After creating the function via the above CLI options, you can change the "Runtime" field inside the -cloudformation-template.json file inside function directory, eg if you want a python lambda function change the runtime to "python3.8". You will also need to create a file called index.py inside your function's directory which has a handler(event, context) function. See example below:
import json
def handler(event, context):
print("Triggered via DynamoDB")
print(event)
return json.dumps({'status_code': 200, "message": "Received from DynamoDB"})
After making these edits, you can run amplify push and, if you open your fxn in the management console online, it should show an attached dynamoDB stream.
Doesn't appear to be available anymore in the CLI codebase - see Supported-service.json deleted and replaced by supported-services.ts
https://github.com/aws-amplify/amplify-cli/commit/607ae21287941805f44ea8a9b78dd12d16d71f85#diff-a0fd8c5607fd81977cb4745b9af3af2c6649ded748991bf9968a7d782b000c6b
https://github.com/aws-amplify/amplify-cli/commits/4e974007d95c894ab4108a2dff8d5996e7e3ce25/packages/amplify-category-function/src/provider-utils/supported-services.ts
Select nodejs and you will be able to view lambda trigger
just add the following to {YOUR_FUNCTION_NAME}-cloudformation-template.json, remember to replace (YOUR_TABLE_NAME) to your table name.
"LambdaTriggerPolicyPurchase": {
"DependsOn": [
"LambdaExecutionRole"
],
"Type": "AWS::IAM::Policy",
"Properties": {
"PolicyName": "amplify-lambda-execution-policy-Purchase",
"Roles": [
{
"Ref": "LambdaExecutionRole"
}
],
"PolicyDocument": {
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Action": [
"dynamodb:DescribeStream",
"dynamodb:GetRecords",
"dynamodb:GetShardIterator",
"dynamodb:ListStreams"
],
"Resource": {
"Fn::ImportValue": {
"Fn::Sub": "${apilanguageGraphQLAPIIdOutput}:GetAtt:(YOUR_TABLE_NAME):StreamArn"
}
}
}
]
}
}
},
"LambdaEventSourceMappingPurchase": {
"Type": "AWS::Lambda::EventSourceMapping",
"DependsOn": [
"LambdaTriggerPolicyPurchase",
"LambdaExecutionRole"
],
"Properties": {
"BatchSize": 100,
"Enabled": true,
"EventSourceArn": {
"Fn::ImportValue": {
"Fn::Sub": "${apilanguageGraphQLAPIIdOutput}:GetAtt:(YOUR_TABLE_NAME):StreamArn"
}
},
"FunctionName": {
"Fn::GetAtt": [
"LambdaFunction",
"Arn"
]
},
"StartingPosition": "LATEST"
}
},
i got them by creating a dummy function using the template that shows up after you choose nodejs and checking compare its -cloudformation-template.json with my own function
I am using Corda version 4.3 and doing all the transactions on the account level by creating accounts for each node. However, I want that whenever I create a node a default account gets created so that no node is created without an account.
I wonder if I can do that in the RPC settings or in the main build.gradle file where I initialize a node like this :
node {
name "O=Node1,L=London,C=GB"
p2pPort 10005
rpcSettings {
address("localhost:XXXXX")
adminAddress("localhost:XXXXX")
}
rpcUsers = [[ user: "user1", "password": "test", "permissions": ["ALL"]]]
}
Try the following:
Create a class and annotate it as #CordaService -which means this class gets loaded as soon as the node starts- (https://docs.corda.net/api/kotlin/corda/net.corda.core.node.services/-corda-service/index.html).
Inside your service class:
Fetch the default account (AccountService class from the Accounts library has methods to fetch and create accounts; it's inside com.r3.corda.lib.accounts.workflows.services).
If the default account is not found, create it.
When trying to submit a Hadoop MapReduce job programmatically (from a Java application using the dataproc library), the job fails immediately. When submitting that exact same job through the UI, it works fine.
I've tried SSHing onto the Dataproc cluster to confirm the file exists, to check permissions, and changed the jar reference. Nothing has worked yet.
The error I'm getting:
Exception in thread "main" java.lang.ClassNotFoundException: file:///usr/lib/hadoop-mapreduce/hadoop-streaming-2.8.4.jar
at java.lang.Class.forName0(Native Method)
at java.lang.Class.forName(Class.java:264)
at com.google.cloud.hadoop.services.agent.job.shim.HadoopRunClassShim.main(HadoopRunClassShim.java:18)
Job output is complete
When I clone the failed job in the console and look at the REST equivalent this is what I see:
POST /v1/projects/project-id/regions/us-east1/jobs:submit/
{
"projectId": "project-id",
"job": {
"reference": {
"projectId": "project-id",
"jobId": "jobDoesNotWork"
},
"placement": {
"clusterName": "cluster-name",
"clusterUuid": "uuid"
},
"submittedBy": "service-account#project.iam.gserviceaccount.com",
"jobUuid": "uuid",
"hadoopJob": {
"args": [
"-Dmapred.reduce.tasks=20",
"-Dmapred.output.compress=true",
"-Dmapred.compress.map.output=true",
"-Dstream.map.output.field.separator=,",
"-Dmapred.textoutputformat.separator=,",
"-Dmapred.output.compression.codec=org.apache.hadoop.io.compress.GzipCodec",
"-Dmapreduce.input.fileinputformat.split.minsize=268435456",
"-Dmapreduce.input.fileinputformat.split.maxsize=268435456",
"-mapper",
"/bin/cat",
"-reducer",
"/bin/cat",
"-inputformat",
"org.apache.hadoop.mapred.lib.CombineTextInputFormat",
"-outputformat",
"org.apache.hadoop.mapred.TextOutputFormat",
"-input",
"gs://input/path/",
"-output",
"gs://output/path/"
],
"mainJarFileUri": "file:///usr/lib/hadoop-mapreduce/hadoop-streaming-2.8.4.jar"
}
}
}
When I submit the job through the console it works. The REST equivalent of that job:
POST /v1/projects/project-id/regions/us-east1/jobs:submit/
{
"projectId": "project-id",
"job": {
"reference": {
"projectId": "project-id,
"jobId": "jobDoesWork"
},
"placement": {
"clusterName": "cluster-name,
"clusterUuid": ""
},
"submittedBy": "user_email_account#email.com",
"jobUuid": "uuid",
"hadoopJob": {
"args": [
"-Dmapred.reduce.tasks=20",
"-Dmapred.output.compress=true",
"-Dmapred.compress.map.output=true",
"-Dstream.map.output.field.separator=,",
"-Dmapred.textoutputformat.separator=,",
"-Dmapred.output.compression.codec=org.apache.hadoop.io.compress.GzipCodec",
"-Dmapreduce.input.fileinputformat.split.minsize=268435456",
"-Dmapreduce.input.fileinputformat.split.maxsize=268435456",
"-mapper",
"/bin/cat",
"-reducer",
"/bin/cat",
"-inputformat",
"org.apache.hadoop.mapred.lib.CombineTextInputFormat",
"-outputformat",
"org.apache.hadoop.mapred.TextOutputFormat",
"-input",
"gs://input/path/",
"-output",
"gs://output/path/"
],
"mainJarFileUri": "file:///usr/lib/hadoop-mapreduce/hadoop-streaming-2.8.4.jar"
}
}
}
I ssh’ed into the box and confirmed that the file is, in fact, present. The only difference I can really see is the “submittedBy”. One works, one doesn't. I'm guessing this is a permission thing, but I cannot seem to tell where the permissions are being pulled from in each scenario. In both cases, the Dataproc cluster is created with the same service account.
Looking at permissions for that jar on the cluster I see:
-rw-r--r-- 1 root root 133856 Nov 27 20:17 hadoop-streaming-2.8.4.jar
lrwxrwxrwx 1 root root 26 Nov 27 20:17 hadoop-streaming.jar -> hadoop-streaming-2.8.4.jar
I tried changing the mainJarFileUri from pointing explicitly to the versioned jar to the link (since it had open permissions), but didn't really expect it to work. And it didn't.
Does anyone with more Dataproc experience have any idea what's going on here, and how I can resolve it?
One common mistake that's easy to make in code is to call setMainClass when you intended to call setMainJarFileUri or vice-versa. The java.lang.ClassNotFoundException you received indicates that Dataproc was trying to submit that jarfile string as a classname rather than a jarfile, so somewhere along the way Dataproc thought you set main_class. You might want to double-check your code to see if this is the bug you encountered.
The reason using "clone job" in the GUI hides this problem is that the GUI tries to be more user-friendly by offering a single text box for setting either main_class or main_jar_file_uri, and infers whether it is a jarfile by looking at the file extension. So, if you submit a job with a jarfile URI in the main_class field and it fails, then you click clone and submit the new job, the GUI will try to be smart and recognize that the new job actually specified a jarfile name, and thus will correctly set the main_jar_file_uri field in the JSON request instead of main_class.
Is there a way to get an export of the store state / actions programmatically in Production that could be imported back into dev tools?
For example I can setup middleware to capture the current state and send that to something like (Trackjs,Sentry, Rollbar) but that lacks all the previous state and actions.
I would like to capture in the same format as exporting from the Redux Dev Tools.
Sample export from Dev Tools
{"monitorState":{},"actionsById":{"0":{"type":"PERFORM_ACTION","action":{"type":"##INIT"},"timestamp":1471017239656},"1":{"type":"PERFORM_ACTION","action":{"type":"INCREMENT"},"timestamp":1471017242004}},"nextActionId":2,"stagedActionIds":[0,1],"skippedActionIds":[],"committedState":5,"currentStateIndex":1,"computedStates":[{"state":5},{"state":6}]}
This is currently in development but you can now push action history right in the extension see https://github.com/zalmoxisus/remotedev-server/pull/20
Another option is to save the actions to a JSON file as an array and import them back in.
That's possible as of https://github.com/zalmoxisus/redux-devtools-extension/issues/173
logger.js
let actions = []
export function logActions (stateSanitizer) {
return store => next => action => {
actions.push(action)
return next(action)
}
}
These actions can be saved to a file or database and can be imported back into the dev tools.
Sample actions
[{
"type": "INCREMENT"
}, {
"type": "DECREMENT"
}, {
"type": "DECREMENT"
}, {
"type": "DECREMENT"
}, {
"type": "DECREMENT"
}]
I created this repo which demos this in action https://github.com/timarney/redux-trackjs-logger it uses middleware to log the actions when an error happens.
I maintain a Redux middleware called Raven for Redux which attaches Redux data to Sentry error reports. Currently it adds the following context to each error report:
The complete state object.
The complete last action object.
The type of all actions that lead up to the current state. These are added as "breadcrumbs".
The Sentry blog has a writeup describing it in more detail: https://blog.sentry.io/2016/08/24/redux-middleware-error-logging.html
You can find the middleware as an NPM package here: https://github.com/captbaritone/raven-for-redux