How do I enable DynamoDB Local Logging? - amazon-dynamodb

More than often DynamoDB Local does not show descriptive error messages, in order to see internal errors you need to enable logging.
What are the steps to enable DynamoDB Local logging on the standard output?

Go to the directory that has the DynamoDBLocal.jar
Create a file called log4j2.xml with these contents:
<?xml version="1.0" encoding="UTF-8"?>
<Configuration status="WARN">
<Appenders>
<Console name="Console" target="SYSTEM_OUT">
<PatternLayout pattern="%d{HH:mm:ss.SSS} [%t] %-5level %logger{36} - %msg%n"/>
</Console>
</Appenders>
<Loggers>
<Logger name="com.amazonaws.services.dynamodbv2.local" level="DEBUG">
<AppenderRef ref="Console"/>
</Logger>
<Logger name="com.amazonaws.services.dynamodbv2.local.shared.access.sqlite.SQLiteDBAccess" level="INFO">
<AppenderRef ref="Console"/>
</Logger>
<Root level="WARN">
<AppenderRef ref="Console"/>
</Root>
</Loggers>
</Configuration>
Remove the existing log4j2.xml from the jar
zip -d DynamoDBLocal.jar log4j2.xml
Add the created log4j2.xml to the jar
zip -u DynamoDBLocal.jar log4j2.xml
or simply edit the log4j2.xml in the DynamoDBLocal.jar using 7-Zip etc. and overwrite it with the xml above and skip the steps 2-4.

Change to the directory with DynamoDBLocal.jar
Create a new file called log4j.properties with the contents:
log4j.rootLogger=DEBUG, stdout
log4j.appender.stdout=org.apache.log4j.ConsoleAppender
log4j.appender.stdout.layout=org.apache.log4j.PatternLayout
log4j.appender.stdout.layout.ConversionPattern=LOG%d %p [%c] - %m%n
Remove the existing log4j.properties files from the jar (there might be two)
zip -d DynamoDBLocal.jar log4j.properties
zip -d DynamoDBLocal.jar log4j.properties
Add the new properties file to the jar
zip -u DynamoDBLocal.jar log4j.properties
Source: https://gist.github.com/mdaley/aaf9b62d90f6817eb72a

Related

How to stop node from logging node.conf during startup

How do I stop the node from printing the node.conf in the logs?
I understand we can change the logging level since the node.conf is printed at INFO level, but I want to avoid that as much as possible since I still want some other information that is at INFO level to be printed out.
The contents of node.conf are printed at INFO level by the net.corda.node.services.config.ConfigHelper class. To prevent the contents of node.conf from being printed to the logs, you'd have to specify a custom logging configuration so that for the net.corda.node.services.config.ConfigHelper class, only messages at WARN or above should be printed to the logs.
The process for providing a custom Log4J2 logging configuration file for your node is documented here. You need to:
Create the custom logging file (e.g. test.xml)
Point your node to the custom logging file when starting the node (e.g. java -Dlog4j.configurationFile=test.xml -jar corda.jar)
Here's an example test.xml that prevents the contents of node.conf being printed to the logs:
<?xml version="1.0" encoding="UTF-8"?>
<Configuration status="INFO">
<Properties>
<Property name="log-path">logs</Property>
<Property name="log-name">node-${hostName}</Property>
<Property name="archive">${log-path}/archive</Property>
</Properties>
<Appenders>
<Console name="Console-Appender" target="SYSTEM_OUT">
<PatternLayout pattern="%highlight{%level{length=1} %d{HH:mm:ss} %T %c{1}.%M - %msg%n}{INFO=white,WARN=red,FATAL=bright red blink}"/>
</Console>
<RollingFile name="RollingFile-Appender"
fileName="${log-path}/${log-name}.log"
filePattern="${archive}/${log-name}.%d{yyyy-MM-dd}-%i.log.gz">
<PatternLayout pattern="[%-5level] %d{ISO8601}{GMT+0} [%t] %c{1} - %msg%n"/>
<Policies>
<TimeBasedTriggeringPolicy/>
<SizeBasedTriggeringPolicy size="10MB"/>
</Policies>
<DefaultRolloverStrategy min="1" max="10">
<Delete basePath="${archive}" maxDepth="1">
<IfFileName glob="${log-name}*.log.gz"/>
<IfLastModified age="60d">
<IfAny>
<IfAccumulatedFileSize exceeds="10 GB"/>
</IfAny>
</IfLastModified>
</Delete>
</DefaultRolloverStrategy>
</RollingFile>
</Appenders>
<Loggers>
<Root level="error">
<AppenderRef ref="Console-Appender"/>
<AppenderRef ref="RollingFile-Appender"/>
</Root>
<Logger name="net.corda" level="error" additivity="false">
<AppenderRef ref="Console-Appender"/>
<AppenderRef ref="RollingFile-Appender"/>
</Logger>
<Logger name="net.corda.node.services.config.ConfigHelper" level="warn" additivity="false">
<AppenderRef ref="RollingFile-Appender"/>
</Logger>
</Loggers>
</Configuration>
Note the final Logger block. We specify that any messages from net.corda.node.services.config.ConfigHelper (e.g. the contents of node.conf) should only be printed if they are at level WARN or above.
#Joel is there any way to point all the nodes to one custom logging file? I.e, when executing runnodes.jar, could you pass a single logfile as a paramater, i.e:
java -jar -Dlog4j.configurationFile=/Users/username/Desktop/Prototype/config/dev/log4j2.xml runnodes.jar
This doesn't seem to work... so curious.

ArangoDB and gremlin: How to reduce LOGGING

I have installed gremlin (v. 2.6.0) and ArangoDB (v. 2.8.11) and when I run any request through the gremlin.sh shell I get all the debug messages like
11:17:39.713 [main] DEBUG com.arangodb.http.HttpManager - [REQ]http-GET: url=http://localhost:8529/_api/gharial/myDB/edge/E/12712, headers={}
11:17:39.716 [main] DEBUG com.arangodb.http.HttpManager - [RES]http-GET: statusCode=200
11:17:39.716 [main] DEBUG com.arangodb.http.HttpManager - [RES]http-GET: text={"error":false,"code":200,"edge":{"_id":"E/12712"," . . .
I see those are DEBUG messages, so I want to suppress in order not to be flooded by those and only get important messages, like errors or warnings.
To reduce logging one has to change the log level on both server and client side.
Complete steps to fix this:
create config PATH_TO/confs/ directory with inside:
a. file confs/arangod.conf
b. file confs/logback.xml
set env. variable CLASSPATH:
export CLASSPATH=PATH_TO/confs
you can start arangod as follows:
arangod --daemon --pid-file /etc/arangodb/arangodb.pid -c PATH_TO/confs/arangod.conf
The content of confs/arangod.conf should contain:
[log]
## info, warning, or error
level = info
file = PATH_TO/logs/arangodb.log
The content of the confs/logback.xml
<configuration>
<!-- Arango log4j conf -->
<appender name="CONSOLE"
class="ch.qos.logback.core.ConsoleAppender">
<!-- deny all events with a level below INFO, that is TRACE and DEBUG -->
<filter class="ch.qos.logback.classic.filter.ThresholdFilter">
<level>INFO</level>
</filter>
<encoder>
<pattern>
%-4relative [%thread] %-5level %logger{30} - %msg%n
</pattern>
</encoder>
</appender>
<root level="DEBUG">
<appender-ref ref="CONSOLE" />
</root>
</configuration>

Oozie shell action issue while creating directories

I am unable to add/delete any files or directories on HDFS from a shell script which I am executing from Oozie workflow.
The username is "scitest" and the hdfs path I am trying to edit/add/delete is
/user/scitest/.
In the shell script I am trying to delete a folder named test123456 from the path /user/scitest/.
---------------Error from oozie log------------------
429737-oozie-oozi-W#shell-node] Launcher ERROR, reason: Main class [org.apache.oozie.action.hadoop.ShellMain], exit code [1]
2016-12-27 05:04:25,553 INFO ActionEndXCommand:520 - SERVER[vscihadoopvm2.manhdev.com] USER[scitest] GROUP[-] TOKEN[] APP[shell.workflow] JOB[0000041-161208225429737-oozie-oozi-W] ACTION[0000041-161208225429737-oozie-oozi-W#shell-node] ERROR is considered as FAILED for SLA
---------shell-script(sample.sh) content----------
#!/bin/bash
echo "`date` hi" > output.log
hadoop fs -mkdir test123456
-------job.properties---------
nameNode=hdfs://vscihadoopvm1.manhdev.com:8020
jobTracker=vscihadoopvm2.manhdev.com:8050
master=yarn-cluster
#user.name=yarn
queueName=default
examplesRoot=oozietest
oozie.use.system.libpath=true
oozie.wf.application.path=${nameNode}/user/${user.name}/${examplesRoot}
---workflow.xml---
<?xml version="1.0" encoding="UTF-8"?>
<workflow-app xmlns="uri:oozie:workflow:0.5" name="shell.workflow">
<start to="shell-node"/>
<action name="shell-node">
<shell xmlns="uri:oozie:shell-action:0.3">
<job-tracker>${jobTracker}</job-tracker>
<name-node>${nameNode}</name-node>
<configuration>
<property>
<name>mapred.job.queue.name</name>
<value>${queueName}</value>
</property>
</configuration>
<exec>sample.sh</exec>
<file>sample.sh#sample.sh</file>
</shell>
<ok to="end"/>
<error to="kill"/>
</action>
<kill name="kill">
<message>Error in Shell.Please refer the Oozie Logs</message>
</kill>
<end name="end"/>
</workflow-app>
#Abhiroy, Maybe its better if you can check with which user your oozie action is getting executed. you can simply place 'id' without quotes in your sample shell script and run the workflow. Then you can trace the oozie job logs to see the container executer for your sample script that oozie is using. Then we can start tracing if you have any permission issues.

Amazon AWS Spring MVC log4j2 logging

Not sure if it is a right place, but will try here.
I have a Spring MVC app deployed on Amazon Elastic Beantalk, using log4j2.
Locally, the logs work well.
this is the config:
<?xml version="1.0" encoding="UTF-8"?>
<Configuration>
<Appenders>
<Console name="STDOUT" target="SYSTEM_OUT">
<PatternLayout pattern="%d %-5p [%t] %C{2} (%F:%L) - %m%n"/>
</Console>
<RollingFile
name="MyFile"
fileName="${sys:catalina.home}/logs/log4j2.log"
filePattern="${sys:catalina.home}/logs/log4j2-%d{MM-dd-yyyy}-%i.log"
immediateFlush="true">
<PatternLayout pattern="%d{yyy-MM-dd HH:mm:ss.SSS} [%t] %-5level %logger{36} - %msg%n"/>
<Policies>
<SizeBasedTriggeringPolicy size="5 MB"/>
</Policies>
<DefaultRolloverStrategy max="2000"/>
</RollingFile>
</Appenders>
<Loggers>
<Logger name="org.apache.log4j.xml" level="debug"/>
<Root level="info">
<AppenderRef ref="STDOUT"/>
<AppenderRef ref="MyFile"/>
</Root>
</Loggers>
</Configuration>
But on EC2 amazon instance, where the ElasticBeanstalk app is running, no appenders working. even SYSTEM_OUT doesn't logged into catalina.out. Absolutely no clue why can that happens :(
Btw before i have experience using ElasticBeanstalk, and usually hadprobs with File appender, but at least log is appended to catalina.out .
Any idea what should i check?

BizTalk ListApp command line

I used the following code in c# to get policies\rules from deployed application in BizTalk server.
BTSTask.exe ListApp -ApplicationName:"EAISolution" -ResourceSpec:"c:\EAISolution.PolicyInf
o.xml" /Server:VHYDTRBELSUP-02 /Database:BizTalkMgmtDb
From above command I got the output as below
<?xml version="1.0" encoding="utf-16" ?>
<ResourceSpec xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:xsd="http://www.w3.org/2001/XMLSchema" ApplicationName="EAISolution" xmlns="http://schemas.microsoft.com/BizTalk/ApplicationDeployment/ResourceSpec/2004/12">
<Resources>
<Resource Type="System.BizTalk:BizTalkAssembly" Luid="EAIOrchestration, Version=1.0.0.0, Culture=neutral, PublicKeyToken=97e0f507fd7fd10d" />
<Resource Type="System.BizTalk:BizTalkAssembly" Luid="EAIServices, Version=1.0.0.0, Culture=neutral, PublicKeyToken=97e0f507fd7fd10d" />
<Resource Type="System.BizTalk:BizTalkAssembly" Luid="FFSchemasTest, Version=1.0.0.0, Culture=neutral, PublicKeyToken=97e0f507fd7fd10d" />
<Resource Type="System.BizTalk:Rules" Luid="RULE/ProcessPurchaseOrder/1.0" />
<Resource Type="System.BizTalk:BizTalkBinding" Luid="Application/EAISolution" />
</Resources>
</ResourceSpec>
and from BizTalk server I got the below output using policy export in BizTalk server administration
<?xml version="1.0" encoding="utf-8" ?>
<brl xmlns="http://schemas.microsoft.com/businessruleslanguage/2002">
<ruleset name="ProcessPurchaseOrder">
<version major="1" minor="0" description="" modifiedby="username" date="2013-05- 27T12:04:55.6121122+05:30" />
<configuration />
<bindings>
<xmldocument ref="xml_31" doctype="RuleTest.PO" instances="16" selectivity="1" instance="0">
<selector>/*[local-name()='PurchaseOrder' and namespace-uri() ='http://EAISolution.PurchaseOrder']/*[local-name()='Item' and namespace-uri()='']</selector>
<selectoralias>/PurchaseOrder/Item</selectoralias>
<schema>....\PO.xsd</schema>
</xmldocument>
<xmldocument ref="xml_32" doctype="RuleTest.PO" instances="16" selectivity="1" instance="0">
<selector>/*[local-name()='PurchaseOrder' and namespace-uri()='http://EAISolution.PurchaseOrder']
</selector>
<selectoralias>/PurchaseOrder</selectoralias>
<schema>....\PO.xsd</schema>
</xmldocument>
</bindings>
<rule name="ApprovalRule" priority="0" active="true">
<if>
<compare operator="less than or equal to">
<vocabularylink uri="3f0e9bcc-6212-4e6a-853c-e517f157a626" element="d4eb2deb-06d3-42c4-af49-ceb21331b1cc" />
<lhs>
<function>
<xmldocumentmember xmldocumentref="xml_31" type="int" sideeffects="false">
<field>*[local-name()='Quantity' and namespace-uri()='']</field>
<fieldalias>Quantity</fieldalias>
</xmldocumentmember>
</function>
</lhs>
<rhs>
<constant>
<int>500</int>
</constant>
</rhs>
</compare>
</if>
<then>
<function>
<xmldocumentmember xmldocumentref="xml_32" type="string" sideeffects="true">
<field>*[local-name()='Status' and namespace-uri()='']</field>
<fieldalias>Status</fieldalias>
<argument>
<constant>
<string>Approved</string>
</constant>
</argument>
</xmldocumentmember>
</function>
</then>
</rule>
</ruleset>
</brl>
So please let me know how to get the output of second using command line.
BTSTask will only export the policy as part of an MSI (see below).
You could then extract the MSI (see How to extract msu/msp/msi fileds from the command line) to get the policy file.
From How to Import a Policy
BTSTask does not provide a specific command for importing (or exporting) policies; however you can use the ExportApp command of BTSTask to selectively export only the policies in an application that you want, including no other application artifacts. Then you can use the ImportApp command to import the .msi file into an application in a different BizTalk group. This is the approach described in this topic. When you do this, the policy is automatically imported and published in the BizTalk group and added to the specified application.
The below steps will get export the policy, but as part of an MSI.
From How to Export a Policy
Use the BTSTask ListApp command with the /ResourceSpec option to generate an XML file that lists the artifacts in the BizTalk application from which you want to export a policy, as described in ListApp Command.
Edit the XML file generated in the previous step, deleting all of the artifacts except for the policy or policies that you want to export.
Use the BTSTask ExportApp command, and specify the modified XML file for the /ResourceSpec parameter. For more information, see ExportApp Command.
BTSTask exports the specified policies and all of their associated vocabularies into an application .msi file.

Resources