how to cluster wso2am-analytics-2.0.0 - wso2-api-manager

Question

We are building wso2am in cluster mode.
Is there any document about building wso2am-analytics cluster ?
I have tried to use wso2das, reference as below.
https://docs.wso2.com/display/DAS310/Working+with+Product+Specific+Analytics+Profiles
But get the error as below
TID: [-1234] [] [2016-12-09 15:00:00,101] ERROR {org.wso2.carbon.analytics.spark.core.AnalyticsTask} - Error while executing the scheduled task for the script: APIM_LATENCY_BREAKDOWN_STATS {org.wso2.carbon.analytics.spark.core.AnalyticsTask}
org.wso2.carbon.analytics.spark.core.exception.AnalyticsExecutionException: Exception in executing query CREATE TEMPORARY TABLE APIMGT_PERHOUR_EXECUTION_TIME USING CarbonAnalytics OPTIONS(tableName "ORG_WSO2_APIMGT_STATISTICS_PERHOUREXECUTIONTIMES", schema " year INT -i, month INT -i, day INT -i, hour INT -i, context STRING, api_version STRING, api STRING, tenantDomain STRING, apiPublisher STRING, apiResponseTime DOUBLE, securityLatency DOUBLE, throttlingLatency DOUBLE, requestMediationLatency DOUBLE, responseMediationLatency DOUBLE, backendLatency DOUBLE, otherLatency DOUBLE, firstEventTime LONG, _timestamp LONG -i", primaryKeys "year, month, day, hour, context, api_version, tenantDomain, apiPublisher", incrementalProcessing "APIMGT_PERHOUR_EXECUTION_TIME, DAY", mergeSchema "false")
at org.wso2.carbon.analytics.spark.core.internal.SparkAnalyticsExecutor.executeQueryLocal(SparkAnalyticsExecutor.java:764)
at org.wso2.carbon.analytics.spark.core.internal.SparkAnalyticsExecutor.executeQuery(SparkAnalyticsExecutor.java:721)
at org.wso2.carbon.analytics.spark.core.CarbonAnalyticsProcessorService.executeQuery(CarbonAnalyticsProcessorService.java:201)
at org.wso2.carbon.analytics.spark.core.CarbonAnalyticsProcessorService.executeScript(CarbonAnalyticsProcessorService.java:151)
at org.wso2.carbon.analytics.spark.core.AnalyticsTask.execute(AnalyticsTask.java:60)
at org.wso2.carbon.ntask.core.impl.TaskQuartzJobAdapter.execute(TaskQuartzJobAdapter.java:67)
at org.quartz.core.JobRunShell.run(JobRunShell.java:213)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.RuntimeException: Unknown options : incrementalprocessing
at org.wso2.carbon.analytics.spark.core.sources.AnalyticsRelationProvider.checkParameters(AnalyticsRelationProvider.java:123)
at org.wso2.carbon.analytics.spark.core.sources.AnalyticsRelationProvider.setParameters(AnalyticsRelationProvider.java:113)
at org.wso2.carbon.analytics.spark.core.sources.AnalyticsRelationProvider.createRelation(AnalyticsRelationProvider.java:75)
at org.wso2.carbon.analytics.spark.core.sources.AnalyticsRelationProvider.createRelation(AnalyticsRelationProvider.java:45)
at org.apache.spark.sql.execution.datasources.ResolvedDataSource$.apply(ResolvedDataSource.scala:158)
at org.apache.spark.sql.execution.datasources.CreateTempTableUsing.run(ddl.scala:92)
at org.apache.spark.sql.execution.ExecutedCommand.sideEffectResult$lzycompute(commands.scala:58)
at org.apache.spark.sql.execution.ExecutedCommand.sideEffectResult(commands.scala:56)
at org.apache.spark.sql.execution.ExecutedCommand.doExecute(commands.scala:70)
at org.apache.spark.sql.execution.SparkPlan$$anonfun$execute$5.apply(SparkPlan.scala:132)
at org.apache.spark.sql.execution.SparkPlan$$anonfun$execute$5.apply(SparkPlan.scala:130)
at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:150)
at org.apache.spark.sql.execution.SparkPlan.execute(SparkPlan.scala:130)
at org.apache.spark.sql.execution.QueryExecution.toRdd$lzycompute(QueryExecution.scala:55)
at org.apache.spark.sql.execution.QueryExecution.toRdd(QueryExecution.scala:55)
at org.apache.spark.sql.DataFrame.<init>(DataFrame.scala:145)
at org.apache.spark.sql.DataFrame.<init>(DataFrame.scala:130)
at org.apache.spark.sql.DataFrame$.apply(DataFrame.scala:52)
at org.apache.spark.sql.SQLContext.sql(SQLContext.scala:817)
at org.wso2.carbon.analytics.spark.core.internal.SparkAnalyticsExecutor.executeQueryLocal(SparkAnalyticsExecutor.java:760)
... 11 more
======================================================
Any suggestion will be appreciated !!

Answer 1

Just to be clear of what you are asking. Are you using DAS or the wso2am-analytics that's shipped along the wso2am-2.0.0?
If you are using wso2am-2.0.0 along with the wso2das-3.1.0 then there is an issue with the spark analytics scripts that comes with the DAS.
This is due to the use of incremental processing. IncrementalProcessing should be changed to IncrementalParams.
You can see this has been fixed by wso2 here, but has not been released yet.
You can update the scripts from the DAS carbon console under main/Batch Analytics/scripts

Answer 2

Tks for ur response!! We need keeping summarized data in statdb,so which table should be purged ? I have found a link as below http://www.rukspot.com/Publishing_APIM_1100_Runtime_Statistics_to_DAS.html
It mentioned that just purge below table
ORG_WSO2_APIMGT_STATISTICS_DESTINATION
ORG_WSO2_APIMGT_STATISTICS_FAULT
ORG_WSO2_APIMGT_STATISTICS_REQUEST
ORG_WSO2_APIMGT_STATISTICS_RESPONSE
ORG_WSO2_APIMGT_STATISTICS_WORKFLOW
ORG_WSO2_APIMGT_STATISTICS_THROTTLE