I wish to pass a file through to my instance, but there is no
file option? Even though documentation says that this should be available?
stack#openstack:/tmp$ nova boot --config-drive true --flavor 8755fc00-4c24-418d-a505-592c809108d9 --image 2ca99129-4b09-4a2a-a43d-b22c76dc4efc --file license=FGVM080000109836.lic --user-data cloudinitfile test
usage: nova [--version] [--debug] [--os-cache] [--timings]
[--os-region-name <region-name>] [--service-type <service-type>]
[--service-name <service-name>]
[--os-endpoint-type <endpoint-type>]
[--os-compute-api-version <compute-api-ver>]
[--endpoint-override <bypass-url>] [--profile HMAC_KEY]
[--insecure] [--os-cacert <ca-certificate>]
[--os-cert <certificate>] [--os-key <key>] [--timeout <seconds>]
[--os-auth-type <name>] [--os-auth-url OS_AUTH_URL]
[--os-system-scope OS_SYSTEM_SCOPE] [--os-domain-id OS_DOMAIN_ID]
[--os-domain-name OS_DOMAIN_NAME] [--os-project-id OS_PROJECT_ID]
[--os-project-name OS_PROJECT_NAME]
[--os-project-domain-id OS_PROJECT_DOMAIN_ID]
[--os-project-domain-name OS_PROJECT_DOMAIN_NAME]
[--os-trust-id OS_TRUST_ID]
[--os-default-domain-id OS_DEFAULT_DOMAIN_ID]
[--os-default-domain-name OS_DEFAULT_DOMAIN_NAME]
[--os-user-id OS_USER_ID] [--os-username OS_USERNAME]
[--os-user-domain-id OS_USER_DOMAIN_ID]
[--os-user-domain-name OS_USER_DOMAIN_NAME]
[--os-password OS_PASSWORD]
<subcommand> ...
error: unrecognized arguments: --file test
Try 'nova help ' for more information.
Please check with alternative command using OpenStack CLI:
***openstack server create --image 2ca99129-4b09-4a2a-a43d-b22c76dc4efc --flavor 8755fc00-4c24-418d-a505-592c809108d9 --config-drive True --file license=FGVM080000109836.lic --user-data cloudinitfile test***
Note:
1. Please also check once the injected files quota.
Yes, --file option is there by which you can pass a file to your instance. You can check available optional parameter using "nova help command".
e.g:
nova help boot
usage: nova boot [--flavor <flavor>] [--image <image>]
[--image-with <key=value>] [--boot-volume <volume_id>]
[--snapshot <snapshot_id>] [--min-count <number>]
[--max-count <number>] [--meta <key=value>]
**[--file <dst-path=src-path>]** [--key-name <key-name>]
[--user-data <user-data>]
[--availability-zone <availability-zone>]
[--security-groups <security-groups>]
[--block-device-mapping <dev-name=mapping>]
[--block-device key1=value1[,key2=value2...]]
[--swap <swap_size>]
[--ephemeral size=<size>[,format=<format>]]
[--hint <key=value>]
[--nic <net-id=net-uuid,v4-fixed-ip=ip-addr,v6-fixed-ip=ip-addr,port-id=port-uuid>]
[--config-drive <value>] [--poll] [--admin-pass <value>]
[--access-ip-v4 <value>] [--access-ip-v6 <value>]
<name>
Boot a new server.
Related
ubuntu#admin[enter image description here][1]:~/orocommerce-application$ symfony console oro:install -vvv --sample-data=y --application-url=https://127.0.0.1:8000 --user-name=admin --user-email=admin#example.com --user-firstname=John --user-lastname=Doe --user-password=admin --organization-name=Oro --timeout=0 --symlink --env=prod -n
In ArrayNode.php line 322:
[Symfony\Component\Config\Definition\Exception\InvalidConfigurationException]
Unrecognized option "amqp" under "oro_message_queue.transport". Available option is "dbal".
Exception trace:
at /home/ubuntu/orocommerce-application/vendor/symfony/config/Definition/ArrayNode.php:322
Symfony\Component\Config\Definition\ArrayNode->normalizeValue() at /home/ubuntu/orocommerce-application/vendor/symfony/config/Definition/BaseNode.php:410
Symfony\Component\Config\Definition\BaseNode->normalize() at /home/ubuntu/orocommerce-application/vendor/symfony/config/Definition/ArrayNode.php:287
Symfony\Component\Config\Definition\ArrayNode->normalizeValue() at /home/ubuntu/orocommerce-application/vendor/symfony/config/Definition/BaseNode.php:410
Symfony\Component\Config\Definition\BaseNode->normalize() at /home/ubuntu/orocommerce-application/vendor/symfony/config/Definition/Processor.php:32
Symfony\Component\Config\Definition\Processor->process() at /home/ubuntu/orocommerce-application/vendor/symfony/config/Definition/Processor.php:46
Symfony\Component\Config\Definition\Processor->processConfiguration() at /home/ubuntu/orocommerce-application/vendor/symfony/dependency-injection/Extension/Extension.php:111
Symfony\Component\DependencyInjection\Extension\Extension->processConfiguration() at /home/ubuntu/orocommerce-application/vendor/oro/platform/src/Oro/Bundle/MessageQueueBundle/DependencyInjection/OroMessageQueueExtension.php:38
Oro\Bundle\MessageQueueBundle\DependencyInjection\OroMessageQueueExtension->load() at /home/ubuntu/orocommerce-application/vendor/symfony/dependency-injection/Compiler/MergeExtensionConfigurationPass.php:76
Symfony\Component\DependencyInjection\Compiler\MergeExtensionConfigurationPass->process() at /home/ubuntu/orocommerce-application/vendor/symfony/http-kernel/DependencyInjection/MergeExtensionConfigurationPass.php:42
Symfony\Component\HttpKernel\DependencyInjection\MergeExtensionConfigurationPass->process() at /home/ubuntu/orocommerce-application/vendor/symfony/dependency-injection/Compiler/Compiler.php:82
Symfony\Component\DependencyInjection\Compiler\Compiler->compile() at /home/ubuntu/orocommerce-application/vendor/symfony/dependency-injection/ContainerBuilder.php:757
Symfony\Component\DependencyInjection\ContainerBuilder->compile() at /home/ubuntu/orocommerce-application/vendor/oro/platform/src/Oro/Bundle/DistributionBundle/OroKernel.php:407
Oro\Bundle\DistributionBundle\OroKernel->initializeContainer() at /home/ubuntu/orocommerce-application/vendor/symfony/http-kernel/Kernel.php:789
Symfony\Component\HttpKernel\Kernel->preBoot() at /home/ubuntu/orocommerce-application/vendor/symfony/http-kernel/Kernel.php:128
Symfony\Component\HttpKernel\Kernel->boot() at /home/ubuntu/orocommerce-application/vendor/oro/platform/src/Oro/Bundle/DistributionBundle/OroKernel.php:231
Oro\Bundle\DistributionBundle\OroKernel->boot() at /home/ubuntu/orocommerce-application/vendor/symfony/framework-bundle/Console/Application.php:168
Symfony\Bundle\FrameworkBundle\Console\Application->registerCommands() at /home/ubuntu/orocommerce-application/vendor/symfony/framework-bundle/Console/Application.php:74
Symfony\Bundle\FrameworkBundle\Console\Application->doRun() at /home/ubuntu/orocommerce-application/vendor/symfony/console/Application.php:171
Symfony\Component\Console\Application->run() at /home/ubuntu/orocommerce-application/bin/console:33
It seems you are trying to set up the amqp transport in config/parameters.yml, which is not available in the community edition of OroCommerce.
You have reset the message queue configuration to the default value that you can find in the config/parameters.yml.dist and clear the cache to apply changes
Here is the document I refer to
1.sample_server.yaml
type: os.nova.server
version: 1.0
properties:
name: cirros_server
flavor: m1.small
image: b86fb462-c5c2-4a08-9fe4-c9f86d05763d
networks:
- network: external-net
2.Execute the following command line
# openstack cluster create --profile pserver --desired-capacity 2 mycluster
# openstack cluster receiver create --type webhook --cluster mycluster --action CLUSTER_SCALE_OUT --params count=2 r_01
# export ALRM_URL01='http://vip:8777/v1/webhooks/aac3433a-40de-4d7d-830c-e0035f2a4d13/trigger?V=1&count=2'
# aodh alarm create --type gnocchi_resources_threshold --aggregation-method mean --name cpu-high --metric cpu_util --threshold 70 --comparison-operator gt --granularity 300 --evaluation-periods 1 --alarm-action $ALRM_URL01 --repeat-actions False --query metadata.user_metadata.cluster_id=$MYCLUSTER_ID --resource-type instance --resource-id f7e0e8a6-51a3-422d-b631-7ddaf65b3dfb
3.log into each cluster nodes and run some CPU burning workloads there to drive the CPU utilization high
I added log output to /usr/lib/python2.7/site-packages/aodh/notifier/rest.py when trigger the alert request
class RestAlarmNotifier(notifier.AlarmNotifier):
def notify(self, action, alarm_id, alarm_name, severity, previous,
current, reason, reason_data, headers=None):
body = {'alarm_name': alarm_name, 'alarm_id': alarm_id,
'severity': severity, 'previous': previous,
'current': current, 'reason': reason,
'reason_data': reason_data}
headers['content-type'] = 'application/json'
kwargs = {'data': json.dumps(body),
'headers': headers}
max_retries = self.conf.rest_notifier_max_retries
session = requests.Session()
LOG.info('#########################')
LOG.info(session)
LOG.info(kwargs)
LOG.info(action.geturl())
LOG.info('#########################')
session.mount(action.geturl(),
requests.adapters.HTTPAdapter(max_retries=max_retries))
resp = session.post(action.geturl(), **kwargs)
LOG.info('$$$$$$$$$$$$$$$$$$$$$$$')
LOG.info(resp.content)
LOG.info('$$$$$$$$$$$$$$$$$$$$$$$')
Some error messages are output in the /var/log/aodh/notifier.log log, as follows:
enter image description here
The reason is the error caused by adding the body request parameter, the direct post request can be successful, for example, using curl request without the body parameter
curl -g -i -X POST 'http://vip:8777/v1/webhooks/34e91386-7176-4b30-bc17-5c3503712696/trigger?V=1'
Aodh related version packages are as follows：
python2-aodhclient-1.1.1-1.el7.noarch
openstack-aodh-api-7.0.0-1.el7.noarch
openstack-aodh-common-7.0.0-1.el7.noarch
openstack-aodh-listener-7.0.0-1.el7.noarch
python-aodh-7.0.0-1.el7.noarch
openstack-aodh-notifier-7.0.0-1.el7.noarch
openstack-aodh-evaluator-7.0.0-1.el7.noarch
openstack-aodh-expirer-7.0.0-1.el7.noarch
Can anyone point me in the right direction? Thanks.
The problem has been solved. Here is the document I refer to
Modify aodh rest.py(aodh/notifier/rest.py)
https://github.com/openstack/aodh/blob/master/aodh/notifier/rest.py#L79
Under the headers['content-type'] , add this line: headers['openstack-api-version'] = 'clustering 1.10'
Restart aodh service
I can't send data to strackdriver. It's simple example.
I'm using syntax what I found in example for collectd.
Stackdriver documentation don't have info about how to send custom data from exec module.
What am I doing wrong?
This is collectd.conf
LoadPlugin exec
<Plugin "exec">
Exec "apache" "/etc/stackdriver/collectd.d/dir.sh"
</Plugin>
This is dir.sh
#!/bin/bash
FOLDER="/var/www/"
while true; do
DU=$(du -shm ${FOLDER} | awk '{print $1}')
echo "PUTVAL \"projects/project_name/custom.googleapis.com/folder/completesolar\" interval=60 N:${DU}"
sleep 60
done
Script output
$/etc/stackdriver/collectd.d/dir.sh
PUTVAL "projects/project_name/custom.googleapis.com/folder/completesolar" interval:60 N:1155
I enabled debug mode and found this error:
[2018-09-21 00:45:55] utils_cmd_putval: handle_putval (fh = 0x3e71d8f040, buffer = PUTVAL "projects/project_name/custom.googleapis.com/folder/completesolar" interval=60 N:1155);
[2018-09-21 00:45:55] No such dataset registered: custom.googleapis.com/folder/completesolar
I created this metric and found it in stackdriver console:
http://joxi.ru/a2XlPGvi1VzJL2
This is json for creating my metric:
{
"name": "projects/project_name/metricDescriptors/custom.googleapis.com/folder/completesolar",
"metricKind": "GAUGE",
"valueType": "DOUBLE",
"unit": "By",
"description": "Folder bytes used",
"displayName": "Folder usage",
"type": "custom.googleapis.com/folder/completesolar",
"metadata": {
"launchStage": "GA",
"samplePeriod": "60s",
"ingestDelay": "0s"
}
}
This should be work.
/etc/stackdriver/collectd.d/dir.conf
LoadPlugin exec
<Plugin "exec">
Exec "apache" "/path/to/dir.sh"
</Plugin>
LoadPlugin target_set
PreCacheChain "PreCache"
<Chain "PreCache">
<Rule "dir">
<Match regex>
Plugin "^exec$"
PluginInstance "^dir$"
</Match>
<Target "set">
MetaData "stackdriver_metric_type" "custom.googleapis.com/folder/completesolar"
# MetaData "stackdriver_metric_type" "custom.googleapis.com/%{plugin_instance}/%{type_instance}"
# MetaData "label:name" "dir usage"
</Target>
</Rule>
</Chain>
/path/to/dir.sh
#!/bin/bash
INTERVAL="$COLLECTD_INTERVAL"
HOSTNAME="$COLLECTD_HOSTNAME"
FOLDER="/var/www/"
while true; do
DU=$(du -sm ${FOLDER} | awk '{print $1}')
# https://collectd.org/documentation/manpages/collectd-exec.5.shtml
# PUTVAL Identifier [OptionList] Valuelist
# Identifier: <host>/<plugin>-<plugin_instance>/<type>-<type_instance>
# Type: See /opt/stackdriver/collectd/share/collectd/types.db
echo "PUTVAL ${HOSTNAME}/exec-dir/gauge-usage/ interval=${INTERVAL} N:${DU}"
sleep "${INTERVAL}"
done
See below if you want to (pre) create or delete a metric.
https://cloud.google.com/monitoring/custom-metrics/creating-metrics
Looks like you are following this guide to ingest exec metrics. I think the rewrite rules did not work so the metric ends up being ingested though a regular agent metrics path, which rejects unrecognized metrics. You might need to modify the metadata for your metric by using filter chain . I would suggest you to revisit the troubleshooting guide.
Custom Function what I made did not work in presto on EMR.
I want to create a simple UDAF that just return 42.
First, my custom function what I wrote a simple functions, but did not work in presto.
A error is following in presto-cli:
presto> select answer_to_life('the universe');
Query 20180324_120433_00000_7n6s6 failed: answer_to_life(varchar):bigint not found
com.facebook.presto.spi.PrestoException: answer_to_life(varchar):bigint not found
at com.facebook.presto.metadata.FunctionRegistry.doGetSpecializedFunctionKey(FunctionRegistry.java:972)
at com.google.common.cache.CacheLoader$FunctionToCacheLoader.load(CacheLoader.java:146)
at com.google.common.cache.LocalCache$LoadingValueReference.loadFuture(LocalCache.java:3716)
at com.google.common.cache.LocalCache$Segment.loadSync(LocalCache.java:2424)
at com.google.common.cache.LocalCache$Segment.lockedGetOrLoad(LocalCache.java:2298)
at com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2211)
at com.google.common.cache.LocalCache.get(LocalCache.java:4154)
at com.google.common.cache.LocalCache.getOrLoad(LocalCache.java:4158)
at com.google.common.cache.LocalCache$LocalLoadingCache.get(LocalCache.java:5147)
at com.google.common.cache.LocalCache$LocalLoadingCache.getUnchecked(LocalCache.java:5153)
at com.facebook.presto.metadata.FunctionRegistry.getSpecializedFunctionKey(FunctionRegistry.java:898)
at com.facebook.presto.metadata.FunctionRegistry.getAggregateFunctionImplementation(FunctionRegistry.java:875)
at com.facebook.presto.sql.planner.LocalExecutionPlanner$Visitor.buildAccumulatorFactory(LocalExecutionPlanner.java:1973)
at com.facebook.presto.sql.planner.LocalExecutionPlanner$Visitor.planGlobalAggregation(LocalExecutionPlanner.java:1984)
at com.facebook.presto.sql.planner.LocalExecutionPlanner$Visitor.visitAggregation(LocalExecutionPlanner.java:955)
at com.facebook.presto.sql.planner.LocalExecutionPlanner$Visitor.visitAggregation(LocalExecutionPlanner.java:596)
at com.facebook.presto.sql.planner.plan.AggregationNode.accept(AggregationNode.java:167)
at com.facebook.presto.sql.planner.LocalExecutionPlanner$Visitor.visitExchange(LocalExecutionPlanner.java:1919)
at com.facebook.presto.sql.planner.LocalExecutionPlanner$Visitor.visitExchange(LocalExecutionPlanner.java:596)
at com.facebook.presto.sql.planner.plan.ExchangeNode.accept(ExchangeNode.java:196)
at com.facebook.presto.sql.planner.LocalExecutionPlanner$Visitor.visitAggregation(LocalExecutionPlanner.java:952)
at com.facebook.presto.sql.planner.LocalExecutionPlanner$Visitor.visitAggregation(LocalExecutionPlanner.java:596)
at com.facebook.presto.sql.planner.plan.AggregationNode.accept(AggregationNode.java:167)
at com.facebook.presto.sql.planner.LocalExecutionPlanner$Visitor.visitOutput(LocalExecutionPlanner.java:638)
at com.facebook.presto.sql.planner.LocalExecutionPlanner$Visitor.visitOutput(LocalExecutionPlanner.java:596)
at com.facebook.presto.sql.planner.plan.OutputNode.accept(OutputNode.java:82)
at com.facebook.presto.sql.planner.LocalExecutionPlanner.plan(LocalExecutionPlanner.java:393)
at com.facebook.presto.sql.planner.LocalExecutionPlanner.plan(LocalExecutionPlanner.java:324)
at com.facebook.presto.execution.SqlTaskExecution.<init>(SqlTaskExecution.java:161)
at com.facebook.presto.execution.SqlTaskExecution.createSqlTaskExecution(SqlTaskExecution.java:121)
at com.facebook.presto.execution.SqlTaskExecutionFactory.create(SqlTaskExecutionFactory.java:71)
at com.facebook.presto.execution.SqlTask.updateTask(SqlTask.java:340)
at com.facebook.presto.execution.SqlTaskManager.updateTask(SqlTaskManager.java:321)
at com.facebook.presto.server.TaskResource.createOrUpdateTask(TaskResource.java:128)
at sun.reflect.GeneratedMethodAccessor311.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.glassfish.jersey.server.model.internal.ResourceMethodInvocationHandlerFactory.lambda$static$0(ResourceMethodInvocationHandlerFactory.java:76)
at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher$1.run(AbstractJavaResourceMethodDispatcher.java:148)
at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.invoke(AbstractJavaResourceMethodDispatcher.java:191)
at org.glassfish.jersey.server.model.internal.JavaResourceMethodDispatcherProvider$ResponseOutInvoker.doDispatch(JavaResourceMethodDispatcherProvider.java:200)
at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.dispatch(AbstractJavaResourceMethodDispatcher.java:103)
at org.glassfish.jersey.server.model.ResourceMethodInvoker.invoke(ResourceMethodInvoker.java:493)
at org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:415)
at org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:104)
at org.glassfish.jersey.server.ServerRuntime$1.run(ServerRuntime.java:277)
at org.glassfish.jersey.internal.Errors$1.call(Errors.java:272)
at org.glassfish.jersey.internal.Errors$1.call(Errors.java:268)
at org.glassfish.jersey.internal.Errors.process(Errors.java:316)
at org.glassfish.jersey.internal.Errors.process(Errors.java:298)
at org.glassfish.jersey.internal.Errors.process(Errors.java:268)
at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:289)
at org.glassfish.jersey.server.ServerRuntime.process(ServerRuntime.java:256)
at org.glassfish.jersey.server.ApplicationHandler.handle(ApplicationHandler.java:703)
at org.glassfish.jersey.servlet.WebComponent.serviceImpl(WebComponent.java:416)
at org.glassfish.jersey.servlet.WebComponent.service(WebComponent.java:370)
at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:389)
at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:342)
at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:229)
at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:841)
at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1650)
at io.airlift.http.server.TraceTokenFilter.doFilter(TraceTokenFilter.java:63)
at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1637)
at io.airlift.http.server.TimingFilter.doFilter(TimingFilter.java:52)
at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1637)
at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:533)
at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)
at org.eclipse.jetty.server.handler.gzip.GzipHandler.handle(GzipHandler.java:454)
at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132)
at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:190)
at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1253)
at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:168)
at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:473)
at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:166)
at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1155)
at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:126)
at org.eclipse.jetty.server.handler.StatisticsHandler.handle(StatisticsHandler.java:169)
at org.eclipse.jetty.server.handler.HandlerList.handle(HandlerList.java:61)
at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132)
at org.eclipse.jetty.server.Server.handle(Server.java:564)
at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:317)
at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:251)
at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:279)
at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:110)
at org.eclipse.jetty.io.ChannelEndPoint$2.run(ChannelEndPoint.java:124)
at org.eclipse.jetty.util.thread.Invocable.invokePreferred(Invocable.java:128)
at org.eclipse.jetty.util.thread.Invocable$InvocableExecutor.invoke(Invocable.java:222)
at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:294)
at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:199)
at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:673)
at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:591)
at java.lang.Thread.run(Thread.java:748)
Code of AggregationFunction is following.
I referenced to this presto code
package sample;
import com.facebook.presto.spi.block.BlockBuilder;
import com.facebook.presto.spi.function.*;
import com.facebook.presto.spi.type.BigintType;
import com.facebook.presto.spi.type.StandardTypes;
import io.airlift.slice.Slice;
#AggregationFunction("answer_to_life")
public final class AnswerToLife {
private AnswerToLife() {
}
#InputFunction
public static void input(#AggregationState NullState state, #SqlType(StandardTypes.VARCHAR) Slice value) {
}
#CombineFunction
public static void combine(#AggregationState NullState state, #AggregationState NullState other) {
}
#OutputFunction(StandardTypes.BIGINT)
public static void output(#AggregationState NullState state, BlockBuilder out) {
BigintType.BIGINT.writeLong(out, 42);
}
}
Detail code is here (https://github.com/asari-mtr/presto-udaf/tree/stackoverflow).
The structure what I deployed is follwoing:
$ ls -1 /usr/lib/presto/plugin/my-udaf/
commons-codec-1.4.jar
guava-21.0.jar
hive-udf-1.0-SNAPSHOT.jar
presto-array-0.197.jar
stats-0.155.jar
presto-udaf-1.0-SNAPSHOT.jar
I use emr-5.12.0(Presto 0.188)
Thank you for your time.
Edited1
list in jar file.
% jar -tf target/presto-udaf-1.0-SNAPSHOT.jar
Picked up _JAVA_OPTIONS: -Dfile.encoding=UTF-8
META-INF/
META-INF/MANIFEST.MF
META-INF/services/
sample/
META-INF/services/com.facebook.presto.spi.Plugin
sample/AnswerToLife.class
sample/AnswerToLifePlugin.class
sample/NullState.class
META-INF/maven/
META-INF/maven/sample/
META-INF/maven/sample/presto-udaf/
META-INF/maven/sample/presto-udaf/pom.xml
META-INF/maven/sample/presto-udaf/pom.properties
sever.log
$ grep answer -i /mnt/var/log/presto/server.log
2018-03-26T06:37:14.213Z INFO main com.facebook.presto.server.PluginManager Installing sample.AnswerToLifePlugin
2018-03-26T06:37:14.214Z INFO main com.facebook.presto.server.PluginManager Registering functions from sample.AnswerToLife
And execute show funcstions
presto> show functions;
Function | Return Type | Argument Types
---------------------------------+-----------------------------------------------------+------------------------------------------------------------------------------
ST_Area | double | Geometry
ST_AsText | varchar | Geometry
...
answer_to_life | bigint | varchar
...
Edited2
Delete src/main/resources/META-INF/services/com.facebook.presto.spi.Plugin
Add <packaging>presto-plugin</packaging> to pom.xml
https://github.com/asari-mtr/presto-udaf/commit/f2a2ddbf0339e08f418b378a7ead511020e98a3b
I deployed zip made by Maven under /usr/lib/presto/plugin/.
But, The contents of the error does not change.
Edited3
I got the source from github (branch 0.188) on my Mac and built presto.
When we placed the above UDAF on its presto, it worked perfectly.
Perhaps there is a mistake in the installation procedure for presto on EMR.
Solved!!
The Custom plugin did not work because it was not a plugin creation, but a way of deploying on EMR was wrong.
Since I did not deploy to all nodes in the procedure I performed, I confirmed the operation by registering the following shell as a bootstrap action.
#/bin/bash
aws s3 cp s3://hogehoge/presto-udaf-1.0-SNAPSHOT.zip /tmp/
sudo mkdir -p /usr/lib/presto/plugin/
sudo unzip -d /usr/lib/presto/plugin/ /tmp/presto-udaf-1.0-SNAPSHOT.zip
10.1. SPI Overview — Presto 0.198 Documentation
Plugins must be installed on all nodes in the Presto cluster
(coordinator and workers).
Create Bootstrap Actions to Install Additional Software - Amazon EMR
You can use a bootstrap action to copy objects from Amazon S3 to each
node in a cluster before your applications are installed. The AWS CLI
is installed on each node of a cluster, so your bootstrap action can
call AWS CLI commands.
From what you wrote there are two things which are missing.
make sure that you used maven packaging presto-plugin in your pom.xml
also implement com.facebook.presto.spi.Plugin which in getFunctions method would return your function.
Take a look in presto-geospatial Presto module and see com.facebook.presto.plugin.geospatial.GeoPlugin and its pom.xml as reference.
I'm trying to customise the deploy scripts to allow me to deploy each of my four API proxies from the command line. It looks very similar to the one provided in the samples on Github:
#!/bin/bash
if [[ $# -eq 0 ]] ; then
echo 'Must provide proxy name.'
exit 0
fi
dirname=$1
proxyname="teamname-"$dirname
source ./setup/setenv.sh
echo "Enter your password for user $username in the Apigee Enterprise organization $org, followed by [ENTER]:"
read -s password
echo Deploying $proxyname to $env on $url using $username and $org
./tools/deploy.py -n $proxyname -u $username:$password -o $org -h $url -e $env -p / -d ./$dirname
echo "If 'State: deployed', then your API Proxy is ready to be invoked."
echo "Run '$ sh invoke.sh'"
echo "If you get errors, make sure you have set the proper account settings in /setup/setenv.sh"
However when I run it, I get the following response:
Deploying teamname-gameassets to int on https://api.enterprise.apigee.com using my-email-address and org-name
Writing ./gameassets/teamname-gameassets.xml to ./teamname-gameassets.xml
Writing ./gameassets/policies/Add-CORS.xml to policies/Add-CORS.xml
Writing ./gameassets/proxies/default.xml to proxies/default.xml
Writing ./gameassets/targets/development.xml to targets/development.xml
Writing ./gameassets/targets/production.xml to targets/production.xml
Import failed to /v1/organizations/org-name/apis?action=import&name=teamname-gameassets with status 500:
{
"code" : "messaging.config.beans.ImportFailed",
"message" : "Failed to import the bundle : java.lang.NullPointerException",
"contexts" : [ ],
"cause" : {
"contexts" : [ ]
}
}
How should I go about debugging when I receive errors during the deploy process? Is there some sort of console I can view once logged in to Apigee?
I'm not sure how your proxy ended up this way, but it looks like the top-level directory is named "gameassets." It should be named "apiproxy". If you rename this directory you should see a successful deployment.
Also, before you customize too much, please try out "apigeetool," which is a more flexible command-line tool for deploying proxies:
https://github.com/apigee/api-platform-tools