Tinkerpop Neptune Config - Running In Docker - gremlin

Does anyone have a gremlin-config.yaml file that would more accurately reflect how Gremlin acts in Neptune?
I am trying to run as much as I can using a local docker container, and I've substituted properties like gremlin.tinkergraph.vertexIdManager=ANY so that the Vertex IDs can be strings.
But I'm still missing details like multiple labels, which I think is only available via the Neo4Js config, unsure what else this will change.
But yeah, generally looking for a config that represents how Neptune functions as closely as possible
Current:
host: 172.17.0.2
port: 8182
evaluationTimeout: 30000
channelizer: org.apache.tinkerpop.gremlin.server.channel.WebSocketChannelizer
graphs: {
graph: conf_local/tinkergraph-custom.properties}
scriptEngines: {
gremlin-groovy: {
plugins: { org.apache.tinkerpop.gremlin.server.jsr223.GremlinServerGremlinPlugin: {},
org.apache.tinkerpop.gremlin.tinkergraph.jsr223.TinkerGraphGremlinPlugin: {},
org.apache.tinkerpop.gremlin.jsr223.ImportGremlinPlugin: {classImports: [java.lang.Math], methodImports: [java.lang.Math#*]},
org.apache.tinkerpop.gremlin.jsr223.ScriptFileGremlinPlugin: {files: [scripts/empty-sample.groovy]}}}}
serializers:
- { className: org.apache.tinkerpop.gremlin.driver.ser.GraphSONMessageSerializerV3d0, config: { ioRegistries: [org.apache.tinkerpop.gremlin.tinkergraph.structure.TinkerIoRegistryV3d0] }} # application/json
- { className: org.apache.tinkerpop.gremlin.driver.ser.GraphBinaryMessageSerializerV1 } # application/vnd.graphbinary-v1.0
- { className: org.apache.tinkerpop.gremlin.driver.ser.GraphBinaryMessageSerializerV1, config: { serializeResultToString: true }} # application/vnd.graphbinary-v1.0-stringd
processors:
- { className: org.apache.tinkerpop.gremlin.server.op.session.SessionOpProcessor, config: { sessionTimeout: 28800000 }}
- { className: org.apache.tinkerpop.gremlin.server.op.traversal.TraversalOpProcessor, config: { cacheExpirationTime: 600000, cacheMaxSize: 1000 }}
metrics: {
consoleReporter: {enabled: true, interval: 180000},
csvReporter: {enabled: true, interval: 180000, fileName: /tmp/gremlin-server-metrics.csv},
jmxReporter: {enabled: true},
slf4jReporter: {enabled: true, interval: 180000}}
strictTransactionManagement: false
idleConnectionTimeout: 0
keepAliveInterval: 0
maxInitialLineLength: 4096
maxHeaderSize: 8192
maxChunkSize: 8192
maxContentLength: 10485760
maxAccumulationBufferComponents: 1024
resultIterationBatchSize: 64
writeBufferLowWaterMark: 32768
writeBufferHighWaterMark: 65536
ssl: {
enabled: false}
gremlin.tinkergraph.vertexIdManager=ANY
gremlin.graph=org.apache.tinkerpop.gremlin.tinkergraph.structure.TinkerGraph
version: '3'
services:
gremlin-server:
container_name: gwent_onboarding_neptune
image: tinkerpop/gremlin-server:3.5.0
user: $USER
volumes:
- ./configuration/gremlin-conf:/opt/gremlin-server/conf_local
# - ./gremlin-console/data:/gremlin-server/scripts
ports:
- 8182:8182
command: ./conf_local/gremlin-serverr.yaml
Edit:
Started down this rabbit hole when I was trying to get tinkerpop to work with multiple labels

It looks as if you have already done the things I usually recommend (such as enabling and using strings for the ID values). Given TinkerGraph does not support transactions currently, to simulate those you would likely need to consider using something like JanusGraph running in "inmemory" mode via a Gremlin Server. For the remaining differences, at the present time, it's mostly a case of avoiding any unsupported features such as meta properties in the code and queries that you write. Other than multiple labels as you mentioned, you should be able to do a lot of development and testing of queries locally but, of course, you will at some point still need to test against an actual Neptune cluster to verify the workloads that you have behave as expected when using the full Neptune cluster architecture.

Related

Error with Gatsby plugin Gatsby-Source-Wordpress

I'm trying to setup my first gatsby + wordpress site. I'm following this tutorial.
I get the site running but at the point where I should get the data from WP I get stuck. I added Gatsby-Source-Wordpress plugin. After I restarted site it throws this error:
success open and validate gatsby-configs - 0.102 s
success load plugins - 0.631 s
success onPreInit - 0.019 s
success initialize cache - 0.053 s
success copy gatsby files - 0.161 s
success onPreBootstrap - 0.040 s
info Creating GraphQL type definition for File
Path: /wp-json
The server response was "404 Not Found"
ERROR #11321 PLUGIN
"gatsby-source-wordpress" threw an error while running the sourceNodes lifecycle:
Cannot read property 'data' of undefined
TypeError: Cannot read property 'data' of undefined
- fetch.js:141 fetch
[gatsby-wordpress]/[gatsby-source-wordpress]/fetch.js:141:21
- next_tick.js:68 process._tickCallback
internal/process/next_tick.js:68:7
warn The gatsby-source-wordpress plugin has generated no Gatsby nodes. Do you need it?
success source and transform nodes - 0.327 s
success building schema - 0.404 s
success createPages - 0.019 s
success createPagesStatefully - 0.090 s
success onPreExtractQueries - 0.022 s
success update schema - 0.079 s
success extract queries from components - 0.595 s
success write out requires - 0.103 s
success write out redirect data - 0.032 s
success Build manifest and related icons - 0.263 s
success onPostBootstrap - 0.308 s
⠀
info bootstrap finished - 6.617 s
⠀
success run static queries - 0.105 s — 3/3 36.11 queries/second
success run page queries - 0.044 s — 5/5 230.97 queries/second
DONE Compiled successfully in 4851ms 10:46:42 AM
⠀
You can now view gatsby-starter-default in the browser.
⠀
http://localhost:8000/
⠀
View GraphiQL, an in-browser IDE, to explore your site's data and schema
⠀
http://localhost:8000/___graphql
⠀
Note that the development build is not optimized.
To create a production build, use npm run build
⠀
ℹ 「wdm」:
ℹ 「wdm」: Compiled successfully.
I run WP locally with Mamp and I'm able to see JSON data here: http://localhost:8888/GatsbyWP/wp-json/ .
Here's my gatsby-config.js file:
module.exports = {
siteMetadata: {
title: `Gatsby wordpress test`,
description: `Testing...`,
author: `#gatsbyjs`,
},
plugins: [
`gatsby-plugin-react-helmet`,
{
resolve: `gatsby-source-filesystem`,
options: {
name: `images`,
path: `${__dirname}/src/images`,
},
},
`gatsby-transformer-sharp`,
`gatsby-plugin-sharp`,
{
resolve: `gatsby-plugin-manifest`,
options: {
name: `gatsby-starter-default`,
short_name: `starter`,
start_url: `/`,
background_color: `#663399`,
theme_color: `#663399`,
display: `minimal-ui`,
icon: `src/images/gatsby-icon.png`,
},
},
{
resolve: "gatsby-source-wordpress",
options: {
baseUrl: `localhost:8888`,
protocol: `http`,
hostingWPCOM: false,
useACF: true,
},
},
`gatsby-plugin-sitemap`,
],
}
I'm stuck and don't have any clue what to do now. I found that other people had similar issue than this but didn't find any good answers or direction where to try figure out my problem.
Thanks in advance!
The options of gatsby-source-wordpress require...
the base URL of the Wordpress site without the trailing slash and the protocol. This is required.
Example : 'gatsbyjsexamplewordpress.wordpress.com' or 'www.example-site.com'
module.exports = {
siteMetadata: {
title: `Gatsby wordpress test`,
description: `Testing...`,
author: `#gatsbyjs`,
},
plugins: [
{
resolve: "gatsby-source-wordpress",
options: {
baseUrl: `localhost:8888/GatsbyWP`,
protocol: `http`,
hostingWPCOM: false,
useACF: true,
},
},
],
}
When Sourcing on hosting from Services Like Godaddy The axios/node clients can be dubious about them and reject the https crt, I have seen this rectified in two ways
1) With adding a third party go daddy crt bundle for example https://ssl-ccp.godaddy.com/repository?origin=CALLISTO, if you have your own server that's fine. So adding this to axios:
var agent = new https.Agent({
ca: fs.readFileSync('ca.pem')
});
axios.get(url, { agent: agent });
// or
var instance = axios.create({ agent: agent });
instance.get(url);
or node
export NODE_EXTRA_CA_CERTS=[your CA certificate file path]
2) But what if you are on Netlify or hosting through gitlab or other, What worked for me was changed my gatsby config protocol to http, this allowed me to source from my site just fine and provided all assets are https anyway, even when I deployed to my https site it all still worked. This stumped me for days, hope this helps someone
{
resolve: `gatsby-source-wordpress`,
options: {
/*
* The base URL of the WordPress site without the trailingslash and the protocol. This is required.
* Example : 'dev-gatbsyjswp.pantheonsite.io' or 'www.example-site.com'
*/
baseUrl: `example.com`,
// The protocol. This can be http or https.
protocol: `http`,
// Indicates whether the site is hosted on wordpress.com.
// If false, then the assumption is made that the site is self hosted.
// If true, then the plugin will source its content on wordpress.com using the JSON REST API V2.
// If your site is hosted on wordpress.org, then set this to false.
hostingWPCOM: false,
Hey you need to make sure you query the post types you pull in and you also need to make sure theres data there to be consumed but here's what you config should look like.
{
resolve: `gatsby-source-wordpress`,
options: {
baseUrl: process.env.API_URL,
protocol: process.env.API_PROTOCOL,
hostingWPCOM: false,
useACF: true,
includedRoutes: [
"**/categories",
"**/posts",
"**/pages",
"**/media",
"**/tags",
"**/taxonomies",
"**/users",
"**/menus",
"**/portfolio",
"**/services",
"**/qualifications",
"**/gallery",
"**/logo",
"**/location",
],
},
},
This may be of help to someone in case all the recommended solutions above do not solve your problems.
It may be that you are using the later version of gatsby and so little changes were made to the way gatsby-source-wordpress plugin works
https://www.gatsbyjs.com/docs/how-to/sourcing-data/sourcing-from-wordpress/
this site was helpful. Check it out too.

DynamoDb re-using ScalableTarget for multiple tables using cloudformation

I am trying to add autoscaling to multiple DDB tables, since all the tables would have the same traffic pattern the autoscaling configuration for them would be identical.
I know of a way to do that by creating ScalableTargets for each table, but since I want all tables to use the same configuration I was wondering if it is possible to re-use the scalable targets
WriteCapacityScalableTarget:
Type: "AWS::ApplicationAutoScaling::ScalableTarget"
Properties:
MaxCapacity: 30
MinCapacity: 5
ResourceId:
Fn::Join:
- /
- - table
- Ref: FooTable
RoleARN:
Fn::GetAtt: DynamoDBAutoScalingRole.Arn
ScalableDimension: "dynamodb:table:WriteCapacityUnits"
ServiceNamespace: dynamodb
WriteScalingPolicy:
Type: "AWS::ApplicationAutoScaling::ScalingPolicy"
Properties:
PolicyName: WriteAutoScalingPolicy
PolicyType: TargetTrackingScaling
ScalingTargetId:
Ref: WriteCapacityScalableTarget
TargetTrackingScalingPolicyConfiguration:
TargetValue: 80
ScaleInCooldown: 10
ScaleOutCooldown: 10
PredefinedMetricSpecification:
PredefinedMetricType: DynamoDBWriteCapacityUtilization
DynamoDBAutoScalingRole:
Type: AWS::IAM::Role
Properties:
AssumeRolePolicyDocument:
Version: '2012-10-17'
Statement:
- Effect: Allow
Principal:
Service:
- application-autoscaling.amazonaws.com
Action:
- 'sts:AssumeRole'
Path: /
Policies:
- PolicyName: root
PolicyDocument:
Version: '2012-10-17'
Statement:
- Effect: Allow
Action:
- 'dynamodb:DescribeTable'
- 'dynamodb:UpdateTable'
- 'cloudwatch:PutMetricAlarm'
- 'cloudwatch:DescribeAlarms'
- 'cloudwatch:GetMetricStatistics'
- 'cloudwatch:SetAlarmState'
- 'cloudwatch:DeleteAlarms'
Resource: "*"
This works fine to enable autoscaling on a single table but when I try the same with multiple tables it fails, I am trying this like -
WriteCapacityScalableTarget:
Type: "AWS::ApplicationAutoScaling::ScalableTarget"
Properties:
MaxCapacity: 30
MinCapacity: 5
ResourceId:
Fn::Join:
- /
- - table
- Ref: FooTable
- table
- Ref: BarTable
RoleARN:
Fn::GetAtt: DynamoDBAutoScalingRole.Arn
ScalableDimension: "dynamodb:table:WriteCapacityUnits"
ServiceNamespace: dynamodb
The above target fails to create, we can re-use the scalable targets for indexes in a same way but is it also possible for multiple tables?
If yes, how to achieve that?

rabbitmq-bundle - symfony3 - how to configure a topic exchange and queues?

I can't find a great configuration for old sound rabbitmq bundle to deal with topics and wildcard.
All I want is a unique exchange that post to multiple queue using wildcard.
Let says for example, i have my exchange name user.update, and i want to post the same message on user.update.address, user.update.profile for a microservice strategy.
do you know how to configure in the configuration file ?
Thx for reading.
Just because you are looking for
... great configuration for old sound rabbitmq bundle ...
visit http://www.inanzzz.com/ and search for "rabbitmq" which will give you what you wish for.
To address your question, you can use config below (I haven't tested it but it should be fine). However, you still need to write whole functionality/classes/consumers/producers etc. so follow this example: RabbitMQ topic example with symfony including 1 Producer & 1 Exchange & 2 Queue & N Worker & 2 Consumer
old_sound_rabbit_mq:
connections:
default:
host: %rabbitmq.host%
port: %rabbitmq.port%
user: %rabbitmq.user%
password: %rabbitmq.pswd%
vhost: /
lazy: true
producers:
user_update_producer:
connection: default
exchange_options: { name: user.update, type: topic }
consumers:
user_update_consumer:
connection: default
exchange_options: { name: user.update, type: topic }
queue_options:
name: user_update_queue
routing_keys:
- 'user.update.address'
- 'user.update.profile'
callback: your_application.consumer.user_update_consumer
It's flow: user.update (P) -> user.update (E) -> [user.update.address & user.update.profile] -> user_update_queue (Q)

Sonata Media CDN Rackspace

I have the next problem with Sonata Media:
I'm trying to use the Rackspace CDN for uploading images:
My config file looks like this based on current documentation:
cdn:
server:
path: %cdn_url%
filesystem:
local:
directory: %kernel.root_dir%/../web/uploads/media
create: false
rackspace:
url: %rackspace.opencloud.host%
secret:
username: %rackspace.opencloud.username%
apiKey: %rackspace.opencloud.api_key%
region: LON
containerName: projectName
create_container: false
replicate:
master: sonata.media.adapter.filesystem.opencloud
slave: sonata.media.adapter.filesystem.local
And on providers config:
providers:
image:
filesystem: sonata.media.filesystem.replicate
cdn: sonata.media.cdn.server
resizer: sonata.media.resizer.square
allowed_extensions: ['jpg', 'png', 'gif', 'jpeg']
allowed_mime_types: ['image/pjpeg','image/jpeg','image/png','image/x-png', 'image/gif']
The problem is(how I discovered this bug)if Rackspace is down or incorrect username/password are provided on every page of the app I'm getting this answer:
Client error response [status code] 401 [reason phrase] Unauthorized [url] https://lon.auth.api.rackspacecloud.com/v2.0/tokens
This is because Gaufrette Opencloud tries to create a connection on Kernel load.
The quickest solution as a temporary fix was to create a compiler pass and check if the authenticate method returns false then replace argument 0 for replicate definition with the local filesystem adaptor.
My questions are:
How can I avoid creating the Rackspace connection on Kernel Load?
In case Rackspace is down how can I swap between Rackspace or other adapter(local or other ftp server)
Thank you in advance and please in case there are not sufficient information provided please leave a comment.
Apparently there is a solution for lazy loading implemented in Gaufrette: https://github.com/KnpLabs/KnpGaufretteBundle/issues/72
All I had to do is:
sonata.media.adapter.open_stack:
class: OpenCloud\Rackspace
arguments: [ %rackspace.opencloud.host%, { username: %rackspace.opencloud.username%, apiKey: %rackspace.opencloud.api_key% }]
sonata.media.adapter.object_store_factory:
class: Gaufrette\Adapter\OpenStackCloudFiles\ObjectStoreFactory
arguments: [ #sonata.media.adapter.open_stack, "LON", ""]
sonata.media.adapter.filesystem.lazyopencloud:
class: Gaufrette\Adapter\LazyOpenCloud
arguments: [ #sonata.media.adapter.object_store_factory, %rackspace.opencloud.container_name%]
And change replicate master to sonata.media.adapter.filesystem.lazyopencloud
Hope it helps :)

SaltStack - grains.filter_by, specify grain key and filter by sub key

I'm not sure if I'm wording this correctly, but I was hoping I could get an example of filtering by matching on grain keys and then filtering by values (or sub key:values). My concern is that another grain could be added some time in the future and be picked up by filter_by incorrectly. Example below...
Example list of grains:
Host1
role:
webserver
secondary:
none
Host2
role:
appserver1
secondary:
appserver2
Host3
role:
appserver1
appserver2
secondary:
none
Example map file:
{% set java = salt['grains.filter_by']({
'default': {
'target': '/some/default/file/path',
},
'appserver1': {
'target': '/app/server1/path',
},
'appserver2': {
'target': '/app/server2/path',
},
},
default='default'
)%}
In this example, imagine secondary was the additional grain that was added at a future time. What would the mapfile pick up for Host2 after this secondary grain is added? I know this isn't the best example, but this came up when code reviewing some states I wrote, and I didn't have a good answer as to how we can target grain keys. In this case, I would want to target the grain 'role' and filter on the values within that grain. How would I do that?
I completely missed this in the docs until I read them multiple times...
Solution is to add grain value to filter on like so:
{% set java = salt['grains.filter_by']({
'default': {
'target': '/some/default/file/path',
},
'appserver1': {
'target': '/app/server1/path',
},
'appserver2': {
'target': '/app/server2/path',
},
},
grain='role',
default='default'
)%}

Resources