Is it a problem if I set spring.json.trusted.packages = '*' while consuming data in Spring Cloud Stream? - spring-kafka

I am using Spring Cloud Stream in my project. I haven't started consuming with functional stream yet. I'm still consuming data with #StreamListener. I also consume the data as batch. Since I consume the data as batch, I have to give deserializer.
I have too many input topics and I have to create new deserializer classes.
However, there is one small point I noticed. If I set spring.json.trusted.packages: '*' , I don't need to create a new deserializer class and I can use JsonDeserializer for any data. So I don't need to create a deserializer for each newly created data.
But before I do this, many questions come to my mind. Does consuming data this way have any effect on performance?
Is there any benefit to me if I provide a separate deserializer? Why do we create new deserializer classes for each data? Does spring.json.trusted.packages:'*' setting do us any harm? Can you please help me with this?
application.yml with PersonDeserializer
public class PersonDeserializer extends JsonDeserializer<Person> {
}
spring:
cloud:
stream:
binders:
bulkKafka:
type: kafka
environment:
spring:
cloud:
stream:
kafka:
binder:
brokers: ${kafka.brokers}
minPartitionCount: ${default-configuration.kafka.partition-count}
autoCreateTopics: true
autoAddPartitions: true
configuration:
max.poll.records: 3000
fetch.min.bytes: 900000
fetch.max.wait.ms: 500
value.deserializer: org.example.PersonDeserializer
bindings:
person-topic-in:
destination: person-topic
contentType: application/json
binder: bulkKafka
group: ${spring.application.name}
consumer:
batch-mode: true
application.yml without PersonDeserializer
spring:
kafka:
consumer:
properties:
spring.json.trusted.packages: "*"
cloud:
stream:
binders:
bulkKafka:
type: kafka
environment:
spring:
cloud:
stream:
kafka:
binder:
brokers: ${kafka.brokers}
minPartitionCount: ${default-configuration.kafka.partition-count}
autoCreateTopics: true
autoAddPartitions: true
configuration:
max.poll.records: 3000
fetch.min.bytes: 900000
fetch.max.wait.ms: 500
value.deserializer: org.springframework.kafka.support.serializer.JsonDeserializer
bindings:
person-topic-in:
destination: person-topic
contentType: application/json
binder: bulkKafka
group: ${spring.application.name}
consumer:
batch-mode: true

If you trust the source of the data, then it is ok; if you are receiving data from untrusted sources, then you should trust only specific packages.

Related

Tinkerpop Neptune Config - Running In Docker

Does anyone have a gremlin-config.yaml file that would more accurately reflect how Gremlin acts in Neptune?
I am trying to run as much as I can using a local docker container, and I've substituted properties like gremlin.tinkergraph.vertexIdManager=ANY so that the Vertex IDs can be strings.
But I'm still missing details like multiple labels, which I think is only available via the Neo4Js config, unsure what else this will change.
But yeah, generally looking for a config that represents how Neptune functions as closely as possible
Current:
host: 172.17.0.2
port: 8182
evaluationTimeout: 30000
channelizer: org.apache.tinkerpop.gremlin.server.channel.WebSocketChannelizer
graphs: {
graph: conf_local/tinkergraph-custom.properties}
scriptEngines: {
gremlin-groovy: {
plugins: { org.apache.tinkerpop.gremlin.server.jsr223.GremlinServerGremlinPlugin: {},
org.apache.tinkerpop.gremlin.tinkergraph.jsr223.TinkerGraphGremlinPlugin: {},
org.apache.tinkerpop.gremlin.jsr223.ImportGremlinPlugin: {classImports: [java.lang.Math], methodImports: [java.lang.Math#*]},
org.apache.tinkerpop.gremlin.jsr223.ScriptFileGremlinPlugin: {files: [scripts/empty-sample.groovy]}}}}
serializers:
- { className: org.apache.tinkerpop.gremlin.driver.ser.GraphSONMessageSerializerV3d0, config: { ioRegistries: [org.apache.tinkerpop.gremlin.tinkergraph.structure.TinkerIoRegistryV3d0] }} # application/json
- { className: org.apache.tinkerpop.gremlin.driver.ser.GraphBinaryMessageSerializerV1 } # application/vnd.graphbinary-v1.0
- { className: org.apache.tinkerpop.gremlin.driver.ser.GraphBinaryMessageSerializerV1, config: { serializeResultToString: true }} # application/vnd.graphbinary-v1.0-stringd
processors:
- { className: org.apache.tinkerpop.gremlin.server.op.session.SessionOpProcessor, config: { sessionTimeout: 28800000 }}
- { className: org.apache.tinkerpop.gremlin.server.op.traversal.TraversalOpProcessor, config: { cacheExpirationTime: 600000, cacheMaxSize: 1000 }}
metrics: {
consoleReporter: {enabled: true, interval: 180000},
csvReporter: {enabled: true, interval: 180000, fileName: /tmp/gremlin-server-metrics.csv},
jmxReporter: {enabled: true},
slf4jReporter: {enabled: true, interval: 180000}}
strictTransactionManagement: false
idleConnectionTimeout: 0
keepAliveInterval: 0
maxInitialLineLength: 4096
maxHeaderSize: 8192
maxChunkSize: 8192
maxContentLength: 10485760
maxAccumulationBufferComponents: 1024
resultIterationBatchSize: 64
writeBufferLowWaterMark: 32768
writeBufferHighWaterMark: 65536
ssl: {
enabled: false}
gremlin.tinkergraph.vertexIdManager=ANY
gremlin.graph=org.apache.tinkerpop.gremlin.tinkergraph.structure.TinkerGraph
version: '3'
services:
gremlin-server:
container_name: gwent_onboarding_neptune
image: tinkerpop/gremlin-server:3.5.0
user: $USER
volumes:
- ./configuration/gremlin-conf:/opt/gremlin-server/conf_local
# - ./gremlin-console/data:/gremlin-server/scripts
ports:
- 8182:8182
command: ./conf_local/gremlin-serverr.yaml
Edit:
Started down this rabbit hole when I was trying to get tinkerpop to work with multiple labels
It looks as if you have already done the things I usually recommend (such as enabling and using strings for the ID values). Given TinkerGraph does not support transactions currently, to simulate those you would likely need to consider using something like JanusGraph running in "inmemory" mode via a Gremlin Server. For the remaining differences, at the present time, it's mostly a case of avoiding any unsupported features such as meta properties in the code and queries that you write. Other than multiple labels as you mentioned, you should be able to do a lot of development and testing of queries locally but, of course, you will at some point still need to test against an actual Neptune cluster to verify the workloads that you have behave as expected when using the full Neptune cluster architecture.

symfony messenger multiple bus

I need to have multiple bus in my application written on Symfony. To do so I'm using messenger bundle.
So I need to have:
external_events_bus - which receiving messages from external systems, which using sqs transport
application_events_bus - for application events (using database driver for now)
domain_events_bus - for domain events (using database driver for now)
command_bus - for commands and queries (CQRS) (currently sync)
so in my messenger.yaml I have:
framework:
messenger:
default_bus: command.bus
buses:
command.bus: ~
domain_event.bus: ~
application_event.bus: ~
external_event.bus: ~
transports:
command_bus: '%env(COMMAND_BUS_DSN)%'
internal_event_bus: '%env(INTERNAL_EVENT_BUS_DSN)%'
external_event_bus: '%(EXTERNAL_EVENT_BUS_DSN)%'
routing:
'App\Application\CommandBus\Command': command_bus
'App\Domain\EventBus\DomainEvent': internal_event_bus
'App\Application\EventBus\ApplicationEvent': internal_event_bus
'App\Infrastructure\ExternalEventBus\ExternalEvent': external_event_bus
and in my services.yaml
services:
_defaults:
autowire: true
autoconfigure: true subscribers, etc.
public: false
_instanceof:
# use tags to avoid coupling handler classes to Messenger's interface
App\Application\CommandBus\CommandHandler:
tags: [ { name: messenger.message_handler, bus: command.bus } ]
App\Interfaces\ExternalEventBus\ExternalEventHandler:
tags: [ { name: messenger.message_handler, bus: external_event.bus } ]
App\Domain\EventBus\DomainEventHandler:
tags: [ { name: messenger.message_handler, bus: domain_event.bus } ]
App\Application\EventBus\ApplicationEventHandler:
tags: [ { name: messenger.message_handler, bus: application_event.bus } ]
... // some other services here
App\Application\CommandBus\CommandBus:
class: App\Infrastructure\CommandBus\CommandBus
arguments: [ "#command.bus" ]
App\Domain\EventBus\DomainEventBus:
class: App\Infrastructure\DomainEventBus\DomainEventBus
arguments: [ "#domain_event.bus" ]
App\Application\EventBus\ApplicationEventBus:
class: App\Infrastructure\ApplicationEventBus\ApplicationEventBus
arguments: [ "#application_event.bus" ]
App\Infrastructure\ExternalEventBus\ExternalEventBus:
class: App\Infrastructure\ExternalEventBus\ExternalEventBus
arguments: [ "#external_event.bus" ]
Want to clarify before I had only command bus and external bus and all was fine, but after I added domain and application bus, somethinkg went wrong.
When in tests I'm using ExternalEventBus, to test if external event handled correctly, like this:
$this->eventBus = self::$container->get(PlatformEventBus::class);
$this->eventBus->dispatch(new SomeExternalEvent());
I'm getting error like
Symfony\Component\Messenger\Exception\NoHandlerForMessageException: No handler for message "SomeExternalEvent".
but when I'm running bin/console debug:messenger for both dev and test env, I can see correct mapping between all events and their handlers.
But when I'm debugging it can see that HandlersLocator::getHandlers() for that SomeExternalEvent for some reason returning list of handlers related to command bus instead of external event bus (maybe because command bus set as a default bus)
I know it's a bit hard to explaing and show topic, but I spend already whole day truing to find what is wrong with it. Any help appreciated, thank you!!!
Sorry for any kind of typos, I'm a bit change name of classes for this example and may did some typos
okay, I figured it out, messenger using autowiryng when you injecting MessageBusInterface depends on variable name, so in case of external event bus it should be $externalEventBus and same for other buses. But I do not understand how it worked before, as I mentioned I have command and external bus for a while and it stop working only when I add more buses...

Symfony 4 enable logging with Monolog's Redis handler

I have a working ELK stack connected to Redis.
I also have a working stateless Symfony 4 application and I want to send all the production logs to my Redis.
I know Monolog has a Redis handler, but I don't know how I'm supposed to tweak the config/prod/monolog.yaml file to accomplish this of if there’s another approach.
This is how it looks right now:
monolog:
handlers:
main:
type: fingers_crossed
action_level: error
handler: nested
excluded_http_codes: [404]
nested:
type: stream
path: "php://stderr"
level: debug
console:
type: console
process_psr_3_messages: false
channels: ["!event", "!doctrine"]
deprecation:
type: stream
path: "php://stderr"
deprecation_filter:
type: filter
handler: deprecation
max_level: info
channels: ["php"]
The approach I took was, first installing the predis client:
composer require predis/predis
Then create a custom service class that extends the RedisHandler class that comes with the Monolog package:
namespace App\Service\Monolog\Handler;
use Monolog\Handler\RedisHandler;
use Monolog\Logger;
use Predis\Client as PredisClient;
class Redis extends RedisHandler
{
public function __construct( $host, $port = 6379, $level = Logger::DEBUG, $bubble = true, $capSize = false)
{
$predis = new PredisClient( "tcp://$host:$port" );
$key = 'logstash';
parent::__construct($predis, $key, $level, $bubble, $capSize);
}
}
Next, activate the service we just created on the services.yml config file:
services:
monolog.handler.redis:
class: App\Service\Monolog\Handler\Redis
arguments: [ '%redis.host%' ]
Be sure the parameter redis.host is set and points to your Redis server. In my case, my parameter value is the IP of my Redis server.
I added other parameters to the class like port and log level. You can set it at the moment of instantiating your service like with the host parameter.
Finally, configure your custom log handler service in your monolog.yaml config file. In my case, I need it only the production logs with the config as follow:
handlers:
custom:
type: service
id: monolog.handler.redis
level: debug
channels: ['!event']

rabbitmq-bundle - symfony3 - how to configure a topic exchange and queues?

I can't find a great configuration for old sound rabbitmq bundle to deal with topics and wildcard.
All I want is a unique exchange that post to multiple queue using wildcard.
Let says for example, i have my exchange name user.update, and i want to post the same message on user.update.address, user.update.profile for a microservice strategy.
do you know how to configure in the configuration file ?
Thx for reading.
Just because you are looking for
... great configuration for old sound rabbitmq bundle ...
visit http://www.inanzzz.com/ and search for "rabbitmq" which will give you what you wish for.
To address your question, you can use config below (I haven't tested it but it should be fine). However, you still need to write whole functionality/classes/consumers/producers etc. so follow this example: RabbitMQ topic example with symfony including 1 Producer & 1 Exchange & 2 Queue & N Worker & 2 Consumer
old_sound_rabbit_mq:
connections:
default:
host: %rabbitmq.host%
port: %rabbitmq.port%
user: %rabbitmq.user%
password: %rabbitmq.pswd%
vhost: /
lazy: true
producers:
user_update_producer:
connection: default
exchange_options: { name: user.update, type: topic }
consumers:
user_update_consumer:
connection: default
exchange_options: { name: user.update, type: topic }
queue_options:
name: user_update_queue
routing_keys:
- 'user.update.address'
- 'user.update.profile'
callback: your_application.consumer.user_update_consumer
It's flow: user.update (P) -> user.update (E) -> [user.update.address & user.update.profile] -> user_update_queue (Q)

Sonata Media CDN Rackspace

I have the next problem with Sonata Media:
I'm trying to use the Rackspace CDN for uploading images:
My config file looks like this based on current documentation:
cdn:
server:
path: %cdn_url%
filesystem:
local:
directory: %kernel.root_dir%/../web/uploads/media
create: false
rackspace:
url: %rackspace.opencloud.host%
secret:
username: %rackspace.opencloud.username%
apiKey: %rackspace.opencloud.api_key%
region: LON
containerName: projectName
create_container: false
replicate:
master: sonata.media.adapter.filesystem.opencloud
slave: sonata.media.adapter.filesystem.local
And on providers config:
providers:
image:
filesystem: sonata.media.filesystem.replicate
cdn: sonata.media.cdn.server
resizer: sonata.media.resizer.square
allowed_extensions: ['jpg', 'png', 'gif', 'jpeg']
allowed_mime_types: ['image/pjpeg','image/jpeg','image/png','image/x-png', 'image/gif']
The problem is(how I discovered this bug)if Rackspace is down or incorrect username/password are provided on every page of the app I'm getting this answer:
Client error response [status code] 401 [reason phrase] Unauthorized [url] https://lon.auth.api.rackspacecloud.com/v2.0/tokens
This is because Gaufrette Opencloud tries to create a connection on Kernel load.
The quickest solution as a temporary fix was to create a compiler pass and check if the authenticate method returns false then replace argument 0 for replicate definition with the local filesystem adaptor.
My questions are:
How can I avoid creating the Rackspace connection on Kernel Load?
In case Rackspace is down how can I swap between Rackspace or other adapter(local or other ftp server)
Thank you in advance and please in case there are not sufficient information provided please leave a comment.
Apparently there is a solution for lazy loading implemented in Gaufrette: https://github.com/KnpLabs/KnpGaufretteBundle/issues/72
All I had to do is:
sonata.media.adapter.open_stack:
class: OpenCloud\Rackspace
arguments: [ %rackspace.opencloud.host%, { username: %rackspace.opencloud.username%, apiKey: %rackspace.opencloud.api_key% }]
sonata.media.adapter.object_store_factory:
class: Gaufrette\Adapter\OpenStackCloudFiles\ObjectStoreFactory
arguments: [ #sonata.media.adapter.open_stack, "LON", ""]
sonata.media.adapter.filesystem.lazyopencloud:
class: Gaufrette\Adapter\LazyOpenCloud
arguments: [ #sonata.media.adapter.object_store_factory, %rackspace.opencloud.container_name%]
And change replicate master to sonata.media.adapter.filesystem.lazyopencloud
Hope it helps :)

Resources