I see people duplicating code to configure kafka for each cluster :(
Is it really necessary to configure Consumer Factory and Producer Factory with different settings every time? And do not use Spring Boot Kafka starter in any way?
If you use similar properties for each cluster, you can override the bootstrap.servers property for each #KafkaListener and/or create multiple #KafkaTemplates with the same override.
Then, the same factories can be used.
See https://docs.spring.io/spring-kafka/docs/current/reference/html/#annotation-properties
#KafkaListener(topics = "myTopic", groupId = "group", properties = {
"max.poll.interval.ms:60000",
ConsumerConfig.MAX_POLL_RECORDS_CONFIG + "=100"
})
/**
* Create an instance using the supplied producer factory and properties, with
* autoFlush false. If the configOverrides is not null or empty, a new
* {#link DefaultKafkaProducerFactory} will be created with merged producer properties
* with the overrides being applied after the supplied factory's properties.
* #param producerFactory the producer factory.
* #param configOverrides producer configuration properties to override.
* #since 2.5
*/
public KafkaTemplate(ProducerFactory<K, V> producerFactory, #Nullable Map<String, Object> configOverrides) {
Spring Boot auto-configuration is by convention for the common microservices use-case: one thing, but simple and clear.
What you are asking is out of Spring Boot scope: the properties configuration is applied only for one ConsumerFactory and one ProducerFactory. As long as you need to connect to different clusters you are on your own. Having a custom ConsumerFactory bean will lead Spring Boot auto-configuration to back off.
You probably can look into a child ApplicationContext configuration with its own provided set of properties. But will it be easier to just have custom configuration for each cluster as you would do without Spring Boot with just a plain Apache Kafka client?
I have no idea if there is some federation solution for Apache Kafka. But that's still out of Spring for Apache Kafka and Spring Boot scope.
Related
we need to send the data to 2 topics which belongs to the two different servers. One is confluent offerings in Azure another is kafka cluster on azure VM.
Now we are using KafkaRoutingTemplate to send message to the 2 different Kafka Offerings.
But we create 2 producers/factory via config class ( coding) - Is there any way to configure 2 producers via application.yml file instead of coding
You can override the bootstrap servers in one of the templates; they can use the same producer factory, but you have to define the templates as beans.
/**
* Create an instance using the supplied producer factory and properties, with
* autoFlush false. If the configOverrides is not null or empty, a new
* {#link DefaultKafkaProducerFactory} will be created with merged producer properties
* with the overrides being applied after the supplied factory's properties.
* #param producerFactory the producer factory.
* #param configOverrides producer configuration properties to override.
* #since 2.5
*/
public KafkaTemplate(ProducerFactory<K, V> producerFactory, #Nullable Map<String, Object> configOverrides) {
Is it possible to replace a private service in the DI container? I need to do this in my test environment, so that I can run integration tests but still mock HTTP calls to external APIs.
For example, I have this code that sets the mock for the HttpClientInterface:
$response = new MockResponse('"some json body"');
$client = new MockHttpClient([$response]);
self::$container->set(HttpClientInterface::class, $client);
// Execute controller / command and perform assertions
// ...
I have already tried to define the HttpClientInterface as a public service for my test environment with the config below, but this does not work as it isn't instantiable (it's an interface).
services:
Symfony\Contracts\HttpClient\HttpClientInterface:
public: true
I resolved this issue myself for my specific case by creating my own MockHttpClient class that doesn't require setting the responses in the constructor. Replacing a service in the container at runtime seemed to be the wrong way to go based on how Symfony have intended for the container to be immutable
I've read hundred of posts and pages but I'm unable to figure the right way to do integration testing mocking just some components.
This is the scenario: I've an application created using Spring Boot (1.2-snapshot) and among various spring libraries, also spring data JPA.
I've several services, for example Service1 and Service2, and they use other components and repositories managed by Spring Data.
If I want to test all the services for a complete integration testing using an embedded hsql database I declare a class this way in my test package:
#Transactional
#RunWith(SpringJUnit4ClassRunner.class)
#SpringApplicationConfiguration(classes = AppConfig.class)
public class IntegrationTest {
#Autowired
private Service1 s1;
#Autowired
private Service2 s2;
[... test methods ...]
}
Where the AppConfig class is instead in my main package, and is defined this way:
#ComponentScan
#Configuration
#EnableAutoConfiguration
public class AppConfig {
public static void main(String[] args) {
SpringApplication.run(AppConfig.class, args);
}
}
During the integration testing a complete spring context get defined, all the spring data repositories are built and instantiated as usuale and everything works fine as expected.
But there 2 are scenarios where I need different goals:
1) I want to test just one service at a time (e.g.: Service1), for example because Service2 is very slow to initialize, and I want to test it in a different test class.
How do I achieve this goal? The problem is that I still need all the Dependency Injection, and in particular all the spring data manages repositories that Service1 autowires on itself. If I was not using spring-data repositories I could new() the Service1 class by myself and then wire all the dependencies by hand, even this would be very cumbersome.
2) While testing Service1, I would like do mock/stub just one of all his dependencies. For example I would simulate a component that in production connects to external services.
I don't know how to selectively inject a stubbed object on the spring context while continuing to use all the others as usual.
Some help on the subject would be very welcome.
Instead of using Mockito mocks you can create different test beans that are set up on different profiles and then execute the tests with those profiles.
FOr example (of course it's just a vision of how you can do it not the exact solution) you could add a profile 'fast' and run your tests with #ActiveProfiles('fast'). Then you would have test configurations annotated with #Profile('fast') that would set up all the beans that you need
I can't find the answer to this...
If I inject the service container, like:
// config.yml
my_listener:
class: MyListener
arguments: [#service_container]
my_service:
class: MyService
// MyListener.php
class MyListener
{
protected $container;
public function __construct(ContainerInterface $container)
{
$this->container = $container;
}
public function myFunction()
{
$my_service = $this->container->get('my_service');
$my_service->doSomething();
}
}
then it works just as well as if I do:
// config.yml
my_listener:
class: MyListener
arguments: [#my_service]
my_service:
class: MyService
// MyListener.php
class MyListener
{
protected $my_service;
public function __construct(MyService $my_service)
{
$this->my_service = $my_service;
}
public function myFunction()
{
$this->my_service->doSomething();
}
}
So why shouldn't I just inject the service container, and get the services from that inside my class?
My list of reasons why you should prefer injecting services:
Your class is dependent only on the services it needs, not the service container. This means the service can be used in an environment which is not using the Symfony service container. For example, you can turn your service into a library that can be used in Laravel, Phalcon, etc - your class has no idea how the dependencies are being injected.
By defining dependencies at the configuration level, you can use the configuration dumper to know which services are using which other services. For example, by injecting #mailer, then it's quite easy to work out from the service container where the mailer has been injected. On the other hand, if you do $container->get('mailer'), then pretty much the only way to find out where the mailer is being used is to do a find.
You'll be notified about missing dependencies when the container is compiled, instead of at runtime. For example, imagine you have defined a service, which you are injecting into a listener. A few months later, you accidentally delete the service configuration. If you are injecting the service, you'll be notified as soon as you clear the cache. If you inject the service container, you'll only discover the error when the listener fails because of the container cannot get the service. Sure, you could pick this up if you have thorough integration testing, but ... you have got thorough integration testing, haven't you? ;)
You'll know sooner if you are injecting the wrong service. For example, if you have:
public function __construct(MyService $my_service)
{
$this->my_service = $my_service;
}
But you've defined the listener as:
my_listener:
class: Whatever
arguments: [#my_other_service]
When the listener receives MyOtherService, then PHP will throw an error, telling you that it's receiving the wrong class. If you're doing $container->get('my_service') you are assuming that the container is returning the right class, and it can take a long time to figure out that its' not.
If you're using an IDE, then type hinting adds a whole load of extra help. If you're using $service = $container->get('service'); then your IDE has no idea what $service is. If you inject with
public function __construct(MyService $my_service)
{
$this->my_service = $my_service;
}
then your IDE knows that $this->my_service is an instance of MyService, and can offer help with method names, parameters, documentation, etc.
Your code is easier to read. All your dependencies are defined right there at the top of the class. If they are scattered throughout the class with $container->get('service') then it can be a lot harder to figure out.
Your code is easier to unit test. If you're injecting the service container, you've got to mock the service container, and configure the mock to return mocks of the relevant services. By injecting the services directly, you just mock the services and inject them - you skip a whole layer of complication.
Don't be fooled by the "it allows lazy loading" fallacy. You can configure lazy loading at configuration level, just by marking the service as lazy: true.
Personally, the only time injecting the service container was the best possible solution was when I was trying to inject the security context into a doctrine listener. This was throwing a circular reference exception, because the users were stored in the database. The result was that doctrine and the security context were dependent on each other at compile time. By injecting the service container, I was able to get round the circular dependency. However, this can be a code smell, and there are ways round it (for example, by using the event dispatcher), but I admit the added complication can outweigh the benefits.
Besides all disadvantages explained by others (no control over used services, run time compilation, missing dependencies, etc.)
There is one main reason, which breaks the main advantage of using DIC - Dependencies replacement.
If service is defined in library, you wont be able to replace it dependencies with local ones filling your needs.
Only this reason is strong enough, to not inject whole DIC. You just break whole idea of replacing dependencies since they are HARDCODED! in service;)
BTW. Don't forget to require interfaces in service constructor instead of specific classes as much as you can - again nice dependencies replacement.
EDIT: Dependencies replacement example
Service definition in some vendor:
<service id='vendor_service' class="My\VendorBundle\SomeClass" />
<argument type="service" id="vendor_dependency" />
</service>
Replacement in your app:
<service id='vendor_service' class="My\VendorBundle\SomeClass" />
<argument type="service" id="app_dependency" />
</service>
This allows you to replace vendor logic with your customized one, but don't forget to implement required class interface. With hardcoded dependencies you're not able to replace dependency in one place.
You can also override vendor_dependency service, but this will replace it in all places not only in vendor_service.
It is not a good idea because you're making your class dependent on the DI. What happens when some day you decide to pull out your class and use it on an entirely different project? Now I'm not talking about Symfony or even PHP, I'm talking generally. So in that case you have to make sure the new project uses the same kind of DI mechanism with the same methods supported or you get exceptions. And what happens if the project does not use DI at all, or uses some cool new DI implementation? You have to go through your whole codebase and change things to support the new DI. In large projects this can be problematic and costly, especially when you're pulling more than just one class.
It is best to make your classes as independent as possible. This means keeping the DI out of your usual code, something like a 3rd person who decides what goes where, points where the stuff should go, but doesn't go there and do it himself. This is how I understand it.
Although, as tomazahlin said, I agree that in Symfony projects in rare occasion it helps prevent circular dependencies. That's the only example where I'd use it and I'd make damn sure that's the only option.
Injecting the whole container is not a good idea in general. Well, it works, but why injecting the whole container, while you only need a few other services or parameters.
Sometimes you want to inject the whole container to avoid circular references, because if you inject the whole container, you get "lazy loading" of the services you require. An example would be doctrine entity listeners.
You can get the container from every class that is "Container Aware" or has access to the kernel.
I'm building a site that relies quite heavily on a third party API so I thought it would make sense to package up the API wrapper as a service, however I'm starting to find instances where it would be useful to have access to it outside of a controller such as in an entity repository.
Also related to that is it would be useful to be able to get access to config values outside of a controller (again such as in an entity repository).
Can anyone tell me if this is possible and if not is there a suggested approach to doing this kind of thing?
thanks for any help
The Symfony distribution relies heavily on dependency injection. This means that usually, dependencies are injected directly into your object via the constructor, the setters or via other means (like reflection over properties). Your API wrapper service is then a dependency for other objects of your application.
That being said, it would be rather difficult to inject this service in an entity repository's constructor because it already requires some other parameters and I think it would not be possible to inject them because of the way we request the repository for an entity.
What you could do is to create another service which will be responsible of doing the work you were about to do in the entity repository. This way, you will be able to inject the entity manager, which will be used to retrieve the entity repository, you custom service and also another service holding your configuration values (There are other ways to share configuration values).
In my use case, I use a Facebook helper service that wraps Facebook API calls. This service is then injected where I need it. My entity repository is only responsible of doing database calls so it receives only the arguments it needs and not the whole dependency. Thus, it will not receive the helper but rather only the arguments needed to do a request, for example, a Facebook user id. In my opinion, this is the way to do it since I think the entity repository should not have dependencies on such helper objects.
Here a small example using YAML as the configuration:
# app/config/config.yml
services:
yourapp.configuration_container:
class: Application/AcmeBundle/Common/ConfigurationContainer
# You could inject configurations here
yourapp.api_wrapper:
class: Application/AcmeBundle/Service/ApiWrapperService
# Inject other arguments if needed and update constructor in consequence
yourapp.data_access:
class: Application/AcmeBundle/Data/Access/DatabaseAccessService
arguments:
entityManager: "#doctrine.orm.entity_manager"
apiWrapperService: "#yourapp.api_wrapper"
configuration: "#yourapp.configuration_container"
# Application/AcmeBundle/Common/ConfigurationContainer.php
public ConfigurationContainer
{
public function __construct()
{
// Initialize your configuration values or inject them in the constructor
}
}
# Application/AcmeBundle/Service/ApiWrapperService.php
public ApiWrapperService
{
public function __construct()
{
// Do some stuff
}
}
# Application/AcmeBundle/Data/Access/DatabaseAccessService.php
public DatabaseAccessService
{
public function __construct(EntityManager $entityManager, ApiWrapperService $apiWrapperService, ConfigurationContainer $configuration)
{
...
}
}
The at sign (#) in the config.yml file means that Symfony should inject another service ,having the id defined after the at sign, and not a simple string. For the configuration values, as I said previously, there is other means to achieve the same goal like using parameters or a bundle extension. With a bundle extension, you could define the configuration values directly into the config.yml and your bundle would read them.
In conclusion, this should give you the general idea of injecting services. Here a small list of documentation on the subject. Alot of links use the XML service definition instead of the YAML definition but you should be able to understand them quite easily.
Symfony Official DI
Fabien Potencier's articles on DI
Richard Miller's articles on DI (Check in his blog for the other DI articles)
Take note that the configuration I'm giving is working for Beta1 of Symfony2. I didn't update yet to Beta2 so there could be some stuff not working as they are in the Beta2 version.
I hope this will help you defining a final solution to your problem. Don't hesitate to ask other questions if you want clarifications or anything else.
Regards,
Matt
I would wrap this kind of behavior in a Symfony service(like a manager).
i would not inject any parameters or logic into the entity repositories, as they should mainly be used to fetch data using object manager queries.
I would put the logic in the services and if the service , require a database access it will call the entity repository to fetch data.