we need to send the data to 2 topics which belongs to the two different servers. One is confluent offerings in Azure another is kafka cluster on azure VM.
Now we are using KafkaRoutingTemplate to send message to the 2 different Kafka Offerings.
But we create 2 producers/factory via config class ( coding) - Is there any way to configure 2 producers via application.yml file instead of coding
You can override the bootstrap servers in one of the templates; they can use the same producer factory, but you have to define the templates as beans.
/**
* Create an instance using the supplied producer factory and properties, with
* autoFlush false. If the configOverrides is not null or empty, a new
* {#link DefaultKafkaProducerFactory} will be created with merged producer properties
* with the overrides being applied after the supplied factory's properties.
* #param producerFactory the producer factory.
* #param configOverrides producer configuration properties to override.
* #since 2.5
*/
public KafkaTemplate(ProducerFactory<K, V> producerFactory, #Nullable Map<String, Object> configOverrides) {
Related
I see people duplicating code to configure kafka for each cluster :(
Is it really necessary to configure Consumer Factory and Producer Factory with different settings every time? And do not use Spring Boot Kafka starter in any way?
If you use similar properties for each cluster, you can override the bootstrap.servers property for each #KafkaListener and/or create multiple #KafkaTemplates with the same override.
Then, the same factories can be used.
See https://docs.spring.io/spring-kafka/docs/current/reference/html/#annotation-properties
#KafkaListener(topics = "myTopic", groupId = "group", properties = {
"max.poll.interval.ms:60000",
ConsumerConfig.MAX_POLL_RECORDS_CONFIG + "=100"
})
/**
* Create an instance using the supplied producer factory and properties, with
* autoFlush false. If the configOverrides is not null or empty, a new
* {#link DefaultKafkaProducerFactory} will be created with merged producer properties
* with the overrides being applied after the supplied factory's properties.
* #param producerFactory the producer factory.
* #param configOverrides producer configuration properties to override.
* #since 2.5
*/
public KafkaTemplate(ProducerFactory<K, V> producerFactory, #Nullable Map<String, Object> configOverrides) {
Spring Boot auto-configuration is by convention for the common microservices use-case: one thing, but simple and clear.
What you are asking is out of Spring Boot scope: the properties configuration is applied only for one ConsumerFactory and one ProducerFactory. As long as you need to connect to different clusters you are on your own. Having a custom ConsumerFactory bean will lead Spring Boot auto-configuration to back off.
You probably can look into a child ApplicationContext configuration with its own provided set of properties. But will it be easier to just have custom configuration for each cluster as you would do without Spring Boot with just a plain Apache Kafka client?
I have no idea if there is some federation solution for Apache Kafka. But that's still out of Spring for Apache Kafka and Spring Boot scope.
I'm refactoring the code in one of my Controller to put it in a service.
In the Controller the entity manager is targeting a db connection called legacy.
The problem is, I did not injected the Entity Manager in my service but just the OrderRepository.
How can I target the good db connection with the repository in my service without the Entity Manager?
OrderController
// In my Controller
$em = $this->getDoctrine()->getManager('legacy');
$em->persist($order);
$em->flush();
OrderService
// In my Service
public function __construct(OrderRepository $orderRepository)
{
$this->orderRepository = $orderRepository;
}
public function updateOrderStatus(Order $order)
{
// some code ...
$this->orderRepository->save($order);
}
Based on some comments it would appear that the basic issue is having multiple entity managers. Nowadays there is a great deal of automated functionality which works well for one entity manager but no so much for multiple managers.
You basically need to define your services manually.
# services.yaml
# assume we have
# doctrine.orm.default_entity_manager and
# doctrine.orm.legacy_entity_manager
# already defined through configuration
# define legacy repository
order_repository.legacy:
class: Whatever\Repository\OrderRepository
factory: ['doctrine.orm.legacy_entity_manager', 'getRepository']
arguments:
- 'Whatever\Entity\Order'
# define default repository
order_repository.default:
class: Whatever\Repository\OrderRepository
factory: ['doctrine.orm.default_entity_manager', 'getRepository']
arguments:
- 'Whatever\Entity\Order'
# then your service
Whatever\Service\MyService:
'#order_repository.legacy'
And you should be good to go. Note that your repository needs to extend EntityRepository and not the doctrine bundle's ServiceEntityRepository.
And if you feel this is too much work then just inject the doctrine bundle's entity manager registry and do what the ControllerTrait::getDoctrine($name) does.
You must forgot that you can always get the entity manager inside your repository just like this:
$em = $this->getEntityManager();
then you can use it normally calling persist, flush etc.
Note that the Repository class itself is agnostic to functionalities of the database access layer like insert or update so it should call entity manager to execute them.
I have setup two entity managers in doctrine.yml
I have to inject repository into service but problem is repository always taken an default entity manager.
How should I give specific entity manager to repository.
In symfony 4 we can treat repository as service using ServiceEntityRepository
You can try to inject Doctrine\Common\Persistence\ManagerRegistry in your construct. And then use $managerRegistry->getManager('your_connection_name');
For example:
//use Doctrine\Common\Persistence\ManagerRegistry;
private $connection;
function __construct(ManagerRegistry $em)
{
$this->connection = $em->getManager('your_connection_name');
}
Rather than lazy/auto loading them, you'd need to setup each as a named service and explicitly configure their loading in your services.yaml file.
I'm using the Symfony web test case to test my endpoints usually. In this case I just wanted to test a single repository call to persist a new entity so I wanted to avoid using the web test client.
I tested using the web test client first and POSTing to the endpoint will create the entity in the SQLite database using the repository. However if I get the repository class in my test and directly call the creation method it fails on persist() with the exception:
Entity of type MyEntity is missing an assigned ID for field 'id'. The identifier generation strategy for this entity requires the ID field to be populated before EntityManager#persist() is called. If you want automatically generated identifiers instead you need to adjust the metadata mapping accordingly.
I've updated the schema using
app/console doctrine:schema:update --force --env=test
But app/console doctrine:schema:validate --env=test still tells me the database is not up to date. This would suggest a problem with the mapping, but if there was a problem with the mapping why would it work in production (which uses MySql) or indeed why would it work if I call it using the web test client?
The doctrine mapping for the ID is:
/**
* #ORM\Id
* #ORM\Column(type="integer")
* #ORM\GeneratedValue(strategy="IDENTITY")
*
* #var int
*/
private $id;
I created a bundle to manage user-group-permissions. I want it to make project independent by moving it into the vendors directory.
To make this bundle immutable I moved the users data into a usermeta bundle.
The main bundle contains username and email only about the user, and usermeta contains everything else (name, birthdate etc. whatever a project require).
The problem is the main user bundle intended to belong to a core bundle group, from which every project using the same.
The user-usermeta relation now created a dependency. So every project will need it.
My question is
- How can I standardize its format, to enforce in every project create it properly.
- How can I make this dependency optional (preferred)
I suggest you only handle a UserInterface instead of a User entity in your bundle.
In case of Symfony UserInterface doesn't implement everything you need (username but no email), create your own UserInterface in your bundle :
namespace YourDomain\YourBundle\Interface;
use Symfony\Component\Security\Core\User\UserInterface as BaseInterface;
/**
* UserInterface is the interface that user classes must implement.
*/
interface UserInterface extends BaseInterface
{
/**
* Returns the email address of the user.
*
* #return string The user email address
*/
function getEmail();
}
And then, in the projects using your bundle, your User entity must implements your specific interface instead of Symfony UserInterface.