We have been using spring kafka 2.1 in our projects .There is a use case to ingest a message to all partitions in a topic so all consumers can identify an event.Is there a way to use kafkaTemplate to send a single record to all partitions of a kafka topic?.
The Partitioner has an architecture like this:
/**
* Compute the partition for the given record.
*
* #param topic The topic name
* #param key The key to partition on (or null if no key)
* #param keyBytes The serialized key to partition on( or null if no key)
* #param value The value to partition on or null
* #param valueBytes The serialized value to partition on or null
* #param cluster The current cluster metadata
*/
public int partition(String topic, Object key, byte[] keyBytes, Object value, byte[] valueBytes, Cluster cluster);
So, as you see there is no out-of-the-box way to determine several partitions at once.
What I only can suggest is something like send the same message to all the partitions manually in the loop.
What you are asking is just out of the Apache Kafka architecture.
Related
If entity A contains multiple entity B and has cascade:persist, how to reuse existing entities B when persisting ?
B entity has one primary key, an integer, and the id of the A parent. The only data it contains is the primary key.
Example:
A has 2 B entities, identified by their id, 14 and 23.
A.Bs = [{id=14, AId=A.id}, {id=23, AId=A.Id}]
Now if I modify this managed entity, to add a B entity to A, with id = 56.
A.Bs = [{id=14, AId=A.id}, {id=23, AId=A.Id}, {id=56}]
Relationships
Entity A
/**
* #var B[]|ArrayCollection
*
* #ORM\OneToMany(targetEntity="B", mappedBy="A", cascade={"persist", "remove"}, orphanRemoval=true)
* #Assert\Valid
*/
private $Bs;
Entity B
/**
* #var A
*
* #ORM\ManyToOne(targetEntity="A", inversedBy="Bs")
* #ORM\JoinColumn(name="A_id", referencedColumnName="A_id")
* #Assert\NotNull()
*/
private $A;
If I try to persist I get Integrity constraint violation, because Doctrine tries to persist the existing entities, that have id 14 and 23.
I understand this is expected behaviour, but how can I make it persist new entities, and reuse existing ones ?
More details:
If I get an existing entity A with $em->find($id) and directly use persist and flush, I will get UniqueConstraintException because it tries to persist the already persisted B entities.
Example code:
/** #var A $existingEntityA */
$existingEntityA = $this->getEntity($id);
$this->serializerFactory->getComplexEntityDeserializer()->deserialize(json_encode($editedEntityADataJson), A::class, 'json', ['object_to_populate' => $existingEntityA]);
$this->entityValidator->validateEntity($existingEntityA);
$this->_em->flush();
Example error : Integrity constraint violation: 1062 Duplicate entry '777111' for key 'PRIMARY'
If I understand your example properly - you're doing something like this:
$b = new B();
$b->setId(56);
$a->getB()->add($b);
and you having a row with primary key 56 into database table that is represented by B?
If my assumption is correct - it is wrong way to go. Reason is that Doctrine internally stores so called "identity map" that keeps track of all entities that either being fetched from database or persisted by calling EntityManager::persist(). Every entity that is scheduled for commit but not available into identity map is considered as "new" and scheduled for insertion. If row with same primary key is already available in database - you're receiving UniqueConstraintException.
Doctrine doesn't handle a case "let me look if there is an entity with such primary key in database" by itself because it will hurt performance significantly and is not needed in most cases. Each such test will result into database query, imagine if you will have thousands of such entities. Since Doctrine doesn't know business logic of your application - it will spend even more resources with attempts to guess optimal strategy so this is intentionally left out of scope.
Correct way for you would be to get your entity by itself before adding to collection:
$newB = $em->find(B::class, 56);
if ($newB) {
$a->getB()->add($newB);
}
In this case new entity will internally have "managed" status and will be correctly handled by Doctrine at a time of commit.
I am trying to send a message to a topic with 4 partitions.And I want the message to go the partition as decided by the DefaultPartitioner(which uses the hash of the key)
kafkaTemplate.sendDefault(DefaultPartitioner(job.getId()),job.getId(),job);
I am not sure how to make the kafkaTemplate use the DefaultPartitioner to get the partition number.
Could someone please help me out with this.
Let's start from JavaDocs!
/**
* The default partitioning strategy:
* <ul>
* <li>If a partition is specified in the record, use it
* <li>If no partition is specified but a key is present choose a partition based on a hash of the key
* <li>If no partition or key is present choose a partition in a round-robin fashion
*/
public class DefaultPartitioner implements Partitioner {
/**
* Send the data to the default topic with the provided key and no partition.
* #param key the key.
* #param data The data.
* #return a Future for the {#link SendResult}.
*/
ListenableFuture<SendResult<K, V>> sendDefault(K key, V data);
So, if you would like to rely on the DefaultPartitioner, all you need is a key for the record. Therefore just use that particular KafkaTemplate method:
kafkaTemplate.sendDefault(job.getId(),job);
I have an interface SupplierInterface with 2 implementations: B2BSupplier (a Doctrine entity), RetailSupplier (a static object).
<?php
namespace MyBundle\Model;
interface SupplierInterface {
const B2B = 'B2B';
const RETAIL = 'Retail';
/**
* #return string
*/
public function getSupplierType();
/**
* #return string
*/
public function __toString();
}
Another entity, Supply has a many-to-one relationship with a Supplier. Normally this isn't problematic. But because RetailSupplier is not a Doctrine entity, I'm a bit flummoxed about how to proceed.
Supply looks like this:
<?php
namespace MyBundle\Entity;
use Doctrine\ORM\Mapping as ORM;
use Doctrine\Common\Collections\ArrayCollection;
use Symfony\Component\Validator\Constraints as Assert;
use Gedmo\Blameable\Traits\BlameableEntity;
use Gedmo\Timestampable\Traits\TimestampableEntity;
/**
* Supply
*
* #ORM\Table(name="cir_supply")
* #ORM\Entity()
*/
class Supply
{
use BlameableEntity;
use TimestampableEntity;
/**
* #var int
* #ORM\Column(name="id", type="integer")
* #ORM\Id
* #ORM\GeneratedValue(strategy="AUTO")
*/
protected $id;
/**
* #ORM\ManyToOne(targetEntity="B2BSupplier")
* #ORM\JoinColumn(name="supplier_id", referencedColumnName="id", nullable=true)
*/
protected $supplier; // <-- PROBLEM, since supplier could be B2BSupplier entity, or it could be vanilla object RetailSupplier
/**
* #ORM\ManyToOne(targetEntity="Chemical", inversedBy="supplies")
* #ORM\JoinColumn(name="chemical_id", referencedColumnName="id", nullable=false)
*/
protected $chemical;
/**
* #ORM\Column(name="external_id", type="string")
*/
protected $externalId;
//getters and setters ...
}
How do I specify a Doctrine relationship when that relationship might not always be valid?
From my experience I'm 99% sure you can't do what you want in your current setup. That being said, there are a few workarounds I can think of. Also before I go into the workarounds. You should think if you really want OneToOne relation on 'supplier' or will ManyToOne work better. OneToOne has some Lazy loading issues and also Workaround 3 work better with ManyToOne.
Workaround 1:
Remove the relation and make the supplier filed contain the id, without having a relation defined.
Extend SupplierRepository 'find' method to handle the cases where id is
2.1 'null' there is no relation in witch case it returns RetailSupplier
2.2 call parent::find for all other cases
2.3 Optional: if null relations are required change 2.1 to use '0' instead of null (adds con 3)
Pros:
fast to achieve from your current setup
keep database foreign key (if step 2.3 is ignored)
Cons:
hidden behavior of the 'find' method
you loose the your doctrine relation
not scalable for other types of Suppliers
source of the information is split between the app and the database
if step 2.3 is required, you loose database foraign key ('0' will not be a foraign key)
Workaround 2:
Modify getSupplier to return RetailSupplier if $this->supplier is null
Modify setSupplier to set null if $supplier is instance of RetailSupplyer
Optinal: Change the first 2 steps to handle '0' as RetailSupplyer and 'null' as no relation
Pros:
fast to achieve from your current setup
keep database foreign key (if step 3 is ignored)
keep doctrine relation
Cons:
hidden behavior of the setter and getter
not scalable for other types of Suppliers
if step 3 is required, you loose database foraign key ('0' will not be a foraign key)
source of the information is split between the app and the database
Workaround 3 (doctrine inheritance mapping):
Create an abstract (called Supplier) this will be inherited by RetailSupplyer and B2BSupplier
Add inheritance metadata to Supplier abstract something like this
Create an entity for RetailSupplyer and a database table with one single line to start (the first RetailSupplier)
Change your database to match your inheritance mapping settings (for more info http://docs.doctrine-project.org/projects/doctrine-orm/en/latest/reference/inheritance-mapping.html)
Change your relation to ManyToOne on $supplier and make it to point to Supplier
Pros:
source of the information is only the database
no hidden behavior in your code
scalable for other types of suppliers and other more retail suppliers
Cons:
harder to achieve from your current setup (database changes, new doctrine setup, possibly some refactor)
pros/cons: Depending on the selected inheritance type you can have full relation path in your database (with foraign key), or you can have no relations. This is up to you ;) after you read the documentation for inheritance mapping.
PS: If I had to choose i will go with Workaround 3. It is hardest to achieve, but solid do it.
Hope this helps and happy coding
Alexandru Cosoi
I am facing a weird problem relating to UUIDs.
I have developed a REST API using Symfony2+FOSRestBundle+JMSSerializer. As I need to update some tables from two sources I thought of using UUID as primary key for one entity.
I did a doctrine:mapping:import to generate entities in my Symfony project. Everything correct. I ended up with the following entity (only exposing the key field and generated getter for simplicity):
<?php
namespace Stardigita\TgaAPIBundle\Entity;
use Doctrine\ORM\Mapping as ORM;
/**
* TgaBookings
*
* #ORM\Table(name="tga_bookings", indexes={[...]})
* #ORM\Entity
*/
class TgaBookings
{
/**
* #var string
*
* #ORM\Column(name="book_cd_booking_UUID", type="blob", length=16, nullable=false)
* #ORM\Id
* #ORM\GeneratedValue(strategy="IDENTITY")
*/
private $bookCdBookingUuid;
/**
* Get bookCdBookingUuid
*
* #return string
*/
public function getBookCdBookingUuid()
{
return $this->bookCdBookingUuid;
}
...
No setter was generated. I can still do it myself and I will, as I will need to know the key beforehand.
The data for this field is correctly stored in the table as a BINARY(16). When I recover the data calling the REST GET method, I get this:
[
{
"book_cd_booking_uuid": "Resource id #1244",
"book_cd_booking": 8,
....
My question is: how can I get the actual data from the field?
I suppose something has to be done in the field getter, but I tried some solutions without any success.
Thanks.
UPDATE:
I've managed to get the actual data logged, modifying the getBookCdBookingUuid method this way:
/**
* Get bookCdBookingUuid
*
* #return string
*/
public function getBookCdBookingUuid()
{
return bin2hex($this->bookCdBookingUuid);
}
and changed the type to "guid" in the property annotation:
/**
* #var string
*
* #ORM\Column(name="book_cd_booking_UUID", type="guid", length=16, nullable=false)
* #ORM\Id
* #ORM\GeneratedValue(strategy="IDENTITY")
*/
private $bookCdBookingUuid;
I have represented the hex UUID correctly in the log before returning the results in the controller:
[2014-11-03 19:52:07] app.ERROR: 1046684e5f6711e4a09f00089bce936a [] []
But still getting an exception relating UTF invalid characters:
request.CRITICAL: Uncaught PHP Exception RuntimeException: "Your data could not be encoded because it contains invalid UTF8 characters." at /var/www/tga_api/vendor/jms/serializer/src/JMS/Serializer/JsonSerializationVisitor.php line 36 {"exception":"[object] (RuntimeException: Your data could not be encoded because it contains invalid UTF8 characters. at /var/www/tga_api/vendor/jms/serializer/src/JMS/Serializer/JsonSerializationVisitor.php:36)"} []
Also I got no response from the service. A 500 error is returned.
Please, I need to solve this issue. Any ideas are welcome.
Thanks.
GeneratedValue
I notice you're using the annotation #ORM\GeneratedValue(strategy="IDENTITY") for the UUID property. IDENTITY means the database should/will use auto-increments, which shouldn't be done when using UUIDs. Please change it to #ORM\GeneratedValue(strategy="NONE") or remove it completely.
Conversion
The string form of a UUID (like 01234567-89ab-cdef-0123-456789abcdef) should be converted to binary when it's persisted in the database, and converted back when fetched from the database.
The easiest way to do this is to introduce a custom type. See here for an example.
Bugs
Doctrine (even master/2.5) has some issues with using UUIDs in associations. I'm attempting to fix these issues in PR #1178.
If you need UUIDs in associations and can't wait till it's fixed, then use regular integer ids and have the UUID is a separate column.
To simplify, two entities are defined: User and Comment. User can post many comments and every comment has only one user assigned, thus Comment entity has:
/**
* #var \Frontuser
*
* #ORM\ManyToOne(targetEntity="Frontuser")
* #ORM\JoinColumns({
* #ORM\JoinColumn(name="ownerUserID", referencedColumnName="id")
* })
*/
private $owneruserid;
However, when in action:
$orm = $this->getDoctrine()->getManager();
$repo = $orm->getRepository('CompDBBundle:Comment');
$repo->findBy(array('owneruserid' => $uid);
Error occured, that there's no such field like owneruserid.
How can I fetch all the user's comments then? The same happens to similar relations in my DB - looks likes you cannot run find() with foreign keys as parameters. I believe a function $user->getComments() should be automatically generated/recognised by Doctrine to allow efficient, quick access to related entities.
The example's simple but, what if there are more entities related to my User in the same way? Do I have to declare repositories for each and try to fetch them by it's owneruserid foreign keys?
Using doctrine, when you define a related entity it's type is the entity class (in this case FrontUser). Therefore firstly your related entity variable name is misleading. It should be e.g.
private $ownerUser;
Then, in order to do a findBy on a related entity field you must supply an entity instance e.g.
$orm = $this->getDoctrine()->getManager();
$userRepo = $orm->getRepository('CompDBBundle:FrontUser');
$user = $userRepo->findById($uid);
$commentRepo = $orm->getRepository('CompDBBundle:Comment');
$userComments = $commentRepo->findByOwnerUser($user);
If you don't have or want to retrieve the user entity you could use a DQL query with the 'uid' as a parameter instead.