I am very new to Drupal and attempting to build a module that will allow admins to tag nodes with keywords to boost nodes to the top of the search results.
I have a separate DB table for the keywords and respective node IDs. This table is UNIONed with the search_index table via hook_query_alter...
function mos_search_result_forcer_query_alter(QueryAlterableInterface &$query) {
if (get_class($query) !== 'PagerDefault') { //<< check this because this function will mod all queries elsewise
return;
}
// create unioned search index result set...
$index = db_select('search_index', 's');
$index->addField('s', 'sid');
$index->addField('s', 'word');
$index->addField('s', 'score');
$index->addField('s', 'type');
$msrfi = db_select('mos_search_result_forcer', 'm');
$msrfi->addField('m', 'nid', 'sid');
$msrfi->addField('m', 'keyword', 'word');
$msrfi->addExpression('(SELECT MAX(score) + m.id / (SELECT MAX(id) FROM {mos_search_result_forcer}) FROM {search_index})', 'score');
$msrfi->addExpression(':type', 'type', array(':type' => 'node'));
$index->union($msrfi);
$tables =& $query->getTables();
$tables['i']['table'] = $index;
return $query;
}
Drupal then generates the almost correct query...
SELECT
i.type AS type, i.sid AS sid, SUM(CAST('10' AS DECIMAL) * COALESCE(( (12.048628015788 * i.score * t.count)), 0) / CAST('10' AS DECIMAL)) AS calculated_score
FROM (
SELECT
s.sid AS sid, s.word AS word, s.score AS score, s.type AS type
FROM
search_index s
UNION SELECT
m.nid AS sid, m.keyword AS word, (
SELECT
MAX(score) + m.id / (SELECT MAX(id) FROM mos_search_result_forcer)
FROM
search_index
) AS score, 'node' AS type
FROM
mos_search_result_forcer m
) i
INNER JOIN node n ON n.nid = i.sid
INNER JOIN search_total t ON i.word = t.word
INNER JOIN search_dataset d ON i.sid = d.sid AND i.type = d.type
WHERE (n.status = '1')
AND( (i.word = 'turtles') )
AND (i.type = 'node')
/* this is the problem line... */
AND( (d.data LIKE '% turtles %' ESCAPE '\\') )
/* ...end problem line */
GROUP BY i.type, i.sid
HAVING (COUNT(*) >= '1')
ORDER BY calculated_score DESC
LIMIT 10 OFFSET 0
...I need that "problem line" to read...
AND( (d.data LIKE '% turtles %' ESCAPE '\\') OR (d.sid IN (SELECT nid FROM mos_search_result_forcer)) )
...what hook can I use to add that OR condition?
I don't want to hack Drupal's core.
I don't want to change the union/subqueries (not my decision).
I will optimize queries later - functionality is more important.
Thanks, smart people!
The basic principle is to grab the conditions array, loop through and find the index for the problem condition, remove it, then re-add an identical condition along with the new one as part of a db_or()
This is un-tested and most likely won't work verbatim, but it should give you a starting point:
$conditions =& $query->conditions();
$index = FALSE;
$removed_condition = NULL;
for ($i = 0, $l < count($conditions); $i < $l; $i++) {
if ($conditions[$i]['field'] == 'd.data' && $conditions[$i]['operator'] == 'LIKE') {
$index = $i;
$removed_condition = $condition;
break;
}
}
if ($index !== FALSE) {
unset($conditions[$index]);
$sub_query = db_select('mos_search_result_forcer')->fields('mos_search_result_forcer', array('nid'));
$new_condition = db_or()
->condition('d.data', $removed_condition['value'], 'LIKE')
->condition('d.sid', $sub_query, 'IN');
$query->condition($new_condition);
}
Thanks to some helpful suggestions from #Clive with hook_module_implements_alter, and a lot of trial and error, I have finally solved this issue.
Here is the final code...
function mos_search_result_forcer_module_implements_alter(&$imps, $hook) {
if ($hook !== 'query_alter' || !array_key_exists('mos_search_result_forcer', $imps)) {
return;
}
$imp = $imps['mos_search_result_forcer'];
unset($imps['mos_search_result_forcer']);
$imps['mos_search_result_forcer'] = $imp;
}
function mos_search_result_forcer_query_alter(QueryAlterableInterface &$query) {
if (get_class($query) !== 'PagerDefault') { //<< check this because this function will mod all queries elsewise
return;
}
// create unioned search index result set...
$index = db_select('search_index', 's');
$index->addField('s', 'sid');
$index->addField('s', 'word');
$index->addField('s', 'score');
$index->addField('s', 'type');
$msrfi = db_select('mos_search_result_forcer', 'm');
$msrfi->addField('m', 'nid', 'sid');
$msrfi->addField('m', 'keyword', 'word');
$msrfi->addExpression('(SELECT MAX(score) + m.id / (SELECT MAX(id) FROM {mos_search_result_forcer}) FROM {search_index})', 'score');
$msrfi->addExpression(':type', 'type', array(':type' => 'node'));
$index->union($msrfi);
$tables =& $query->getTables();
$tables['i']['table'] = $index;
// needs special "or" condition to keep from filtering out forced resutls...
class MSRFPagerDefaultHelper extends PagerDefault { //<< override to gain access to protected props
static function msrfHelp(PagerDefault &$pagerDefault) {
$searchQuery =& $pagerDefault->query;
MSRFSearchQueryHelper::msrfHelp($searchQuery);
}
}
class MSRFSearchQueryHelper extends SearchQuery { //<< override to gain access to protected props
static function msrfHelp(SearchQuery &$searchQuery) {
$conditions =& $searchQuery->conditions;
$condition = db_or()->condition($conditions)->condition('d.sid', db_select('mos_search_result_forcer')->fields('mos_search_result_forcer', array('nid')), 'IN');
$searchQuery->conditions = $condition;
}
}
MSRFPagerDefaultHelper::msrfHelp($query);
return $query; //<< i don't think this is needed as var is reffed - just for good measure, i guess
}
Related
I am currently trying to get data from a repository to show it in a Bootgrid, but I always exceed the maximum execution time (120 seconds).
I use the following code:
In my Javascript:
$params.table.bootgrid({
ajax: true,
url: $params.authoritiesDataPath //the url is correct here, I verified it
})
In my controller:
/**
* Authorities Data
*
* #param Request $request
* #return JsonResponse
*/
public function dataAction(Request $request)
{
$this->denyAccessUnlessGranted(RoleVoterHelper::SECTION_COMPANY_VIEW);
try {
$data = $this->getDoctrine()->getRepository('BugTrackerModelBundle:Authority')->findByParameters(
$request->request->all()
);
} catch (\Exception $e) {
$data = [
'status' => 'error',
'error' => $e->getMessage(),
'rows' => [],
'current' => 1,
'rowCount' => 0,
'total' => 0,
];
}
return new JsonResponse($data);
}
in my repository:
public function findByParameters(array $parameters)
{
$queryBuilder = $this->createQueryBuilder('a')
->select('COUNT(a.id)');
if (!empty($parameters['searchPhrase'])) {
$queryBuilder->where('a.name LIKE :search')
->setParameter('search', '%'.$parameters['searchPhrase'].'%');
}
$parameters['rows'] = array();
$parameters['current'] = isset($parameters['current']) ? (int)$parameters['current'] : 0;
if ($parameters['total'] = (int)$queryBuilder->getQuery()->getSingleScalarResult()) {
$queryBuilder->select('a.id', 'a.name', 'COUNT(DISTINCT c.id) as companies',
'COUNT(DISTINCT u.id) as users', 'COUNT(DISTINCT dl.id) as deviceLists',
'COUNT(DISTINCT d.id) as devices', 'a.name as authority', 'a.enabled')
->leftJoin('BugTrackerModelBundle:Company', 'c', Join::WITH, 'c.authority = a.id')
->leftJoin('BugTrackerModelBundle:Device', 'd', Join::WITH, 'd.authority = a.id')
->leftJoin('BugTrackerModelBundle:Device\DeviceList', 'dl', Join::WITH, 'dl.authority = a.id')
->leftJoin('BugTrackerModelBundle:User', 'u', Join::WITH, 'u.company = c.id')
->groupBy('a.id');
if (!empty($parameters['sort'])) {
$order = reset($parameters['sort']) ?: 'ASC';
switch (key($parameters['sort'])) {
case 'name':
$queryBuilder->orderBy('c.name', $order);
break;
case 'users':
$queryBuilder->orderBy('users', $order);
break;
case 'company':
$queryBuilder->orderBy('companies', $order);
break;
case 'enabled':
$queryBuilder->orderBy('a.enabled', $order);
break;
default:
$queryBuilder->orderBy('c.id', $order);
}
}
if (isset($parameters['rowCount']) && $parameters['rowCount'] > 0) {
$queryBuilder->setFirstResult(($parameters['current'] - 1) * $parameters['rowCount'])
->setMaxResults($parameters['rowCount']);
}
$parameters['rows'] = $queryBuilder->getQuery()->getArrayResult();
}
return $parameters;
}
I tried returning arrays pretty much everywhere to find where the loop was (I'm guessing that it's a loop since my code is usually very fast), and it seems to come from the following line
$parameters['rows'] = $queryBuilder->getQuery()->getArrayResult();
I tried returning $queryBuilder->getQuery() and it took a few seconds, so the issue is with getArrayResult
The query it returned when I printed the sql:
SELECT a0_.id AS id_0, a0_.name AS name_1, COUNT(DISTINCT c1_.id) AS sclr_2, COUNT(DISTINCT u2_.id) AS sclr_3, COUNT(DISTINCT d3_.id) AS sclr_4, COUNT(DISTINCT d4_.id) AS sclr_5, a0_.name AS name_6, a0_.enabled AS enabled_7 FROM authority a0_ LEFT JOIN client_company c1_ ON (c1_.authority_id = a0_.id) LEFT JOIN device d4_ ON (d4_.authority_id = a0_.id) LEFT JOIN device_list d3_ ON (d3_.authority_id = a0_.id) LEFT JOIN user u2_ ON (u2_.company_id = c1_.id) GROUP BY a0_.id LIMIT 5 OFFSET 0
Here is the "explain" when I ran the query in PhpMyAdmin:
Explain
I really don't understand why it takes so much time, I never had issues with getting data from my database and there is no loop in all that code that could cause it. Is there any way for me to test other things to understand why it takes so much time and change it?
have you indexed the correct columns? this should reduce the querying time, additionally, it could be that the results set is so complex it's taking a while to hydrate the array which is causing the time duration you're seeing.
Looking at the query produced, you have 4 joins in it and it hydrates 5 entities BugTrackerModelBundle:Authority, BugTrackerModelBundle:Company, BugTrackerModelBundle:Device, BugTrackerModelBundle:Device\DeviceList and BugTrackerModelBundle:User.
That `s a lot of joins,as "the process of hydration becomes extremely expensive when more than 2 LEFT JOIN operations clauses". See Reference. In short, the result set is large and doctrine takes too long to map it to Entities.
My assumption is that the ORM is taking too long to normalize the result set returned by the query.
My advice is to split the query to 2 with maximum 2 joins each:
$queryBuilder = $this->createQueryBuilder('a')
->select('a.id', 'a.name',
'COUNT(DISTINCT c.id) as companies',
'COUNT(DISTINCT u.id) as users',
'a.name as authority', 'a.enabled')
->leftJoin('BugTrackerModelBundle:Company', 'c', Join::WITH, 'c.authority = a.id')
->leftJoin('BugTrackerModelBundle:User', 'u', Join::WITH, 'u.company = c.id')
->groupBy('a.id');
$queryBuilder = $this->createQueryBuilder('a')
->select('a.id', 'a.name',
'COUNT(DISTINCT dl.id) as deviceLists',
'COUNT(DISTINCT d.id) as devices', 'a.name as authority', 'a.enabled')
->leftJoin('BugTrackerModelBundle:Device', 'd', Join::WITH, 'd.authority = a.id')
->leftJoin('BugTrackerModelBundle:Device\DeviceList', 'dl', Join::WITH, 'dl.authority = a.id')
->groupBy('a.id');
Hope this helps.
I'm using DBAL and I want to execute multiple insert query. But I have the problem: bindValue() method not working in loop. This is my code:
$insertQuery = "INSERT INTO `phonebook`(`number`, `company`, `user`) VALUES %s
ON DUPLICATE KEY UPDATE company=VALUES(company), user=VALUES(user)";
for ($i = 0; $i < count($data); $i++) {
$inserted[] = "(':number', ':company', ':user')";
}
$insertQuery = sprintf($insertQuery, implode(",", $inserted));
$result = $db->getConnection()->prepare($insertQuery);
for ($i = 0; $i < count($data); $i++) {
$result->bindValue($data[$i]["number"]);
$result->bindValue($data[$i]["company"]);
$result->bindValue($data[$i]["user"]);
}
$result->execute();
As result I received one-line table with fields: :number, :company, :user.
What am I doing wrong?
Thanks a lot for any help!
The problem you're having is that your binding has no way to determine to which placeholder it should be doing the binding with. To visualize it better, think on the final DBAL query you're generating:
INSERT INTO `phonebook`(`number`, `company`, `user`) VALUES
(':number', ':company', ':user'),
(':number', ':company', ':user'),
(':number', ':company', ':user');
When you do the binding, you're replacing all the parameters at the same time, ending up with a single row inserted.
One possible solution would be to give different parameter names to each row and then replace each one accordingly.
It would look like something similar to this:
public function randomParameterName()
{
return uniqid('param_');
}
...
$parameters = [];
for ($i = 0; $i < count($data); $i++) {
$parameterNames = [
'number' => $this->randomParameterName(),
'company' => $this->randomParameterName(),
'user' => $this->randomParameterName(),
];
$parameters[$i] = $parameterNames;
$inserted[] = sprintf("(':%s', ':%s', ':%s')",
$parameterNames['number'],
$parameterNames['company'],
$parameterNames['user']
);
}
$insertQuery = sprintf($insertQuery, implode(",", $inserted));
$result = $db->getConnection()->prepare($insertQuery);
foreach ($parameters as $i => $parameter) {
$result->bindValue($parameter['number'], $data[$i]["number"]);
$result->bindValue($parameter['company'], $data[$i]["company"]);
$result->bindValue($parameter['user'], $data[$i]["user"]);
}
You could probably extend your $data variable and incorporate the new parameter names into it. This would remove the need of yet another array $parameters to hold reference to the newly created parameter names.
Hope this helps
There is another alternative:
$queryStart = "INSERT INTO {$tableName} (" . implode(', ', array_keys($buffer[0])) . ") VALUES ";
$queryRows = $params = $types = [];
foreach ($rowBuffer as $row) {
$rowQuery = '(' . implode(', ', array_fill(0, count($row), '?')) . ')';
$rowParams = array_values($row);
list($rowQuery, $rowParams, $types) = SQLParserUtils::expandListParameters($rowQuery, $rowParams, $types);
$queryRows[] = $rowQuery;
$params = array_merge($params, $rowParams);
}
$query = $queryStart . implode(', ', $queryRows);
$connection->executeQuery($query, $params, $types);
Two doctrine2 entities, Photo and Tag, are linked by a many-to-many relationship, and mapped accordingly.
Each tag has a key and a value, so an example key is 'photo-type' and an example value 'people'.
I have created a custom repository PhotoRepository.php, to facilitate easy searching for photos with either an array (or a comma-separated list*) of tag pairs, extract below:
public function getQueryByTags($tags = null, $limit =0, $start =-1, $boolean ="and")
{
...
$qb->select('p')
->from('MyBundle\Entity\Photo', 'p')
->join('p.tags','t');
# ->join('a.tags','t')
if ($boolean == "and") { $qb->where('1 = 1'); }
$i = 1;
foreach ($tags as $tag) {
if ($boolean == "and") {
$qb->andWhere('t.key = ?'.$i.' AND t.value = ?'.($i+1));
} elseif ($boolean == "or") {
$qb->orWhere('t.key = ?'.$i.' AND t.value = ?'.($i+1));
}
$qb->setParameter($i, $tag['key'])->setParameter(($i+1), $tag['value']);
$i += 2;
}
...
return $qb->getQuery();
}
This works fine for a single tag. However, once tags are multiple (e.g. searching for 'photo-type'=>'people', 'person'=>'Bob'), the boolean logic breaks down and no results are return.
My suspicion is this is something to do with putting together andWhere/(orWhere) clauses from the joined Tag entity with the Doctrine2 queryBuilder(). (Since the same Tag cannot be both 'photo-type'=>'people' AND 'person'=>'Bob', although the same Photo should be).
$photos = $em->getRepository('MyBundle:Photo')->
findByTags(array(
array('key' => 'context','value' => $context),
array('key' => 'photo-type','value' => 'field'),
));
I tried to construct a JOIN WITH query instead, but this seems to require a very complex construction to create the expression, which I haven't been able to figure out:-
public function getQueryByTags($tags = null, $limit =0, $start =-1, $boolean ="and")
{
...
$qb->select('p')
->from('MyBundle\Entity\Photo', 'p');
$i = 1;
$expr = $qb->expr();
foreach ($tags as $tag) {
if ($boolean == "and") {
$expr = $expr->andX($qb->expr()->eq('t.key', '?'.$i),$qb->expr()->eq('t.value', '?'.($i+1)));
} elseif ($boolean == "or") {
}
$qb->setParameter($i, $tag['key'])->setParameter(($i+1), $tag['value']);
$i += 2;
}
$qb->join('p.tags','t',Doctrine\ORM\Query\Expr\Join::WITH,$expr);
# ->join('a.tags','t')
...
return $qb->getQuery();
}
EDIT: Ultimately the result set I want is either:
"and" search: SELECT all Photos which have (Tag with key(A) AND value(B) ) AND (another Tag with key(C) AND value(D))
"or" search: SELECT all Photos which have (Tag with key(A) AND value(B) ) OR (another Tag with key(C) AND value(D))
in which: A, B is the first unique tag 'pair' (e.g. 'photo-type'='people' or 'photo-type'='animal') and C, D is another unique tag 'pair' (e.g. 'person'='Bob', 'person'='Tiddles' or, theoretically 'animal'='Tiddles')
Can anyone help me either figure out how to construct this complex JOIN WITH expression?
Or, if it seems like I'm barking up the wrong tree, can anyone suggest an alternative more elegant way to do things?
*NB: If $tags is received as comma-separated string (e.g. $tags="photo-type=people,person=Bob") it is first converted into an array.
EDIT#2: the Tag entity, on request from #Wilt:
Tag.yml
MyBundle\Entity\Tag:
type: entity
table: tag
fields:
id:
id: true
type: integer
unsigned: true
nullable: false
generator:
strategy: IDENTITY
key:
type: string
length: 20
fixed: false
nullable: true
value:
type: string
length: 50
fixed: false
nullable: true
manyToMany:
photos:
targetEntity: Tag
mappedBy: tags
lifecycleCallbacks: { }
Photo.yml (extract only)
MyBundle\Entity\Photo:
type: entity
repositoryClass: MyBundle\Entity\PhotoRepository
table: photo
fields:
sha1:
....
manyToMany:
tags:
targetEntity: Tag
inversedBy: photos
joinTable:
name: x_photo_tag
joinColumns:
photo_sha1:
referencedColumnName: sha1
inverseJoinColumns:
tag_id:
referencedColumnName: id
Your code looks way too complicated for something like this.
You can use native doctrine solutions for this checking of your tag type with a in array solution:
$array = [];
foreach ($tags as $i => $tag) {
$array[] = $tag['value'];
}
$qb = $this->createQueryBuilder('p')
->innerJoin('p.tags','t')
->where('t.type IN(:array)')
Or am I misunderstanding your case? Then try to be a bit more clear on what result set you actually want.
EDIT
I think you can do something like this:
// Main query
$qb = $this->createQueryBuilder('p')
->innerJoin('p.tags','t');
// Get tag repository to make tag query (you can also use FROM instead)
$tagRepository = $this->getEntityManager()->getRepository('MyBundle\Entity\Tag');
// Choose your where
$where = 'orWhere'; //`OR` query:
$where = 'andWhere'; //`AND` query:
// Add a in sub query expression for each tag
foreach ($tags as $i => $tag){
$alias = 't' . $i;
$sub = $tagRepository->createQueryBuilder($alias);
$sub->where($alias . '.key = :key' . $i);
$sub->andWhere($alias . '.value = :value' . $i);
$qb->setParameter('key' . $i, $tag['key']);
$qb->setParameter('value' . $i, $tag['value']);
$qb->$where($qb->expr()->in('t.id', $sub->getDQL()));
}
// get your resultset
return $qb->getQuery();
A client has requested that I make the characters "S" and "$" interchangeable in the search function, i.e. "Search Query" and "$earch Query" should return identical results.
Is there a way to accomplish this?
Change it before WP queries the database:
$the_replacements = array(
'$' => 'S',
);
function modify_search_vars($search_vars) {
global $the_replacements;
if (!empty($search_vars['s']) && !empty($the_replacements[$search_vars['s']])) {
$search_vars['s'] = $the_replacements[$search_vars['s']];
}
return $search_vars;
}
add_filter('request', 'modify_search_vars', 99);
This seems to work well enough I guess..
add_action('pre_get_posts', 'modified_search');
function modified_search($query){
global $wp_query;
if($query->is_search){
global $wpdb;
$original_query = get_search_query();
$modified_query = preg_replace("/(s|S)/", "$", $original_query);
$new_query = "
SELECT $wpdb->posts.ID
FROM $wpdb->posts
WHERE $wpdb->posts.post_status = 'publish'
AND (($wpdb->posts.post_title LIKE '%$original_query%') OR ($wpdb->posts.post_content LIKE '%$original_query%') OR ($wpdb->posts.post_title LIKE '%$modified_query%') OR ($wpdb->posts.post_content LIKE '%$modified_query%'))
ORDER BY $wpdb->posts.post_date DESC
LIMIT 0, 10
";
$results = $wpdb->get_results($new_query);
$post_ids = array();
foreach ($results as $post_id){
$post_ids[] = $post_id->ID;
}
$query->set('post__in', $post_ids);
}
}
I'm sure there are ways to improve that, and I'm more than happy to implement suggestions, but this appears to return results using both search terms, so it's good enough.
How to get list of terms of a node ( by node id) belongs to a particular vocabulary. Is there any drupal function ?
taxonomy_node_get_terms function.
http://api.drupal.org/api/function/taxonomy_node_get_terms/6
Or also:
taxonomy_node_get_terms_by_vocabulary
http://api.drupal.org/api/function/taxonomy_node_get_terms_by_vocabulary/6
I know there is api for getting list of vocabularies But i am nto sure, one api exist for gettign list of terms of vocabularies.
However, you can try this function. It will work.
function myutils_get_terms_by_vocabulary($vname, $tname = "") {
$sql = "select td.*
from term_data td
inner join vocabulary v on td.vid = v.vid
where v.name = '%s'";
if($tname) {
$result = db_query($sql . " and td.name = '%s'", $vname, $tname);
return db_fetch_object($result);
} else {
$result = db_query($sql, $vname);
}
$terms = array();
while ($term = db_fetch_object($result)) {
$terms[$term->tid] = strtolower($term->name);
}
return $terms;
}
Basically i created a 'myutils' module for such common functions and added this function there. so that i can use them in all similar scenarios.