Symfony 3.1 Cache - DeleteItem not working in prod - symfony

I've been using Symfony 3.1 with new Cache Component (https://symfony.com/doc/current/components/cache.html), I'm using the redis adapter
config.yml
cache:
app: cache.adapter.redis
default_redis_provider: "redis://127.0.0.1:6379"
Basically, I save data in redis when I do a GET to a specific resource, and I remove it from redis, when I do a POST..
With symfony in dev mode data is stored/removed from cache as I expected.
But when I change it to prod, the 'deleteItem' no longer removes the item from redis cache..
I cannot find any error in logs, so I'm getting a little bit lost with it..
This is a sample of how I'm using the cache
protected function getCache(){
return $this->get('cache.app');
}
public function getAction(){
$cacheItem = $this->getCache()->getItem('example-key');
$data = ... // Check cacheItem isHit() ...
$cacheItem->expiresAfter($this->defaultCacheTime);
$cacheItem->set($data);
$this->getCache()->save($cacheItem);
}
public function postAction() {
...
$this->getCache()->deleteItem('example-key');
}
Update - I've found what might be causing this issue
This is part of the code of symfony AbstractAdapter and RedisAdapter:
public function deleteItem($key)
{
return $this->deleteItems(array($key));
}
public function deleteItems(array $keys)
{
$ids = array();
foreach ($keys as $key) {
$ids[$key] = $this->getId($key);
unset($this->deferred[$key]);
}
try {
if ($this->doDelete($ids)) {
return true;
}
} catch (\Exception $e) {
}
$ok = true;
// When bulk-delete failed, retry each item individually
foreach ($ids as $key => $id) {
try {
$e = null;
if ($this->doDelete(array($id))) {
continue;
}
} catch (\Exception $e) {
}
CacheItem::log($this->logger, 'Failed to delete key "{key}"', array('key' => $key, 'exception' => $e));
$ok = false;
}
return $ok;
}
protected function doDelete(array $ids)
{
if ($ids) {
$this->redis->del($ids);
}
return true;
}
This is part of code from Predis StreamConnection.php:
public function writeRequest(CommandInterface $command)
{
$commandID = $command->getId();
$arguments = $command->getArguments();
$cmdlen = strlen($commandID);
$reqlen = count($arguments) + 1;
$buffer = "*{$reqlen}\r\n\${$cmdlen}\r\n{$commandID}\r\n";
for ($i = 0, $reqlen--; $i < $reqlen; $i++) {
$argument = $arguments[$i];
$arglen = strlen($argument);
$buffer .= "\${$arglen}\r\n{$argument}\r\n";
}
$this->write($buffer);
}
When I call deleteItem('example-key'), it then calls the deleteItems(..) to remove that key..
The thing is, deleteItems() is calling doDelete() and passing an array like
'example-key' => 'prefix_example-key'
The doDelete(), then calls the Redis client, passing that same array string => string, when I think it should be, index => string,eg: [0] => 'prefix_example-key' instead of ['example-key'] => 'prefix_example-key'
Then the redis client when processing the command to execute, receives that array as $arguments, and in the for loop, it does this:
$argument = $arguments[$i]; since the array is in string => string format, it won't work, in dev mode, it shows Notice undefined offset 0 error
This is the strange part
In 'dev' mode, it will throw an error, and so, the deleteItems() will catch it, and try to delete the item again, this time, sending the arguments properly
In 'prod' mode, the Notice undefined offset 0 don't know why, but it does not throw an exception, so deleteItems(..) won't catch it, returns right there..
I've found a way to make it work for me, if I add array_values in the doDelete method, it works:
protected function doDelete(array $ids)
{
if ($ids) {
$this->redis->del(array_values($ids));
}
return true;
}
I don't know if all of this is making sense or not, I think I'll open an issue in symfony bug tracker

My bad, this was caused by an outdated version of Predis, I tought I had the latest version of Predis, but I didn't
Everything is working fine with the latest version

Related

Symfony Messenger with RabbitMQ JsonSerialize problem with requeue

I have created 2 queues in Rabbit. First if main, second should be used when Exception is thrown in consumer. On older Symfony with ampq bundle is all ok, but in 6.2 with Messenger not. There is problem with JsonSerializer. I "copy/paste" solution recomended by manual of Symfony.
Lets have first queue named queue with this parameters (second is queue_retry, but it doesnt matter at this time):
arguments:
x-dead-letter-exchange: 'queue_retry'
x-dead-letter-routing-key: ''
And class for JsonSerializer
class RabbitMessageMessengerJsonSerializer implements SerializerInterface
{
public function decode(array $encodedEnvelope): Envelope
{
$body = $encodedEnvelope['body'];
$headers = $encodedEnvelope['headers'];
$data = json_decode($body, true);
$message = new RabbitMessageMessenger($data);
$envelope = new Envelope($message);
$stamps = [];
if (isset($headers['stamps'])) {
$stamps = unserialize($headers['stamps']);
}
return $envelope->with(... $stamps);
}
public function encode(Envelope $envelope): array
{
$message = $envelope->getMessage();
if ($message instanceof RabbitMessageMessenger) {
$data = $message->getContent();
} else {
throw new \Exception('Unsupported message class');
}
$allStamps = [];
foreach ($envelope->all() as $stamps) {
$allStamps = array_merge($allStamps, $stamps);
}
return [
'body' => json_encode($data),
'headers' => [
'stamps' => serialize($allStamps),
],
];
}
}
When I run consumer and throw Exception, I got this message:
In RabbitMessageMessengerJsonSerializer.php line 46:
[Exception] Serialization of 'Closure' is not allowed
In this case is message "copied" and exists in both queues. I need to take it from "queue" and when it fails "move" to "queue_retry". And without problems.
If I remove stamps (set to empty array) message is copied into "queue" many times until I brak running of consumer.
What is wrong?
Thank you very much

Validate the API key and the city name by the API request

I created a custom module where in the block I display the weather using data from https://openweathermap.org/
Code of this block:
https://phpsandbox.io/n/sweet-forest-1lew-1wmof
Also I have WeatherForm.php file with the form which adds in a configuration city and an API key for which it is necessary to display weather.
I needed to Add form validation:
fields should not be empty
City name should not contain numbers
I did it this way:
public function validateForm(array &$form, FormStateInterface $form_state) {
$pattern = '/[0-9]/';
if (empty($form_state->getValue('weather_city'))) {
$form_state->setErrorByName('weather_city', $this->t('Fields should not be empty'));
}
if (preg_match($pattern, $form_state->getValue('weather_city'))) {
$form_state->setErrorByName('weather_city', $this->t('City name should not contain numbers'));
}
}
But I got these remark after the code review:
Also, will be good to validate the API key and the city name by the API request.
I found an example of how to implement this:
public function validateWeatherData(string $city_name, $api_key):bool {
try {
$url = "https://api.openweather.org/data/2.5/weather?q=$city_name&appid=$api_key";
$response = $this->client->request('GET', $url);
if ($response->getStatusCode() != 200) {
throw new \Exception('Failed to retrieve data.');
}
$reg_ex = "#^[A-Za-z-]=$#";
return preg_match($reg_ex, $city_name);
}
catch (GuzzleException $e) {
return FALSE;
}
}
But I don't know how to integrate the example code into my function validateForm. What my code should look like so that it also implements validate the API key and the city name by the API request?
All code of my Form:
https://phpsandbox.io/n/spring-mountain-gdnn-emozx
Why not use both, brainstorm with me something along the lines of..
function validateWeatherData($city, $apikey) {
try {
$url = "https://api.openweather.org/data/2.5/weather?q=$city_name&appid=$api_key"; // Build the URL
$response = file_get_contents($url); // You can use cURL here as well incase CORS blocks file_get_contents
return $response; // Return the data from the call made above
}
catch (Exception $e) {
return $e;
}
}
function validateForm(array &$form, FormStateInterface $form_state) {
$pattern = '/[0-9]/';
if (empty($form_state->getValue('weather_city'))) {
$form_state->setErrorByName('weather_city', $this->t('Fields should not be empty'));
return false; // Failed to validate city, return false go back start again
}
if (preg_match($pattern, $form_state->getValue('weather_city'))) {
$form_state->setErrorByName('weather_city', $this->t('City name should not contain numbers'));
return false; // Failed to validate city, return false go back start again
}
$apikey = "ABCDEFG"; // API key (can be conditional based on city via CASE/IF when needed)
$weatherdata = validateWeatherData($form_state->getValue('weather_city'), $apikey); // Validate weather data
return $weatherdata; // Return validateWeatherData's response or do something else with it
}

Get initial value of entity in FormExtension

In my update form, I want to add a data attribute on the inputs that will contains the initial value of the entity. This way, I will be able to highlight the input when the user will modify it.
In the end, only the input modified by the users will be highlighted.
I want to use this only in update, not in creation.
To do so, I created a form extension like this:
class IFormTypeExtension extends AbstractTypeExtension
{
...
public static function getExtendedTypes()
{
//I want to be able to extend any form type
return [FormType::class];
}
public function configureOptions(OptionsResolver $resolver)
{
$resolver->setDefaults([
'is_iform' => false,
'is_iform_modification' => function (Options $options) {
return $options['is_iform'] ? null : false;
},
]);
$resolver->setAllowedTypes('is_iform', 'bool');
$resolver->setAllowedTypes('is_iform_modification', ['bool', 'null']);
}
public function buildView(FormView $view, FormInterface $form, array $options)
{
if (!$options['is_iform'] && !$this->isParentIForm($form)) {
return;
}
//We need to add the original value in the input as data-attributes
if (is_string($form->getViewData()) || is_int($form->getViewData())) {
$originValue = $form->getViewData();
} elseif (is_array($form->getViewData())) {
if (is_object($form->getNormData())) {
$originValue = implode('###', array_keys($form->getViewData()));
} elseif (is_array($form->getNormData()) && count($form->getNormData()) > 0 && is_object($form->getNormData()[0])) {
$originValue = implode('###', array_keys($form->getViewData()));
} else {
$originValue = implode('###', $form->getViewData());
}
} else {
//There's no value yet
$originValue = '';
}
$view->vars['attr'] = array_merge($view->vars['attr'], ['data-orig-value' => $originValue]);
}
private function isParentIForm(FormInterface $form)
{
if (null === $form->getParent()) {
return $form->getConfig()->getOption('is_iform');
}
return $this->isParentIForm($form->getParent());
}
}
As you can see in the buildView method, I get the originValue from the ViewData.
In a lot of cases, this works well.
But if I have any validation error in my form OR if I reload my form through AJAX, the ViewData contains the new information and not the values of the entity I want to update.
How can I get the values of the original entity?
I don't want to make a DB request in here.
I think I can use the FormEvents::POST_SET_DATA event, then save the entity values in session and use these in the buildView.
I could also give a new Option in my OptionResolver to ask for the initial entity.
Is it possible to have the original data of the entity directly form the buildView? (If I'm not wrong, this means the form before we call the handleRequest method).
Someone wanted to have an example with a controller. I don't think it's really interresting, because with the FormExtension, the code will be added automatically. But anyway, here is how I create a form in my controller :
$form = $this->createForm(CustomerType::class, $customer)->handleRequest($request);
And in the CustomerType, I will add the 'is_iform' key with configureOptions() :
public function configureOptions(OptionsResolver $resolver)
{
$resolver->setDefaults([
"translation_domain" => "customer",
"data_class" => Customer::class,
'is_iform' => true //This line will activate the extension
]);
}
It's probably an opinionated answer. There also might be better approaches.
I'm not a big fan of your form extension, since it's really convoluted and unclear what's happening, at least to my eyes.
What I'm proposing: When the form submit happened, in your controller you should do the following
// ((*)) maybe store customer, see below
$form = $this->createForm(CustomerType::class, $customer);
$form->handleRequest($request);
if($form->isSubmitted() && $form->isValid()) {
// easy case, you got this.
$em->flush();
return $this->redirect(); // or another response
} elseif($form->isSubmitted()) {
// form was submitted with errors, have to refresh entity!
// REFRESH - see discussion below for alternatives
$em->refresh($customer);
// then create form again with original values:
$form = $this->createForm(CustomerType::class, $customer);
}
// other stuff
return $this->render(..., ['form' => $form->createView(), ...]);
so, essentially, when the form validation fails, you refresh the entity and recreate the form, avoiding the problem with the changed state of your entity. I believe this approach ultimately is easier then hacking the form to magically not update values or re-set older values.
Now the question remains: how to refresh an entity? Simplest approach: reload from database:
$em->refresh($customer); // easiest approach, will likely run another query.
Alternatives:
Instead of giving $customer to the form, you create a customer DTO that contains the same values but on change doesn't automatically change the original object. If the form validation fails, you can just re-generate the DTO.
Instead of refresh($customer), which will most likely run another query (except maybe not, if you have a cache), you could cache the customer yourself via a DefaultCacheEntityHydrator, you would have to create your own EntityCacheKey object (not really hard), generate a cache entry (DefaultCacheEntityHydrator::buildCacheEntry() at the ((*)) above) and restore the entry for when you need to restore it. Disclaimer: I don't know if/how this works with collections (i.e. collection properties, the entity might have).
That being said ... if you really really want a form extension for whatever reason, you might want to form event with a PRE_SET_DATA handler that stores the data in the form type object, then on buildView uses those values. I wouldn't store something in the session for I don't see the necessity ... your aversion to db queries is baffling though, if that's your main reason for all the shenanigans
In the end, I managed to make it work BUT I'm not fully convinced by what I did.
It was not possible to get the original data from the form OR add a new property (the form is read only in the form extension).
public function buildForm(FormBuilderInterface $builder, array $options)
{
$builder->addEventListener(
FormEvents::POST_SET_DATA,
function (FormEvent $event) {
$form = $event->getForm();
if ('_token' === $form->getName()) {
return;
}
$data = $event->getData();
$this->session->set('iform_'.$form->getName(), is_object($data) ? clone $data : $data);
}
);
}
What I do here, is simply register the form values by its name in the session.
If it's an object, I need to clone it, because the form will modify it later in the process and I want to work with the original state of the form.
public function configureOptions(OptionsResolver $resolver)
{
$resolver->setDefaults([
'is_iform' => false,
'is_iform_modification' => function (Options $options) {
return $options['is_iform'] ? null : false;
},
]);
$resolver->setAllowedTypes('is_iform', 'bool');
$resolver->setAllowedTypes('is_iform_modification', ['bool', 'null']);
}
The configure options did not change.
And then, depending on the value type, I create my "data-orig-value" :
public function buildView(FormView $view, FormInterface $form, array $options)
{
if (!$options['is_iform'] && !$this->isParentIForm($form)) {
return;
}
$propertyValue = $this->session->get('iform_'.$form->getName());
$originValue = '';
try {
if (null !== $propertyValue) {
//We need to add the original value in the input as data-attributes
if (is_bool($propertyValue)) {
$originValue = $propertyValue ? 1 : 0;
} elseif (is_string($propertyValue) || is_int($propertyValue)) {
$originValue = $propertyValue;
} elseif (is_array($propertyValue) || $propertyValue instanceof Collection) {
if (is_object($propertyValue)) {
$originValue = implode('###', array_map(function ($object) {
return $object->getId();
}, $propertyValue->toArray()));
} elseif (is_array($propertyValue) && count($propertyValue) > 0 && is_object(array_values($propertyValue)[0])) {
$originValue = implode('###', array_map(function ($object) {
return $object->getId();
}, $propertyValue));
} else {
$originValue = implode('###', $propertyValue);
}
} elseif ($propertyValue instanceof DateTimeInterface) {
$originValue = \IntlDateFormatter::formatObject($propertyValue, $form->getConfig()->getOption('format', 'dd/mm/yyyy'));
} elseif (is_object($propertyValue)) {
$originValue = $propertyValue->getId();
} else {
$originValue = $propertyValue;
}
}
} catch (NoSuchPropertyException $e) {
if (null !== $propertyValue = $this->session->get('iform_'.$form->getName())) {
$originValue = $propertyValue;
$this->session->remove('iform_'.$form->getName());
} else {
$originValue = '';
}
} finally {
//We remove the value from the session, to not overload the memory
$this->session->remove('iform_'.$form->getName());
}
$view->vars['attr'] = array_merge($view->vars['attr'], ['data-orig-value' => $originValue]);
}
private function isParentIForm(FormInterface $form)
{
if (null === $form->getParent()) {
return $form->getConfig()->getOption('is_iform');
}
return $this->isParentIForm($form->getParent());
}
Maybe the code sample will help anyone !
If anyone have a better option, don't hesitate to post it !

How to cache a JSON response in the Silverstripe Controller?

We have a Silverstripe 4 project which acts as a headless CMS returning a group of complex data models formatted as JSON.
Here's an example of the code:
class APIController extends ContentController
{
public function index(HTTPRequest $request)
{
$dataArray['model1'] = AccessPointController::getModel1();
$dataArray['model2'] = AccessPointController::getModel2();
$dataArray['model3'] = AccessPointController::getModel3();
$dataArray['model4'] = AccessPointController::getModel4();
$dataArray['model5'] = AccessPointController::getModel5();
$dataArray['model6'] = AccessPointController::getModel6();
$this->response->addHeader('Content-Type', 'application/json');
$this->response->addHeader('Access-Control-Allow-Origin', '*');
return json_encode($dataArray);
}
The problem we're having is the data models have got so complex the generation time for the JSON is running into seconds.
The JSON should only change when site content has been updates so ideally we'd like to cache the JSON & rather than dynamically generating it for each call.
What is the best way to cache the JSON in the above example?
Have you looked at the silverstripe docs about caching?
They do provide a programmatic way to store things in cache. And configuration options what back-ends are to be used to store the cache.
A simple example might be:
I've extended the cache live time here, but still you should note that this cache is not intended for storing generated static content, but rather to reduce load. Your application will still have to compute the api response every 86400 seconds (24 hours).
# app/_config/apiCache.yml
---
Name: apicacheconfig
---
# [... rest of your config config ...]
SilverStripe\Core\Injector\Injector:
Psr\SimpleCache\CacheInterface.apiResponseCache:
factory: SilverStripe\Core\Cache\CacheFactory
constructor:
namespace: "apiResponseCache"
defaultLifetime: 86400
<?php // app/src/FooController.php
class FooController extends \SilverStripe\Control\Controller {
public function getCache() {
return Injector::inst()->get('Psr\SimpleCache\CacheInterface.apiResponseCache');
// or your own cache (see below):
// return new MyCache();
}
protected function hasDataBeenChanged() {
// alternative to this method, you could also simply include Page::get()->max('LastEdited') or whatever in your cache key inside index(), but if you are using your own cache system, you need to handle deleting of old unused cache files. If you are using SilverStripe's cache, it will do that for you
$c = $this->getCache();
$lastCacheTime = $c->has('cacheTime') ? (int)$c->get('cacheTime') : 0;
$lastDataChangeTime = strtotime(Page::get()->max('LastEdited'));
return $lastDataChangeTime > $lastCacheTime;
}
public function index() {
$c = $this->getCache();
$cacheKey = 'indexActionResponse';
if ($c->has($cacheKey) && !$this->hasDataBeenChanged()) {
$data = $c->get($cacheKey);
} else {
$dataArray['model1'] = AccessPointController::getModel1();
$dataArray['model2'] = AccessPointController::getModel2();
$dataArray['model3'] = AccessPointController::getModel3();
$dataArray['model4'] = AccessPointController::getModel4();
$dataArray['model5'] = AccessPointController::getModel5();
$dataArray['model6'] = AccessPointController::getModel6();
$data = json_encode($dataArray);
$c->set($cacheKey, $data);
$c->set('cacheTime', time());
}
$this->response->addHeader('Content-Type', 'application/json');
$this->response->addHeader('Access-Control-Allow-Origin', '*');
return json_encode($dataArray);
}
}
If you are looking for a permanent/persistent cache, that will only ever update when data changed, I suggest you look for a different back-end or just implement a simple cache yourself and use that instead of the silverstripe cache.
class MyCache {
protected function fileName($key) {
if (strpos($key, '/') !== false || strpos($key, '\\') !== false) {
throw new \Exception("Invalid cache key '$key'");
}
return BASE_PATH . "/api-cache/$key.json";
}
public function get($key) {
if ($this->has($key)) {
return file_get_contents($this->fileName($key));
}
return null;
}
public function set($key, $val) {
file_put_contents($this->fileName($key), $val);
}
public function has($key) {
$f = $this->fileName($key);
return #file_exists($f);
}

Laravel 4 Model Events don't work with PHPUnit

I build a model side validation in Laravel 4 with the creating Model Event :
class User extends Eloquent {
public function isValid()
{
return Validator::make($this->toArray(), array('name' => 'required'))->passes();
}
public static function boot()
{
parent::boot();
static::creating(function($user)
{
echo "Hello";
if (!$user->isValid()) return false;
});
}
}
It works well but I have issues with PHPUnit. The two following tests are exactly the same but juste the first one pass :
class UserTest extends TestCase {
public function testSaveUserWithoutName()
{
$count = User::all()->count();
$user = new User;
$saving = $user->save();
assertFalse($saving); // pass
assertEquals($count, User::all()->count()); // pass
}
public function testSaveUserWithoutNameBis()
{
$count = User::all()->count();
$user = new User;
$saving = $user->save();
assertFalse($saving); // fail
assertEquals($count, User::all()->count()); // fail, the user is created
}
}
If I try to create a user twice in the same test, it works, but it's like if the binding event is present only in the first test of my test class. The echo "Hello"; is printed only one time, during the first test execution.
I simplify the case for my question but you can see the problem : I can't test several validation rules in different unit tests. I try almost everything since hours but I'm near to jump out the windows now ! Any idea ?
The issue is well documented in Github. See comments above that explains it further.
I've modified one of the 'solutions' in Github to automatically reset all model events during the tests. Add the following to your TestCase.php file.
app/tests/TestCase.php
public function setUp()
{
parent::setUp();
$this->resetEvents();
}
private function resetEvents()
{
// Get all models in the Model directory
$pathToModels = '/app/models'; // <- Change this to your model directory
$files = File::files($pathToModels);
// Remove the directory name and the .php from the filename
$files = str_replace($pathToModels.'/', '', $files);
$files = str_replace('.php', '', $files);
// Remove "BaseModel" as we dont want to boot that moodel
if(($key = array_search('BaseModel', $files)) !== false) {
unset($files[$key]);
}
// Reset each model event listeners.
foreach ($files as $model) {
// Flush any existing listeners.
call_user_func(array($model, 'flushEventListeners'));
// Reregister them.
call_user_func(array($model, 'boot'));
}
}
I have my models in subdirectories so I edited #TheShiftExchange code a bit
//Get all models in the Model directory
$pathToModels = '/path/to/app/models';
$files = File::allFiles($pathToModels);
foreach ($files as $file) {
$fileName = $file->getFileName();
if (!ends_with($fileName, 'Search.php') && !starts_with($fileName, 'Base')) {
$model = str_replace('.php', '', $fileName);
// Flush any existing listeners.
call_user_func(array($model, 'flushEventListeners'));
// Re-register them.
call_user_func(array($model, 'boot'));
}
}

Resources