I have a large multi-tenanted Symfony2 application using doctrine and around 40 to 50 database tables in MySQL.
Has anybody had experience setting up an audit logging process for, at a minimum, tracking all Doctrine actions that create, edit or delete data in a large application?
Ideally it should "just happen" so that other developer don't need to worry about making their new entities or code "loggable".
I don't have specific requirements to meet (HIPAA etc) but it would be nice to be as close to any ISO27000 best practices as I can be.
My inital plan is to look at using some of the Doctrine lifecycle callbacks to grab the information and push it off into a a seperate MySQL database with no DELETE or UPDATE permissions. I'm concerned this method might have a big performance impact.
Has anybody done this before or anything similar that has any tips or words of warning abot my planned approach?
If it matters the whole stack is running on AWS and I can use any of their services in the EU-WEST-1 region. I am already using things like RDS, Elasticache and SQS.
Thanks!
OroPlatform is built in Symfony 2.8 and has a bundle called OroDataAuditBundle that audits all Doctrine entity operations (create, update, delete) at a deep level in just two tables: oro_audit and oro_audit_field.
And it looks like that in the UI:
You will not probably be able to use this bundle in your project - since it is built for OroPlatform - but, you could at least get some inspiration from their architecture.
If you want to check this audit system working, they have an online demo in https://demo.orocrm.com. Just log in as administrator and play around creating entries, and then you will be able to check all audit entries in https://demo.orocrm.com/audit.
Related
We are working on a version of Sylius 0.9 in which we made several huge changes, overriding a lot of core logics.
Doing so we broke a bit the wired in logging or history functionality.
Lurking the code it looks like that this functionality is automagically working based on the Doctrine orm logging offered by the Gedmo bundle
It also looks like that this logging of any update to the model is declared in the models in the vendor.
My question is:
is there a way to disable this kind of logging?
It pollutes a table named ext_log_entries with around 1'000'000 records each week, which I'm manually truncating every now and then since we're not using this functionality and I don't want to have hundreds of MB in my DB for nothing.
I'm new to Azure and started an Azure Mobile App Quick-Start (.NET) project.
I'm studying on this blog wrote by Adrian Hall:
https://shellmonger.com/2016/05/09/30-days-of-zumo-v2-azure-mobile-apps-day-18-asp-net-authentication/. However, I've been confused by the explanation saying that:
Since this is Entity Framework, I would normally need to do an Entity
Framework Code First Migration to get that field onto my database. You
can find several walk-throughs of the process online. This isn’t an
Entity Framework blog, so I’ll leave that process to better minds than
mine. Just know that you have to deal with this aspect when using the
ASP.NET backend.
On the next page, https://shellmonger.com/2016/05/11/30-days-of-zumo-v2-azure-mobile-apps-day-19-asp-net-table-controllers/
The demonstration was using SQL syntax to create/manage the Azure SQL Database, such as: CREATE TABLE... and CREATE TRIGGER... and etc.
So, my question is, with either SQL(like the blog sample shown) or Entity Framework:1) Are they both able to do the exact same stuff?2) Should I only choose one of those methods only?
All the methods discussed (code first, model first, database first) provide you with a way to create a SQL database and update the data within it. The end result is the same - a database with data.
The method you pick tends to rely on where you prefer the 'intelligence' for your data to live - with the database or with the code.
Do you already have an existing database you'll be using?
Go database first, so you can automatically generate code and classes from the database.
Do you know SQL? Do you prefer to use your database for data only (i.e. no stored procedures or validating data inside the database)?
Go code first, you have full control of your model from the Code, and its a bit easier to keep databases in sync with your application.
There are a few more considerations that can help skew you to a different method. You can take a look at this blog post from Roland about the pros and cons of each approach.
There's also a StackOverflow thread that summarizes the differences between the methods.
Is it possible to build an ASP.NET website using EF where each customer logging in has separately stored data? We have customers demanding that their data won’t be stored in the same tables as other customers’ data.
I’ve read that EF can’t work with several databases but is it possible to switch database at runtime depending on input parameters? I have a feeling it won’t be possible since the migration features are tightly connected to the database being used, but I'm not sure.
One solution could be to have a separate website deployment and database for each customer. They’ll get separate domains to access but that’s not a problem. But this solution feels a bit clumsy if you’re having many customers, especially with deployment and future upgrades.
Am I missing some smart ways of solving this or is this a very tricky issue?
is structure (of the db) the same ?
if so you could switch connections - not w/o issues though, but should work. For details on how that should be done check the long discussion we've had here (and linked previous questions etc.)...
Code first custom connection string and migrations without using IDbContextFactory
We have an ASP.NET app with SQL Server & it is a photo & video sharing site.
Details of photos and videos are stored in tables & the files are in the file system.
Database has 75 tables and 225 stored procedures. The app will be ready for production deployment within next 6 months.
Due to longer time growth concerns, we decided to switch to NoSQL (MongoDB) database.
We have few questions regarding the best way to approach this:
Is it better to deploy the app with SQL Server backend and migrate to NoSQL later?
OR re-architecture now and rewrite/recreate database, tables, procedures and data layer
How difficult will it be re-architecture/recode with MongoDB? Any tools or BKMs?
EDIT:
Our app is Youtube+Flickr type site where user will share photos and videos with lots of comments, tags and ratings (photo\video & comments).
Is NoSQL a better database to move to? Reason for moving: cost + read query speed
Please help me with you valuable advise.
Thank you very much.
Change is always exponentially more expensive the later it is introduced to a project. This is a core principle of software engineering. You should do this now.
That said, I question your long-term vision. Relational databases, used properly, have a lot of performance in them.
This question raises more questions than answers.
Have you benchmarked your current implementation in terms of requests/responses?
Why MongoDB out of all possible NoSQL databases? (Don't get me wrong, I love Mongo, but love and hype should not weigh in technology choices)
Are you certain you will get the large userbase you're expecting? Why are you so certain?
Using stored procs seems to tip off that you aren't using an ORM? Why not?
Generally, I'm against these types of re-architectures. Firstly, you need to get your whole team acclimated to how Mongo affects development. Secondly, your ops team needs to get acclimated to how to deploy and maintain a Mongo installation. More likely than not, this will prevent you from launching in a timeline you want to launch.
I'd say that you should probably launch as is, fix the ORM part if you aren't using one, benchmark your app, benchmark a prototype of your app backed by Mongo and if the performance advantages are so big that it warrants the pain of re-architecture do it.
To your latter question, there aren't any tools right now, as far as I can tell, that'll automate or semi-automate the database import/export from SQL Server to Mongo. There are barely tools to do that for MySQL.
I've done such a migration a few month ago, during the early developement stage of a website in ASP.NET. It was a hard decision, but I could concentrate on that migration. The reason why I did this migration was the ORM that I couldn't trust anymore and some very slow queries that I had no idea how to optimize.
During coding phase, what I figured out was : I was spending a lot of time with the data model in SQL Server (using Entity) and all the plumbery code.
Now, no more store procedures (C# and Linq code instead), no more 2 layers to maintain (the code is the model).
My small experience says : The earlier the better but don't get me wrong, before migrating you really have to think in Document rather than in RDBMS. This means you may have to partially change the businness DataModel to correctly utilize MongoDB features, otherwise you could get bad performances and Mongo DB is useless for bad models.
Another point is the admin stuff. You'll have to quickly learn Mongo DB admin to be up to speed. And even if the tools are good, they completely differ from SQL Server tools.
In conclusion, If you're convinced MongoDB is your future data store and search database,
(and it was in my case), read documentation, take time to do some Proof Of Concept. Then you can think Document and load test you new model.
Your core question appears to be whether to make the switch to MongoDB now, or deploy on SQL and go to MongoDB in a future release.
You do not appear to be using an ORM (e.g. NHibernate, Entity Framework.) Setting other concerns aside, if you're convinced that you want to go to NoSQL, then I would do it now rather than later. Unless you integrate a Provider model for your data access, changing the underlying data access strategy after it is already established would be difficult.
I agree. Switching now is better, if only to avoid the data migration headache switching post-deployment will require.
How can we extend the Alfresco database? I need to add new tables to the existing database structure.
Does alfresco support this?
thanks in advance,
sri..
I think changing the alfresco db model is never a good solution. Some alfresco upgrades are made using Schema Upgrade Scripts, and that could get messy.
Have you tried to extend the alfresco content model?
Alfresco support some data types, allowing you to persist data. The Web Script framework allow you manipulate all your data inside your content model.
If your data is not suitable for a "content model", I think you should create a new database to hold your data.
Well, it is just a database. So you can create as many new tables as you want just like you would in any other database.
Obviously Alfresco won't use them because it doesn't know them, but you can query the tables as you like.
Advices from alfresco engineers are do not touch alfresco database. Please take a look at this page.
http://forums.alfresco.com/forum/general/non-technical-alfresco-discussion/where-alfreso-user-details-are-stored-i-alfresco
Changing alfresco database is not recommended.Content Model will be the good way.If such requirement is mandatory than,
You can use spring with hibernate for database connection.Properties which is required for connecting database are all ready declared inside alfresco-global.properties which is located inside "tomcat/shared/classes/".
For Spring bean injection you can declare beans inside any file which ends with "-context" which resides inside "tomcat/shared/classes/alfresco/extension" folder.
I will still recommend developer to use content model.
Depending on your use case, you may or may not need to play directly with the[/a] data base. I think your use case should fall in one of the following:
Use case 1:
You need to setup some metadata on folders and/or documents. You may have to nest multiple levels of nodes with different sets of custom metadata on each level.
You probably need to extend alfresco models in order to define custom document/folder models that best suits your business requirements. Please check jpotts' tutorial to learn how to do so.
Use case 2:
You need to define multiple lists with different sets of properties, those lists may or may not be linked to some content in your alfresco repo.
You probably need to learn more about alfresco sites' datalists, once you do so you may be interested in learning how to extend OOTB alfresco content model, jpotts' tutorial would be a good starting point, and then you should be checking this tutorial in order to learn how to manage datalists in stand alone aikau apps/share pages.
Use case 3:
You need to leverage a relational database in order to define and leverage you complex business logic that do not fall in any of the use cases defined above.
Are you sure you do not want to code a brand new app leveraging a technology that you are familiar with and have it communicate with alfresco using RESTfull api/cmis/.... ?
Are you sure alfresco is THE way to go ? If so, and you still want to have your custom complex business model in a bare relational database:
Please consider using a separate database instance / database for your custom extension, this way you would be sure any new patch/upgrade to alfresco that may change database structure won't affect your extension (or at least wont give you hard time upgrading it)
If you are really tied to only 1 database instance / 1 database schema, you will probably want to precede your table names with some prefix and hope none of alfresco future upgrades would have new tables with the same prefix. You probably also need to make sure to manage your database config wisely (connection pools ..) so neither your alfresco instance nor your custom extension have to starve. (make sure you close the connections you are opening)
Alfresco and Activiti come with a database. It is not good to access the database directly. Doing so can cause unexpected locking behavior or exhaust connection pools on the DB. This turns out into performance problems and other kinds of issues are possible too. In case you want to update Alfresco or Activiti you can do it through APIs. Easy to extend, easy to customize and hassle free integration capability are some of the reasons which has made http://loganwinson.doodlekit.com/blog/entry/4249216/top-things-to-know-about-alfresco-development>Alfresco web development popular among businesses.