Wordpress's datamodel provides extensibility through it's "meta" tables (aka, wp_postmeta, wp_commentmeta, wp_usermeta). This name/value pair linking is extremely flexible and supported by Wordpress's API. All good reasons to celebrate it but it also can get quite non-performant if you have lots of extension attributes hanging off of your transactions. Let's say, for instance, that I have a Custom Post Type called "Blood Test" and I want it to be able to capture 30-40 different measurements coming out of the blood analysis. Is it practical to use this wp_postmeta joining for each test? Probably not.
I could go completely bespoke and build tables left and right but what I'm wondering is, is there not a way to at least build an "80/20 rule" and have a generic extension table that provides a static number of additional columns to put SQL-searchable attributes and then end with a column for a JSON object that would allow for non-SQL searchable attributes to extend to an almost unlimited amount? Something like what is diagrammed below:
I was thinking that by doing this one could also extend the Wordpress API so that most development could largely be ignorant of this structural difference. Something like this:
EXAMPLE OF REGISTRATION API:
$my_meta_ext = new WP_Meta_Extension( 'post' );
$my_meta_ext->add_tran_type( 'blood-measurement' , ( 'col1' => 'total-cholesterol' , 'col2' =>
'triglycerides', 'col3' => 'etc');
EXAMPLE OF USAGE API:
add_ext_meta ( $term_id, 'blood-measurement.total-cholesterol', $meta_value );
Ok, so here are my explicit questions for the group:
Does this make sense? Do you see value in this? Does anything like this already exist in a 'plugin' form (or otherwise)?
If you were to use one extension table across all types (aka, posts, comments, users, etc.) would you be able to build an index in mySQL that was efficient? Is it better just to have a table per type?
Does anyone have any metrics -- even if they're not fully proven out -- on when the default wordpress extension model starts to degrade in performance?
If you've done something like this already ... any key lessons learned when extending the model? How complicated of a task is this?
I brought this topic into a specialist WordPress working group in London and while we didn't arrive on an answer there was pretty clear agreement that this type of solution would be highly valuable. Furthermore, I have decided to start a GIT repository with the intent of developing a plugin to address this (hopefully with the generous support of the group in London or any of you who are reading this and feel you could help).
The repository can be found here:
https://bitbucket.org/ksnyder/wp-database-extension/
Please understand that at this moment there is really just an idea, a high-level design, an interface spec, but no real code. That will change of course and it will change more quickly if you're willing to help us with this endeavour (mind you I have no personal experience running distributed development but you have to learn somewhere).
Related
I'm creating my main project functionality right now so it's kind of a big decision to make in my project, I want efficient & scalable solution. I use different API's to fetch users products ultimately for 1 collection to display products information inside a table with possible merge by SKU TITLE from different sources.
I have thought of 2 approaches (In both approaches we add Meteor.userId() to collection insert so each users has it's own products:
1) to create each API it's own collection and fetch the products to it, after or in middle of the API query where I insert it to sourceXProducts also add the logic of merge products by sku and add it to main usersProducts Only the fields I need, and we have the collection of the sourceXproducts if we ever need anything we didn't really include to main usersProducts we can query it and get it so we basically keep all the information possible (because it can come handy)
source1Products = new Meteor.Collection('source1Products');
source2Products = new Meteor.Collection('source2Products');
usersProducts = new Meteor.Collection('usersProducts');
Pros: Honestly I'm not sure, It makes it organized also the way I learned Meteor it seems to be used a lot.
Cons: Meteor collection joins is not supported in core yet, So I have to use a meteor package such as: meteor-publish-composite which seems good but this way might hit performance
2) Create 1 collection and just insert everything the API resonse has and additional apiSource field so we can choose products from X user X api.
usersProducts = new Meteor.Collection('usersProducts');
Pros: No joins, possibly better performance
Cons: Not organized, It can become a large collection maybe it's not good for mongodb
3) Your ideas? :)
First, you should improve the question. You do not tell us anything precise about your schema. What are the entities you have and what type of relations are there and what type of joins do you think you will be doing. How often you will be doing them?
Second, you should rethink your schema and think in the terms of a non-relational database. I see many people coming from SQL world and then they simply design their schema in the same way. Wrong. MongoDB is not SQL and things you learned there you should not try to just reuse here. You should start using features like subdocuments and arrays which can help you solve many basic things you would do in SQL with joins. So, knowing your schema would help us help you design the schema. For example, see this answer and especially the comments for the discussion for a similar type of question you are asking here.
Third, have you evaluated various solutions which exist out there? There are many, but you have not shown us that you tried any of them and how it worked for you. What were pros and cons of them, for you and your project?
Fourth, if you are lazy to evaluate, you can just use peerlibrary:peerdb and peerlibrary:related. They are simply perfect. You should trust me. I am their author.
We have a requirement to allow customising our core product and adding additional fields on a per client basis e.g. People entity some client wants to record their favourite colour etc. As far as I know we can't add properties to EF at runtime as it needs classes defined at startup. Each customer has their own database but we are deploying the same solution to all customers with all additional code. We are then detecting which customer they are and running customer specific services etc.
Now the last thing I want is to be forking my project or alternatively adding all fields for all clients. This would seem likely to become a nightmare. Also more often than not the extra fields would only be required in a very limited amount of place. Maybe some reports, couple of screens etc.
I found this article from Jermey Miller http://codebetter.com/jeremymiller/2010/02/16/our-extension-properties-story/ describing how they are adding extension properties and having them go from domain to the web front end.
Has anyone else implemented anything similar using EF? How did it work out? Are there any blogs/samples that anyone has seen? I am not sure if I am searching for the right thing even if someone could tell me the generic name for what we want to do that would help. I'm guessing it is a problem that comes up for other people.
Linked question still requires some forking or implementing all possible extensions in single solution because you are still creating strongly typed extensions upfront (= you know upfront what extensions customer wants). It is not generally extensible solution. If you want generic extensible solution you must leave strongly typed world and describe extensions as data.
You will need to use some metamodel. Your entity classes will contain only properties used by all customers and navigation property to special extension entity (additional table per every extensible entity) where you will be able to put additional properties as name / value pair (you can add other columns like type, validation, etc. if needed).
This will in general moves part of your model from hardcoded scenario to configuration based scenario and your customers will even be allowed to define extensions at runtime (if you implement such feature).
I'd like to learn about using catalogs correctly.
I have about 30 useful content types, about 50 indexes in catalog.xml, and about 45 metadatas. There are just three types which account for most of the site's data - and I may need millions of these. I've been reading, and there's lots to do, but I want to have the basic configuration right before I begin all that.
This page told me that any non-default indexes should not be added to the portal_catalog. I've even read people explaining how removing one, or two of the default indexes makes a performance difference.
My question is: what are the rules for dividing up the indexes into different catalogs, and for selecting which catalog(s) index which type(s)?
So far I have created one additional catalog, used to catalog all indexes for my 'site-setup' objects (which I have caused to no longer be indexed in portal_catalog). The site-setup indexes are very often used, but more rarely modified than others, so I thought it was correct to separate them from objects which are reindexed more often. I'm not sure if that's the main consideration though.
Another similar question (a good example of the kind of thing I want to solve): how would you handle something like secondary workflow review_state variables? I give each workflow's review_state variable an index (and search on them quite often), but some of my workflows are only used on just a few types. (my most prolific objects have secondary workflows...)
I'd be very grateful for advice!
Campbell
This won't cover everything but I'll bring up some points..
Anything not in the portal_catalog won't work with collections, folder_contents view, getFolderContents method, search, portlet collections, related items(I think) and anything else the assumes you're using the portal_catalog.
I like to use an additional catalog when I need to be able to query the data but it only affects a sub-set of the content objects.
Use collective.indexing to speed up indexing operations.
Mount the catalogs on their own mount points so you can cache them differently from the rest of the site(so you can cache the whole catalog). Then, you can even serve the the catalogs from dedicated zeoserver.
Also, if your content doesn't have to be cataloged by the portal_catalog(with all the constraints listed), you may even want to think about if you need it as a full-fledged (archetype|dexterity) type in the first place. You can use a more slim repoze.catalog to catalog arbitrary objects(which could be very simple data) for whatever your purpose is and get even more performance. Or better yet, look into Solr for indexing it for VERY good performance.
On more thing, depending on the type of data you're storing, you could even look into using a relational database for a data store. But I don't know what kind of queries, indexes, data, etc you have...
30 different types seems like a lot but I don't know what your use case is. Care to share? Perhaps there is a better way to do it.
I am working with the commerce module to create an online store. I am modifying the products .install file to create a content type (as I have been told this is required) and as part of that content type, I need to create lots of fields. The list will be around 50-60 different pieces of information.
Ideally I would like to store these in a single table with the productID at the beginning and all the other information along, but this doesn't seem to be the case; all the fields are stored in different tables.
I noticed that the "Address" module that is also used with commerce creates a field-type that has about 15 different values all stored in the same box. How is this possible? I noticed that if I set the cardinality up to 5 for example, it creates different rows. I just want a table with the following:
ID - value1 - value2 - value3 etc etc.
I also don't need any modules/extensions as this all needs to be written in the files. I also don't think that changing to the mongoDB ( I think ) is an option, so what are my options in this situation?
That's not how the Drupal field system works I'm afraid, one field == one table (well actually 2 tables if you include the revision table for each field).
The Address module uses hook_field_schema() to define several columns for that particular field (have a look in address.install and you'll see what I mean).
So if you want to put everything in one table you'll simply have to define your own field type (see the examples module, specifically field_example for help with that).
Bear in mind though that the number of columns you define in hook_field_schema() will be static once the module is installed, and the only way you're going to be able to increase/decrease it is with an _update hook for your custom module.
Also, if you're hacking at files that are included in the Commerce module...stop!: Commerce is still very much in it's infancy and you will likely have to update it soon...once you've done that your code changes will be gone and there's a good chance your site will be in an inconsistent state.
The whole point to Drupal is that everything is hooked/farmed out so that it can be altered by other parts of the system. There's nothing you can change in product.install that can't be done by implementing a Drupal hook in another module.
If you're unsure, post another question detailing what you're trying to accomplish by directly editing a contrib module file and one of the Drupal gurus on SO will point you in the right direction :-)
EDIT
Just to say I've been working with Ubercart in Drupal 7 for quite some time now and find it a very, very good solution (a lot of Commerce contributed modules are still in dev/alpha/beta; this is less so for Ubercart contributed modules). It might be worth a look.
Some more info
I think you've basically got two options here but either way you'll need to create a custom module (excellent set of instructions here).
Option 1: Create a custom field
If you're a Drupal coding beginner I'd suggest this is probably the easiest way to accomplish what you want, but it's still not totally straight forward. Grab the field_example module from the Drupal Examples module link above and have a look in the .install file, specifically the field_example_field_schema() function. That defines the columns that will be in the table for that field. Then have a look in field_example.module...pretty much every function that's commented with Implements hook_x is one that you're going to want to copy into your module and tweak for your own needs.
I think this will be easier because Drupal will handle the table/form field creation for you
so you don't have to mess with the database, schema or form APIs.
Option 2: Create a custom module
This option involves implementing your own table (like you suggest in your comment) where the primary key would be the entity ID of the product and would also contain all of your custom columns. (See the Schema API documentation for help with this).
Then you'd implement hook_form_alter() to add the form fields necessary for a user to input the data, and then implement hook_node_insert() and hook_node_update() to persist this data to your database table. It's quite hard to go into any more detail without actually writing code and it's quite a bit of code!
Hope that helps, sorry I can't be any more specific but it's not easy without knowing all the ins and outs of the situation
I am building a real estate website. I have a table for properties (i.e. "houses"), a table for pictures, a table for features, etc. So each property's data comes from three, not only one table.
I need to provide a function that retrieves requested property data depending on a search criteria, for example:
property of the week
featured properties
properties on offer
the simple search
I have multiple drop-down lists for selecting country, region, area, bedrooms, price range, type, etc.
I already built many functions each dealing with one of the above cases, but I was wondering, if there was any other way of building one flexible mechanism that will retrieves whatever you want in a smart way, I mean one function that works for all cases?
I know my question is kind of vague and too broad, but bear with me please.
Thanks in advance.
PS: I am using ASP.NET 3.5, VB.NET, Visual Web Developer 2008 Express, SQL Server 2005 Express
I think you're looking to build an Advanced Search functionality that integrates with the default/simple search provided.
Why not build a single stored procedure to encapsulate all possible types of searches by specifying default values of parameters that are not passed.
In this manner, the simple search would be using all parameter defaults. Whenever any criteria is provided to narrow the search, the passed parameter value will be used by the Stored Procedure.
Is Lucene an option? Lucene.net is a great way to provide a quick and flexible search engine to your site.
When you use Lucene you also many other advantages, like fuzzy search etc...
http://www.codeproject.com/KB/library/IntroducingLucene.aspx
The question is twofold:
Can you logically unite the search for "properties of the week" and the search for "properties in this specific region, costing 100,000 or less" into a single database search?
Does it buy you anything to think of/create something that does 1.)? (Especially when you already have something that works.)
If you would answer both questions with "Yes", then I'd say you must extend the simple search. I suppose you already have a flag for each of your special cases ("property of the week", "property on offer", etc). Just add support for these flags to your simple search function.
Provide reasonable defaults for the search criteria and you can use simple search for all the cases you described.