Is it reasonable to use a recursive relationship in an ER-diagram? I've made an example below:
For instance, if a customer orders 3 fish dishes, 3 chips and 2 drinks, where each fish dish itself consist of 1 fish and 2 vegetables which themselves are menu items, and so on:
group1 : fish_dish_group(3), chips(3), drinks(2)
fish_dish_group : fish(1), vegetable_group(2)
vegetable_group : vegetable_1, vegetable_2
However, is such a relationship even permissible, as grouping and food are different types? I would think not, so what are the alternatives to recursive relationships?
an entity type can have a relation with it self and this is absolutely valid in ER. in your example, an alternative can be introducing a secondery entity type "sub-group" and link it to the original "group" by an n:m relationship.
Related
I am trying to solve an optimization which looks similar to a knapsack-problem. The setting is the following:
I am having a pool of ~80,000 players of which I want to build the cheapest squad of exactly 11 players. Each player has multiple attributes, the main position he is playing in, nation, club, league and rating.
The players not only need to be selected but also assigned to a position in the formation:
Stating the following problem:
The first constraint is a minimum rating of the squad, which can simply be formulated as a linear constraint. The second and third constraint make sure that exactly one player is selected for each position and each player can only be selected once.
There are several other linear constrains that can occur like a minimum amount of players from one nation or at most three players from a specific club etc.
The chemistry of a squad is a non-linear constraint with a step function.
A players individual chemistry is the product of his position & link bonus.
The position bonus is defined by what the players main position is and where in the formation he is placed in. A central defender placed in the according position gets 3 points, used as a striker he gets 0 points. The bonuses can be seen in the next table.
This part of the constraint still can be formulated linearly. The link bonus is the non linear component. Each position/node in the formation/graph has a weight between [0-3], two adjacent players have a weight of 1 if they are from the same nation, league or club. Sharing two attributes is a weight of 2 and for three respectively. The bonus for a specific position is the average of all edges multiplied by a factor 3.
This bonus is plugged into a step function, which can be seen in the next figure (mapping values between [0-1] to 0.9 etc.). The link bonus is multiplied by the position bonus and capped to 10. The team chemistry is defined as the sum of the individual player chemistries.
I implemented it as described with miniZinc solving it with the osicbc solver, but even for a player pool of ~100 players this is not really feasible to compute, depending on the additional constraints.
Now I am looking for an implementation that can approximate the solution. I was thinking about a simulated annealing or genetic algorithm. However, due to this chemistry constraint these approaches produce a lot of invalid solutions, wandering around in the dark.
Does anyone have an approach that might be applicable to my problem?
I'm trying to analyze goal-scoring networks in hockey. I have data for the player who scored the goal and the player who assisted on that goal. My issue is that some goals do not have an assist, so I'm not sure what I should do in those situations.
So, an example for my data looks like this:
scorer <- c("Lidstrom", "Yzerman", "Fedorov", "Yzerman", "Shanahan")
assister <- c("", "Lidstrom", "Yzerman", "Shanahan", "Lidstrom")
mydata <- data.frame(scorer, assister)
And the output is:
scorer assister
1 Lidstrom
2 Yzerman Lidstrom
3 Fedorov Yzerman
4 Yzerman Shanahan
5 Shanahan Lidstrom
When I'm dealing with unassisted goals, does it make sense to act as if the assist goes to the scorer?
EX:
scorer assister
1 Lidstrom Lidstrom
2 Yzerman Lidstrom
3 Fedorov Yzerman
4 Yzerman Shanahan
5 Shanahan Lidstrom
Or does it make sense to create a new name "unassisted" for unassisted goals?
EX:
scorer assister
1 Lidstrom UNASSISTED
2 Yzerman Lidstrom
3 Fedorov Yzerman
4 Yzerman Shanahan
5 Shanahan Lidstrom
Here's the rest of my code for the PageRank, assuming that something is filled in for the blank assister space:
library(igraph)
library(dplyr)
my_network <- mydata %>%
as.matrix() %>%
graph.edgelist(directed = TRUE)
page_rank(my_network, directed = TRUE)$vector
I can't just remove goals that are unassisted, so I'm trying to come up with some solution that doesn't defy any major graph theory principles (of which I'm not knowledgeable). Any ideas?
I agree with the suggestion of #emilliman5 outlined in the comments: for unassisted goals, just make an edge from the scorer to itself. Then use PageRank for finding the most influential players. Actually, PageRank can be a particularly good choice here because the principles underlying the PageRank score bear some similarity to what is going on in a "real" hockey match.
Let me elaborate on this a bit. PageRank was originally invented for modeling the behaviour of a randomly chosen Internet user browsing the pages on the web. In each time step, the user can choose to follow a link on the web page currently being viewed, or surf to another, unrelated page, chosen uniformly from the set of all pages on the Internet. There is a fixed probability value that decides whether the user is going to follow a link (typically 0.85) or the user is going to "teleport" to a randomly chosen page (typically 0.15). The idea behind PageRank is that the most important pages are where the user is likely to spend a lot of time when following the rules above. The behaviour of the user is essentially a random walk over the set of webpages.
Now, in a hockey game, the "user" is the hockey puck that is being passed from player to player. At each pass, the puck is either passed from one player to another, or a goal is scored, or the puck is accidentally passed to the opposing team. In the latter two cases, the puck ends up at the opposing team, and eventually it is returned to the first team at a randomly chosen player. (This is a first approximation; if you want to go deeper, you could keep on "tracking" the puck for the opposing team as well). I think you can start seeing the similarities here. The assister-to-scorer network that you have captures a fragment of this, namely the last pass before each goal. From this point of view, I think it totally makes sense to think about unassisted goals as events where the player passed to himself before scoring.
Of course you would have a much better understanding of the team dynamics if your dataset contained all the passes, not only the ones that resulted in a goal. In fact, in that case, you could add an additional node called "GOAL" to your network, draw edges from scorers to the "GOAL" node, and then calculate the so-called personalized PageRank vector for the "GOAL" node, which would give you the most influential nodes from which the "GOAL" node is the easiest to reach. But this is more like a research question from this point onwards, and it is probably not a good fit for further discussion on Stack Overflow.
Closed. This question does not meet Stack Overflow guidelines. It is not currently accepting answers.
This question does not appear to be about programming within the scope defined in the help center.
Closed 7 years ago.
Improve this question
I have a large sales database of a 'home and construction' retail.
And I need to know who are the electricians, plumbers, painters, etc. in the store.
My first approach was to select the articles related to a specialty (wires [article] is related to an electrician [specialty], for example) And then, based on customer sales, know who the customers are.
But this is a lot of work.
My second approach is to make a cluster segmentation first, and then discover which cluster belong to a specialty. (this is a lot better because I would be able to discover new segments)
But, how can I do that? What type of clustering should I occupy? Kmeans, fuzzy? What variables should I take to that model? Should I use PCA to know how many cluster to search?
The header of my data (simplified):
customer_id | transaction_id | transaction_date | item_article_id | item_group_id | item_category_id | item_qty | sales_amt
Any help would be appreciated
(sorry my english)
You want to identify classes of customers based on what they buy (I presume this is for marketing reasons). This calls for a clustering approach. I will talk you through the entire setup.
The clustering space
Let us first consider what exactly you are clustering: either orders or customers. In either case, the way you characterize the items and the distances between them is the same. I will discuss the basic case for orders first, and then explain the considerations that apply to clustering by customers instead.
For your purpose, an order is characterized by what articles were purchased, and possibly also how many of them. In terms of a space, this means that you have a dimension for each type of article (item_article_id), for example the "wire" dimension. If all you care about is whether an article is bought or not, each item has a coordinate of either 0 or 1 in each dimension. If some order includes wire but not pipe, then it has a value of 1 on the "wire" dimension and 0 on the "pipe" dimension.
However, there is something to say for caring about the quantities. Perhaps plumbers buy lots of glue while electricians buy only small amounts. In that case, you can set the coordinate in each dimension to the quantity of the corresponding article (presumably item_qty). So suppose you have three articles, wire, pipe and glue, then an order described by the vector (2, 3, 0) includes 2 wire, 3 pipe and 0 glue, while an order described by the vector (0, 1, 4) includes 0 wire, 1 pipe and 4 glue.
If there is a large spread in the quantities for a given article, i.e. if some orders include order of magnitude more of some article than other orders, then it may be helpful to work with a log scale. Suppose you have these four orders:
2 wire, 2 pipe, 1 glue
3 wire, 2 pipe, 0 glue
0 wire, 100 pipe, 1 glue
0 wire, 300 pipe, 3 glue
The former two orders look like they may belong to electricians while the latter two look like they belong to plumbers. However, if you work with a linear scale, order 3 will turn out to be closer to orders 1 and 2 than to order 4. We fix that by using a log scale for the vectors that encode these orders (I use the base 10 logarithm here, but it does not matter which base you take because they differ only by a constant factor):
(0.30, 0.30, 0)
(0.48, 0.30, -2)
(-2, 2, 0)
(-2, 2.48, 0.48)
Now order 3 is closest to order 4, as we would expect. Note that I have used -2 as a special value to indicate the absence of an article, because the logarithm of 0 is not defined (log(x) tends to negative infinity as x tends to 0). -2 means that we pretend that the order included 1/100th of the article; you could make the special value more or less extreme, depending on how much weight you want to give to the fact that an article was not included.
The input to your clustering algorithm (regardless of which algorithm you take, see below) will be a position matrix with one row for each item (order or customer), one column for each dimension (article), and either the presence (0/1), amount, or logarithm of the amount in each cell, depending on which you choose based on the discussion above. If you cluster by customers, you can simply sum the amounts from all orders that belong to that customer before you calculate what goes into each cell of your position matrix (if you use the log scale, sum the amounts before taking the logarithm).
Clustering by orders rather than by customers gives you more detail, but also more noise. Customers may be consistent within an order but not between them; perhaps a customer sometimes behaves like a plumber and sometimes like an electrician. This is a pattern that you will only find if you cluster by orders. You will then find how often each customer belongs to each cluster; perhaps 70% of somebody's orders belong to the electrician type and 30% belong to the plumber type. On the other hand, a plumber may only buy pipe in one order and then only buy glue in the next order. Only if you cluster by customers and sum the amounts of their orders, you get a balanced view of what each customer needs on average.
From here on I will refer to your position matrix by the name my.matrix.
The clustering algorithm
If you want to be able to discover new customer types, you probably want to let the data speak for themselves as much as possible. A good old fashioned
hierarchical clustering with complete linkage (CLINK) may be an appropriate choice in this case. In R, you simply do hclust(dist(my.matrix)) (this will use the Euclidean distance measure, which is probably good enough in your case). It will join closely neighbouring items or clusters together until all items are categorized in a hierarchical tree. You can treat any branch of the tree as a cluster, observe typical article amounts for that branch and decide whether that branch represents a customer segment by itself, should be split in sub-branches, or joined with a sibling branch instead. The advantage is that you find the "full story" of which items and clusters of items are most similar to each other and how much. The disadvantage is that the outcome of the algorithm does not tell you where to draw the borders between your customer segments; you can cut up the clustering tree in many ways, so it's up to your interpretation how you want to identify your customer types.
On the other hand, if you are comfortable fixing the number of clusters (k) beforehand, k-means is a very robust way to get just any segmentation of your customers in k distinct types. In R, you would do kmeans(my.matrix, k). For marketing purposes, it may be sufficient to have (say) 5 different profiles of customers that you make custom advertisement for, rather than treating all customers the same. With k-means you don't explore all of the diversity that is present in your data, but you might not need to do so anyway.
If you don't want to fix the number of clusters beforehand, but you also don't want to manually decide where to draw the borders between the segments afterwards, there is a third possibility. You start with the k-means algorithm, where you let it generate an amount of cluster centers that is much larger than the number of clusters that you hope to end up with (for example, if you hope to end up with somewhere about 10 clusters, let the k-means algorithm look for 200 clusters). Then, use the mean shift algorithm to further cluster the resulting centers. You will end up with a smaller number of compact clusters. The approach is explained in more detail by James Li over here. You can use the mean shift algorithm in R with the ms function from the LPCM package, see this documentation.
About using PCA
PCA will not tell you how many clusters you need. PCA answers a different question: which variables seem to represent a common underlying (hidden) factor. In a sense, it is a way to cluster variables, i.e. properties of entities, not to cluster the entities themselves. The number of principal components (common underlying factors) is not indicative of the number of clusters needed. PCA can still be interesting if you want to learn something about the predictive value of each article about a customer's interests.
Sources
Michael J. Crawley, 2005. Statistics. An Introduction using R.
Gerry P. Quinn and Michael J. Keough, 2002. Experimental Design and Data Analysis for Biologists.
Wikipedia: hierarchical clustering, k-means, mean shift, PCA
A very basic question here:
Example rule (suppose its generated from WEKA) :
bread=t 10 ==> milk=t 10 conf:(1)
Which means that "from 10 instances, everytime people buy bread, they also buy milk". (ignore the support)
Does this rule can be read both ways? Like, "every time people buy milk, they also buy bread?"
Another example
Physics101=A ==> Superphysics401=A
Can it be read both ways like this:
"If people got A on Physics101, they also got A on Superphysics401"
"If people got A on Superphysics401, they also got A on Physics101" ?
If so, what makes WEKA generate the rule in that order (Physics ==> Superphysics), why not the other way? Or does the order not relevant?
Does this rule can be read both ways? Like, "everytime people buy milk, they also buy bread?"
No, it can only be read one way.
This follows from the rules of implication. A -> B and B -> A are different things. Read former as "A is a subset of B", thus, whenever you are in A, you are in B. B -> A, also called converse of A -> B, can be interpreted in similar way. When both of these hold, we say that A <-> B which means that A and B are essentially the same.
If the above looks like too much jargon, keep the following in mind:
Rain -> Clouds is true. Whenever there is rain, there will be clouds, But Clouds -> Rain is not always true. There may be clouds but no rain.
If so, what makes WEKA generate the rule in that order (Physics ==>
Superphysics), why not the other way? Or does the order not relevant?
The dataset leads to the rules. Here is an example :
Milk, Bread, Waffers
Milk, Toasts, Butter
Milk, Bread, Cookies
Milk, Cashewnuts
Convince yourself that Bread -> Milk, but Milk ! -> Bread.
Note that we may not be always interested in rules that either hold or do not hold. Thus, we try to add a notion of confidence to the rules. A natural way of defining confidence for A->B is P(B|A) i.e. how often do we see B when we see A.
This can be calculated by dividing the count of B and A appearing together and dividing by the count of A appearing alone.
In our example,
P(Milk | Bread) = 2 / 2 = 1 and
P(Bread | Milk) = 2 / 4 = 0.5
You can now sort list of rules on the basis of confidence and decide which ones do you want to use.
Does anyone know how to replicate the (pg_trgm) postgres trigram similarity score from the similarity(text, text) function in R? I am using the stringdist package and would rather use R to calculate these on a matrix of text strings in a .csv file than run a bunch of postgresql quires.
Running similarity(string1, string2) in postgres give me a number score between 0 and 1.
I tired using the stringdist package to get a score but I think I still need to divide the code below by something.
stringdist(string1, string2, method="qgram",q = 3 )
Is there a way to replicate the pg_trgm score with the stringdist package or another way to do this in R?
An example would be getting the similarity score between the description of a book and the description of a genre like science fiction. For example, if I have two book descriptions and the using the similarity score of
book 1 = "Area X has been cut off from the rest of the continent for decades. Nature has reclaimed the last vestiges of human civilization. The first expedition returned with reports of a pristine, Edenic landscape; the second expedition ended in mass suicide, the third expedition in a hail of gunfire as its members turned on one another. The members of the eleventh expedition returned as shadows of their former selves, and within weeks, all had died of cancer. In Annihilation, the first volume of Jeff VanderMeer's Southern Reach trilogy, we join the twelfth expedition.
The group is made up of four women: an anthropologist; a surveyor; a psychologist, the de facto leader; and our narrator, a biologist. Their mission is to map the terrain, record all observations of their surroundings and of one anotioner, and, above all, avoid being contaminated by Area X itself.
They arrive expecting the unexpected, and Area X delivers—they discover a massive topographic anomaly and life forms that surpass understanding—but it’s the surprises that came across the border with them and the secrets the expedition members are keeping from one another that change everything."
book 2= "From Wall Street to Main Street, John Brooks, longtime contributor to the New Yorker, brings to life in vivid fashion twelve classic and timeless tales of corporate and financial life in America
What do the $350 million Ford Motor Company disaster known as the Edsel, the fast and incredible rise of Xerox, and the unbelievable scandals at GE and Texas Gulf Sulphur have in common? Each is an example of how an iconic company was defined by a particular moment of fame or notoriety; these notable and fascinating accounts are as relevant today to understanding the intricacies of corporate life as they were when the events happened.
Stories about Wall Street are infused with drama and adventure and reveal the machinations and volatile nature of the world of finance. John Brooks’s insightful reportage is so full of personality and critical detail that whether he is looking at the astounding market crash of 1962, the collapse of a well-known brokerage firm, or the bold attempt by American bankers to save the British pound, one gets the sense that history repeats itself.
Five additional stories on equally fascinating subjects round out this wonderful collection that will both entertain and inform readers . . . Business Adventures is truly financial journalism at its liveliest and best."
genre 1 = "Science fiction is a genre of fiction dealing with imaginative content such as futuristic settings, futuristic science and technology, space travel, time travel, faster than light travel, parallel universes, and extraterrestrial life. It often explores the potential consequences of scientific and other innovations, and has been called a "literature of ideas".[1] Authors commonly use science fiction as a framework to explore politics, identity, desire, morality, social structure, and other literary themes."
How can I get a similarity score for the description of each book against the description of the science fiction genre like pg_trgm using an R script?
How about something like this?
library(textcat)
?textcat_xdist
# Compute cross-distances between collections of n-gram profiles.
round(textcat_xdist(
list(
text1="hello there",
text2="why hello there",
text3="totally different"
),
method="cosine"),
3)
# text1 text2 text3
#text1 0.000 0.078 0.731
#text2 0.078 0.000 0.739
#text3 0.731 0.739 0.000