Given many rectangles, whats the best approach to group by line - math

Question

TLDR: How to find boxes that are lined up horizontally
Given I have the data from an image like this:
We can visually see that we have two lines:
Tare: 11700 kg 10:40:58 am 16-May
Gross: 21300 kg 12:49:34 pm 9-Aug
The data I have for each blue box shown in the image is:
Top
Left
Width
Height
Coordinates for each corner of the box (X, Y)
My main thought is to start from the top of my "grid" and loop through each value of y, and then group boxes where they share the largest amount of matching "y" values, but it seems very over the top for something that seems simple.
Unsure really where to go from here
Example data set
I was able to get boxes lined up using this bit of code (in JavaScript), it essentially finds the first "most top left" box, and then finds any boxes that "intersect" with a line that starts from the middle of that first box
We don't care what order we get the boxes in, so as long as we start with the most left on any line we are golden.
function getMostTopLeftBox(boxes) {
const sorted = boxes.slice()
.sort(
(a, b) => {
if (a.Left === b.Left) {
return a.Top < b.Top ? -1 : 1;
}
return a.Left < b.Left ? -1 : 1;
}
);
return sorted[0];
}
function getAlignedBoxesFromSet(boxes) {
const mostTopLeftBox = getMostTopLeftBox(boxes);
const line = mostTopLeftBox.Top + (mostTopLeftBox.Height / 2);
return boxes
.filter(({ Top, Height }) => Top < line && (Top + Height) > line)
.sort(({ Left: a }, { Left: b }) => a < b ? -1 : 1)
}
function getAlignedBoxes(boxes) {
let remaining = boxes;
const lines = [];
const next = () => {
const line = getAlignedBoxesFromSet(remaining);
lines.push(line);
remaining = remaining.filter(box => line.indexOf(box) === -1);
if (!remaining.length) {
return;
}
return next();
};
next();
return lines;
}
The above code with the data set provided above gives us this result
However, it doesn't account for slight angles on the boxes, for example this image:
Another example of different boxes, with senstive information removed:
You can see from the above that the values below should be considered to be on the same line:
Product: [type]
Num Of [type]: 0
[value]: [value]
I may make a new question for this, but part of the answer to this is to figure out the actual curve of a line, and not just assume that the median angle of all lines is the actual "curve" of the line, so if I was to start with the most left box, then progress to the second box, now I have two distinct lines that I would want to find the smoothed curve for, which I would then use to find the next box, as I find each box I would want to adjust this curve to find the complete line, I will investigate this one further, if anyone has any hints, please do mention it.

Answer 1

I've managed to solve this, with a variant of the code posted in the question.
Here is a code sandbox of the solution, I will do a full write up of this, but here it is for now: https://codesandbox.io/s/102xnl7on3
Here is an example of grouped boxes based on angled lines calculated from the angle of all horizontal lines, if all the boxes were to be straight, then the result would be the lines being straight as well, so it should work in all scenarios.
Here is also an example where the lines are straight:
You can see the lines from the box before intersecting with the next box, it does this each time until it can find a complete line of boxes (till no more line up), this works out better than using an average angle from the entire data set.
I would like to be able to generate a mathematical curve for the already found boxes and apply that to find the next box, but for now, using the previous box as the anchor works pretty well.