Best way to find least standard deviation - math

Question

I have a spreadsheet where I put numbers that represent number of verses on each paragraph of a book.
I manually distribute sequential paragraphs by number of verses, so in the spreadsheet I'll have something like this:
Verses Day
5 1
6 1
3 1
10 2
8 3
4 3
2 3
6 4
3 4
10 5
3 5
2 6
5 6
10 7
= 2,7080128015
By summing the total of verses for each day - in this case, 7 days - I get the standard deviation and try to reduce it for a better distribution of paragraphs.
The question is: what is the best way to find the least standard deviation?
I thought on using brute force to generate all possible combinations, but that is not a good idea if the number increases.
EDIT: The standard deviation is based on total number of verses of each day, which are identified sequentialy. Day 1 has total of 14 verses, day 2, 10 and so on.
1 14
2 10
3 14
4 9
5 13
6 7
7 10
= 2,7080128015

Answer 1

Since the total number of verses and the number of days is constant, you want to minimize
sum (avg verse count - verse count of day i)^2
i
avg verse count is a constant and simply the total number of verses divided by the number of days.
This problem can be solved with a dynamic program over the days. Let us build the partial solution function f(days, paragraph) that gives us the minimal sum of squares for distributing paragraphs 0 through paragraph over days days. We are interested in the last value of this function.
We can build the function incrementally. Calculating f(1, p) for any p is straight-forward since we just need to calculate the differences to the average and square. Then, for all other days, we can calculate
f(d, p) = min f(d - 1, i) + (avg verse count - sum verse count of paragraph j)^2
i<p j:i+1..p
That means, we check the solutions for one day less and fill up the current day with the paragraphs between the previous day's end paragraph and p. While we calculate this function, we keep a pointer to the chosen minimum element (as usual for a dynamic program). When we are done calculating the entire function, we just follow the pointers back to the start, which will give us the partitioning.
The algorithm has a running time of O(d * p^2), where d is the number of days and p is the number of paragraphs.
Example Code
Here is some example C# code that implements the above algorithm:
struct Entry
{
public double minCost;
public int predecessor;
}
public static void Main()
{
//input data
int[] versesPerParagraph = { 5, 6, 3, 10, 8, 4, 2, 6, 3, 10, 3, 2, 5, 10 };
int days = 7;
//calculate constants
double avgVerses = (double)versesPerParagraph.Sum() / days;
//set up DP table (f(d,p))
int paragraphs = versesPerParagraph.Length;
Entry[,] dp = new Entry[days, paragraphs];
//initialize table
int verseCount = 0;
for(int p = 0; p < paragraphs; ++p)
{
verseCount += versesPerParagraph[p];
double diff = avgVerses - verseCount;
dp[0, p].minCost = diff * diff;
dp[0, p].predecessor = -1;
}
//run dynamic program
for(int d = 1; d < days; ++d)
{
for(int p = d; p < paragraphs; ++p)
{
verseCount = 0;
dp[d, p].minCost = double.MaxValue;
for(int i = p; i >= d; --i)
{
verseCount += versesPerParagraph[i];
double diff = avgVerses - verseCount;
double cost = dp[d - 1, i - 1].minCost + diff * diff;
if(cost < dp[d, p].minCost)
{
dp[d, p].minCost = cost;
dp[d, p].predecessor = i - 1;
}
}
}
}
//reconstruct the partitioning
{
int p = paragraphs - 1;
for (int d = days - 1; d >= 0; --d)
{
int predecessor = dp[d, p].predecessor;
//calculate number of verses, just to show them
verseCount = 0;
for (int i = predecessor + 1; i <= p; ++i)
verseCount += versesPerParagraph[i];
Console.WriteLine($"Day {d} ranges from paragraph {predecessor + 1} to {p} and has {verseCount} verses.");
p = predecessor;
}
}
}
The output is:
Day 6 ranges from paragraph 13 to 13 and has 10 verses.
Day 5 ranges from paragraph 10 to 12 and has 10 verses.
Day 4 ranges from paragraph 9 to 9 and has 10 verses.
Day 3 ranges from paragraph 6 to 8 and has 11 verses.
Day 2 ranges from paragraph 4 to 5 and has 12 verses.
Day 1 ranges from paragraph 2 to 3 and has 13 verses.
Day 0 ranges from paragraph 0 to 1 and has 11 verses.
This partitioning gives a standard deviation of 1.15.