Processing large amounts of arrays and memory exhaustion - arduino

Question

First: I've inherited this project from someone who couldn't complete it due to time constraints.
The code contains a little over 100 declared arrays, each one containing a set of INTs. The arrays are all unique.
byte arr_foo[] = {2, 5, 6, 8, 3};
byte arr_bar[] = {1, 7};
byte arr_baz[] = {6, 10, 9, 11, 7, 8, 3};
Those INTs relate to a specific LED on a board - there are 11 total. And the arrays represent a specific sequence that these LEDs should light up.
What they were attempting to do was to write a routine that, when given an array name, would then go fetch the contents of that array and process the INTs. Well, passing an array name as a string to then be matched with a variable doesn't work. And this is where they passed it on, saying they don't have time to figure it out.
So I'm looking at this and thought, why not a 2-dimensional array? I quickly ran into trouble there.
byte seqs[][7] = {
{2, 5, 6, 8, 3},
{1, 7},
{6, 10, 9, 11, 7, 8, 3}
}
While in principle this works, the issue here is that it pads each array with trailing zeros because I told it each one has [7] elements. This results in a lot of memory being wasted and the thing running out of memory.
So I am stuck. I'm not sure how to deal with 100+ separate arrays, other than to write 100+ separate routines to be called later. Nor can I figure out how to make it more efficient.
Then there's the issue of, I may still run out of memory at a later time as more sequences are added. So then what? Add an external i2c flash memory, and shove things in there? Having never dealt with that, I'm not sure how to implement that, in what format to store the values, and how to do it. Am I correct that one has to first write a program that loads all the data in memory, upload that and run it, then put the actual program that's going to process that data on the micro controller?
So I guess I'm asking for two things: What's a better way of handling lots and lots of (small) arrays and be able to use them within a routine that calls them, and if I'm better off shoving this data into an external flash, what format should they be stored in?

Answer 1

Putting the data into 2D arrays wont save any space at all.
Right now, you're storing these values into your 2k of SRAM. Change these declarations to use the PROGMEM keyword, so they're stored where there's much more space.
Using the PROGMEM instructs the compiler to load this data into the flash portion of memory:
const PROGMEM uint8_t arr_foo[] = { 2, 5, 6, 8, 3 };
However the data needs to be accessed with a function call, you can't just use it directly.
for (byte k = 0; k < 5; k++)
{
uint8_t next_led = pgm_read_byte_near( arr_foo + k );
// Do something with next_led
}

Answer 2

If these arrays form a pattern of leds that should be lit, while the other ones are switched off, you could store the state of all leds in an uint16_t and have an array of those in PROGMEM. (As in Kingsley's answer)
If you're not familiar with HEX notation you could use the binary format.
const PROGMEM uint_16_t patterns[] = {
// BA9876543210 Led Pins
0b000101101100, //foo: 2, 5, 6, 8, 3
0b000010000010, //bar: 1, 7
0b111111001000, //baz: 6, 10, 9, 11, 7, 8, 3
// ...
};
I wonder about the order of your numbers, so I'm not sure if this guess is correct at all. So no more details how to work with this approach rigth now

Answer 3

Update
To me, your comments changed the intention of your question completely.
As I read it now, there's no need for a special kind of "name" data to identify an array. What you want seems to be just to pass different arrays around as function arguments.
This is usually done via pointers, and there are two things to note:
Most of the time, arrays "decay" to pointers automatically. That means that in most places an array variable can be used in place of a pointer. And a pointer can be used like an array.
An array in C does not carry any length information at runtime. The length of an array needs to be held/passed separately. Alternatively, you can define a struct (or class in C++) which contains both the array and its length as members.
Example:
If you want to pass an array of elements of type T to a function, you can declare the function to accept a pointer to T:
void somefunc(uint8_t* arr, uint8_t arrLength) {
for ( uint8_t i = 0; i < arrLength; i++ ) {
uint8_t value = arr[i];
value = *(arr+i); // equivalent to arr[i]
}
}
or equivalently
void somefunc(uint8_t arr[], uint8_t arrLength) {
...
}
then call that function by simply passing the array variable and the corresponding array's length, like
uint8_t arr_foo[] = { 1,2,3,4,5 };
uint8_t arr_bar[] = { 1,2 };
somefunc(arr_foo,5);
somefunc(arr_bar,2);
The arrays' constant data can be put into PROGMEM to save RAM, but, as others have noted, read accesses are a little more complex, requiring pgm_read_...() calls in C++. (AVR gcc does support __flash-qualified data only in C, not in C++.)
Then there's the issue of, I may still run out of memory at a later
time as more sequences are added.
Notice that the "Arduino" AVR has 32kb of flash memory. If each sequence consumes 15 bytes, it could probably still hold 1000 or 2000 of these items along with your program.
then what? Add an external i2c flash memory, and shove things in
there? Having never dealt with that, I'm not sure how to implement
that, in what format to store the values, and how to do it.
If you actually run out of flash at some point you can still resort to any form of external storage.
A common solution is SPI flash memory, which is readily available in the mega-bit range. Winbond is a well-known supplier. Just search for "Arduino SPI flash" modules and libraries.
A more complex approach would be to support SD cards as external memory. But probably not worth it unless you actually want to store gigabytes of data.
Am I correct that one has to first write a program that loads all the
data in memory, upload that and run it, then put the actual program
that's going to process that data on the micro controller?
That's definitely one way to do it. If your code space permits, you can alternatively include the routines to write to the external flash memory in your application, like some kind of bootloader so that you can switch into "upload external flash data" mode without re-flashing the microcontroller.