I usually use base(hardware's UART) serial communication for debugging during development.
This means that most of the text snend to print() will not be send in a final product (limited to repair level only messages).
with mine debugging messages being of a considerable info with loot of tabs and stated variables (and their descriptions), I find that mine 1K5 lined project spends several times more RAM on debugging messages then on program itself.
With one letter being one byte 2000 letters is nothing.
Most of mine non-debugging serial communication (during development is using software serial Tx) and using write function, works with bytes themselves and does not send actual text. (currently mine serial communication routine function and structure uses 6 byte blocks including addressing).
To the point: I use Streaming.h to speed the text addition to serial sending.
It is annoying to keep putting Text strings into F() every single time
F() Function slows the operation of the device because rather then Globally wasting RAM it reads it from flash every time its used and without it mine debuging messages use too much SRAM (arduino loads them as a global variables)
Is there a way of making print() use F() function without editing the Wire.h library ? (which would block me from being able to automatically update the header files)
You should use F() to store you text in FLASH memory, you can't avoid this.
You can define the macro:
#define FPRINT(x) print(F(x))
Serial.FPRINT("text");
Or even like that:
#define SFPRINT(x) Serial.print(F(x))
SFPRINT("test");
Of course, you can replace FPRINT with anything you want, that is not pre-defined (in this case you will get compiler warning).
You can also use printf_P function from <stdio.h> and PSTR macro from <avr/pgmspace.h> (they should in included by default to you program in Arduino IDE).
Typical use with text stored in RAM:
int a = 5;
printf("This is my variable: %d", a);
Result:
Sketch uses 1946 bytes (6%) of program storage space.
Global variables use 39 bytes (1%) of dynamic memory
Use with text stored in FLASH:
int a = 5;
printf_P(PSTR ("This is my variable: %d"), a);
Result:
Sketch uses 1958 bytes (6%) of program storage space.
Global variables use 15 bytes (0%) of dynamic memory
Related
I am coding an offline, battery-powered esp32 to take periodic sensor readings and store them until a hotspot is found, in which it connects and pushes the data elsewhere. I am relatively new to esp32 and ask for suggestions on the best way to do this.
I was thinking of storing the reading and DateTime in SPIFFS memory and running a webserver that starts when a network is found, checking every minute or so. Since it is battery-powered, I would also like to deep sleep the board to save power. Does the setup() function run again when the board comes out of deep sleep or would I need to have my connectToWiFi function inside the loop?
Is this viable? And are there any better routes to take? I've seen things on asynchronous servers and using the esp32 as an access point that could maybe work. Is it best to download the file through a web server or send the file line by line through a free online database?
Deep sleep on the ESP32 is almost the equivalent of being power cycled - the CPU restarts, and any dynamic memory will have lost its contents. An Arduino program will enter setup() after deep sleep and will have to completely reinitialize everything the program needs to run.
There is a very small area (8Kbytes) of static memory associated with the real time clock (RTC) which is retained during deep sleep. You can directly reference variables stored there using a special decorator (RTC_DATA_ATTR) when you declare the variable.
For instance, you could use a variable stored in this area to count the number of times the CPU has slept and woken up.
RTC_DATA_ATTR uint64_t sleep_counter = 0;
void setup() {
sleep_counter++;
Serial.begin(115200);
Serial.print("ESP32 has woken up ");
Serial.print(sleep_counter);
Serial.println(" times");
}
Beware that it's generally not safe to store objects in this area - you don't necessarily know whether they've allocated memory that won't persist during deep sleep. So storing a String in this memory won't work. Also storing a struct with pointers generally won't work as the pointers won't point to storage in this area.
Also beware that if the ESP32 loses power, RTC_DATA_ATTR will be wiped out.
The RTC static RAM also has the advantage of not costing as much power to write to as SPIFFS.
If you need more storage than this, SPIFFS is certainly an option. Beware that ESP32's generally use cheap NAND flash memory which is rated for a maximum of maybe 100,000 writes.
SPIFFS performs wear-leveling, which will help avoid writing to the same location in flash over and over again, but eventually it will still wear out. This isn't a problem for most projects but suppose you're writing to SPIFFS once a minute for two years - that's over a million writes. So if you're looking for persistent storage that's frequently written to over a very long time you might want to use a better quality of flash storage like an external SD card.
If an SD card is not an option (beware you don’t pull out the SD while writing!) I would write to SPIFFS or a direct write with esp_partition_write().
For the latter: if you use predefined structs with you sensor data (plus time etc) and you start the partition with a small table with the startvalue which has to be updated next time until the other (a mini fat) it’s easy to retrieve data (no fuzz with reading lines). Keep in mind that every time you wipe the flash the wear counts for that whole block! So if you accept old data to be ignored but present, this could dramatically reduce wear.
For example say you write:
Struct:
Uint8_t day;
Uint8_t month;
Uint8_t year; //year minus 2000, max 256
Uint8_t hour;
Uint8_t minutes;
Uint8_t seconds;
Uint8_t sensorMSB;
Uint8_t sensorLSB;
That’s 8 bytes.
The first struct (call it the mini fat):
Uint8_t firstToProcessMSB;
Uint8_t firstToProcessLSB;
Uint8_t amountToProcessMSB;
Uint8_t amountToProcessLSB;
Uint8_t ID0;
Uint8_t ID1;
Uint8_t ID2;
Uint8_t ID3;
Also eight bytes. For the ID you can use some values to recognize a well written partition. But that’s up to you. It’s only a suggestion!
In an 65.536 byte sized partition you can add 8192 (-1) bytes before you have to erase. Max 100.000 times…
When your device makes contact you read out the first bytes. Check if it’s ok, read the start byte. With fseek you step 8 bytes every hop to the end position and read all values in one time until you reached the end. If succesfull, you change the startposition to the end + 1 and only erase when things go tricky. (Not enough space)
It’s wise to do before the longest suspected amount of time to run out. Otherwise you will also lose data. Or you just make the partition bigger.
But if in this case you could write every minute for 5 days.
I am connecting a SD card to an Arduino which is then communicating over serial to Visual studio. Everything works fine independently and 99% collectively. Now if i write this code in the setup in works fine. If i pop it into a function which is called when a specific character is sent from visual studio I get the strange characters at the bottom.
I have debugged each step of the code and nothing seems abnormal, unfortunately I cannot the code as
1) it's far too long...
2) it's confidential...
:(
I understand without code I cannot get a complete solution but what are those characters! why in the setup does it work perfectly and in a function I get all kinds of randomness?
myFile = SD.open("test.txt");
if (myFile) {
Serial.println("test.txt:");
// read from the file until there's nothing else in it:
while (myFile.available()) {
Serial.write(myFile.read());
}
// close the file:
myFile.close();
} else {
// if the file didn't open, print an error:
Serial.println("error opening test.txt");
}
}
整瑳湩ⰱ㈠⸳ࠀ -- Copied straight from the text file
整瑳湩%E2%81%A7ⰱ㈠%E2%80%AC⸳ࠀ -- Output when pasted into google
Its the arduino DUE and yes lots of "String" including 4 x 2D string arrays we are using to upload to a tft screen. I ran into memory issues with the UNO but thought we would be ok with the DUE as its got considerably more ram?
Well, there's your problem. I like how you ended that with a question mark. :) Having extra RAM is no guarantee that the naughty String will behave, NOT EVEN ON SERVERS WITH GIGABYTES OF RAM. To summarize my Arduino forum post:
Don't use String™
The solutions using C strings (i.e., char arrays) are very efficient. If you're not sure how to convert a String usage to a character array, please post the snippet. You will get lots of help!
String will add at least 1600 bytes of FLASH to your program size and 10 bytes per String variable. Although String is fairly easy to use and understand, it will also cause random hangs and crashes as your program runs longer and/or grows in size and complexity. This is due to dynamic memory fragmentation caused by the heap management routines malloc and free.
Commentary from systems experts elsewhere on the net:
The Evils of Arduino Strings (required reading!)
Why is malloc harmful in embedded systems?
Dr Dobbs Journal
Memory Fragmentation in servers (MSDN)
Memory Fragmentation, your worst nightmare (nice graphics)
Another answer of mine.
I am using the readstream interface to sample at 100hz, I have been able to integrate the interface into Oscilloscope application. I just have a doubt in the way I pass on the buffer value on to the packet to be transmitted . Currently this is how I am doing it :
uint8_t i=0;
event void ReadStream.bufferDone( error_t result,uint16_t* buffer, uint16_t count )
{
if (reading < count )
i++;
local.readings[reading++] = buffer[i];
}
I have defined a buffer size of 50, I am not sure this is the way to do it as I am noticing just one sample per packet even though I have set Nreadings=2.
Also the sampling rate does not seem to be 100 samples/second when I check.I am not doing something right in the way I pass data to the packet to be transmitted.
I think I need to clarify a few things according to your questions and comments.
Reading a single sample from an accelerometer on micaZ motes works as follows:
Turn on the accelerometer.
Wait 17 milliseconds. According to the ADXL202E (the accelerometer) datasheet, startup time is 16.3 ms. This is because this particular hardware is capable of providing first reading not immediately after being powered on, but with some delay. If you decrease this delay, you will likely get a wrong reading, however, the behavior is undefined, so you may sometimes get a correct reading or the result may depend on environment conditions, such as ambient temperature. Changing this 17-ms delay to a lower value is certainly a bad idea.
Read values (in two axes) from the Analog to Digital Converter (ADC), which as an MCU component that converts analog output voltage of the accelerometer to the digital value (an integer). The speed at which ADC can sample is independent from the parameters of the accelerometer: it is another piece of hardware.
Turn off the accelerometer.
This is what happens when you call Read.read() in your code. You see that the maximum frequency at which you can sample is once every 17 ms, that is, 58 samples per second. It may be even a bit smaller because of some overhead from MCU or inaccuracy of timers. This is true when you sample by calling Read.read() in a loop or every fixed interval, because this call itself lasts no less than 17 ms (I mean the delay between the command and the event).
What you may want to do is:
Turn on the accelerometer.
Wait 17 ms.
Perform series of reads.
Turn off the accelerometer.
If you do so, you have one 17-ms delay for a set of samples instead of such delay for each sample. What is important, these steps have nothing to do with the interface you use for performing readings. You may call Read.read() multiple times in your application, however, it cannot be the same implementation of the read command that is already implemented for this accelerometer, because the existing implementation is responsible for turning on and off the accelerometer, and it waits 17 ms before reading each sample. For convenience, you may implement the ReadStream interface instead and call it once in your application.
Moreover, you wrote that ReadStream used a microsecond timer and is independent from the 17-ms settling time of the ADC. That sentence is completely wrong. First of all, you cannot say that an interface uses or does not use a timer. The interface is just a set of commands and events without their definitions. A particular implementation of the interface may use timers. The Read and ReadStream interfaces may be implemented multiple times on different platforms by various hardware components, such as accelerometers, thermometers, hygrometers, magnetometers, and so on. Secondly, the 17-ms settling time refers to the accelerometer, not the ADC. And no matter which interface you use, Read or ReadStream, and which timers a driver uses, milli- or microsecond, the 17-ms delay is always required after powering on the accelerometer. As I mentioned, you probably want to make this delay once per multiple reads instead of once per a single read.
It seems that the TinyOS source code already contains an implementation of the accelerometer driver providing the ReadStream interface which allows you to sample continuously. Look at the AccelXStreamC and AccelYStreamC components (in tos/sensorboards/mts300/).
The ReadStream interface consists of two commands. postBuffer(val_t *buf, uint16_t count) is called to provide a buffer for samples. In the accelerometer driver, val_t is defined as uint16_t. You may post multiple buffers, one by one. This command does not yet start sampling and filling buffers. For that purpose, there is a read(uint32_t usPeriod) command, which directs the device to start filling buffers by sampling with the specified period (in microseconds). When a buffer is full, you get an event bufferDone(error_t result, val_t *buf, uint16_t count) and a component starts filling a next buffer, if any. If there are no buffers left, you get additionally an event readDone(error_t result, uint32_t usActualPeriod), which passes to your application a parameter usActualPeriod, which indicates an actual sampling period and may be different (especially, higher) from a period you requested when calling read due to some hardware constraints.
So the solution is to use the ReadStream interface provided by AccelXStreamC and AccelYStreamC (or maybe some higher-level components that use them) and pass an expected period in microseconds to the read command. If the actual period is lower than one you expect, this means that sampling at higher rate is impossible either due to hardware constraints or because it was not implemented in the ADC driver. In the second case, you may try to fix the driver, although it requires good knowledge of low-level programming. The ADC driver source code for this platform is located in tos/chips/atm128/adc.
I have recently bought an Arduino Uno, and now I am experimenting a bit with it. I have a couple of 18B20 sensors and an ENC28J60 network module connected to it, then I am making a sketch to allow me to connect to it from a browser and read out the temperatures either as a simple web page or as JSON. The code that makes the web pages looks like this:
client.print("Inne: ");
client.print(tempin);
client.println("<br />");
client.print("Ute: ");
client.print(tempout);
client.print("<br /><br />");
client.println(millis()/1000);
// client.print("j");
The strange thing is: if I uncomment the last line, the sketch compiles fine, uploads fine, but I cannot get to connect to the board. The same thing happens if I add on some more characters in some of the other printouts. Thus, it looks to me as if I'm running into some kind of memory limit (the total size of the sketch is about 15 KB, and there are some other strings used elsewhere in the code - and yes I know, I will rewrite it to use an array to store the temporaries, I've just stolen some code from an example).
Is there any limit on how much memory I can use to store strings in an Arduino and are there any way to get around that? (using GUI v 1.0.1 on a Debian PC with GCC-AVR 4.3.5 and AVR Libc 1.6.8).
The RAM is rather small, as the UNO's 328 is only 2K. You may just be running out of RAM. I learned that when it runs out, it just kind of sits there.
I suggest reading the readme from this library to get the FreeRAM. It mentions how the ".print" can consume both RAM and ROM.
I always now use (for Arduino IDE 1.0.+)
Serial.print(F("HELLO"));
versus
Serial.print("HELLO");
as it saves RAM, and this should be true for lcd.print. Where I always put a
Serial.println(freeMemory(), DEC); // print how much RAM is available.
in the beginning of the code, and pay attention. Noting that there needs to be room to run the actual code and re-curse into its subroutines.
For IDE's prior to 1.0.0 the library provides getPSTR()).
IDE 1.0.3 now starts to display the expected usage of RAM at the end of the compile. However, I find it is often short, as it is only an estimate.
I also recommend that you look at Webduino as it has a library that supports JSON. Its examples are very quick to get going. However it does not directly support the ENC28J60.
I use the following code to get free available RAM
int getFreeRam()
{
extern int __heap_start, *__brkval;
int v;
v = (int) &v - (__brkval == 0 ? (int) &__heap_start : (int) __brkval);
Serial.print("Free RAM = ");
Serial.println(v, DEC);
return v;
}
You can check the memory usage with a small lib called memoryFree.
When there is ram left, you might be pushing the serial buffer limit instead of the ram limit. If so, you can increase SERIAL_BUFFER_SIZE in HardwareSerial.cpp
(C:\Program Files (x86)\Arduino\hardware\arduino\cores\arduino on a windows machine)
Be carefull though, ram and serial buffer are both stored on the SRAM. Increasing the serial buffer will result in less available memory for your variables.
For playing with JSON on the arduino there is a really nice lib, called aJson.
add this function and call it in setup and every now and then in your loop to make sure RAM is not being used up.
// Private function: from http://arduino.cc/playground/Code/AvailableMemory
int freeRam () {
extern int __heap_start, *__brkval;
int v;
return (int) &v - (__brkval == 0 ? (int) &__heap_start : (int) __brkval);
}
You need to call it for example inside a print:
Serial.println(freeRam());
Can someone please explain how the Arduino bootloader works? I'm not looking for a high level answer here, I've read the code and I get the gist of it.
There's a bunch of protocol interaction that happens between the Arduino IDE and the bootloader code, ultimately resulting in a number of inline assembly instructions that self-program the flash with the program being transmitted over the serial interface.
What I'm not clear on is on line 270:
void (*app_start)(void) = 0x0000;
...which I recognize as the declaration, and initialization to NULL, of a function pointer. There are subsequent calls to app_start in places where the bootloader is intended to delegate to execution of the user-loaded code.
Surely, somehow app_start needs to get a non-NULL value at some point for this to all come together. I'm not seeing that in the bootloader code... is it magically linked by the program that gets loaded by the bootloader? I presume that main of the bootloader is the entry point into software after a reset of the chip.
Wrapped up in the 70 or so lines of assembly must be the secret decoder ring that tells the main program where app_start really is? Or perhaps it's some implicit knowlege being taken advantage of by the Arduino IDE? All I know is that if someone doesn't change app_start to point somewhere other than 0, the bootloader code would just spin on itself forever... so what's the trick?
Edit
I'm interested in trying to port the bootloader to an Tiny AVR that doesn't have separate memory space for boot loader code. As it becomes apparent to me that the bootloader code relies on certain fuse settings and chip support, I guess what I'm really interested in knowing is what does it take to port the bootloader to a chip that doesn't have those fuses and hardware support (but still has self-programming capability)?
On NULL
Address 0 does not a null pointer make. A "null pointer" is something more abstract: a special value that applicable functions should recognize as being invalid. C says the special value is 0, and while the language says dereferencing it is "undefined behavior", in the simple world of microcontrollers it usually has a very well-defined effect.
ATmega Bootloaders
Normally, on reset, the AVR's program counter (PC) is initialized to 0, thus the microcontroller begins executing code at address 0.
However, if the Boot Reset Fuse ("BOOTRST") is set, the program counter is instead initialized to an address of a block at the upper end of the memory (where that is depends on how the fuses are set, see a datasheet (PDF, 7 MB) for specifics). The code that begins there can do anything—if you really wanted you could put your own program there if you use an ICSP (bootloaders generally can't overwrite themselves).
Often though, it's a special program—a bootloader—that is able to read data from an external source (often via UART, I2C, CAN, etc.) to rewrite program code (stored in internal or external memory, depending on the micro). The bootloader will typically look for a "special event" which can literally be anything, but for development is most conveniently something on the data bus it will pull the new code from. (For production it might be a special logic level on a pin as it can be checked nearly-instantly.) If the bootloader sees the special event, it can enter bootloading-mode, where it will reflash the program memory, otherwise it passes control off to user code.
As an aside, the point of the bootloader fuse and upper memory block is to allow the use of a bootloader with no modifications to the original software (so long as it doesn't extend all the way up into the bootloader's address). Instead of flashing with just the original HEX and desired fuses, one can flash the original HEX, bootloader, and modified fuses, and presto, bootloader added.
Anyways, in the case of the Arduino, which I believe uses the protocol from the STK500, it attempts to communicate over the UART, and if it gets either no response in the allotted time:
uint32_t count = 0;
while(!(UCSRA & _BV(RXC))) { // loops until a byte received
count++;
if (count > MAX_TIME_COUNT) // 4 seconds or whatever
app_start();
}
or if it errors too much by getting an unexpected response:
if (++error_count == MAX_ERROR_COUNT)
app_start();
It passes control back to the main program, located at 0. In the Arduino source seen above, this is done by calling app_start();, defined as void (*app_start)(void) = 0x0000;.
Because it's couched as a C function call, before the PC hops over to 0, it will push the current PC value onto the stack which also contains other variables used in the bootloader (e.g. count and error_count from above). Does this steal RAM from your program? Well, after the PC is set to 0, the operations that are executed blatantly "violate" what a proper C function (that would eventually return) should do. Among other initialization steps, it resets the stack pointer (effectively obliterating the call stack and all local variables), reclaiming RAM. Global/static variables are initialized to 0, the address of which can freely overlap with whatever the bootloader was using because the bootloader and user programs were compiled independently.
The only lasting effects from the bootloader are modifications to hardware (peripheral) registers, which a good bootloader won't leave in a detrimental state (turning on peripherals that might waste power when you try to sleep). It's generally good practice to also fully initialize peripherals you will use, so even if the bootloader did something strange you'll set it how you want.
ATtiny Bootloaders
On ATtinys, as you mentioned, there is no luxury of the bootloader fuses or memory, so your code will always start at address 0. You might be able to put your bootloader into some higher pages of memory and point your RESET vector at it, then whenever you receive a new hex file to flash with, take the command that's at address 0:1, replace it with the bootloader address, then store the replaced address somewhere else to call for normal execution. (If it's an RJMP ("relative jump") the value will obviously need to be recalculated)
Edit
I'm interested in trying to port the bootloader to an Tiny AVR that doesn't have separate memory space for boot loader code. As it becomes apparent to me that the bootloader code relies on certain fuse settings and chip support, I guess what I'm really interested in knowing is what does it take to port the bootloader to a chip that doesn't have those fuses and hardware support (but still has self-programming capability)?
Depending on your ultimate goal it may be easier to just create your own bootloader rather than try to port one. You really only need to learn a few items for that part.
1) uart tx
2) uart rx
3) self-flash programming
Which can be learned separately and then combined into a bootloader. You will want a part that you can use spi or whatever to write the flash, so that if your bootloader doesnt work or whatever the part came with gets messed up you can still continue development.
Whether you port or roll your own you will still need to understand those three basic things with respect to that part.