Using Program Memory (PROGMEM)

As I was building my DateTime class (Arduino library) I realised I had hard-coded strings for the names of the days of the week and the months of the year. I know that this is a huge drain on the Arduino's minimal amount of SRAM. It was finally time to tackle program memory and get an understanding of it. This is the process I went through ...

Get code here: https://github.com/prawnhead/ArduinoDateTime

I already understand that the Arduino has three different kinds of memory. If you're not familiar with them, have a look here: http://arduino.cc/en/Tutorial/Memory. I found some resources. I'm going to read through them and trying to pick up an understanding of program memory.

• Microcontroller memory (Arduino Playground)
• Arduino PROGMEM page
• pgmspace.h documentation (AVR-LibC)
• Mikal Hart's Flash library post (Arduino Forum) 
• AVR-LibC FAQ
• Arduino Playground - printf

The PROGMEM page on the Arduino site explains about using the data types defined by pgmspace.h. However, the pgmspace.h documentation shows these to be deprecated. So the documentation is out of date on the Arduino site and the code samples they show should only be used if you don't mind rewriting your code at some point in the future when these features stop working. That's a bummer.

The Arduino Memory page mentioned the use of the F() syntax. This goes back to Mikal's Flash library. It looks like it is this library that has been incorporated since 1.0 of Arduino. Need to do some digging and find the code for it! Ok, here it is: https://github.com/johnmccombs/arduino-libraries/blob/master/Flash/Flash.h. The F() syntax looks really good. If you want to print a long string and don't want it taking up SRAM, you just wrap it in the F (flash) function F("any long string you want"). The print() functions have been extended to happily work with pointers to flash memory. So that's useful straight away.

Because I'm dealing with DateTimes, I really want to be able to build a string of the form "Monday 24 September 2013 23:07:13". This means I'll need to not only store the names of the days of the week and the months of the year in flash, but I'll need to be able to convert them to String types for concatenation with other strings.

I created this function as a test:

String& DateTime::dayOfWeekToString() {
  static String* output;
  if (output) delete(output);
  output = new String(F("Hello World"));
  return *output;
}

It's using the F() function to ensure a string is stored in flash, then trying to create a String object from it. It's not valid. Here's the error I got:

DateTime.cpp: In member function 'String& DateTime::dayOfWeekToString()':
DateTime.cpp:425: error: call of overloaded 'String(const __FlashStringHelper*)' is ambiguous

Usually I've found the "call of overloaded ... is ambiguous" errors to mean that if I can cast to the right kind of type I can get away with it. Alas, all the casts I tried didn't work. This just shows I don't understand what I'm doing. Looking up the reference to FlashStringHelper to find the code (searching the Arduino folder with Notepad++) I found the __FlashStringHelper class defined in WString.h as:

class __FlashStringHelper;
#define F(string_literal) (reinterpret_cast<const __FlashStringHelper *>(PSTR(string_literal)))

Also Print.h and Print.cpp have print() and println() functions to handle __FlashStringHelper pointers. I don't know where to go with this. I'll look elsewhere.

What I want is a string stored in flash memory F("Hello world") to be returned from a function.

String& toString() {
  return F("Hello World");
} // doesn't compile

From https://github.com/johnmccombs/arduino-libraries/blob/master/Flash/Flash.h, F(str) is defined thus:

#define F(str) (_FLASH_STRING(PSTR(str)).Printable())

Which instantiates a _FLASH_STRING object (see the _FLASH_STRING class in the same file) with the argument (PSTR(str)) and then calls the .printable() function on that object. PSTR(s) is defined in pgmspace.h as:

# define PSTR(s) (__extension__({static char __c[] PROGMEM = (s); &__c[0];}))

... and I'm out of my depth again. What the hell is that? Ah, I'll pick it apart and see what I can know about it. Curly braces are code blocks. Semicolons terminate lines of code. So in the curly braces we have two lines of code: the first allocates a character array called '__c' in program memory and initialises it with the value of 's'; the second line returns the address of the first character in the array. What does __extension__ do? This would suggest it only prevents warnings when the GCC compiler is run with the -pedantic option.

OK. So expanding the macro PSTR("Hello world") becomes ...

{
static char __c[] PROGMEM = ("Hello world");
&__c[0];
}

Since this is returning a value, PSTR(s) can have member functions called on it, or be assigned to a variable. So now F("Hello world"); becomes:

_FLASH_STRING({
static char __c[] PROGMEM = ("Hello world");
&__c[0];
}).Printable()

OK. So _FLASH_STRING("Hello world") calls the constructor on the _FLASH_STRING class and passes in the address of a char in PROGMEM. The constructor accepts a const prog_char *arr so this works, although the prog_char* type is one of those deprecated ones. All the constructor does is passes the pointer to an initialiser for _arr, like this:

_FLASH_STRING::_FLASH_STRING(const prog_char *arr) : _arr(arr) { }

_arr is the only private member variable (pointer to program memory character) in the _FLASH_STRING class. Having created an object of the _FLASH_STRING class, the .Printable() method is called on it. So the line:

#define F(str) (_FLASH_STRING(PSTR(str)).Printable())

... creates an array of characters in program memory, creates a _FLASH_STRING object with the address of the array, and calls the .Printable() method on that object. The circle is complete.

---

In my reading I've come to understand a few key things. One is that a normal string of characters resides in both SRAM and flash memory. The code of your sketch is downloaded onto the microcontroller's flash memory. But because strings may be manipulated by the program or just passed around as parameters, they are also copied into SRAM. So there's an immutable copy in flash and a "useful" copy in SRAM. This is the normal behaviour. Tagging a variable declaration with the PROGMEM mnemonic says "I know I'm not going to try to modify this variable, or if I use references to it I acknowledge I have to do it in certain ways, don't make an SRAM copy of it!".

Now that we can tell the compiler not to make an SRAM copy of a variable, we can see why this work has to be done before the program is even loaded on the microcontroller. This necessarily reflects back to the code. We can't wait until the code is in execution to create a PROGMEM string. It's too late at that point. Executing code can't change the flash memory. So when we declare our PROGMEM data, we have to do it outside a function, outside a class, outside any executable unit of code. It has to be a global-level operation.

The second important macro I've stumble across is PSTR(). This is defined in the pgmspace.h file. It places a string in program memory and returns a const PROGMEM char * to it. I tried just using PSTR to put my strings in program memory. This didn't work:

#include <pgmspace.h>
void setup() {
  Serial.begin(115200);
  Serial.print("setup()");
  Serial.print(month(1));
  Serial.print(month(2));
}
void loop() {
}
String month(byte number) {
  switch (number) {
    case 1:
      return PSTR("January");
      break;
    case 2:
      return PSTR("February");
      break;
  }
}

However, this works (after reading from http://www.fourwalledcubicle.com/AVRArticles.php):

#include <pgmspace.h>
const char month01[] PROGMEM = "January\0February\0March\0April";
const byte months[] = {0, 8, 17, 23};
void setup() {
  Serial.begin(115200);
  Serial.println("setup()");
  char buffer[10];
  char* month;
  for (int i = 0; i < 4; i++) {
    month = (char*)(month01 + months[i]);
    strcpy_P(buffer, month);
    Serial.println(buffer);
  }
}
void loop() {
}

OUTPUT:

setup()
January
February
March
April

OK, so I'm seeing a plan here:

1. Create PROGMEM strings outside the class


2. use the provided strcpy_P() to copy the string from flash to SRAM.


3. use the string any way you like once it's in SRAM

Actually, just reading Dean Camera's PDF on the subject (link to "AVR-GCC and the PROGMEM attribute" on http://www.fourwalledcubicle.com/AVRArticles.php) really helped me cement my knowledge.

A few important points:

• You can't just use "normal" pointers to access flash memory. There are some bugs with this. You can see in pgmspace.h (on my machine: C:\Program Files (x86)\arduino-1.0.3\hardware\tools\avr\avr\include\avr\pgmspace.h) there is a set of assembler macros used to "translate". So, don't go diving into accessing flash memory directly! #using <pgmspace.h> is the proper way.


• As much as I'd like my strings to be declared as part of the class they belong to, as you would expect in C++, this is not possible. My limited understanding of this is that the PROGMEM operations were written for C not C++ and while they will work in compiled C++ code must be treated as they would in a C program.


• I'm still using the strcpy_P function which uses the deprecated prog_char data type. The pgmspace.h file included with Arduino doesn't say this is deprecated, but the documentation does. I'll have to look at fixing that some day I think.


• I haven't gone into the distinction between "C-strings" and String objects. I've tried to use (capital S) Strings when I mean objects.


• The single most helpful thing I found to aid in understanding was Dean Camera's PDF about PROGMEM (from here: http://www.fourwalledcubicle.com/AVRArticles.php). If time is short, read just that.

---

After a few hours coding I've now implemented this learning in the DateTime class. I found on the way through that rather than having separate PROGMEM variables declared for each month or day of the week that I can just insert null characters into a single string as a delimiter. As is the norm with C-strings, once one of those is read, the string ends. So I only need to access the correct starting location to get the full word out of the larger string.

After worrying a lot about how much memory was used by each DateTime object in holding on to static Strings, I realised I can fold them all together. By having a single getProgMemString() function that declares a static String* internally, every function that returns a String shares the same pointer and so only ever uses enough memory to hold on to one string. The two-step process this function uses to read flash to SRAM as a C-string, then convert the C-string into a String could be simplified if only the String() constructor could accept a const PROGMEM char * but it can't. It would certainly be possible to add PROGMEM capabilities to the String class and I think I could work that out. But as much as I love rabbit holes, I'm staying out of this one!

DateTime isn't finished yet, but it's looking good to me! What's your opinion?

That is all.