Basic GPS Data Logger (By Request)

Premise

I got the following comment on my EM-406A GPS From Scratch - Part 3 post. I hope you don't mind Milton, but I thought it was a great idea. I hope to help you and document a solution for my own future use. Maybe it'll help others too?



Hi Christian.

These are three great posts. I'm trying to figure out how to write the code for the smallest possible GPS data logging to an SD card that I can. I went through Jeremy Blum's tutorial but he uses a Mega and with an Uno the sketch gets close enough to memory capacity that it glitches and gets a Programmer Not in Sync code. I got rid of the boot loader on the chip by using a programmer on the ICSP pins but still wasn't enough. I don't need most of the formatting and such.

My goal is a very small sketch that writes NMEA data to the SD card with the ability to change sampling intervals in the software (something like your Part I sketch but writing to an SD card). I'm working on a small, long duty cycle device that is as spare as possible. If you have any ideas, that would be great. I'm going to work off of what you've done in these three post and see if I can make progress. 

So far I've been trying to eliminate things from the bloated sketches but your minimalist approach is likely more sound.
Milton 

It's a great idea and I have the equipment. Challenge accepted!

SD Card Test

Hardware

• Arduino Duemilanove
• Sparkfun microSD Shield
• Nokia micro SD card 256MB

I formatted the micro SD card using an SD card adaptor. My microSD shield has a mini breadboard stuck to it; this will be useful later. I plugged it all together and fired up Arduino 1.0.3 on my PC.

Software

In the Arduino IDE I went straight for the SD card Examples. CardInfo was the first I tried. Upload. Open Serial Monitor - bingo!

Initializing SD card...Wiring is correct and a card is present.

Card type: SD1

Volume type is FAT16

Volume size (bytes): 254799872

Volume size (Kbytes): 248828

Volume size (Mbytes): 242

Files found on the card (name, date and size in bytes): 

So the card and shield work fine. Good start. Let's try something a bit harder. ReadWrite from the SD card Examples. Upload. Open Serial Monitor - hmmm.

Initializing SD card...initialization done.

error opening test.txt

test.txt:

I know the SD card was empty when I started. So I hope the 'error opening test.txt' is not a problem. Running it again I get:

Initializing SD card...initialization failed!

Something's up. Every time I restart the Serial Monitor or press the reset button on the microSD shield it's the same. By reading around a few things I found a comment in the previous project, CardInfo, that states the chip select (CS) should be pin 8 for the Sparkfun SD shield. I made the following code change in ReadWrite:

  if (!SD.begin(8)) { // Was 4. Should be 8 for SparkFun microSD shield.

After uploading, the result is good:

Initializing SD card...initialization done.
Writing to test.txt...done.
test.txt:
testing 1, 2, 3.
testing 1, 2, 3.

So it looks like I need CS to be 8. This project successfully reads and writes to the SD card so I reckon I'll use this code in a little bit. I'll save this as project Milton01. The SD card test is complete.

GPS Test

Enter the trusty EM-406A GPS receiver. I've previously documented my issues with the Sparkfun GPS shield, so I'm just going to use jumper wires and a GPS cable I've cut one plug off and tinned the ends of.

Hardware

Note: EM-406A pins are numbered right to left when looking into the socket with the PCB below the socket. See previous posts for links to the manual or Google it.

• 6 wires from EM-406A into breadboard
• Pin 1 GPS GND to GND on the Arduino
• Pin 2 GPS VIN to 5V on the Arduino
• Pin 3 GPS RX (not connected)
• Pin 4 GPS TX to RX on the Arduino
• Pin 5 GPS GND (not connected)
• Pin 6 GPS 1PPS (not connected)

IMPORTANT! We've done something 'silly' here. The little microcontroller only has one serial port. It can only communicate with one device at a time. But we now have two devices connected to one serial port. The GPS is connected and so is the PC. When we need to upload a new sketch, we need to lift off the jumper connecting the GPS serial line to the Arduino. When the upload is done, we can put it back. Milton, could this be what's happening when you say "it glitches and gets a Programmer Not in Sync code"? If I leave my GPS connected and try to upload a sketch, this is what I get:

Software

I test the GPS using a super basic sketch. The four line sketch from EM-406A GPS From Scratch - Part 1 will do. Don't forget to pull the GPS data line off for the upload and replace it afterwards. You'll also need to change the serial port speed to 4800 to synch with the GPS speed.

The Sketch:

void setup()
{
  Serial.begin(4800);
}


void loop()
{
  if (Serial.available()) Serial.write(Serial.read());
}

The Result:

vƒ;; c F6†F–æ6F†ÆÆÆÆÆ Æ ÆÆÆÖc æ ÆÖcÆ ¦V&k
$GPGSA,A,1,,,,,,,,,,,,,,,*1E
$GPGSV,3,1,12,24,89,000,,09,55,048,,12,53,240,,04,30,089,*7F
$GPGSV,3,2,12,02,26,040,,17,21,135,,25,21,261,,15,21,341,*7B
$GPGSV,3,3,12,14,17,223,,29,02,321,,26,00,007,,22,-9,253,*60
$GPRMC,043849.348,V,,,,,,,260113,,*25
$GPGGA,043850.357,,,,,0,00,,,M,0.0,M,,0000*5D
$GPGSA,A,1,,,,,,,,,,,,,,,*1E
$GPRMC,043850.357,V,,,,,,,260113,,*23
$GPGGA,043851.348,,,,,0,00,,,M,0.0,M,,0000*52
$GPGSA,A,1,,,,,,,,,,,,,,,*1E
$GPRMC,043851.348,V,,,,,,,260113,,*2C
$GPGGA,043852.346,,,,,0,00,,,M,0.0,M,,0000*5F

Data galore! It's working. The first line is full of glitches, but that works itself out by the second line. So all the hardware's set up. We just need to combine a sketch for the SD card with a sketch for the GPS to achieve the functionality we need.

Coding the Main Sketch

Functions for SD Card Operations

The first thing I want to do is change the code for the SD card so that writing to the card is a function. I want to:

1. open the SD card
2. write text to the card
3. close the card

This, I hope, means the file is open only when needed and there's less chance of the data file getting corrupted if power is lost while the file is open. So it's back to the project I called Milton01 earlier. Looking at the code you can see all the work is done in setup(). This is unusual. You'd expect a few lines of setup() and a whole lot of things going on in loop(). But it makes sense in this scenario.

We need to divide the code in setup() into the code that:

• configures the SD card for use
• writes to the card
• reads back data from the card

The latter two can be placed into functions. So I'm going to grab what's in setup() below Serial.println("initialization done."); and move that to a new function called sdWrite(). Where I stole the code from setup() I've added a call to sdWrite();

After uploading and testing to see that the code still works the same, I copied sdWrite() to make sdRead() and changed each function:

• in sdWrite() I removed the code after the comment // re-open the file for reading:
• in sdRead() I removed the code above the same comment.

Again, I added a call into the setup() function to call sdRead(). The code is now functionally equivalent to the standard SD card example ReadWrite, but uses functions instead. This code is easier to understand and simpler to transplant into another sketch, which we'll do later.

Actually, while we're at it, why don't we take more code from setup() and make sdStart()? Again, it's functionally equivalent to ReadWrite but calls three functions to perform operations on the SD card. The only extra work we've had to do is to ensure that if sdStart() fails, the other two functions are not called; they'll create errors if they do. So sdStart() returns a boolean to indicate if it succeeded or failed. An if statement uses this result to decide if sdWrite() and sdRead() should be called.

Here's the full code at this stage:

#include <SD.h>;

File myFile;

void setup()
{
 // Open serial communications and wait for port to open:
  Serial.begin(9600);
   while (!Serial) {
    ; // wait for serial port to connect. Needed for Leonardo only
  }

  if (sdStart()) {
    sdWrite();  
    sdRead();
  }
}

void loop()
{
  // nothing happens after setup
}

boolean sdStart()
{
  Serial.print("Initializing SD card...");
  // On the Ethernet Shield, CS is pin 4. It's set as an output by default.
  // Note that even if it's not used as the CS pin, the hardware SS pin 
  // (10 on most Arduino boards, 53 on the Mega) must be left as an output 
  // or the SD library functions will not work. 
   pinMode(10, OUTPUT);
   
  if (!SD.begin(8)) { // Was 4. Should be 8 for SparkFun microSD shield.
    Serial.println("initialization failed!");
    return false;
  }
  Serial.println("initialization done.");
  return true;
}

void sdWrite()
{
  // open the file. note that only one file can be open at a time,
  // so you have to close this one before opening another.
  myFile = SD.open("test.txt", FILE_WRITE);
  
  // if the file opened okay, write to it:
  if (myFile) {
    Serial.print("Writing to test.txt...");
    myFile.println("testing 1, 2, 3.");
 // close the file:
    myFile.close();
    Serial.println("done.");
  } else {
    // if the file didn't open, print an error:
    Serial.println("error opening test.txt");
  }
}

void sdRead()
{
  // re-open the file for reading:
  myFile = SD.open("test.txt");
  if (myFile) {
    Serial.println("test.txt:");
    
    // read from the file until there's nothing else in it:
    while (myFile.available()) {
     Serial.write(myFile.read());
    }
    // close the file:
    myFile.close();
  } else {
   // if the file didn't open, print an error:
    Serial.println("error opening test.txt");
  }
}

The Final Sketch

Surely there isn't much left to do? I started a new Arduino project and called it Milton03. I grabbed all the code from EM-406A GPS From Scratch - Part 2 and pasted that in. Because I haven't tested that lately, I ran it as-is to see if it would work.

Start!
INVALID: C,051750.360,V,,,,,,,260113,,
Valid:   $GPGGA,051751.360,,,,,0,00,,,M,0.0,M,,0000
Valid:   $GPGSA,A,1,,,,,,,,,,,,,,,
Valid:   $GPRMC,051751.360,V,,,,,,,260113,,
Valid:   $GPGGA,051752.360,,,,,0,00,,,M,0.0,M,,0000
Valid:   $GPGSA,A,1,,,,,,,,,,,,,,,
Valid:   $GPRMC,051752.360,V,,,,,,,260113,,
Valid:   $GPGGA,051753.360,,,,,0,00,,,M,0.0,M,,0000

We're in business! I mean, the point of me writing these blog post is so that I can grab old code, use it successfully and understand why I wrote it the way I did. If I can't just grab it and have it work straight away I'm going something wrong!

So we have two Arduino sketches. Milton03 can read NMEA sentences out of the GPS and report if they're valid or not. Milton02 can read/write to the SD card. Let's smash them together and (hopefully) get a working combination of the two!

These are the code pieces I copied over from Milton02 (SD card read/write sketch) to Milton03 (GPS data sketch):

• the #include statement at the top
• the declaration File myFile; near the top
• the three functions
• sdStart()
• sdWrite()
• sdRead()

The updated Milton03 sketch complies (sorry, verifies) so I save it. We need to call the three new SD card functions at the appropriate time and ensure sdWrite() writes the GPS sentence, not some arbitrary junk like it currently does.

I added a global variable called sdOK to record if the SD card started properly and put the line sdOK = sdStart(); in setup(). Now we ensure the SD card is started from setup() and we know anywhere in the sketch if the SD card is OK to use. I added if (!sdOK) return; to the top of sdWrite() and sdRead() to force those functions to quit if the SD card didn't start properly.

In getGpsData() I added a call to sdWrite() and sdRead() just for testing and changed the arbitrary text in sdWrite to say "Valid NMEA data received.". Now, when a valid GPS sentence is received, "Valid NMEA data received." is written to the SD card and read back in. Let's test it out.

Alright. It works as expected. I won't post the code just yet. I just need to ensure the GPS data is written out to the SD card before I do that. By taking a look at the code in printSentence(char *sentence, byte length) and incorporating that into sdWrite(), I can write the GPS data out to the SD card.

Uploading - again with the GPS data line disconnected then reconnecting after. Running the program - it works! The only issue is the text file isn't getting line breaks when it needs them. A quick addition to sdWrite() will fix that. While I'm in there I'll seach-and-replace the "test.txt" file name with "gps_log.txt". Retest. Brilliant!

So here's the full code:

#include <SD.h>;
File myFile;

static const byte GPS_BUFFER_SIZE = 100;
                  // The maximum size allowed for an NMEA sentence.
char sentence[GPS_BUFFER_SIZE];
                  // The character array where the NMEA sentence
                  // is stored.
byte sentenceIndex;
                  // The position in the sentence where the next
                  // character will be appended.
boolean checking; // Determines if characters arriving from GPS
                  // are X-OR'd for comparison to check digits.
boolean sdOK;
char checkHex1, checkHex2;
                  // The last two characters of the NMEA sentence.
int checksum, checkDigit;
                  // checksum: the running total of X-OR operations
                  // on all the characters arriving from the GPS
                  // checkDigit: which of the check digits is
                  // is arriving next

//---------------------------------------------------------------------------------------------
void setup()
{
  Serial.begin(4800);
  Serial.println("Start!");
  sentenceIndex = 0;  // resetting all variables
  checking = false;
  checksum = 0;
  checkHex1 = 0;
  checkHex2 = 0;
  checkDigit = 0;
  sdOK = sdStart();
}

//---------------------------------------------------------------------------------------------
void loop()
{
  getGpsData();
  // This gets called repeatedly. As soon as it's finished
  // it's called again.
}

//---------------------------------------------------------------------------------------------
void getGpsData()
{
  while (Serial.available())
  // When there's no data available this function quits.
  {
    char character = Serial.read(); // get a single character
    switch (character) // decide based on the character received
    {
    case 10: // ASCII line feed 'LF'
      continue;  // just ignore it, restart at while()
    case 13: // ASCII carriage return 'CR'
      // This is the end of the sentence. Process the entire
      // sentence and restart for the next sentence.
      if (isValid()) // see function below
      {
        Serial.print("Valid:   ");
        sdWrite(sentence, sentenceIndex);
        sdRead();  // Use for debugging only. Comment out for normal use.
      }
      else
        Serial.print("INVALID: ");
      printSentence(sentence, sentenceIndex);
      Serial.println();
      sentenceIndex = 0;  // restart the sentence position
      checksum = 0;  // reset the checksum
      continue; // go back to while()
    case 42: // ASCII asterisk '*'
      // Stop adding characters to the sentence and instead
      // collect the check digit information.
      checking = false; // don't add remaining characters to 'checksum'
      checkDigit = 1; // next character will be first check digit
      continue;
    default:
      break;
    }
    if (checkDigit == 1)
    {
      checkHex1 = character;
      checkDigit++; // next character will be checkDigit 2
    }
    else if (checkDigit == 2)
    {
      checkHex2 = character;
      checkDigit = 0; // no more check digits
    }
    else sentence[sentenceIndex++] = character;
      // Append the current character to the sentence
    if (sentenceIndex == GPS_BUFFER_SIZE)
      // Check we haven't overrun the sentence array, if so, restart
    {
      sentenceIndex = 0;
      checksum = 0;
      continue;
    }
    if (checking) checksum ^= character;
      // X-OR character with previous X-OR sum
    if (character == 36) checking = true; // ASCII Dollar '$'
      // Leading $ is not included in checksum calculation, but
      // does indicate when to start calculating the checksum.
  }
}

//---------------------------------------------------------------------------------------------
boolean isValid()
// Compares the calculated [chechsum] to the check digits [checkHex1, checkHex2]
// supplied by the GPS. If the calculation matches the sentence is valid.
{
  return (charToHex(checkHex1) == (checksum >> 4) && charToHex(checkHex2) == (checksum & 0x0F));
}

//---------------------------------------------------------------------------------------------
byte charToHex(char in)
// Converts a hexadecimal character 
// {'0', '1', '2', '3', '4', '5', '6', '7', '8', '9', 'A', 'B', 'C', 'D', 'E', 'F'}
// into its numeric equivalent
// { 0,   1,   2,   3,   4,   5,   6,   7,   8,   9,  10,  11,  12,  13,  14,  15}
{
  if (in >= '0' && in <= '9') return in - '0';
  if (in >= 'A' && in <= 'F') return in - 'A' + 10;
  return 0;
}

//---------------------------------------------------------------------------------------------
void printSentence(char *sentence, byte length)
// Prints a specified number of characters from a character array
// and shows any non-printing characters by their ASCII number.
{
  for(int i = 0; i < length; i++)
  {
    if (sentence[i] < 32)
    {
      Serial.print('<');
      Serial.print(sentence[i], DEC);
      Serial.print('>');
    }
    else
    {
      Serial.print(sentence[i]);
    }
  }
}

boolean sdStart()
{
  Serial.print("Initializing SD card...");
  // On the Ethernet Shield, CS is pin 4. It's set as an output by default.
  // Note that even if it's not used as the CS pin, the hardware SS pin 
  // (10 on most Arduino boards, 53 on the Mega) must be left as an output 
  // or the SD library functions will not work. 
   pinMode(10, OUTPUT);
   
  if (!SD.begin(8)) { // Was 4. Should be 8 for SparkFun microSD shield.
    Serial.println("initialization failed!");
    return false;
  }
  Serial.println("initialization done.");
  return true;
}

void sdWrite(char *sentence, byte length)
{
  if (!sdOK) return;
  // open the file. note that only one file can be open at a time,
  // so you have to close this one before opening another.
  myFile = SD.open("gps_log.txt", FILE_WRITE);
  
  // if the file opened okay, write to it:
  if (myFile) {
    Serial.print("Writing to file ...");
    for(int i = 0; i < length; i++)
    {
      myFile.print(sentence[i]);
    }
    myFile.println();
    // close the file:
    myFile.close();
    Serial.println("done.");
  } else {
    // if the file didn't open, print an error:
    Serial.println("error.");
  }
}

void sdRead()
{
  if (!sdOK) return;
  // re-open the file for reading:
  myFile = SD.open("gps_log.txt");
  if (myFile) {
    Serial.println("Reading from file:");
    
    // read from the file until there's nothing else in it:
    while (myFile.available()) {
     Serial.write(myFile.read());
    }
    // close the file:
    myFile.close();
  } else {
   // if the file didn't open, print an error:
    Serial.println("Error.");
  }
}

Parting Remarks

We have not implemented a nominal delay between writes to the SD card. But this is a one-line addition to the sdWrite() or sdRead() functions, delay(10000) for example. A few additions to the code would make it more maintainable in the future:

• Making the name of the text file a variable
• Making the delay value a variable
• Commenting the call to sdRead() in setup() to make it run at real-time speed

I don't believe the solution I've offered here is the "smallest possible", but it's light weight and could be trimmed. I hope you've had success Milton. If not, I hope this helps!

[That is all]