Doin' Uni'

In the midst of the campus, in the midst of a bar, in the midst of an nth beer, in the midst of a lecture I'm not attending, in the midst of an assignment for said subject who's lecture I'm not attending ... a moment of clarity.
University is about two kinds of time. Spending time learning. Spending time proving that you've learned.
Actually that is probably leaving out the main reason some (most) are at Uni. Spending time doing neither of the above. I should invest more time in the third. I'm pressured constantly to achieve academically. This is good for graduating with an impressive transcript, but in the meantime I'm missing life. I'll grab another beer.
While my outlook on life is improving, the quality of my assignment is declining. Oh well. One day out of 5 (days) * 12 (weeks) * 2 (semesters/year) * 3 (years) = 360 ain't bad.
Tomorrow I'm digging holes in my backyard to build my office. I hope this is the start of something new and wonderful. Time will tell. It always does.
That is all.

Micro SD Card Duplicator 6

I’ve been stuck, not on what to do next but on time. Late nights, sick kids, friends over. I can’t complain really, but at the same time I haven’t made any progress on this project in over a week!

I figure there are two ways forward. I can trace through the code and see how the ‘usual’ file read and write operations work using the standard classes, or I can start from MULITPLE_BLOCK_WRITE in the Sd2Card class and work out how to expose and utilise that through the FAT file system. Other options?

Actually, I could switch to .NET Micro Framework and use my FEZ Panda II.

Tracing SD.write()

No time to write a detailed explanation. A quick summary of what I’ve done:

1. Committed SD_Speed_3 to Unfuddle
2. SD_Speed_3.ino calls SD.begin() to initialise the card
3. This calls card.init() with parameters
4. card is private member Sd2Card of class SDClass in SD.h
5. Sd2Card is not included by SD.h but SdFat.h and SdFatUtil.h are. Checking last blog post, SdFat.h includes Sd2Card.h.
6. First parameter in card.init() sets clock rate via Sd2Card::init() –> Sd2Card::setSckRate()
7. End of analysis of card start-up. Looking at SD.open().
8. SD_Speed_3.ino declares a File myFile; which is of the FIle class in SD.h – a wrapper class over the SdFile class
9. The SdFile class is delcared in SdFat.h and implemented in SdFile.cpp
10. Jumping to myFile.write().
11. SdFat.h defines three File::write() functions. One for a single byte, one for a pointer buffer of defined length and one for a c-string. My sketch is currently writing only a single byte.
12. write() for a single byte calls write() for a string of defined length 1.
13. Reading size_t SdFile::write(const void* buf, uint16_t nbyte). Sucky, what’s with the use of goto?
14. Now we’re down into blocks and clusters. I can see what’s going on, kinda. But I really need to have a definition of blocks and clusters in my head. Need FAT specification.
15. Read FAT32 File System Specification and various other source. This one was good.

So to sum up a whole load of stuff I’m writing a byte at a time. Each byte I write causes a block of 512 bytes to be read from the card, updated, and written back to the card. I’m incurring massive overhead by not writing more than a byte at a time.

I rejigged my sketch from my SD Card Duplicator 4 post and have got writes up to 47kB/s by writing 10 bytes at a time. That’s not a huge improvement, but it’s better. Now I’m reading a block from the SD card, making ten changes to it, then writing it back. It’s more efficient, but not by much. At 50 bytes per write, I’m hitting 55kB/s.

So I have learned a few things. Looking to use MULTIPLE_BLOCK_WRITE will likely improve performance, but I’m dealing with the wrong level of abstraction there. The fact is that the SD card is working in blocks and so, separately, is the FAT file system. Writing ASCII characters to the card this way incurs one or more read, update, write-back cycles if at least 512 bytes each.

FAT

I’ve spent a lot of the day reading about the FAT file system. It’s not too complex, but it’s also not much fun. It was invented in, what, the late 70’s? It’s almost as old as me! FAT works in clusters and blocks too, and it’s interesting to see how the SD card’s blocks and FAT blocks are not the same thing. FAT generally formats data in 512 byte blocks. SD cards too usually work in 512 byte blocks. But neither is constrained to this size, which is interesting.

I guess if the SD card was working with 1024 byte blocks and FAT was using 512 byte blocks (just for example) then every read from and write to an SD card would move two FAT blocks. Conversely, if FAT was using 2048 byte blocks and the SD card was using 1024 byte blocks, then reading a FAT block would require two read operations on the SD card.

It seems that FAT and SD are both usually working on 512 byte blocks. But I read something that talked about the FAT format “not necessarily observing the SD block boundaries”. So that might mean even at the same block size, multiple reads/writes are required to properly manipulate FAT blocks.

Writing from Scratch?

I’m tempted to start from scratch. Really. A class to handle SPI operations (like, say, the existing SPI class included with Arduino) then one to handle the SD card SPI protocol. From there up to the FAT file system. I’d like to be able to do this:

SDCard cardSource = New SDCard(PIN_1);
SDCard cardDestination1 = New SDCard(PIN_2);
SDCard cardDestination2 = New SDCard(PIN_3);
SDCard* cards[] = {cardDestination1, cardDestination2};
File file = cardSource->getFirstFile();
while(file) {
  file.copyTo(cards);
  file.getNext();
}

That way the source file’s blocks are only read once and are written to as many destination cards as required. Only one buffer would be required too. But oh the work! I can’t imagine the hours of slashing through code to interpret FAT boot sectors and file allocation tables. There’s enough of that kind of code already, but there are a few critical issues with what I’ve found so far.

First, nobody seems to have considered the need to access more than one SD card. Existing code makes a single object to read/write from what is assumed to be the only card attached to the system. This means it would require a significant re-write to allow the instantiation of multiple card objects.

Second, and this gets bonus points for being my pet peeve. Code is very ‘C’ and not very ‘C++’. I can’t help but feel that this scenario is perfect for OO techniques. Then again, if I had some more C experience I might think not.

I’m at a crossroads. I’ll either decide to sink over a hundred hours into this project, or I’ll scrap it. I have other time pressures and I think it’s quite likely I’ll get up to my armpits in it before conceding I can’t finish it. What to do?

[That is all]

Micro SD Card Duplicator 5

So, getting down to brass tacks it is. I’ve printed all the code from the C:\Program Files (x86)\arduino-1.0.3\libraries\SD folder. Wasting paper isn’t my favourite pastime, but I had the chance to do some reading away from any of my computers, so I killed a tree.

The first thing I realised was that the files link together in odd ways. I needed to be able to imagine the hierarchy of which file calls which file, so I could understand where they fit into the family tree. If I can do that, I can generally see the layers of abstraction peeled away.

Micro SD Card Duplicator 4

I need another shower. Inspiration is failing me.

I took a couple of days to come back to this. I just couldn’t see a way through. In the end I sat down to graph what was going on and a plan presented itself. I don’t know if it’s going to lead to success, but it’s at least an approach. Anything’s better than being stalled.

Micro SD Card Duplicator 3

The best data write speed I've achieved so far is 21.5kB/s. That's not much. A 1GB file would take, oh, 14 hours to write to an SD card. That's not good.

It looks like the base speed for SD cards is 12.5MB/s. That's mega bytes per second. So we know SD card technology can move about a thousand times faster than we're currently going. I’ve already mentioned that SD cards have two communications protocols: SPI (SSI) and SDIO. However we don’t really have the option to use SDIO (the native protocol) so we’re just left with pushing SPI as fast as we can get it. If that doesn’t work, the gig is up.

Micro SD Card Duplicator 2

Yesterday I began speed testing reads and writes on micro SD cards. Much as I did for the Basic GPS Data Logger, I began by using Examples code in the Arduino editor and breaking it down into functions. I soon had this code. For brevity I have deleted comments, which doesn't help understanding I'm sorry. Look at the ReadWrite and Datalogger examples code under SD in the Arduino editor.

Micro SD Card Duplicator 1 (By Request)

Another by request post!

As a computer science student I'm excited to be given 'real' work to do. Microcontrollers and coding are my passion - something I don't get to do at all in my day job as an electrical engineer. Things are about to change though. I have resigned from my job and am returning to uni full time this year. The target is 8 high distinctions. After that I will have only 3 subjects remaining in my degree program. I hope my grades will earn me a place in the honours programme and from there who knows. I hope that in the gaps between my studies I'll develop some real-world useful projects, like this one.