Mars Rover Recovering From Memory Problems

New Scientist ^ | 1-28-2004 | David L Chandler

[Mannaggia l'America]

Also, why couldn't the number of files storable on a given memory be a SW configurable parameter upon reset?

It's not that easy if they are using the FAT filesystem. It's a hard limit.

The FAT filesystem stores the root directory differently than subdirectories. It has a fixed size and goes in a specific location on the media. Even MS couldn't expand it without breaking existing software (but the FAT32 and NTFS filesystems do not have this limit).

Now, if they are indeed using the FAT filesystem and ran into the root directory limit, they could easily solve it by storing the files in subdirectories under the root - the number of files allowed in subdirectories is limited only by the amount of space available.

[Consort]

...a seep of ice...

Yes. I posted the other day that it resembled cracked ice. Of course, it could be the remains of flotsam, jetsam, or ligan.

[longtermmemmory]

R'Ver i seek the creator. (/h)

[muskogee]

8-bit,16-bit,32-bit,64-bit,or 128-bit file system I wonder? The write command probably failed without giving an error. Easy to miss in testing if you are spot checking reads and writes.

[HamiltonJay]

Miracles?

They just shot 2 slugs into space, landed them on a foreign planet that 1/2 the attempts to reach have failed, had them land on the planet almost exactly where we wanted them to.... And despite travelling for 8 months in space and how many millions of miles they landed, and for the most part are operating properly....

I don't know what counts as miracles in your book, but do the statistical odds that we would get one of these things to even get to mars and answer, much less both and both still work?

[meanie monster]

I saw on PBS that a lot of software was programmed during flight from earth to mars. They can program them from earth.

[discostu]

Welcome to the real world. No ammount of testing will find more than about 1/3 of the bugs. The goal of testing is to find the same 1/3 that would be encountered in the field. They missed one, it happens, a lot, on more expensive and more important systems than this. The good news is it seems to be easily recovered from has an easy work around.

[NotJustAnotherPrettyFace]

Mars Rover has Alzheimers?!

[Bikers4Bush]

#46 would explain a lot.

[OrioleFan]

And here I thought Spirit was transmitting back "Renegotiate" until it found out Opportunity had landed and it wasn't the only game in town.

[Nick Danger]

The longest test was 9 days!

Maybe I'm just a forgiving kind of guy, but I could see how this could happen. The operating system in use here is a fairly old one with a good track record. It's probably one of the last places they expected to have an issue. They have this huge electro-mechanical machine full of moving parts and bleeding-edge technology for imaging, telecommunications, and locomotion on uncertain surfaces. If you're drawing up the testing budget, how much time do you allocate to debug an OS that's been in service in ten million little widgets around the world for a decade?

The thing is, embedded systems do not often have the problem of managing lots of "files." That's not really what they are about. Embedded-system OS's are about compact size, interrupt latency, and managing lots of little tasks. "File management" is what they do in IT shops; it hardly ever comes up on the factory floor or in a heart-lung machine.

So, they got surprised. It happens to everybody. People who haven't made any mistakes haven't tried enough new things.

I would love to have been on the team that figured out what this was. When it finally dawned on somebody what was causing this, they must have whooped for joy, because this is an easy fix and the $400 million machine is safe. There are a lot of ways this could have turned out worse.

[discostu]

Fat32 has a limit for a single folder but it's pretty absurd (I think is 2048, might be 10,000ish), only time I ever run into is dumping things through the print spooler because those files are tiny and you can fill the spool directory before filling the harddrive. NTFS is wide open though.

[discostu]

That's SOP for these Mars missions. Most if not all of our landers have had the software loaded on the fly as it got to Mars. The orbitter than was mixed between English and metric measurements and tried to go through the planet had it's orbiting routine loaded minutes before it started the manuever. Fully writing and testing the software has to happen after the hardware is made, so you can either launch with incomplete software and try to get it finished before your object completes it's journey or delay the mission a year and a half and launch then. In general this way is better, though some complications can arise.

[Bikers4Bush]

I can see the benefits. The only problem being it would seem the hardware guys aren't talking to the software guys.

Think about the advancements in hardware that occur over a 2 year period. That has to make the job for the guys writing the software tough when they aren't thinking about the hardware limitations initially.

[reagan_fanatic]

Maybe they just need to upgrade their current version of Windows. Reminds me of an old joke:

There are three engineers in a car; an electrical engineer, a chemical engineer and a Microsoft engineer. Suddenly the car just stops by the side of the road, and the three engineers look at each other wondering what could be wrong. The electrical engineer suggests stripping down the electronics of the car and trying to trace where a fault might have occurred. The chemical engineer, not knowing much about cars, suggests that maybe the fuel is becoming emulsified and getting blocked somewhere.Then, the Microsoft engineer, not knowing much about anything, comes up with a suggestion: "Why don`t we close all the windows, get out, get back in, open the windows again, and maybe it`ll work !?"

[js1138]

OK, that's another way of saying the same thing. A file handle is just an address. The handle is a number that takes memory to store. Someone determinined the maximum number of files and allocated a certain amount of memory to store the file handles. There appears to have been no check to prevent exceeding this number.So when the one-too-manyth file handle is added it overwrites something important in memory. Result is a reboot.

As an old "c" programmer, I know this one quite intimately. When you have to squeeze a lot of code into a tiny amount of memory, you put the obligation on the user not to break things.

[Howlin]

Mars Rover Recovering From Memory Problems

Gee, I wonder if JPL can help me?

[scouse]

Nice picture, but it gives rise to a few questions: For example; Why did they send a wood burning stove and where are they going to find the logs?

[discostu]

They probably are but somebody probably forgot. Assume for a second it's because the flash card uses FAT with it's well documented file limit, probably at some point the hardware guys mentioned the flash card was using FAT but probably nobody thought about the implications of that, most users that go back to the FAT days were taught good file management techniques and probably never ran into the file limit (which remember isn't actually a bug, it's a documented design limitation), I've been using computers since DOS 3.1 and didn't run into the file limit until last year and only hit that because the software I test professionally can (if you configure it inefficiently) run massive quantities of files through the Windows print spooler, and if I'd setup that machine as NTFS instead of FAT32 I wouldn't have run into then either. You're not going to write your software to keep track of the number of files that's what an OS is for, keeping from running over the file limit is really a user thing that's system management and maintenance that on lean systems (which this would be to maximize storage capacity) is supposed to be done by hand.

In the end I'd gues this is something that didn't make the "to do" list of the operator sending instructions to the rover, somewhere on there was an entry to check free space and delete files as necessary, now a new entry just got penciled in.

[DMCA]

I have several CF cards that do not use FAT, they use various other flash tuned file systems. Since these probes run VxWorks the flash more than likely has a file system on it that spread out the writes to ensure that no one section of the flash is written to more that the others. This makes that flash last longer.