Mars Rover Recovering From Memory Problems

New Scientist ^ | 1-28-2004 | David L Chandler

[blam]

Mars rover recovering from memory problems

13:35 28 January 04

NewScientist.com news service

A full revival of the Mars rover Spirit from its electronic ailments now seems highly likely. Engineers now think there is no real hardware or software problems, but something much easier to fix - a simple overload of files in its onboard memory.

If further testing confirms this diagnosis, that will be very good news for Spirit's twin, Opportunity. Any software bug or hardware weakness would probably be present in both rovers and might require weeks of analysis and repair.

But if, as it appears, the problem is a previously unrecognised limit on the number of files that can be stored in the craft's flash memory, then Opportunity's data collection and file management can be planned to prevent the problem.

This would avoid the bleak situation faced by engineers when Spirit fell silent for more than a day and failed to respond to commands. Having initially described the Spirit's troubles as "critical", mission manager Jennifer Trosper says "the patient is now in rehab".

However, Opportunity has developed a problem of its own, according to another mission manager Jim Erickson. The rover is losing power, apparently due to a heating unit that is switching itself on when it should not. What this will mean for the rover's mission and whether it can be fixed are not yet known.

Coaxed communication

Spirit's controllers have been coaxing the rover back into communication since it ended its silence on Thursday with a single bleep. The engineering data returned has allowed them to piece what had happened.

The rover first failed halfway through a test of a moving mirror that directs light to the mini-TES instrument. The high-gain antenna was also being used at the time, and the spacecraft entered a "safe mode" associated with antenna problems.

Later data returns showed the craft had entered a repeating cycle of resetting its computer system, preventing it from carrying out anything but the simplest commands. At last count, it had rebooted itself more than 120 times. This constant resetting prevented it from entering its night sleep cycle, needed to conserve its batteries.

But detailed analysis of the start of each reset cycle eventually led to the apparent answer to the mystery. The problem was clearly associated with the handling of files being written to one of its three types of internal memory: a non-volatile 256 megabyte flash memory.

Testing on Monday and Tuesday suggests that it is not the flash memory itself that is at fault, but the software's file-handling system. Unbeknownst to the engineers, there seems to be a limit on the number of files that can be simultaneously stored in the flash memory, even though the overall memory capacity is not full.

The solution is likely to be simply deleting unneeded files, many of which were accumulated during the eight-month journey to Mars. It will require some skillful programming to get the computer to do this without falling back into its resetting cycle, but Trosper says a full recovery is now expected.

David L Chandler

[Tallguy]

Testing on Monday and Tuesday suggests that it is not the flash memory itself that is at fault, but the software's file-handling system. Unbeknownst to the engineers, there seems to be a limit on the number of files that can be simultaneously stored in the flash memory, even though the overall memory capacity is not full.

Good grief! This isn't Rocket Surgery y'know!

[Lokibob]

On the JPL site this am, they said that Spirit transmitted 2 pictures overnight. They were optimistic for the recovery.

[Bikers4Bush]

So as I understand it we sent a probe to a planet millions of miles away and never bothered to have some computer geeks do a dry run of the hardware and software before we sent it?

Is there any reasonable explanation for not having run some continuous testing before we shot this sucker into space?

[js1138]

Need to use the move command instead of copy.

[Tallguy]

Is there any reasonable explanation for not having run some continuous testing before we shot this sucker into space?

Yeah, and you have to wait 1-1/2 years for the darned things to arrive at Mars after launching, too! Just what were the so-called systems experts doing during that time? Twiddling their thumbs?

BTW, most operating systems have a limit when it comes to the number of files. So why should the Rover's OS be any different?

[brownsfan]

Is there any reasonable explanation for not having run some continuous testing before we shot this sucker into space?

No, the explaination is not reasonable, but troubling. The rovers were designed to last 90 days, (and travel 8 months). The longest test was 9 days! NASA is still infested by "faster, cheaper". I can't even imagine who would be dumb enough to let these guys send them to Mars!

[N. Theknow]

Rovezhiemer's?

[lormand]

I couldn't imagine the latency involved with uploading the software fix.

How long do you think that it takes a packet to reach the Mars Rover from Earth?

[Post Toasties]

As a hardware designer, I could've told NASA that nine out of ten of these problems are software:)

[BigWaveBetty]

"Where am I? Who am I?"

[Slicksadick]

This mornings briefing starts at 9:20 am Pacific time, and can be viewed at JPL's site.

http://www.jpl.nasa.gov/index.html

[Paradox]

SHEESH! First of all, this is GREAT NEWS, and then, its is BAD NEWS.

Great news is that the Spirit is Lifted!

Bad news is that some IDIOT forgot to check this out BEFORE LAUNCH, or heck, even in transit!

Dear numbnutz, its mv, not cp!

[HamiltonJay]

Its not a matter of dry run of the hardware or software, its the fact that despite the best efforts of thousands of individuals, all for the most part highly intelligent, unforseen situations arrise.....

This is very common in software, to find problems in underlying systems that are not documented or known or even acknowleged by the manufacturer, or other underlying system.

Or even if documented aren't really considered because no one bothered to consider the contingency.

Fortunately it appears salvagable which is good... the other rovers heater though, now that's going to drain the batteries long before it should.. I hope they get that resolved quickly.

[DeepDish]

"But if, as it appears, the problem is a previously unrecognized limit on the number of files that can be stored in the craft's flash memory, then Opportunity's data collection and file management can be planned to prevent the problem."

It sounds like more of a problem in the flash memory's file limitation. It could be as simple as the programmer not reading the specs on the memory, or the memory manufacturer may have mis-published the spec, or it may be an undocumented feature(bug) in the memory that the manufacturer was unaware of.

So far they have handled this very well IMO. Mariner 1 was blown up at launch by the range safety officer when it flew off course at launch because of a misplayed comma in the guidance coding.

[Mannaggia l'America]

Almost sounds like they are using the MSDOS FAT filesystem, which has a limit of 512 files in the root directory.

Memory cards like those found in digital cameras use FAT. (Which is probably why Microsoft recently decided to assert it's patent on FAT - it seemed odd because DOS is obsolete, but the FAT filesystem is used in lots of places.)

[NCjim]

I seem to recall a limitation in DOS that there was a maximum number of files that could be stored in the root directory. Therefore you had to make subdirectories to circumvent this limitation. Deja vu?

[DeepDish]

misplayed=misplaced oops!

Sorry about the spelling but when the misplaced comma misplayed the program the mission was lost.

[js1138]

As a hardware designer, I could've told NASA that nine out of ten of these problems are software:)

As a software designer (lightweight) I am amused by the prospect that software can be repaired and rebuilt from millions of miles away.

This looks to me like a classic off-by-one error in allocating memory for the file allocation table. We shall see. Either that or they simply assumed that file memory would fill up before there were too many files for the index.

[brownsfan]

Its not a matter of dry run of the hardware or software, its the fact that despite the best efforts of thousands of individuals, all for the most part highly intelligent, unforseen situations arrise.....

I respectfully disagree. I don't think the proper effort was given to "dry runs". I think that testing was pushed aside, 9 days isn't enough. On a larger scale, I'm concerned that NASA no longer has the dedication and drive to work "miracles" as it once did. Here's hoping they get both rovers repaired. I support NASA's mission, I'm just not sure about the current NASA administration.