Mars Rover Recovering From Memory Problems

New Scientist ^ | 1-28-2004 | David L Chandler

[blam]

Mars rover recovering from memory problems

13:35 28 January 04

NewScientist.com news service

A full revival of the Mars rover Spirit from its electronic ailments now seems highly likely. Engineers now think there is no real hardware or software problems, but something much easier to fix - a simple overload of files in its onboard memory.

If further testing confirms this diagnosis, that will be very good news for Spirit's twin, Opportunity. Any software bug or hardware weakness would probably be present in both rovers and might require weeks of analysis and repair.

But if, as it appears, the problem is a previously unrecognised limit on the number of files that can be stored in the craft's flash memory, then Opportunity's data collection and file management can be planned to prevent the problem.

This would avoid the bleak situation faced by engineers when Spirit fell silent for more than a day and failed to respond to commands. Having initially described the Spirit's troubles as "critical", mission manager Jennifer Trosper says "the patient is now in rehab".

However, Opportunity has developed a problem of its own, according to another mission manager Jim Erickson. The rover is losing power, apparently due to a heating unit that is switching itself on when it should not. What this will mean for the rover's mission and whether it can be fixed are not yet known.

Coaxed communication

Spirit's controllers have been coaxing the rover back into communication since it ended its silence on Thursday with a single bleep. The engineering data returned has allowed them to piece what had happened.

The rover first failed halfway through a test of a moving mirror that directs light to the mini-TES instrument. The high-gain antenna was also being used at the time, and the spacecraft entered a "safe mode" associated with antenna problems.

Later data returns showed the craft had entered a repeating cycle of resetting its computer system, preventing it from carrying out anything but the simplest commands. At last count, it had rebooted itself more than 120 times. This constant resetting prevented it from entering its night sleep cycle, needed to conserve its batteries.

But detailed analysis of the start of each reset cycle eventually led to the apparent answer to the mystery. The problem was clearly associated with the handling of files being written to one of its three types of internal memory: a non-volatile 256 megabyte flash memory.

Testing on Monday and Tuesday suggests that it is not the flash memory itself that is at fault, but the software's file-handling system. Unbeknownst to the engineers, there seems to be a limit on the number of files that can be simultaneously stored in the flash memory, even though the overall memory capacity is not full.

The solution is likely to be simply deleting unneeded files, many of which were accumulated during the eight-month journey to Mars. It will require some skillful programming to get the computer to do this without falling back into its resetting cycle, but Trosper says a full recovery is now expected.

David L Chandler

[Post Toasties]

I wonder how much of NASA's other hardware is constrained by this type of OS limitation? Also, why couldn't the number of files storable on a given memory be a SW configurable parameter upon reset? Then, if it turns out to be an inappropriate size, simply change that parameter rather than have to rewrite and revalidated the whole OS, which is vastly more time consuming.

[brownsfan]

This looks to me like a classic off-by-one error

Actually, they said it was the difference between the number of files the flash could handle vs. the physical size of the files. That is: The memory isn't full, but it's out of file handles.

That's what they say... but this is the same crew that said: "ooops, we calculated in feet but programmed in meters!" (Or vice versa).

[xp38]

I think it was Jawas but on examining Spirit they concluded Spirit wouldn't be sellable to a moisture farmer.

[Daus]

However, Opportunity has developed a problem of its own

NASA has just announced they have changed the names of the Rovers to Pinto and Nova.

[Consort]

A hardware/software glitch is toooo booring. I prefer to think that they found/saw/are testing/are implementing/are planning/are installing.... someting that they don't want to share with us — something......Art Bellian.

[Consort]

someting = something

[DeepDish]

That is the problem is it not. It almost takes a miracle to pull off a successful mission. A flawless mission would be like the second coming of Christ with no casualties. No way, and no how.

One thing JPL said, before the problems cropped up, was these landers were put together fast, not cheap, but fast because of the launch window. An extra year for design analysis would have been nice.

[from occupied ga]

Is there any reasonable explanation for not having run some continuous testing before we shot this sucker into space?

From the same people who brought you the Hubble and didn't test it either, but what the heck. It's only taxpayer money.

[N3WBI3]

In most systems there is a limint in the number of files (in a given Linux box it is the number of inodes assigend to a partitions, NTFS has a fixed limit of 4,294,967,295 files.

[brownsfan]

it seemed odd because DOS is obsolete, but the FAT filesystem is used in lots of places

Not so odd. FAT is well known, and readily available. In a situation like a digital camera, (or a rover?) it is reliable enough. Modern filesystems expand performance, and security. FAT is fine for small applications.

[DeepDish]

Yeah, like it turns out that the rock ledge with the vertical fractures is really the remains of a wall. </tinfoil off>

[dljordan]

Huh? Wha' happened? Where am I? Millions of dollars that could have been used to help undocumented workers were wasted. It is to weep.

[tm22721]

As a hardware designer, I could've told NASA that nine out of ten of these problems are software:)

And I can tell you as an EE with 30 years of digital hardware and software design experience that many late stage software patches result from bad or poorly specified hardware. Software changes are the quick fix that avoid the emabarrassment of changing out faulty hardware on the customer's premises.

BUMP

[ClearCase_guy]

Opportunity has developed a problem of its own

Latest news: transient problem with Opportunity. This problem occurred once so far. It seems that Opportunity transmitted a disturbing banging sound. Engineers do not expect this problem to recur, having noted that Opportunity knocks but once.

/rimshot

[lilylangtree]

What's that drug for treatment of alzehiemers(sp?)?

[Consort]

Yeah, like it turns out that the rock ledge with the vertical fractures is really the remains of a wall.

The wall or slab was probably was a Universal Rosetta Stone that the space craft destroyed beyond use by the constant bouncing and rolling that was reported during the landing phase.

[gesully]

Maybe the engineers built the system on Windows ME and the problem is just a simple blue screen of death.

[FourtySeven]

How long do you think that it takes a packet to reach the Mars Rover from Earth?

I've heard it takes 10 minutes for radio signals to travel......one way. I don't know how much information can be transmitted in one radio "burst" though.

[DeepDish]

My best science fiction hope is that the "wall" turn out to be a seep of ice, once the dust is knocked off. To me that would be as stunning as a universal Rosetta stone.

[Bikers4Bush]

Basic computer knowledge and NASA seems to be devoid of it.

Great.