I have seen many car stereos cause that same problem too; the internals (usually the memory line) gets a short and drains the battery. Even if the radio works just fine otherwise, it can still cause the problem. If you are having troubles narrowing it down, completely disconnect the stereo and continue testing. One last possibility is that the battery is just failing, I had another car where that was the problem.

Steve