When you click on links to various merchants on this site and make a purchase, this can result in this site earning a commission. Affiliate programs and affiliations include, but are not limited to, the eBay Partner Network.
It's most likely your master cylinder. It has 2 boots, one for the front brakes and one for the rear brakes, fluid is going past the first boot allowing the pedal to drop with no external leaks. I've experienced this a few times.
The first thing I would do is to pull both rear drums to see that everything is OK. If rear shoes, parking brake linkage, etc. is all OK I would suspect the master cylinder. It is very unlikely to be the booster. When these fail the pedal usually goes hard.