Yes, I would pull the pan and clean up the lower end. Look carefully for bearing debris. Hope you don't find any. Be careful working inside the pan. Most of those engines used leaded gasoline and the lead would settle in the bottom of the pan. Wear nitrile gloves and minimize your exposure.
Replace the pan using new gaskets and rope seals.
Next I would remove the heads and look closely for cracks around the valve seats. Wire brush around the valves and elsewhere in the combustion chamber to expose any cracks. If you can afford it take it to an engine shop that has experience with flatheads and have them inspect for cracks.
If it passes inspection look at the cylinder bores and pistons for excessive wear or scoring. Check for proper valve action too. Unstick any stuck valves too.