Keep the catalytic converter and EGR. Modern high flow converters are very efficient and can be found for less than $100. Ash can the old school myth of smaller exhaust pipes retaining torque at lower RPMs. Anything beyond the exhaust manifold exit or header collector will add very little torque. Too small an exhaust will kill any chance of gaining any torque throughout the RPM band once you start increasing airflow through an engine. The stock computer can compensate for small changes in airflow/efficiency due to better intakes, mild cams, exhaust, headers etc. Going too far outside it's design parameters will result in poor performance and possible engine damage.
Changing the exhaust to flow better air is an excellent way to improve mileage and performance. As noted before the Feds frown upon changing emission components unless the originals are faulty. In your case, the catalytic converter has been violated. Mine was the same way on my '92 F350. Completely hollow from the previous owner. The entire exhaust was rusted out and falling apart. I replaced everything including the manifolds.
I have posted before my combination on my 5.8L:
Dorman replacement manifolds (cheap until I want to spend more $$)
Bassani 58150B2 SS y-pipe ( designed for a Bronco w/5.8L)
Magnaflow Spun Series Catalytic converter p/n 59929
Magnaflow 3" In/Out Center/Center SS muffler 12279
Dynomax/Walker 45683 3" aluminized steel mandrel bent w/two hangers attached ( designed for a mid-90's Chevy truck)
Dynomax 3" SS band clamps 33272
Now I have a very free flowing mandrel bent 3" exhaust. All the parts, including manifolds, were ~$600. I installed it all in my home garage without a welder. The muffler has a 24" body so even though it is a straight through design, it is quiet with a deep rumble. I also took the opportunity to replace the missing EGR tube to make the EGR functional again. There are several manufacturers who make emission friendly headers for our trucks at a reasonable price. Keep the cat and EGR. Once you replace your old cat with a modern 3-way you can ditch the air tube system, keep the relays connected to make the computer happy. We all have to breath this air, try to keep it as clean as possible.