The fuel charge in any engine does its work by expanding as it burns, pushing the pistons down in the cylinders. Once the fuel charge has accomplished this task, the main thing you want to do with the leftovers of combustion is get them out of your car as efficiently as possible - although turbocharged cars scavenge a bit more energy out of the gas on the way out. This very simple function is accomplished by your exhaust system, and that is the most effective place where you can easily bolt on some relatively low-cost horsepower.
As the name implies, a cat-back exhaust system replaces your factory exhaust from the last catalytic converter to the exhaust tips at the rear of the car. This section of the exhaust system is where the mufflers are found. Desirable features in this part of the power system are velocity and free flow with as few bends as possible.
Because of differing chassis underbody configurations and rear bumper designs, Corvette exhaust systems are not shared among different model eras. Make sure the system you buy will fit your car! Cat-back exhausts are readily available for all Corvette models, and you can reasonably expect to gain 15-30 horsepower from a good cat-back exhaust.
You can get a cat-back exhaust for your C4 Corvette from about $300 for a Magnaflow up to about $1,500 for a Borla system for a C6. You can install these in your own garage in an afternoon. For older cars, you can get headers and a wide variety of performance exhausts, including the ever-popular side-pipes - that's what I'm putting on the About.com Project Corvette!