Depends on where the cap is being placed in the circuit as to whether you'll hear any differences. Capacitors have characteristics like ESR, inductance, resistance, a percent tolerance of value accuracy, etc.
When introduced into a circuit, a different component having these different characteristics can affect the frequency response in a different manner. Not so much because of the material, but because of these different electronic characteristics inherent to it.
I will say that tantalums are stupid sensitive to correct polarity - to the point of violently exploding if the polarity is incorrectly crossed. Electrolytics are also polarized, but depending on where they are installed in a circuit, the polarization won't harm them.