Is this supposed to be able to be used with a phantom-powered mic, or a line-level output? It does matter, as there are different trade-offs in these two applications when working out a variation to improve the level of "offness".
Also, a good place to start would be to figure out the target level of attenuation that would be acceptable, and measure a little more precisely what you have now, perhaps play a 1KHz tone through a speaker and into the mic, so you can take a proper reading. From there, if you know the source impedance of the microphone, you can get an idea of where the problem may be. For instance, if you really are getting about -50dB attenuation, and the mic has a 150 ohm source impedance . . . this means that the switch has a around 0.45 ohm of impedance across the audioband when it's closed . . . and this is pretty high! If it's rated to have a low contact resistance, then perhaps it's defective . . . or perhaps you're really getting more like -70dB and it's still not enough, or your mic (or other driving source) has a lower source impedance . . .
In any case, nailing down some of these variables would help immensely. With a design goal of "-infinite dB" from "any source", then the only way to solve it is with "crazy overkill".