My only audio recording device is my webcam and it is incapable of being in use without the light being on as far as I am aware. So how would it send them audio data without the audio device realize it is being used?
I take the view that anything under software control or the control of a chip I can't open is suspect, I've taped the webcam on laptops and physically disconnected built in microphones (I use a headset, built in ones suck).
I'm not really happy about my Nexus 4 at all either, I think my next phone will be a dumb mobile.
The sad thing is that I try to avoid paranoia with this stuff but the threat landscape is so large it's practically a full time job staying up to date with whats going on.