Here's a fun fact: Every time you do a voice search, Google records it. And if you're an Android user, every time you say "OK Google," the company records that, too. Don't freak out, though, because Google lets you hear (and delete) these recordings. Here's how. Image: Google
Head over to Google's Voice and Audio Activity page and start deleting all those recordings. You can delete them individually or all at once, just click the More > Delete Options > Advanced to get there. Each file will also have a plaintext transcript and recording information associated with it for your perusal. While you're shoring up privacy settings, you can also tell Google to stop tracking which web pages you visit here, or turn off everything from a map of stored locations to YouTube watch history on this page.
While voice recordings can be "paused", they will be reactivated the next time you use Virtual Assistant, Voice Search, or say "OK Google." Which kind of sucks! So stay diligent about deleting stuff, especially if you say anything embarrassing and/or incriminating to your phone.
Again, the recording feature is definitely more of a hindrance for Android users. That said, having the recordings available for users to listen to is markedly more transparent than similar services like Siri (which stores your data for up to two years, unless you turn off the service) or Amazon Echo (which may or may not be wiretapped by the FBI).