Unfortunately you pretty much can't, since Google needs to know about the file. To be honest it's Google being pretty stupid thinking that showing that file in SERPs is ever useful to end users.
One possibility is to use a gzipped file instead of XML, i.e. sitemap.xml.gz. You'd need to customize the sitemap plugin for that, though.