What do you mean by "directly"? If you mean they typed in the URL, or are using a script of some kind to download your site, they will have a blank Referer. You can block that by removing the last RewriteCond line in your example there. As far as I know that won't cause any problems with regular browsers.
Otherwise it's not really possible. I find the best solution is to keep an eye on your logs and block hotlinking sites (using a blacklist, rather than a whitelist as you have now).