In your stylesheet you'll want to look for these styles and change any or all of them:
.qa-voting and .qa-a-count - reduce the padding on top & bottom, and change the colors if need be.
.qa-vote-buttons - reduce the width and/or padding.
.qa-netvote-count - reduce the width.
.qa-netvote-count-data and .qa-a-count-data - reduce the font size.