Instead of a simple average why not implement a median rating? That would discount very low and very high number and come closer to what people true think it is rated.
Potential problem if there are a lot of votes it may take time to sort those in order to take the median. Perhaps it might be faster to find and discart the highest and lowest votes then do simple averge for what remain. If less then 3 vote then just do simple averge.