I'm good with that idea myself, Spacenewt. That's how The Sims Resource would reckon their artists 'pay'. And they also had a sort of committee deal to approve new submissions. The committee was, admittedly, pretty lenient, but I think in all, quite fair.
I'd also suggest that works in progress not be eligible, simply by the nature of the beast. It's not a finished artwork or whatever.
I'll also put out that it is not that difficult to find people who are violating copyright and the like, even if it is material one person is unaware of: Someone will be, and having a reporting system for suspected copyright violation is easy enough to put in. This means the staff doesn't have to be familiar with 2904023420398098523423 forms of various material.
That's easily abused by people, too, so once a report comes in, the staff can google and check for themselves. Then it can be acted on.
Ratings and comments are great, but most people don't bother with them. It's not that they're mean or bad people, they just literally forget or don't notice. (For instance, when was the last time you remembered to fill out the "how was the service here?" cards at the local coffee shop?)
I don't have much of a burning want or need for the portfolio things, I really do not think that I'd be getting a job from making mat poses. There's people who have the schooling and all for graphic arts, I certainly don't!