A discussion on Review Ratings

[DonkeyKonga]

Personally, I think it's more a case of whether our work is likely to get seen. I see many new releases going by with very little attention and it's sad. I want those editors to feel the encouragement I have. I'd watch some of them myself but I have a constant battle with the clock XD

That's not the only issue. The other issue is popularity of the source material and the rules. There are tons of edits I'd like to see, but I have no interest in owning the source material and paying money just to watch a fanedit. I only purchase films I intend on watching more than once. I just cant spare the $ and I'm probably not the only one. And I feel like purchasing second hand doesnt count, because that in no way supports the original artists lol. (but it's the cheapest option)

Which automatically means that if people take the rules seriously, more obscure edits wont be watched often. In fact even some more popular stuff wont be watched often. I did my TDK edit after the massive interest in TDKR but it didnt even get nearly the same amount of requests. Probably because people think the original is good enough, or they already found an edit they deemed *their definitive version*. Also my Interstellar edit was barely seen by anyone, which had to do with timing. Mine came two months after a terrific interstellar fanedit was released.

I didnt mind, though because I edit for myself first and foremost. I just share in case someone shares my personal taste.

you'll always have underappreciated gems, that's just the way it is. I dont see a way to fix that, tbh.

 

[DigModiFicaTion]

And I feel like purchasing second hand doesnt count, because that in no way supports the original artists lol. (but it's the cheapest option)

Used copies are legal copies of a movie. Yes that counts for OTS

Which automatically means that if people take the rules seriously, more obscure edits wont be watched often.

As you share, that's not something that should stop someone from doing an edit of something they like.

you'll always have underappreciated gems, that's just the way it is. I dont see a way to fix that, tbh.

Is that a real issue that needs to be fixed though?

 

[DonkeyKonga]

Used copies are legal copies of a movie. Yes that counts for OTS

As you share, that's not something that should stop someone from doing an edit of something they like.

Is that a real issue that needs to be fixed though?

I know it counts hahaha it's just my own logic that I'm basically using against myself

It doesnt need fixing, I'm answering tremault in saying that addressing the ratings wont stop certain edits from being overshadowed. That's just part of the game.

 

[Wraith]

Out of my last 50 reviews which I have submitted, only nine, are striaght 10s.

 

[DonkeyKonga]

Out of my last 50 reviews which I have submitted, only nine, are striaght 10s.

Sounds like you desperately need one more 10/10 review.

 

[Adabisi]

Out of my last 50 reviews which I have submitted, only nine, are striaght 10s.

Only? That's a lot.

 

[DigModiFicaTion]

Sounds like you desperately need one more 10/10 review.

 

[DonkeyKonga]

I offer myself.as tribute! I'm a selfless guy what can I say?

 

[Wraith]

I offer myself.as tribute! I'm a selfless guy what can I say?

I don't own the source...and I read that you created someting quite remarakable....

(hmmm....where's my coin jar...)

 

[krausfadr]

Ratings are subjective to each viewer. And the average reviewer doesn’t seem to notice overly compressed audio, background color banding artifacts. Many viewers also don’t notice mediocre audio transitions.

Hence a sea of perfect 10’s.

When some reviewers do notice these things and point them out some editors fix them but most don’t.

Other factors for unwarranted high scores of course is when the reviewer doesn’t want to offend the editor or is fearful they will be denied access to edits in the future.

Providing access to edits should never be tied to leaving a review.

 

[Gieferg]

This must be the most pointless discussion I've ever read here. You cant set the rules for ratings that always were and always will be a subjective thing.

 

[DigModiFicaTion]

This must be the most pointless discussion I've ever read here. You cant set the rules for ratings that always were and always will be a subjective thing.

Feel free to step out if you feel it's pointless.

 

[WilliamRedRobin]

To be honest, I'm not sure what the problem we're trying to address here is. A lot of people seem to be worried about getting "unfair" review scores, but there's no way to avoid that. People have different standards and their scores will always reflect that. Some people are sticklers for high video quality, personally I can barely tell the difference between HD and qHD. Is it unfair for them to rate an edit low for colour banding, or unfair for me to rate it high because I don't see a problem with it?
There's no guidelines that can be written to guard against that. At the end of the day a review is just, like, your opinion, man.

The other thing people seem to be worried about is every edit getting 9-10 stars. This is almost the opposite problem as the first one, but it has the same result of edits falling through the cracks and ending up under-appreciated. And the reason for this, as Donkey said, is that 8 is considered a low score on IFDB. "Flawless" sounds like a high bar for an edit to achieve, but it really just means "I didn't see a problem" and for most people that's a very low bar.
Sure, if you're someone who edits as a hobby, and you intently watch an edit looking for any issues it might have, you'll almost always find something, but that's not how most people watch movies. I've seen million dollar Hollywood productions with moments of terrible editing in them, but I almost never see editing mentioned in ammeter reviews (apart from Rise of Skywalker).
I've done reviews in the past where I've rated all 10s because I don't see a problem, and then I feel like I have to rate enjoyment a 10 too even though I'm only lukewarm on it, because I'd feel bad discounting a perfect score just because of my personal tastes, and I know I'd end up getting a pm along the lines of "c'mon, man".
A lot of people would say that's the the system working as intended because enjoyment is "subjective", but that's the reason everything is getting rated 10 these days.
People don't want to be mean, and giving someone a 7 is like spitting in their face.

That's my last input in this conversation. We've had multiple threads now that are either all about the same debate or five different debates all happening at once.
Until we actually figure out what the problem is, there's zero chance of reaching a productive solution, and I know more than a few people are getting sick of the endless talking.

 

[DonkeyKonga]

To be honest, I'm not sure what the problem we're trying to address here is. A lot of people seem to be worried about getting "unfair" review scores, but there's no way to avoid that. People have different standards and their scores will always reflect that. Some people are sticklers for high video quality, personally I can barely tell the difference between HD and qHD. Is it unfair for them to rate an edit low for colour banding, or unfair for me to rate it high because I don't see a problem with it?
There's no guidelines that can be written to guard against that. At the end of the day a review is just, like, your opinion, man.

The other thing people seem to be worried about is every edit getting 9-10 stars. This is almost the opposite problem as the first one, but it has the same result of edits falling through the cracks and ending up under-appreciated. And the reason for this, as Donkey said, is that 8 is considered a low score on IFDB. "Flawless" sounds like a high bar for an edit to achieve, but it really just means "I didn't see a problem" and for most people that's a very low bar.
Sure, if you're someone who edits as a hobby, and you intently watch an edit looking for any issues it might have, you'll almost always find something, but that's not how most people watch movies. I've seen million dollar Hollywood productions with moments of terrible editing in them, but I almost never see editing mentioned in ammeter reviews (apart from Rise of Skywalker).
I've done reviews in the past where I've rated all 10s because I don't see a problem, and then I feel like I have to rate enjoyment a 10 too even though I'm only lukewarm on it, because I'd feel bad discounting a perfect score just because of my personal tastes, and I know I'd end up getting a pm along the lines of "c'mon, man".
A lot of people would say that's the the system working as intended because enjoyment is "subjective", but that's the reason everything is getting rated 10 these days.
People don't want to be mean, and giving someone a 7 is like spitting in their face.

That's my last input in this conversation. We've had multiple threads now that are either all about the same debate or five different debates all happening at once.
Until we actually figure out what the problem is, there's zero chance of reaching a productive solution, and I know more than a few people are getting sick of the endless talking.

This is the single best post in this thread because it accurately depicts how there is no 'right' answer. Extra points for quoting the Dude. It's time to put the matter to rest for good.

 

[krausfadr]

A scale of 1-10 for each category is way too much and loses meaning.

To get more accurate reviews, it should be 1 to 5 stars for each category with no half stars. The final calculated score should then give half stars or can easily be translated to a scale of 1-10.

The guidelines should also be simplified and shown right in the review page:

Easier guidelines:

^^^^^^

5 = Excellent
4 = Good
3 = Average
2 = Bad
1= Awful

Note: Edits with noticeable video compression artifacts, jarring cuts, or confusing narratives not in the original source should not receive a perfect score.

^^^^^^

 

[Wraith]

To be honest, I'm not sure what the problem we're trying to address here is.
People don't want to be mean, and giving someone a 7 is like spitting in their face.

That's my last input in this conversation.

This will also be my last post.

My next public review will illustrate subjectivity.

I also believe in proportionality too.

If an editor maker several hundred changes, and a handfull of audio transitions are not quite perfect, or the source material just does not permit for a better result...the several hundred alterations win out for me, and if the other 489 transitions are v good to perfect, and nine are a touch off, I will award a 10.

If 150 transitions out of 500 are a touch off and source material means that was the best possible result, and maybe an 8 or 9...
to me 9/500 is excellent..even 30/500....

THIS is W, "over and out".

 

[krausfadr]

If an editor maker several hundred changes, and a handfull of audio transitions are not quite perfect…

Reviewers typically will never understand the breadth of cuts made in an extensive edit. So while this sentiment might have some merit, it can’t really ever be a factor we expect reviewers to use.

 

[Wraith]

Reviewers typically will never understand the breadth of cuts made in an extensive edit. So while this sentiment might have some merit, it can’t really ever be a factor we expect reviewers to use.

Agree...but I do use that factor...whenever I am able or aware...

oops...I made another post!

 

[DonkeyKonga]

Reviewers typically will never understand the breadth of cuts made in an extensive edit. So while this sentiment might have some merit, it can’t really ever be a factor we expect reviewers to use.

Reviewers need to be left alone to subjectively review as they see fit. As WilliamRedRobin said there is no need for all of this debate. I mean if we can't expect reviewers to understand the breath of cuts in an extensive edit, can we expect them to understand the standards for visual quality and audio quality? The answer is no. I mean we are only seeing disagreements in the thread. There is no objectivity when it comes to a review. A review is an opinion.

Can we really expect every single reviewer to understand if an edit could have been better with an L or J cut? Not really. Gatekeeping the review process for people who just want to enjoy the fanedits they watch and show their appreciation is pointless. We need to strive for high standards for our editors, not our (re)viewers.

 

[DigModiFicaTion]

To be honest, I'm not sure what the problem we're trying to address here is. A lot of people seem to be worried about getting "unfair" review scores, but there's no way to avoid that. People have different standards and their scores will always reflect that. Some people are sticklers for high video quality, personally I can barely tell the difference between HD and qHD. Is it unfair for them to rate an edit low for colour banding, or unfair for me to rate it high because I don't see a problem with it?
There's no guidelines that can be written to guard against that. At the end of the day a review is just, like, your opinion, man.

The other thing people seem to be worried about is every edit getting 9-10 stars. This is almost the opposite problem as the first one, but it has the same result of edits falling through the cracks and ending up under-appreciated. And the reason for this, as Donkey said, is that 8 is considered a low score on IFDB. "Flawless" sounds like a high bar for an edit to achieve, but it really just means "I didn't see a problem" and for most people that's a very low bar.
Sure, if you're someone who edits as a hobby, and you intently watch an edit looking for any issues it might have, you'll almost always find something, but that's not how most people watch movies. I've seen million dollar Hollywood productions with moments of terrible editing in them, but I almost never see editing mentioned in ammeter reviews (apart from Rise of Skywalker).
I've done reviews in the past where I've rated all 10s because I don't see a problem, and then I feel like I have to rate enjoyment a 10 too even though I'm only lukewarm on it, because I'd feel bad discounting a perfect score just because of my personal tastes, and I know I'd end up getting a pm along the lines of "c'mon, man".
A lot of people would say that's the the system working as intended because enjoyment is "subjective", but that's the reason everything is getting rated 10 these days.
People don't want to be mean, and giving someone a 7 is like spitting in their face.

That's my last input in this conversation. We've had multiple threads now that are either all about the same debate or five different debates all happening at once.
Until we actually figure out what the problem is, there's zero chance of reaching a productive solution, and I know more than a few people are getting sick of the endless talking.

It was clear what the intention of this thread was. A few members are hijacking it with claims that the system is flawed, broken and unable to calibrate. That's not helpful. Your post here implies the issue, a lack of integrity in rating. That much is clear. But it's counter productive to post such and feed that narrative when an attempt to address it is made. Especially when, as you stated, there are already multiple threads about this where people can share their concerns.

We've attempted to hear and listen to the community. We ask now that the community try to callibrate instead of obfuscating the situation.

To be clear. The rating will remain 1-10. The purpose of this thread was to calibrate how to give each score. Due to the lack of focus and engagement this thread will be temporarily locked. When reopened, or a new thread is made, we ask that people engage in the task of calibration. There are already three other threads for expressing why one thinks the model is wrong.

Also, just because one doesn't see how integrity of the rating system can be obtained doesn't mean it can't happen. When we resume, feel free to observe if all you have to share is doubt.