A discussion on Review Ratings

[DigModiFicaTion]

Can people send a pm to me with their videos? I can then use those to facilitate and guide the discussion so we aren't all over the place with examples. Thank you to tall who volunteer!

 

[Adabisi]

However stereo for more modern films? They usually have 5.1 mixed ready and available. Nothing is stopping you from editing in 5.1 other than learning how to. I dont have a 5.1 audio setup myself but all my edits are 5.1 simply because I want people who have the luxury of owning such hardware to also enjoy the edits without feeling like it's a step down. You can edit in 5.1 even if you dont own the hardware to play it.

I would recommend not editing in 5.1 if you can't properly review your work. I've mentioned it a bunch of times, but there are ton of approved edits on this site by experienced editors that have absolutely terrible 5.1 mixes.

 

[DonkeyKonga]

Its meaningless if only 9s or 10s matter. You also kind of contradict yourself with the stereo statement getting a 9. I dont think stereo should get a low rating perse, but I also dont consider an 8/10 a low rating for proper stereo sound. It's kind of silly that proper stereo would be able to get the same rating as a 5.1 mix if the source has a 5.1 mix available.

Regarding the ' more channels doesnt equal better sound' argument. There is a reason we dont mix in mono. More channels does equal better sound to an extend depending on the scene/source.

I think a better example for visual quality shouldnt nessecarily be: 'compared to the source' used, but 'compared to industry standard/best source available for this project'

An example of this is dr Dre's lotr projects that are actually an upscaled 4k image that looks better than the official 4k release (which in turn looks worse than the 1080p release)

I would recommend not editing in 5.1 if you can't properly review your work. I've mentioned it a bunch of times, but there are ton of approved edits on this site by experienced editors that have absolutely terrible 5.1 mixes.

Mine must be one of them because all my 5.1 work was done poorly (though unnoticed by anyone without a 5.1 setup lol). I now know how to 'read' the audio tho so I know it's done properly now. Not being able to hear it doesnt mean you cant verify it.

 

[DigModiFicaTion]

Its meaningless if only 9s or 10s matter. You also kind of contradict yourself with the stereo statement getting a 9. I dont think stereo should get a low rating perse, but I also dont consider an 8/10 a low rating for proper stereo sound. It's kind of silly that proper stereo would be able to get the same rating as a 5.1 mix if the source has a 5.1 mix available.

How, if it's almost undetectable it would be able to get a 9. You are free to rate it an 8 8f it's slightly noticeable. 8 is a solid score.

 

[tremault]

Regarding the ' more channels doesnt equal better sound' argument. There is a reason we dont mix in mono. More channels does equal better sound to an extend depending on the scene/source.

This could be a misunderstanding but there's two ways I look at this.

firstly, quality is about how clear(?) the audio is, not about how many different channels there are. 5 channels with an arbitrary quality rating of 5, are the same as one channel with an arbitrary quality rating of 5. the quality rating is the same, there are just more of them. if I take 6 scones of the same quality and put one of them on the left, the one on the left is the same quality as the 5 on the right. quantity does not equal quality.

The second way I look at it is more of a consideration though. an audio outputs at 600kbps is going to have higher quality with less channels. a 600kbs audio with one channel will have 600kbps per channel, but a 5.1 version would have 100kbps per channel. That would be noticably bad.

 

[DonkeyKonga]

How, if it's almost undetectable it would be able to get a 9. You are free to rate it an 8 8f it's slightly noticeable. 8 is a solid score.

Anything is undetectable if you dont have the equipment to view it. That isnt an argument at all if were looking for accurate review ratings.

If you have a 5.1 system (I dont) you'll immediately hear the difference between 5.1 and stereo. When I review an edit that has 5.1 I assume it will sound like proper 5.1 with the right setup. But now that I think of it I probably shouldnt be able to review 5.1 since I dont have means to check its execution. Or my review should have a disclaimer that I dont own a 5.1 setup but it sounded good on my stereo setup.

I understand this might be controversial. But searching for a good review system only works if we are honest across the board. Otherwise you're just cherry picking stuff that suits you.

 

[Wraith]

ONE can't say that VIDEO QUALITY is a 10 ONLY if 4k.
That is Nonsense

Many older edits were made before the advent of BD or 4K...Are they to be downgraded?
It can' be done...

VIDEO QUALITY, should be in relation to THE SOURCE.

Since Everyone should OWN THE SOURCE, if an edits is DVD sourced, THAT is what it is compared against. 10 means IDENTICAL to its source.
PERIOD IMHO

Otherwise, the whole historic rating archive becomes an anachronistic mess.

 

[Wraith]

Anything is undetectable if you dont have the equipment to view it. That isnt an argument at all if were looking for accurate review ratings.

If you have a 5.1 system (I dont) you'll immediately hear the difference between 5.1 and stereo. When I review an edit that has 5.1 I assume it will sound like proper 5.1 with the right setup. But now that I think of it I probably shouldnt be able to review 5.1 since I dont have means to check its execution. Or my review should have a disclaimer that I dont own a 5.1 setup but it sounded good on my stereo setup.

I understand this might be controversial. But searching for a good review system only works if we are honest across the board. Otherwise you're just cherry picking stuff that suits you.

So by your standards, how would you score CITIZEN KANE?

I am planning an edit on that one...and a few other MONO classics, shot in Academy Ratio....many only available on DVD even to this day with no streaming service carrying them...
Just curious?

Should I even bother ?

 

[DonkeyKonga]

ONE can't say that VIDEO QUALITY is a 10 ONLY if 4k.
That is Nonsense

Many older edits were made before the advent of BD or 4K...Are they to be downgraded?
It can' be done...

VIDEO QUALITY, should be in relation to THE SOURCE.

Since Everyone should OWN THE SOURCE, if an edits is DVD sourced, THAT is what it is compared against. 10 means IDENTICAL to its source.
PERIOD IMHO

Otherwise, the whole historic rating archive becomes an anachronistic mess.

Reviews shouldnt be altered retrospectively, that's a pretty bad idea. I can only trust my eyes. If I own the bluray for something and I watch a dvd edit, I dont think it's fair for me to be forced to give it a 10/10 when the bluray is available. I agree that 4k is unrealistic, if only because of the disk space required.

Visual quality and visual editing are two different categories. Visual editing shouldnt be influenced by the source material, but visual quality? I mean just get rid of this category altogether if it's going to be only compared to the source that was used and not the best source available

 

[Wraith]

Reviews shouldnt be altered retrospectively, that's a pretty bad idea. I can only trust my eyes. If I own the bluray for something and I watch a dvd edit, I dont think it's fair for me to be forced to give it a 10/10 when the bluray is available. I agree that 4k is unrealistic, if only because of the disk space required.

Visual quality and visual editing are two different categories. Visual editing shouldnt be influenced by the source material, but visual quality? I mean just get rid of this category altogether if it's going to be only compared to the source that was used and not the best source available

You say that, but I had a review that dinged me on VISUAL QUALITY, and the render was a STRAIGHT PASSTHOUGH from the source, with actually higher settings.

One cant expect reviewers will have the level of diligence to make these comparisons to the source...that just takes the fun out of it.

I think we are in danger of over thinking this and making a rod for everyones back.

I have analysed many of the edit scoring frequencies etc...(OCD DOES THAT)...

There is a propensity for Franchises with fervent followings to score highly and disproportionately, and for more academic, preserving or technical works to score more diligently. This reflects the audience that watch the edits.

I won't bore you all with examples, but I am sure you all know what I am describing.

 

[DonkeyKonga]

So by your standards, how would you score CITIZEN KANE?

I am planning an edit on that one...and a few other MONO classics, shot in Academy Ratio....many only available on DVD even to this day with no streaming service carrying them...
Just curious?

Should I even bother ?

I already addressed this multiple times: it should probably be compared to the best source available. I'd probably give it an 8/10 unless upscaled very well.

Ofcourse you can bother. I'd probably describe it like this: Visual quality is good and close to the dvd source hence the 8/10. This edit is a joy to watch. Let's hope one day Wraith can update this edit with a proper HD source! Citizen Cane deserves it.

Edit: in fact my return of the jedi edit is an example of this. There are one or two reviews that took points from visual quality because I used a scan instead if the official bluray release which has more clarity. Which I felt was fair, if they felt that way. However a few also gave it a 10/10 because they loved how authentic it looked compared to the bluray. Both are valid arguments.

TLDR: There really isnt a right way to review and we should just leave it as is. If more freedom of interpretation means more reviews I think we need to leave things as is!

 

[tremault]

I have a practical example.
here is a dvd screenshot of a movie.

I edited it slightly and saved the results. how do you rate how closely i matched the original in terms of quality?

Nobody's asking you to rate something in terms of how good it could possibly be. if you think it looks as good as the original, then surely that makes it a 10?

 

[DonkeyKonga]

I have a practical example.
here is a dvd screenshot of a movie.

I edited it slightly and saved the results. how do you rate how closely i matched the original in terms of quality?

Nobody's asking you to rate something in terms of how good it could possibly be. if you think it looks as good as the original, then surely that makes it a 10?

See my previous post. I think an 8/10 because its dvd quality is as much a valid argument as yours is. I think more freedom = more reviews which is best for all parties. Just leave it to interpretation, and let the written part of the review elaborate on scoring.

Certainly an 8/10 with praise about how it was well edited and looks just like the dvd source wouldnt trigger you?

 

[DigModiFicaTion]

One cant expect reviewers will have the level of diligence to make these comparisons to the source...that just takes the fun out of it.

I have faith in the competency of reviewers. If we define, we can follow.

I think we are in danger of over thinking this and making a rod for everyones back.

I would disagree. Calibration is necessary if we are continue to say any score has integrity.

There is a propensity for Franchises with fervent followings to score highly and disproportionately, and for more academic, preserving or technical works to score more diligently. This reflects the audience that watch the edits.

I think the bigger issue is the propensity for reviewers to rate base on friendly or relationship based measures.

 

[tremault]

. I think more freedom = more reviews which is best for all parties.

that was kinda my point. you were saying a dvd version shouldn't be able to get a 10. I was arguing for the freedom to be able to do so.

Certainly an 8/10 with praise about how it was well edited and looks just like the dvd source wouldnt trigger you?

well it would and it did. somebody reviewed saying my video was flawless compared to the bluray but because it wasn't 4k I can't get higher than an 8. that was a bit of a kick in the teeth.

 

[DonkeyKonga]

I have faith in the competency of reviewers. If we define, we can follow.


I would disagree. Calibration is necessary if we are continue to say any score has integrity.


I think the bigger issue is the propensity for reviewers to rate base on friendly or relationship based measures.

Based on friendship? People cant be friends and review? That's a slippery slope right there. I think realistically FotM is already partly a popularity contest. Which makes sense considering the rules. You need to own a movie to watch an edit. The more popular a movie the more likely it is to win FotM.

Same goes for popular people. That's just the way humans work.

What you're searching for is a seperate review system where only a select group of reviewers review an edit. 'Official' reviews for a lack of better words. It's a bit much for a fanedit community tbh.

 

[Wraith]

I have faith in the competency of reviewers. If we define, we can follow.

I think the bigger issue is the propensity for reviewers to rate base on friendly or relationship based measures.

There are a number of reviewers, who I have good reletionships with...and THEY are my HARSHEST critics...

So I disagree about the FRIENDLY connection...in fact I value their challenge, observations and constructive comments which make me, and other editors who get such feedback, better editors making better edits...

However,
The problem is that it is hard eough to get reviews as things are. Less than 10% of requests I get turn in a review.

We shouldn't be policing the freedom to give a review as viewers see fit based on their subjective opinion. Guiding them is fine, sure...but nobody polices their systems; not IMDB, RotTom etc. People may stop reviewing altogether.

Telling people when they can give a perfect score and when they can't is almost a form of censorship and is a very bad idea.

Guding them, sure, we should be doing that, but this is and always will be a subjective human involvement, otherwise, we may as well wait for someone to develop an app, and we can then run the file through it and generate a score, and remove reviews completely...

I for one, despite its flaws and challenges, am happy with how things are overall...by all mean lets flesh out the guidelines...and be cogniscent that the phrase is not

HARDLINES but...
GUIDE....lines

 

[Moe_Syzlak]

I’ve said this in previous discussions, but as a non-editor, I find the numerical scores to be completely useless. Worse yet, I think those scores often lead to reviewers not feeling the need to elaborate on those technical areas in the narrative reviews. As a result you end up with reviews that read like this:

“This was great edit. I didn’t notice any editing flaws and the cuts were mostly good. It replaces my theatrical version. Great work!”

That tells me very little and honestly is how most reviews end up reading more or less. If you scrap the number ratings altogether and replace them with separate narrative windows perhaps you’ll get more detailed reviews and a more useful guide for viewers.

 

[DonkeyKonga]

that was kinda my point. you were saying a dvd version shouldn't be able to get a 10. I was arguing for the freedom to be able to do so.


well it would and it did. somebody reviewed saying my video was flawless compared to the bluray but because it wasn't 4k I can't get higher than an 8. that was a bit of a kick in the teeth.

Glad to hear we are on the same page! Though please dont take this personally but if someone says he have it an 8/10 based on his own personal standards and that upset you that sounds a bit fragile. I mean if he had given it a 4/10, sure. But an 8/10 is a respectable score. If I get an 8/10 for anything I'm ecstatic lol.

You cant preach for more freedom then expect 10/10 reviews because it suits your narrative.

 

[DonkeyKonga]

I’ve said this in previous discussions, but as a non-editor, I find the numerical scores to be completely useless. Worse yet, I think those scores often lead to reviewers not feeling the need to elaborate on those technical areas in the narrative reviews. As a result you end up with reviews that read like this:

“This was great edit. I didn’t notice any editing flaws and the cuts were mostly good. It replaces my theatrical version. Great work!”

That tells me very little and honestly is how most reviews end up reading more or less. If you scrap the number ratings altogether and replace them with separate narrative windows perhaps you’ll get more detailed reviews and a more useful guide for viewers.

Hear, hear. I love reading elaborate reviews. A 8/10 review with 4 paragraphs tells me more than a 10/10 with one sentence!