Pass or Fail or Skip: A simple, fast, and controlled model ranking feature

Pass or Fail or Skip

A simple, fast, and controlled model ranking feature

This is an idea proposal for CivitAI to implement as a new feature for the site, since it already has a well established userbase that could make use of it and many checkpoint models that would benefit from a more precise ranking system that this would provide.

Pass or Fail (think the "hot or not" site in The Social Network) feature for rating images based on their content in relation to the model and prompt that generated them.

Users would be presented with a page that randomly displays a generated image that they would either Pass or Fail based on comparing the image to the prompt that generated it.

An option to Skip would be good to have, since not all users will understand what the prompt is supposed to generate, so if they are unsure there should be an option for them to Skip the image.

When they Pass or Fail or Skip an image, the page would immediately update with a new image for them to pass or fail.

The images should be rendered using the recommended settings provided in the description of each model, or a baseline setting for models that do not provide that information, such as Euler Ancestral, Normal, 50 steps, random seed.

This could be used to better rank the performance of model in a controlled way based on their strengths and weaknesses in the various categories and challenges of a known fixed dataset.

As for the prompts, a dataset like PartiPrompts provided by Google-Research would be a good starting place for a fixed set of simple through complex prompts that have various categories and challenges for each category, this dataset could easily be expanded as necessary.

PartiPrompts dataset
https://github.com/google-research/parti
https://github.com/google-research/parti/blob/main/PartiPrompts.tsv

I have converted the .tsv file to JSON and made it available below at GitHub Gist:

PartiPrompts.json
https://gist.github.com/jesterjunk/ce42fe81b905c3385a33c24db6de0286


Screenshot source:
Reviewing & Rating 50 SDXL models
https://www.youtube.com/watch?v=GQNiKKq2EP4
YouTube Channel: Render Realm
Date Published: Oct 21, 2023