[FOR REVIEW] - Guideline impact ratings and scoring

[fershad]

Hi everyone.

One of the things to come out of TPAC this year was the need to make progress on some kind of metrics/measurement work to accompany the guidelines. The guiding question, at least for now, is one that @TzviyaSiegman has mentioned a few times in calls through the year. Something along the lines of ...

People are asking "where should I begin? what are the most impactful things I can do, now, with these guidelines?"

The metrics task force was working on this through the middle of the year, but progress stalled since late September. During discussions at TPAC, I agreed to complete a first pass at this. I am creating this issue to share that first version and open it up for feedback and discussion.

TL;DR

Here's a summary of what has been done:

• Each guideline has been assigned an impact rating ("low", "medium", "high") for the categories of "People", "Planet", and "Prosperity".
• Each guideline has also been assigned an impact timeframe ("short", "medium", "long") indicating how long the impact of that guideline might take to be realised.
• These inputs are weighted and added together to give each guideline an overall impact score.
• For each guideline, where appropriate:
  • Additional context/rationale has been included in the form of a short sentence.
  • Potential metrics for tracking the impact of that guideline have been provided.
  • Machine and human testable categorisations have been reevaluated.

Note: This work has been done on the current published version of the guidelines, and does not incorporate any guideline consolidation/removal work that may be ongoing in the background.

For review

To review the impact ratings and scores, you should:

1. Read this Google Doc (view only). It introduces the work that has been done, giving context for what you will see when reviewing the document containing the ratings later.
2. Read the note below on how to give feedback.
3. Review the impact ratings, scores, rationales, and metrics in this Google Spreadsheet (view only).

How to give feedback

Firstly, I want to acknowledge that no one reading this will agree with every single rating I have assigned for each guideline. That is okay, it is a feature not a bug. This is a difficult topic, and the broadness of the Web Sustainability Guidelines leaves their impacts very much open to interpretation.

Important - At this time, the main area of feedback we're looking for is does this approach make sense? and is this approach something we are comfortable with publishing?

You can leave constructive feedback as a comment to this GitHub issue.

• Please bear in mind the key area of feedback we're after at this stage.
• If you have very strong opinions about any of the ratings in the spreadsheet which you really need to get off your chest, then please do so. Please be sure to include a rationale for any proposed changes.
• If you know of any suitable metrics for a particular guideline, please feel free to share those!
• If you have suggestions for a better tallying system, please share those thoughts.

Remember that this is a first go at this. We don't need it to be perfect, and we can leave plenty of room for ourselves to improve and iterate on it in the future as well.

Please note that while comments will be seen, it may take time for any feedback to be incorporated. The IG Chairs will need to work out how to proceed with how best to proceed with this work in terms of incorporating feedback, gathering consensus and publication.

Other related issues:

• Metrics taskforce: Impact and prioritisation efforts sustainableweb-wsg#96
• Utilise other metrics beyond just CO2 for prioritisation and impact assessment sustainableweb-wsg#91

[susannah-hill]

Thank you very much for doing this important work, Fershad. Ryan and I and a few co-conspirator significantly revised section 4. The S.C.s have changed and are different from what is reflected in the spreadsheet you shared. How can Ryan and I help you revise—to minimize the disruption to your project?

[systemstree]

Than you, @fershad and everyone on the metrics task force. This is an amazing framework, and it makes things much easier. It is clear that all your hard work has paid off.

I've the following suggestions:

We need core principles for operationalizing certain terms because they might mean different to different people. What does "positive" mean? What does "Medium" really mean? Can we quantify the timeframe. In the document we've given the example of a month but can we make it specific. For instance, when we talk about "Medium," does that mean less than six months? I understand, it might change according to context and the kind of projects, but can we have some defined criteria to come up with the duration?

Can we classify measurability into observed or quantified, even if it involves relative comparisons?

I propose the following enhancement to the existing definitions. We don't have to follow them exactly, but this might help us bring more clarity to these definitions:

Impact definitions:

• Low: The guideline results in small positive impact (minor improvements) on the given category. The benefits are easily overshadowed by other factors or are not readily apparent. It addresses a very minor issue.
• Medium: The guideline produces noticeable and measurable positive changes within the category. It addresses a key issue or improves an important aspect of the category's performance.
• High: The guideline delivers substantial, transformative positive changes within the category. It significantly improves performance and sustainability, addresses a critical issue, or unlocks new capabilities.

Regarding the timeframe, I feel we need to change the word since people might confuse it with project timeframe. We can call it "Time to Notice Impact (TNI)" or something similar to make it sound more direct.

Timeframe definitions:

• Short: The impact of the guideline will be noticed quickly, typically within hours or days of implementation. The changes are direct and readily apparent.
• Medium: The impact of the guideline will be noticed after few days. The effects are less direct and require some time for users or systems to adapt.
• Long: The impact of the guideline will be noticed after a lengthy period such as several weeks or months.

Have we planned to create a visual representation of the framework? A step-by-step process to guide users in measuring the impact might help further and users will get a ready format in which to present their findings? For example, in training evaluation, we use frameworks that help evaluators follow a step-by-step process to evaluate training programs. Something similar can be done here, as it will make the process easier. It would be great if W3C only comes up with such a framework; otherwise, we would be bombarded by multiple frameworks, and people might get confused about which one to apply.

[timfrick]

The rationale for these ratings and scoring system are clear, as is the explanation. Thanks for the clarity and attention to detail, @fershad.

Reading through the various documents also underscored the importance of of highlighting (as often as possible) how different organizations will implement these differently based on team size, experience, resources/budget, tech stack, and many other variables. This could influence the timeframe component of overall scoring.

Also, impact ratings could be similarly influenced by a variety of variables, such as amount of traffic, for instance. The metrics column helps with this, but I wonder if we can be more explicit when relevant.

For example, variations on this statement are often included under Rationale:

The direct impacts as a result of this guideline a minimal or uncertain.

If we provide potentially known influencing factors alongside metrics, does that help guideline users or add to potential confusion?

Also, I generally agree with @systemstree's suggestions above as well.

Next steps and questions:

• Should the spreadsheet be updated based on the Dec. 15, 2025 release?
• If so, does that need to happen before anything else is done?
• How might we be more explicit in answering the "where should I begin?" question?
• Who else should review this (and when) to ensure we make the proposed scoring consensus deadline of Feb 15, 2026?

Also, this is a very minor thing, but the use of 'medium' as a measurement criteria in both impact and timeframes could be confusing. Perhaps use midrange or something similar for timeframe? Just a thought.

[fershad]

@susannah-hill thanks for that. I think if yourself, Ryan, and anyone else involved with Section 4 would want to create an impacts table just for the updated version of that section, following the format I've used, then I can look to incorporate those updates into my version eventually.

[fershad]

@systemstree all good points you've made, and definitely part of the reason for getting this out there for group review. I think January 8th has been set as a date to come together and work out a way forwards, so between now and then I think the group might consider your suggestions alongside any others made.

[fershad]

@timfrick the downside of the broadness of the guidelines is that we won't have a single set of "influencing" factors we can point to as being consistent across all guidelines. That's why I've left them out of consideration when going through this first round.

What it sounds like might be useful though, might be to add an additional information column like I've got for metrics titled "Influencing factors" where we can list the factors that might influence the impact of a given guideline with some additional context.

To this specific questions:

How might we be more explicit in answering the "where should I begin?" question?

That's the purpose of the overall impact score - to allow for guidelines to be "ranked" based on potential impact.

[fershad]

One other general question for the group - something that's not in the current impact scales but could be incorporated is the acknowledgement that some guidelines may have no impact at all on a given category. Should this be captured by adding a fourth impact rating of "None"? Right now, I've defaulted to "Low" in cases like this.

E.g. 3.14 Use Standards-based JavaScript and APIs likely has no direct impact on "Prosperity".

[fershad]

Also, a note about how we talk about these after watching the meeting from yesterday - these are neither metrics nor measurements and we should probably steer clear of referring to them with either of these terms. Yes, there's a metrics field in the spreadsheet aimed at giving folks a steer on how they might look to track impact, but it's a steer only.

It'd feel more appropriate to call these what they are - "impact ratings". And as was pointed out by Tzivya & Tim, we should be sure to capture the subjectivity of them if they (or a future iteration) do end up being published.

[timfrick]

Really great point, @fershad. Thanks for the reminder.

[systemstree]

Thank you @fershad. I've been working on a checklist (google sheets format). My idea is to incorporate the scores once they are finalized. It would be helpful for teams to use it as a checklist and take into consideration the guidelines that have a greater impact. This contains the latest version of the guidelines: Google spreadsheet (view only).

[systemstree]

I've tried to further operationalize the impact definitions in relation to the categories. Was there any discussion about whether we can provide similar justifications for the ratings given per category for each guideline?

Low Impact:
People: Addresses minor improvements in user experience, accessibility, inclusivity, and overall well-being. Doesn't significantly impact a large number of people.
Planet: Results in minimal environmental benefit. May reduce smallest amount of carbon emissions, using slightly less energy than before, or making hardly any changes to resource consumption. Difficult to measure. Might lead to awareness in users to reduce their consumption.
Prosperity: Lead to marginal economic gains or cost savings. May involve very minor improvements to work efficiency, small reductions in operational expenses, or minimal contributions to local economies.

Medium Impact:
People: Results in noticeable and positive improvements in user experience, accessibility, inclusivity, and overall well-being. Involves addressing common usability issues, implementing broader accessibility standards (e.g., WCAG), improving content clarity, and making features more inclusive. Requires moderate effort and resources. Likely to impact a significant number of users.
Planet: Achieves measurable environmental benefits. Could involve reducing carbon emissions by a noticeable amount, improving energy efficiency, significantly reducing usage of natural resources such as water, or minimizing wear and tear of hardware. Actively promote user awareness of consumption reduction.
Prosperity: Contributes to meaningful economic gains and cost savings. May involve significant improvements in work efficiency, reduced operational expenses, or positive contributions to local communities and economies. Requires substantial effort and resources.

High Impact:
People: Leads to transformative changes in user experience, accessibility, inclusivity, and overall well-being. Addresses systemic barriers, promotes equity, significantly enhances usability for a large audience, and ensures universal access. Requires significant investment, strategic planning, and ongoing commitment. Can have widespread societal benefits.
Planet: Results in substantial and demonstrable environmental improvements. Could involve significantly reducing carbon emissions, achieving carbon neutrality, avoiding usage of natural resources (such as water) on which local communities depend on, promoting circular economy principles, protecting biodiversity, or mitigating climate change impacts. Requires large-scale initiatives, technological innovation, and policy advocacy.
Prosperity: Generates significant economic benefits, creates new opportunities. May involve driving sustainable innovation, creating jobs, promoting economic sustainability in underserved communities, or reduce income inequality. Requires considerable effort and often involves enforced implementation of government policies.