Section 3.3.1: can a measure of support effectiveness be included?

[jasonjgw]

The ultimate measure of an organization's support program is its effectiveness, and I think this should be captured in the proof points.
For example, are employees/clients with disabilities satisfied that the support provided is meeting their needs and overcoming barriers to access?
Consistent satisfaction/dissatisfaction by those who are supposed to benefit indicates more/less maturity in the organization's support offerings.

An intuitive way of thinking about it is: there should be evidence of whether the support is "working" or not, and to what extent. I think this should be included in the proof points by adding an appropriate criterion.

[swinehartganderson]

@Helixopp @jeffkline
Example structure proposal (from our maturity meeting today), using the forthcoming updates:

• Include goals and metrics in the maturity level descriptions for launch, integrate, and optimize.
• Include a description of what evidence might be used for goals and metrics in the proof points. Could be a new proof point rather than a bullet inside an existing proof point.

[jeffkline]

Hi Stacey,Thank you for the quick note.So just a little food for thought… I don’t think that we should try to include in the maturity-model or presume any metrics that organizations may want to use as the topic is very broad and organizations are as well. Only that they should establish some and use them to measure progress and improve maturity.When I wrote the accessibility for rules for Texas State government, that’s exactly how I structured it, leaving it up to the agencies to establish their own as suited their organization. We or somebody could always provide some possible suggestions on what goals and metrics to establish, but I would not want to put it in concrete terms in the maturity-model. Again, I believe that the best approach is for each dimension should carry a proof point for establishing metrics and goals for the dimension to drive progress.Thanks again. I look forward to additional ***@***.*** 5  1  2   .   4  2  6   .   9   7  7  9On May 22, 2024, at 3:24 PM, Stacey Swinehart Ganderson ***@***.***> wrote: @Helixopp @jeffkline Example structure proposal (from our maturity meeting today), using the forthcoming updates: Include goals and metrics in the maturity level descriptions for launch, integrate, and optimize. Include a description of what evidence might be used for goals and metrics in the proof points. Could be a new proof point rather than a bullet inside an existing proof point. —Reply to this email directly, view it on GitHub, or unsubscribe.You are receiving this because you were mentioned.Message ID: ***@***.***>

[swinehartganderson]

@jeffkline To clarify, I didn't mean that WE (maturity task force) list what goals and metrics. It'd be more of a general description, like we've been doing with the rest of them.

So an example might be for launch: plans for accessibility goals and metrics are in place

An example for integrate might be: accessibility goals and metrics are in place and teams are starting to follow them and include in prioritization

(not the best, I know, but you get the drift)

I think the convo in the next meeting will be good. I'm also looking forward to it :)

[jeffkline]

Thanks, Stacey for the clarification,That description example actually would meet the Maturity stages of a proof point, don’t you think? But we don’t have to spell it out or define it beyond that,as the maturity stages really take care of that just as they do for all the other proof points. Anyway, we should have further discussion on the call next week. JeffOn May 22, 2024, at 6:46 PM, Stacey Swinehart Ganderson ***@***.***> wrote: @jeffkline To clarify, I didn't mean that WE (maturity task force) list what goals and metrics. It'd be more of a general description, like we've been doing with the rest of them. So an example might be for launch: plans for accessibility goals and metrics are in place An example for integrate might be: accessibility goals and metrics are in place and teams are starting to follow them and include in prioritization (not the best, I know, but you get the drift) I think the convo in the next meeting will be good. I'm also looking forward to it :) —Reply to this email directly, view it on GitHub, or unsubscribe.You are receiving this because you were mentioned.Message ID: ***@***.***>

[jasonjgw]

Having dealt with too many inadequate support processes over the years, I am not comfortable leaving the goals and metrics wide open for organizations to stipulate as they wish. It's too easy (intentionally or not) to define the goals and metrics in such a way that they do not capture the extent to which actual support problems are adequately addressed to the satisfaction of the people with disabilities whose needs are meant to be served.
I think there needs to be language in the proof points requiring the goals and metrics to be grounded in the lived experience of people with disabilities and whether the problems are being addressed from their perspective. The goals should lead to concrete improvement, and the metrics should reflect the adequacy of the support in fact provided.
I am not opposed to flexibility, but I am opposed to unconstrained flexibility for bureaucrats to specify goals/metrics that stop short of measuring and improving what they're meant to address, and then claiming a high level of maturity for processes with which people with disabilities are generally dissatisfied.

[jasonjgw]

It became clear at today's meeting that the problem should be addressed for the document as a whole, and not just for this maturity dimension.

There are multiple potential solutions. I suggest one as a starting point for discussion (probably bound to generate controversy):

• Add a subsection after section 2.2 that discusses the different types of evidence and recommends according priority to direct evidence of benefits to users with disabilities and qualitative improvements to ICT systems. It should be suggested that implemeters of the Maturity Model define goals and metrics that take advantage of the most provative sources of evidence reasonably available to the organization in assessing concrete effects of the accessibility program upon employees, customers and other parties with disabilities.

• Add criteria to the Ratings for Evaluation sections of different dimensions that refer to goals and metrics being defined, and progress made in achieving those goals, as appropriate to each level. I would suggest including this component at the Integrate level and above, as it doesn't seem to make much sense to apply below this level.

• As appropriate, add proof points to relevant dimensions of the Model that reflect direct (non-circumstantial) evidence, e.g., user feedback, survey findings, accessibility/usability study findings, software defect resolution to the satisfaction of the users/reporter, conformity to regional accessibility standards and trends therein over time/across products, etc. Aspects of this are already addressed in proof points. The task here would be to tighten this up and add examples as appropriate.
  I think these measures would address the issue for the whole document.