Contextualizing definitions for export

[KDean-GS1]

Description of problem

The terminology include file is being processed, but restrictReferences() is failing, which results in unused terms appearing in the final document, each of which generates a warning.

URL to affected spec or repo:

https://w3c.github.io/vc-use-cases/

What happened (e.g., it crashed)?:

ReferenceError: $ is not defined
    at restrictReferences (common.js:123:5)
    at ln (utils.js:294:28)
    at Ls (data-include.js:54:16)
    at data-include.js:98:7
    at async Promise.all (/vc-use-cases/index 34)
    at async Ps (data-include.js:107:6)
    at async Object.run (data-include.js:77:8)
    at async base-runner.js:72:27

Expected behavior (e.g., it shouldn't crash):

Unused terms should be removed.

Optional, steps to reproduce:

See https://github.com/w3c/vc-use-cases/blob/2023-08-25-cleanup/index.html and https://github.com/w3c/vc-use-cases/blob/2023-08-25-cleanup/terms.html.

Fails under Chrome Version 116.0.5845.96 (Official Build) (x86_64) on MacBook Pro, macOS Monterey 12.6.7.

[jandrieu]

Also see tobie/pr-preview#144

[marcoscaceres]

This appears to be an issue with restrictReferences() function in that spec's included common.js stuff (not part of ReSpec). It's trying to use "$", which I guess is jQuery? However, the main document is not including jQuery (ReSpec doesn't use or include jQuery anymore, and hasn't for many years).

[marcoscaceres]

@jandrieu (and maybe @msporny), it would be amazing if there is stuff that is needed in common.js from the VC specs could move into ReSpec proper... I don't know what common.js is doing, but it's better if we just get ReSpec to handle whatever that is.

[marcoscaceres]

@jandrieu, and just as an aside.... I think to fix the $ issue, just add <script src="https://ajax.googleapis.com/ajax/libs/jquery/3.7.1/jquery.min.js"></script> to the doc above the ReSpec script.

(but there should really be no reason to be using jQuery nowadays)

[msporny]

@jandrieu, and just as an aside.... I think to fix the $ issue, just add <script src="https://ajax.googleapis.com/ajax/libs/jquery/3.7.1/jquery.min.js"></script> to the doc above the ReSpec script.

We removed jquery a while ago in the "latest" common.js... we should just update the vc-use-cases repo w/ the new restrictReferences() function to fix this particular issue:

https://github.com/w3c/vc-data-model/blob/main/common.js#L131-L185

IOW, this is a common.js issue, not a ReSpec issue.

[msporny]

@jandrieu (and maybe @msporny)

Also, /cc @gkellogg (who, I believe, wrote the initial restrictReferences() function)... and @jyasskin who noted this issue recently when reviewing the VCDM specification.

it would be amazing if there is stuff that is needed in common.js from the VC specs could move into ReSpec proper... I don't know what common.js is doing, but it's better if we just get ReSpec to handle whatever that is.

Yes, agreed. I haven't suggested moving some of this to ReSpec yet because it's unclear if common.js is doing some sort of ugly hack, if ReSpec already has this feature, or if this is a general feature that other specifications might find useful. Also, lack of time to engage in a respectful way to @marcoscaceres time and availability (he's super busy with many other Really Important Things(tm)).

I'll try to describe the feature and then see if ReSpec already does this (it might, the common.js "hack" went in over 7 years ago, IIRC).

In general, we wanted to define ONE terminology file that multiple specs in the WG used. We want a shared terminology file, which is pulled in via data-include, because we didn't want the definitions to drift between the specs over time.

The problem with including one terminology file is that not every spec uses all of the terminology, and that results in ReSpec linting warnings. So, the restrictReferences() function goes through every <dfn> tag and removes references that are not used in the specification.

There are at least two problems w/ this approach:

• In PR Preview, since the terminology file is pulled in relative to the spec (./terms.html), that file isn't included in the PR Preview build and so all the terminology definitions fail to be included. PR Preview is still useful, but has a bunch of red underlines under every terminology reference (since all the terminology is "broken" in the PR Preview). This could be "fixed" by setting base or providing an absolute URL to the terminology file and common.js (maybe? I don't know, haven't tried it yet).
• We are copy-pasting restrictReferences() all over the place, the code is drifting, and resulting in issues like this being raised.

The "ideal" solution, which might already be implemented in ReSpec, or might be so ideal that it's impossible to achieve, is to be able to pull terminology definitions into a specification from any other specification and have them appear in the "Terminology" section of a specification.

I don't believe this is currently possible because not every specification defines their terminology like you would find in a dictionary, with the term listed and then a brief description provided. Many definitions are provided in-line in a paragraph that talks about other things. I think W3C has made a few attempts at a "global glossary", and I'm not sure those attempts were resounding successes?

I do know that ReSpec does have the ability to reference terms in other specifications, which is useful, but not quite the feature we're looking for here. Mainly because people tend to complain when we cross-link too heavily between specs, especially when reading a single paragraph sends you off into 3-4 other specifications (hooray for proper architectural layering?). The grumbling typically goes like this: "Ugh, why are you making me jump all around to all these specifications when all I want to do is find out how to implement this thing you've specified." (which is a legitimate gripe).

So, I think the solution we're looking for here provides for an option to transclude the terminology that we need into the specification and put each term in, verbatim, to a particular Terminology section. Does ReSpec provide this feature today? If not, is this a "generally useful feature"?

[jyasskin]

I think it's an antipattern to define the same terms across multiple specifications (sorry VC folks), but it would be generally useful to be able to find full definitions and copy them into the local spec. @tabatkins should also be involved, since it'd be good for the design to work across both Respec and Bikeshed.

Right now, as Manu says, definitions are marked with <dfn id data-export>, and they can appear anywhere inside a paragraph, which makes it hard to know what to copy out to the referring specification. (<h1>-<h6> elements can also be definitions, which makes it easy to know what to copy, but the section can be too long to copy anyway.)

What if we

1. Allow <dt> to also hold definitions, as if it were a heading element
2. Define an attribute on <p> and maybe a few other block-level elements to say "this is the definition for a particular <dfn>'ed term." Maybe data-defines="#id-of-the-definition", which spec processors could compute from defines="[=shorthand=]", or even infer entirely when a paragraph includes a single definition.

Then referring specs could generate a Terminology section that included the actual definitions, instead of just a list of "Terms defined by reference". We could also have clicks on references pop up an inline definition (speced/bikeshed#1321) instead of always linking away to the defining spec.

[gkellogg]

The restrictReferences() function was used in the JSON-LD specifications (three documents) to allow for a shared term glossary, although terms are defined in only one of those three specs. There is some cleanup work to remove terms not actually used in any given specification, so that they can be maintained in a given place. However, this is fragile and can be disrupted by downstream changes to ReSpec. Note that definitions defined in another spec still appear, and can be easily referenced, but they dereference to the specification that holds the actual definition.

All of this is challenging to handle before FPWD, when mutual references are published. But, there are many other challenges of managing a document series.

Some specs choose to define terms in a given section, others scatter them through the document, and both ways are reasonable, IMO. But, now that ReSpec supports automating the terms defined and used in the specification using <section id="index"></section> this might be obsolete. What's missing is a way to provide some more context around those references, and perhaps some extension method could be provided to add further description. If that was the case manually editing term definition sections might not be as important.

@jyasskin's ideas about something like <p data-defines="#id-of-the-definition">Narrative definition goes here</p> sound interesting.

[marcoscaceres]

Basically, what @jyasskin said. Defining things across multiple specs should no longer necessary (and bit of an anti-pattern). We have exceptional tooling now for definitions across with "xref" and a bunch of stuff that W3C has built. Even with data-include, there is still a risk of things getting out of sync (specially on TR), so it's best to just define in one place and let ReSpec do the linking to the canonical definition.

What's missing is a way to provide some more context around those references, and perhaps some extension method could be provided to add further description. If that was the case manually editing term definition sections might not be as important.

Agree. But that's basically what the Web's standard <a href=""> is for. It would be nicer of course, if we could pop-over the definitions (like Wikipedia does), but unfortunately, we can't because of how things are defined in different documents.

@jyasskin proposal is compelling.
Thought I think we just want something simpler, like:

<p>
  <span class="def">The <dfn>whatever thing</dfn> is the whatever to use</span>.
  However, <span class="def"><dfn>this other thing</dfn> is also defined here, and could be show elsewhere</span>
</p>

And we just have dt's <dd> automatically used, with some syntax to handle exceptional cases (such as the case for headings, which ReSpec doesn't support anyway because of the reason that headings are usually not great for definitions).

[tidoust]

And we just have dt's <dd> automatically used, with some syntax to handle exceptional cases (such as the case for headings, which ReSpec doesn't support anyway because of the reason that headings are usually not great for definitions).

CSS specs often use of the <dt> pattern to define and detail values of a CSS property. The definition can be somewhat long, with code, examples, notes and issues. Some of it is easy to filter out, but without further indication, it's easy to end up with a definition that does not make sense or remains too long. I wouldn't necessarily call that an "exceptional" case ;)

A couple of examples of somewhat lengthy <dd> linked to a definition in <dt>:
https://drafts.csswg.org/css-will-change-1/#valdef-will-change-custom-ident
https://drafts.csswg.org/css-fonts/#valdef-font-size-relative-size

FWIW, Reffy already tries to extract textual definitions from CSS specs when it cannot find anything better (see code). That remains clumsy and essentially unusable, except to provide contextual information to readers reviewing the data.

The ability to extract definitions in HTML, instead of text, has been in the back of our mind for some time. This would also be useful for reference documentation projects such as MDN.

[tabatkins]

Right, there is no way to extract "the definition" of a term automatically, given current spec-writing practices. It would have to be something spec authors explicitly indicate, and given the very large body of standards that are not actively maintained, it would not be something one could rely on for the vast majority of terms.

You could probably do a halfway reasonable approach by taking the entire paragraph a term is defined in, or the first paragraph of the dd corresponding to a definition dt, but it would be a very rough approach, and would fail in many cases. And when the definition is a heading, I don't think there's any automatic extraction that would work.

[dontcallmedom]

discussing this with @tidoust, we would be willing to run an experimentation in this space to see if there is a pattern that could usefully be generalized to other specs.

This would mean:

• define an initial markup convention - I would suggest starting with something as simple as possible while fitting the particular specs we're targetting. @gkellogg @msporny any preference between @marcoscaceres 's Contextualizing definitions for export  #4522 (comment) and @jyasskin 's Contextualizing definitions for export  #4522 (comment) convention? The convention should be clean enough that we would feel reasonably confident it could serve as a basis to an update to https://github.com/speced/spec-dfn-contract
• we would update reffy/webref to do the extraction based on that convention; this will very likely need some rules to clean up the markup being extracted (e.g. removing UI markup generated by ReSpec) - ideally these rules would also serve as a basis for shared conventions in ReSpec and Bikeshed (we already have to do some of this ad-hoc clean-up in reffy in other contexts)
• respec would then add the support to display in-line one way or another (some of?) the definitions content
• the VC specs would be updated to use the new mechanism
• once we gain experience with it, we try to generalize it, making it usable in bikeshed, updating webdex, etc

does this feel compelling? the respec work would include both client-side and server-side changes, so getting some sense if people would be ready to put in the work would be useful before we put this in motion.

[msporny]

does this feel compelling? the respec work would include both client-side and server-side changes, so getting some sense if people would be ready to put in the work would be useful before we put this in motion.

Yes, it sounds compelling.

If we had this feature, I can volunteer to update the W3C Verifiable Credential and Data Integrity specs to use it instead of what we're doing today... and advocate for other specs to upgrade to the new mechanism as well.

[tabatkins]

Yeah, I think this is a reasonable direction to explore. Given that I already have a CSS Index spec auto-genned from Bikeshed, I'd appreciate a more complete definition for them too. ^_^

[tidoust]

• define an initial markup convention - I would suggest starting with something as simple as possible while fitting the particular specs we're targetting. @gkellogg @msporny any preference between @marcoscaceres 's Contextualizing definitions for export  #4522 (comment) and @jyasskin 's Contextualizing definitions for export  #4522 (comment) convention? The convention should be clean enough that we would feel reasonably confident it could serve as a basis to an update to https://github.com/speced/spec-dfn-contract

In the generated spec, we are going to need something like a data-defines="#id-of-the-definition" attribute in any case, because some definitions will not contain the related <dfn> term (be it only when the <dfn> is in a <dt> while the corresponding prose definition is in the following <dd>). In other words, we need a way to relate the definition text with the <dfn> in the general case. For consistency, if we can have that everywhere, that's going to simplify processing and avoid having to introduce heuristics at the processing stage.

From an authoring perspective, the easier, the better. For the same reason, authors will need some way to relate definition text with a <dfn> to handle cases where the text does not wrap the <dfn>. A class="def" attribute would not work for that. Something like a data-defines attribute would work.

Authoring tools could start by only supporting two cases for that attribute:

1. No value? The definition text is for the <dfn> enclosed in the element. Over time, that logic may be extended to cover cases such as "look for a definition in the previous <dt>", "look for a definition in the closest heading", etc.).
2. Value specified? Interpret the value as a reference to the <dfn>. What makes that a bit clumsy is that the value will need to contain the shorthand to clarify what term is being targeted when there's some ambiguity. So "[=term=]" to reference a term, "{{term}}" to reference some Web IDL construct, etc. Shorthands could perhaps be omitted when there is no ambiguity.

Regardless, one possible source of confusion for authors is that the data-dfn-for attribute will look a lot like that new attribute.

[tabatkins]

Agree with (1) - a data-defines or similar attribute, with no value but with a single dfn inside of it, should be taken as marking the confines of the definition for that dfn.

I'm not certain what you're referring to with your (2). You're referring to the dfn by ID, how could there be any ambiguity?

[tabatkins]

Only half-joking: data-dfn-longdesc

[tidoust]

I'm not certain what you're referring to with your (2). You're referring to the dfn by ID, how could there be any ambiguity?

I was thinking that, in the source, it might be more convenient for authors to refer to the dfn without using IDs, because IDs only appear in the generated specification.

For instance, an IDL interface Foo will likely create an ID dom-foo, while a CSS property foo will create an ID propdef-foo. In the generated spec, their definitions would end up with data-defines="dom-foo" and data-defines="propdef-foo" attributes. In the source, it seems easier for authors to be able to say data-defines="foo"... but that's only going to work when there is no ambiguity. If there is ambiguity, data-defines="{{Foo}}" or data-defines="'foo'" would make the intent explicit, and would be consistent with the mechanisms that authors already need to use and know to refer to definitions from other parts of the specification.

[tabatkins]

I would not try to invent a new reference mechanism for this one location. If you don't want to guess at the generated ID, you can create the ID manually on the definition.

[marcoscaceres]

Happy to see where this goes. Might be good to just start with some very simple cases (inline "dfn" or "dt > dfn + dd" cases) and go from there.

[msporny]

Happy to see where this goes. Might be good to just start with some very simple cases (inline "dfn" or "dt > dfn + dd" cases) and go from there.

I offer the VC Data Model v2.0 and Data Integrity specifications as a sacrifice. We do the inline "dt > dfn + dd" thing in those specs (so, it should be fairly straightforward).

[tidoust]

Via w3c/reffy#1444 (comment). @jyasskin's proposal suggests a data-defines="#id" construct. Any reason to have # in there? Could it be data-defines="id" instead? I suppose that's meant to make it look like a href. I don't see why we would ever need to use absolute URLs or something more complex.

The only advanced scenario that comes to mind would be that of a multi-page spec. But that seems bad practice to have the actual definition of a term in a different page from the term itself anyway.

[tabatkins]

Yeah, I'd just take it as an id. The definition should always live next to the term being defined.

[jyasskin]

I see two reasons:

1. It leaves lexical space to decide to put shorthands in that field later. This could be as simple as data-defines="term" vs data-defines="#term". It's unlikely an ID would collide with the string form of a different definition, but I don't like putting unlikely failures in my programming languages. Using a # for IDs here doesn't commit us to adding the shorthands, but it allows it.
2. I'm always confused by the aria-labelledby sort of attribute that refer to other elements without that being apparent in their syntax. Using a # makes it clear that this is a reference to an ID. Making it shorter isn't always better.

[dontcallmedom]

@msporny with #4620 merged and released, could you look into updating the relevant specs to get them marked up in a way that enable the extraction of the definitions? This would give us a basis to work on the next step (getting these extracted definitions to be re-importable in a respec doc)

[jyasskin]

Specifically, it looks like the request is to put class=definitions on the <dl>s that contain definitions? This should get documented in respect.org, but it could make sense to wait on that until the full pipeline is working.

[msporny]

@msporny with #4620 merged and released, could you look into updating the relevant specs to get them marked up in a way that enable the extraction of the definitions?

Oooh, excellent, thank you!

I'll do a pass this weekend on the Verifiable Credentials Data Model spec as well as the Data Integrity spec, which re-uses many of the definitions. I think I might be able to exercise both markup styles provided by the new feature.

[msporny]

@msporny with #4620 merged and released, could you look into updating the relevant specs to get them marked up in a way that enable the extraction of the definitions?

I believe I did the necessary bits in "vc-data-model" here:

w3c/vc-data-model@e577060#diff-0eb547304658805aad788d320f10bf1f292797b5e6d745a3bf617584da017051R647

... and in the rest of the spec here:

w3c/vc-data-model@e577060

... and now I think I need to wait 6 hours to a day for those refs to show up in webref?

Once that's done, I plan to start cross-referencing from the vc-data-integrity specification to the vc-data-model specification... and if that works, will be ready to attempt some sort of auto-generate-the-terminology-section PR.

I will note that there is another approach that feels compelling, which is showing the terminology definition when you hover over the term, like what's done in Spec-Up (mouse over "verifiable credential" link to see the preview): https://hyperledger.github.io/anoncreds-spec/#terminology -- which obviates the need to click and jump to a separate specification.

I'll note that the Spec-Up approach might make it so that auto-generating the terminology section is less ideal (because it removes the need to click and go "somewhere else" to figure out what the term means). We would (most likely?) want to preserve the link that jumps to the definition for a11y reasons. We might also want to include a "Go to definition" link if we choose to take the "term preview" approach that Spec-Up uses.

[tabatkins]

Hm, yeah, is the idea that we'd generate a terminology section for links with these detected definitions, and have the link jump you to there (with a link to the actual source there, next to the definition)? Similar to how biblios work today?

Bikeshed has recently gained the ability to show info panels on links; see flexbox, for example. (Have to be on a device with a pointer; the panel shows up on mousedown. Drag into the panel before releasing to keep it up.) I could definitely embed the definition there, to make it accessible without disrupting the link itself.

[dontcallmedom]

The VC terms and definitions have landed in webref: https://github.com/w3c/webref/blob/main/ed/dfns/vc-data-model-2.0.json.

List of definitions

"/claims: An assertion made about a subject."
"/credential: A set of one or more claims made by an issuer. The claims\nin a credential can be about different subjects. The definition of\ncredential used in this specification differs from,\nNIST's definitions of\ncredential."
"/holders: A role an entity might perform by possessing one or more\nverifiable credentials and generating verifiable presentations\nfrom them. A holder is often, but not always, a subject of the\nverifiable credentials they are holding. Holders store their\ncredentials in credential repositories."
"/issuers: A role an entity can perform by asserting claims about one or\nmore subjects, creating a verifiable credential from these\nclaims, and transmitting the verifiable credential to a\nholder."
"/presentation: Data derived from one or more verifiable credentials, issued by one or\nmore issuers, that is shared with a specific verifier."
"/credential repository: A program, such as a storage vault or personal verifiable credential\nwallet, that stores and protects access to holders'\nverifiable credentials."
"/selective disclosure: The ability of a holder to make fine-grained decisions about what\ninformation to share."
"/subjects: A thing about which claims are made."
"/claim validation: The assurance that a claim from a specific issuer satisfies the\nbusiness requirements of a verifier for a particular use. This\nspecification defines how verifiers verify verifiable credentials and\nverifiable presentations.
\nIt also specifies that verifiers validate claims in verifiable credentials before relying on them. However, the means for such validation\nvary widely and are outside the scope of this specification. It is expected\nthat verifiers will trust certain issuers for certain claims and\napply their own rules to determine which claims in which credentials\nare suitable for use by their systems."
"/vc: A verifiable credential is a tamper-evident credential that has authorship that\ncan be cryptographically verified. Verifiable credentials can be used to build\nverifiable presentations, which can also be cryptographically verified."
"/verifiable data registries: A role a system might perform by mediating the creation and verification\nof identifiers, keys, and other relevant data, such as\nverifiable credential schemas, revocation registries, issuer public keys,\nand so on, which might be required to use verifiable credentials. Some\nconfigurations might require correlatable identifiers for subjects. Some\nregistries, such as ones for UUIDs and public keys, might just act as namespaces\nfor identifiers."
"/vp: A verifiable presentation is a tamper-evident presentation encoded in such a way\nthat authorship of the data can be trusted after a process of cryptographic\nverification. Certain types of verifiable presentations might contain data that\nis synthesized from, but do not contain, the original verifiable credentials (for example, zero-knowledge proofs)."
"/verify: The evaluation of whether a verifiable credential or verifiable presentation is an authentic and current statement of the issuer or\npresenter, respectively. This includes checking that: the credential (or\npresentation) conforms to the specification; the proof method is satisfied; and,\nif present, the status check succeeds. Verification of a credential does not\nimply evaluation of the truth of claims encoded in the credential."
"/verifier: A role an entity performs by receiving one or more\nverifiable credentials, optionally inside a\nverifiable presentation for processing. Other specifications might refer\nto this concept as a relying party."

(generated via jq '.dfns[]|select(.access =="public" and .htmlProse)|.for[0] +"/" + .linkingText[0] + ": " + .htmlProse' vc-data-model-2.0.json)

The terms are thus already auto-linkable from any Respec or Bikeshed spec.

The next steps to enable the automatic inclusion of the definition prose in ReSpec would be to:

• update https://github.com/w3c/respec-web-services/ to expose the new htmlProse field from webref in xref
• update ReSpec to make use of it (for pop-over boxes on auto-links, and/or for the terminology index)

While the second step won't work until the first one is deployed, I think it's likely already possible to draft a pull request to experiment with various markup options to achieve the intended result (taking accessibility into account in particular).

Any taker for the first step? I'll get to it at some point if nobody else does, but certainly wouldn't mind someone else stepping up :)

[dontcallmedom]

with w3c/respec-web-services#401 merged, xref now exposes htmlProse in its results

[msporny]

@tabatkins wrote:

Hm, yeah, is the idea that we'd generate a terminology section for links with these detected definitions, and have the link jump you to there (with a link to the actual source there, next to the definition)? Similar to how biblios work today?

Yes, that's one potential way it could work. The other is allowing a "mouseover" affordance so the individual doesn't even need to jump to the terminology section (but that has a11y challenges that will need to be worked through).

Bikeshed has recently gained the ability to show info panels on links; see flexbox, for example. (Have to be on a device with a pointer; the panel shows up on mousedown. Drag into the panel before releasing to keep it up.) I could definitely embed the definition there, to make it accessible without disrupting the link itself.

Yep, exactly. I think the jury is still out on the fine-tuning / end result, but the bits required are more or less there now to build the experience.

[dontcallmedom]

FWIW, I've updated webdex to include the definition along with the existing data there, see e.g. https://dontcallmedom.github.io/webdex/c.html#claim%20validation%40%40vc-data-model%25%25dfn

[msporny]

FWIW, I've updated webdex to include the definition along with the existing data there, see e.g.

Excellent, looks great @dontcallmedom... and that's effectively the same sort of functionality that we'd need for building terminology sections in specs, right?

I've got an action item (this weekend) to convert the Data Integrity spec over to use the new format introduced here and the interesting thing in that spec is that it has a terminology section, but we'd want to include terminology linked to in other specs (such as infra and VC data model). I expect we'd need effectively the same sort of code used in webdex to transclude those external definitions into the existing definitions in the existing terminology section.

Perhaps that would be done through another CSS class like?

      <dl class="termlist definitions external-definitions">

... and then some of the questions become:

• Should there be an option to only transclude a specific number of external definitions?
• Is that termlist alphabetized or not (or should it be an option)?