Documentation generation is flaky

[owengray-google]

Describe the bug
When running dokka, some files are not generated. This can apparently happen to any file, though it seems orders of magnitude more common for index.html and other files near the root of the output hierarchy. I suspect that this is due to file lock contention by the output writer.

We first discovered this by adding code that enforces that all expected files in our dackka integration tests ("goldens") were actually generated (previously we had only enforced that all generated files were expected).

This seems like it would be a huge problem for us and for you, because it could result in missing files in developer.android.com/kotlinlang.org (depending on how your update process works and how it removes files that are intended to cease being generated from one version to the next.)

From a test and a rough estimate, out of 130,000 files generated by running unmodified upstream dokka on androidx framework/support, 400 files were generated in the second run that were not generated in the first. Around a quarter of these (100 files) were index.html, representing a flake rate for index.html of around 20%, and for other files of 0.2%.

• Dokka version: 1.7.10

Replication process:
-Copy frameworks/support to an upstream-dokka checkout.
--Delete compose/ui/ui-test/src/androidAndroidTest because dokka fails if you don't (throws in org.jetbrains.kotlin.types.expressions--looks like the compiler).
-run ./gradlew shadowJar in the upstream checkout.
-run java -jar ./runners/cli/build/libs/dokka-cli-1.7.10-SNAPSHOT.jar -pluginsClasspath "./integration-tests/cli/build/libs/fat-base-plugin-1.7.10-SNAPSHOT.jar" -sourceSet "-src ./frameworks/support/" in the upstream checkout.
-add dokka/root to git or otherwise copy it for later comparison.
-run java -jar ./runners/cli/build/libs/dokka-cli-1.7.10-SNAPSHOT.jar -pluginsClasspath "./integration-tests/cli/build/libs/fat-base-plugin-1.7.10-SNAPSHOT.jar" -sourceSet "-src ./frameworks/support/" again.
-notice that there is a diff (e.g. git status) between the docs generated by each run. I found roughly 400 out of 130,000 .html files that were generated by the second run but not by the first.

[owengray-google]

When we run this through dackka, we notice the heightened flake rate for all files in the root of the package tree like index.html, such as package-list, classes.html, and packages.html, which upstream dokka does not seem to generate.
I think this points further towards file lock contention.

[owengray-google]

If a quick fix happened to slow down the file writing step, I don't think that would be too much of an issue for us, based on the times that dokka prints out for us on the androidx checkout (unless rendering is mostly file-writing already):

                           Initializing plugins:    465
                                 Validity check:      0
                  Creating documentation models: 373132
Transforming documentation model before merging:   1960
                   Merging documentation models:     30
 Transforming documentation model after merging:    510
                                 Creating pages:  14222
                             Transforming pages:   4366
                                      Rendering: 107674
                           Running post-actions:     11

[owengray-google]

I notice that you have no test for written files, only for files stored in the FileWriter and nodes in the model, so it does look like the issue is after the tree in the FileWriter is generated.

[owengray-google]

This bug was in error; I misread a git diff.

The actual issue upstream is not missing files, but changed files. We had a missing file issue I have since fixed. The bug title still applies, however.

I see changes primarily in link resolution, but also in API type resolution. For example, I found, in pages.json, as a diff between two runs:

,{"name":"abstract fun forLocale(locale: Locale, type: PluralRules.PluralType): PluralRules","description":"androidx.core.i18n.messageformat_icu.simple.PluralRules.Factory.forLocale","location":"root/androidx.core.i18n.messageformat_icu.simple/-plural-rules/-factory/for-locale.html","searchKeys":["forLocale","abstract fun forLocale(locale: Locale, type: PluralRules.PluralType): PluralRules","androidx.core.i18n.messageformat_icu.simple.PluralRules.Factory.forLocale"]}
->
,{"name":"abstract fun forLocale(locale: Locale, type: Any): PluralRules","description":"androidx.core.i18n.messageformat_icu.simple.PluralRules.Factory.forLocale","location":"root/androidx.core.i18n.messageformat_icu.simple/-plural-rules/-factory/for-locale.html","searchKeys":["forLocale","abstract fun forLocale(locale: Locale, type: Any): PluralRules","androidx.core.i18n.messageformat_icu.simple.PluralRules.Factory.forLocale"]}

Roughly 400 files had diffs (as above), most of which were link resolution in docs. I believe the API resolution issue is another case of external/classpath references not resolving, though instead of becoming UNRESOLVED_TYPE becoming Any is common.

The heightened rate of difference appearing in index.html files is expected given that apparently the flakes happen on a roughly per-function level. They will also be localized into certain projects, as flakes are on a per-reference(-to-external-type?) level, which tend to selectively occur in certain projects.

You do still seem to have no tests for written files, though, which could be a gap in coverage.

[vmishenev]

I have investigated this. There are flaky html pages, including pages.json, because of unresolved link.
The reason of the link unresolving is multiple duplication the same class. Dokka logger printed: e.g. "WARN: For BiometricPromptTest: expected 1 page, but got 15".
To sum up, there are symlinks to a folder (a lot of symlink to buildSrc) in your repository androidx framework/support. It cause a duplication of files. Dokka analysis knows nothing about symlinks and considers it as a regular folder.

[owengray-google]

It looks like the pages.json and flaky html files here are two complete different issues.

The first one is due to us lumping all multiplatform files into a single sourceSet at the moment.
The second one is due to dokka trying to produce type links for generic type params like T, and looking in very strange places all across the unimported parts of the classpath, which resolves flakily.