Generative AI for Kubernetes Docs

[reylejano]

Discussion about using generative AI for Kubernetes docs started on the SIG Docs biweekly meeting on July 11, 2023. This GH discussion continues the conversation from the July 11 meeting. The following are notes taken from the July 11, 2023 meeting notes:

[Robert Reeves, Linux Foundation] Generative AI for Kubernetes Docs
Do we want support for tooling from specific vendors in this space? If so, LF might be able to help

[Marcelo] What about the issue of trawling sources that “aren’t allowed” in terms of generating this content?

[Keegan] We can have an instance that is trained on only the things we want
[Nate] Our use case wouldn’t likely be solely about generating docs from scratch, however, as Keegan mentioned, our instance would be specific to our project/docs

[Tim] Using Large Language Models (LLMs) to produce warnings on certain docs or improvements to be made across our docs

[Natali] Spitballing here, but using LLMs to help us with smaller, keyword-based checks against our style guide/spell checking in order to help ease the workload of reviewers, lower-hanging fruit kind of work

[Tim] Looking into the future for new contributors, this could come in the form of a suggested feedback/style guide information on a contributor’s PR

[Rey] We can make a GitHub Discussion about this to help collect more ideas and get back to Robert and the LF about what we want to do and where we want support

[sftim]

Can we use a generative AI to suggest improvements to our example manifests?
(and then validate that the updated manifest is still valid)

[divya-mohan0209]

One of the challenges we've faced (and continue to face) is the generation and maintenance of API reference documentation - whether it be during releases or otherwise. What if we had LLMs trained to document the APIs and/or their respective changelogs? There are tools available in the market, like documatic/mintlify, that can do this, but we will need to explore the possibility of integration.

[tpepper]

I'm likely very naive on all of this, but in a “document our stuff” and quality check style against our style space it does feel like a model trained on our stuff could be ok. It’s not like we need a model we can ask to create some documents about a talking lion, some witches, an animated scarecrow, and a young girl swept up with her dog in a tornado and how Kubernetes solves their predicament.

[a-mccarthy]

The localization subproject also talked about this topic in our August 7, 2023 meeting. In terms of localization, these tools could help do draft translations, but a human reviewer and approver is still needed. We've had issues before with folks submitting content entirely generated and the content being unusable.

[r2-lf]

Thanks, @nate-double-u ! I plan on taking the three most recent blogs in https://github.com/kubernetes/website/tree/main/content/en/blog/_posts to Spanish using translation tools "manually". I'm selecting Spanish as I have experience with that language. I'll PR them for the es folder.

My goal is to get feedback on the translation from a native speaker. Once this is working, I can automate with GitHub Actions.

My strategy is do the work first. Then, figure out automation.

[sftim]

@r2-lf we're not looking to use GitHub Actions to support generative AI, at least not right now.

[r2-lf]

Roger, @sftim . What ever automation the team eventual decides, we'll use that.

[sftim]

TL;DR: until the evidence can convince a whole localization team otherwise, we still need humans for this work.

@r2-lf now I've seen the example PR, it looks like you're suggesting a workflow:

• someone (or an automated agent) makes a branch using a large language model to translate an existing page
• the same person or agent turns that into a pull request, unedited
• reviewers provide feedback

If that's the idea: we've tried it, and it doesn't scale. There are two problems:

• in our experience, machine translations are good but not perfect. It's fine if a contributor uses tooling to help them do their work but if the feedback cycle is from machine to reviewer, that's a longer feedback cycle than with a human in the loop, and it burns up our scarce reviewer resources. The main opportunity cost of correcting the machine translation is being able to review [more] work by people who actually speak Spanish.
  Each generation of ChatGPT has got better; I think we'd want evidence that a native / fluent Spanish feature speaking contributor considered the work pretty much unimprovable before we took the author out of the loop.
• because the machine is almost correct, it's hard to persuade people who speak some but not much of the language to step away from submitting changes (and we've had to do this - in fact, we already ended up turning off one localization after the approvers admitted they couldn't speak it properly)

I would love to see tooling where an author is writing a translation and they see a plausible tab-completion as they do. Or a script that produces a draft translation in that team's usual format (for an example of a team-specific format, look at https://github.com/kubernetes/website/blob/main/content/zh-cn/_index.html - you'll see that the English text is preserved as well), ready for polishing.

[electrocucaracha]

Hola a todos, I've been leading the Spanish translation team for some years. During that time, we've seen non-native Spanish speakers trying to contribute to the documentation with the unique interest to increase their metrics (maybe motivated by scholarships). Even when any contribution is welcome, our experience has been not the most pleasant, having to provide a lot of suggestions in multiple cycles. That's why we limit the contributors to native Spanish speakers or few rarely exceptions.

In the another hand, Latam is growing in terms of open source culture, and we use the localization process as a way to provide a safer environment to initiate an open source journey, several open source enthusiasts remember their roots in documentation activities.

My two cents on this is that I'm in favor of using Generative AI tools as tools for improving the contributor experience and facilitate the reviewing process, but I'm fully disagreed to produce massive load for reviewers. BTW, I'm open to participate in the validation of this in case that is needed.

[jpegleg-k8s]

I'm still thinking that the docs syntax and style is a good target.

PR CI triggers call to private instance, output from CI is optionally used by people.

Syntax, formatting, “style”, trained/tuned on our docs, seems to be a good target. These suggestions or outputs can be artifacts in CI which people can reference or leverage if they want to.

We could perhaps have a private instance, limited to our docs scope, tuned for syntax in our requirements. This can be done with Azure OpenAI. The challenge is to get the data assembled, and cost of the private instance, as well as the copyright situation.

[sftim]

I wonder if our entire style guide is small enough to fit in the prompt. My guess is: yes.

[jpegleg-k8s]

We may be able to get away with prompt prepending of style/syntax information, I agree. This can be automatically inserted in the API call, inserted at the start of a chunk of data. This likely isn't optimized compared to embeddings input to tuned instances, but seems like a much easier start. Simple is good.

We might also be able to get more out if we can tune with a larger dataset of good examples.

[jpegleg-k8s]

We could skip the private instance effort, and just use OpenAI API directly.

A simple demo might be to do that and prepend style information.

[a-mccarthy]

I think this might be a good topic for our Docs sprint at the upcoming Kubecon contributor summit 2023. During the docs sprint we'd have a lot of folks from the community in the same room to talk through and put together demos

[SpringWiz11]

I have lately come across the usage of Generative-AI in the field of orchestration. Must say, it is a splendid topic of research. I have made the following study which might be helpful for the world of opensource:

1. Traffic Routing and Load Balancing: AI can aid in real-time traffic routing and load balancing decisions. By analyzing incoming traffic patterns and container performance metrics, AI models can direct traffic to the most suitable and available instances.
2. Configuration Optimization: AI can analyze configuration parameters of containers and services to optimize their performance, security, and resource consumption. This can involve suggesting the best settings for various parameters based on the current workload and historical data.
3. Fault Prediction and Recovery: AI models can predict potential failures or anomalies in containerized applications based on historical data and system metrics. This proactive approach enables orchestration platforms to take preventive actions or quickly initiate recovery processes.
4. Auto-scaling: AI models can monitor the incoming traffic and performance metrics of containers and services. Based on this data, they can automatically scale the number of container instances up or down to meet demand. This ensures efficient resource utilization and maintains consistent performance.

[sftim]

Thanks for the comment.
I'm not sure that these help with the documentation work.

[rolfedh]

U.S. Copyright Office

• Copyright Registration Guidance: Works Containing Material Generated by Artificial Intelligence (pdf)
• Webinar: Registration Guidance for Works Containing AI-generated Content
• Copyright and Artificial Intelligence landing page

[rolfedh]

AI-generated summary from Copyright Registration Guidance: Works Containing Material Generated by Artificial Intelligence:

• Understanding of Copyright Eligibility: AI-generated content is not eligible for copyright protection. Only the human-contributed aspects of a work can be copyrighted.
• Clear Disclosure in Applications: When submitting a work for copyright registration that includes AI-generated content, writers must clearly identify and describe both the human-authored content and the AI-generated material.
• Exclusion of AI Content: AI-generated material that significantly contributes to the work must be explicitly excluded in the application. A brief description of the AI-generated content should be provided under the "Material Excluded" heading.
• Correction of Previous Submissions: If a writer has already submitted an application that inadequately disclosed AI-generated material, they must correct the information, either through contacting the Copyright Office's Public Information Office (for pending applications) or by submitting a supplementary registration (for processed applications).
• Potential Risks of Non-disclosure: Failing to properly disclose AI-generated content may lead to the cancellation of the registration or disregarding of the registration in legal proceedings.
  Awareness of Ongoing Changes: Writers should stay informed about any future guidance or changes in copyright law as it relates to AI-generated content, as the Office continues to monitor developments in this area.
• Ethical Use of AI Tools: Though not explicitly mentioned, an underlying theme is the responsible use of AI in the creative process, understanding that the technology serves as a tool rather than a co-author.

These insights align with the Copyright Office's position that human creativity is the cornerstone of copyright protection. Writers incorporating AI into their creative works should approach the registration process with transparency and attention to the specific guidelines concerning the disclosure of AI-generated content.

[nate-double-u]

The Linux Foundation has just published Guidance Regarding Use of Generative AI Tools for Open Source Software Development. (Might be relevant to this conversation)

[a-mccarthy]

We talked about this topic at the KubeCon NA 2023 contributor summit Docs sprint. Here is some notes on our discussion: https://hackmd.io/G3Lv4qIfQOetPgJQzz65kQ?view.

In summary, using AI tools could provide a lot of help on several use cases within SIG Docs, unfortunately we don't have the people to take on the research, testing, implementation and maintenance to tackle the larger use cases, like reviewing PRs or translations. If we start using any AI tool, it should be focused on smaller use cases and workflows, like video transcription and meeting note summaries.

We also came up with a list of some AI writing tools:

• claude.ai
• docsbot.ai
• writer.com
• jasper.ai
• fireflies.ai
• www.deepl.com
• chat gpt
• transcription tools:
  • otter.ai
  • fasterwhisper - https://github.com/guillaumekln/faster-whisper

We had a really lively discussion on this topic! Please continue to reach out if you are interested in helping with this :)

[dims]

FYI there's guidance that is being worked on for CNCF (Nate pointed to the LF link above) that is more focused for our use. no ETA yet.

[a-mccarthy]

thanks @dims!