[Data stream lifecycle] Conditionally display effective retention

[gmarouli]

While the global retention is work in progress, we would like to display the effective retention conditionally. For this reason we added an extra requirement to determine if we are going to add the field in XContent serialisation.

A sample yaml test for this could be:

---
"Get effective retention from component template":

  - do:
      cluster.put_component_template:
        name: test-component-template
        body:
          template:
            lifecycle:
              data_retention: "10d"
  - is_true: acknowledged

  - do:
      cluster.get_component_template:
        name: test-component-template
  - match: { component_templates.0.component_template.template.lifecycle.effective_retention: "10d" }
  - match: { component_templates.0.component_template.template.lifecycle.retention_determined_by: "data_stream_configuration" }

---
"Get effective retention from index template":

  - do:
      indices.put_index_template:
        name: test-composable-template
        body:
          index_patterns: "my-data-stream"
          template:
            lifecycle:
              data_retention: "10d"
          data_stream: {}
  - is_true: acknowledged

  - do:
      indices.get_index_template:
        name: test-composable-template
  - match: {index_templates.0.index_template.template.lifecycle.effective_retention: "10d"}
  - match: {index_templates.0.index_template.template.lifecycle.retention_determined_by: "data_stream_configuration"}

---
"Get effective retention from data stream":
  - do:
      indices.put_index_template:
        name: ds-template
        body:
          index_patterns: "my-data-stream"
          template:
            lifecycle:
              data_retention: 10d
          data_stream: { }
  - is_true: acknowledged

  - do:
      indices.create_data_stream:
        name: my-data-stream
  - is_true: acknowledged

  - do:
      indices.get_data_lifecycle:
        name: "my-data-stream"

  - match: { data_streams.0.lifecycle.effective_retention: "10d" }
  - match: { data_streams.0.lifecycle.retention_determined_by: "data_stream_configuration" }

  - do:
      indices.get_data_stream:
        name: "my-data-stream"

  - match: { data_streams.0.lifecycle.effective_retention:  "10d" }
  - match: { data_streams.0.lifecycle.retention_determined_by: "data_stream_configuration" }

---
"Get effective retention from simulating index template":

  - do:
      indices.put_index_template:
        name: test
        body:
          index_patterns: te*
          template:
            lifecycle:
              data_retention: "10d"
          data_stream: {}
  - do:
      indices.simulate_index_template:
        name: test

  - match: {template.lifecycle.effective_retention: "10d"}
  - match: {template.lifecycle.retention_determined_by: "data_stream_configuration"}

[elasticsearchmachine]

Pinging @elastic/es-data-management (Team:Data Management)

[masseyke]

You told me this when we spoke offline and it made sense, but now I can't remember -- why can't we just check PATH_RESTRICTED in toXContent(), and not worry about remembering to call maybeAddEffectiveRetentionParams?

[gmarouli]

No problem it's a lot to keep in mind. We want only the specific APIs to add the effective retention to their response, at the DataStreamLifecycle level we do not know who called us and probably multiple places set the PATH_RESTRICTED. So, we are trying to ensure we do not expose this more than we should by explicitly choosing when to set the maybeAddEffectiveRetentionParams.

Is this clear now?