Skip to content

Commit

Permalink
(+Doc) Flush out Data Tiers
Browse files Browse the repository at this point in the history
πŸ‘‹πŸ½  howdy, team!

I highly value the content on this [Data Tiers](https://www.elastic.co/guide/en/elasticsearch/reference/current/data-tiers.html) page. Thanks for writing it! In my experience, some users may become slightly confused by its golden nuggets due to its brevity. This PR attempts to flush out common questions while remaining concise. 

The main changes are in the first and second-to-last sections; however, I do attempt some heading restructuring to make the TOC idea-groupings more clear for easier scan-throughs. 

The specific clarifications I'd like to push in order of appearance:

- There's content tier (for "data category" > "content" as we've dubbed it on the higher page) and the data temperature tiers (for time series). That the temperature tiers group together is technically not stated so users end up asking about when they'd go hot>warm vs content>warm, etc. I suspect this confusion is only because users come straight to this page instead of starting at the hierarchy-parent page so have linked up. 
- (Main) Frozen being accessed/searched "rarely" should imply, well rarely. I wrote 1% in the PR `[TIP]` guideline section as a discussion starting point. Frequently we see users not understanding either that they actually have been or that they shouldn't have β‰₯25% of all searches hitting frozen tier. This comes up because of architecture bugs (e.g. frozen indices with future timestamps) but also just happenstance (e.g. 01605242 where of searches they hit majority hot, ~5% cold, but then again hit 75% frozen).
- There's a slew of "how do I check that?", "how do I change that (at creation/later)?", "what if I set it null?" questions we get about `_tier_preference` so just extended the existing section already about it. 

TIA! πŸ™
  • Loading branch information
stefnestor committed Apr 27, 2024
1 parent 4664ced commit 94d72a8
Showing 1 changed file with 94 additions and 39 deletions.
133 changes: 94 additions & 39 deletions docs/reference/datatiers.asciidoc
Expand Up @@ -2,40 +2,66 @@
[[data-tiers]]
== Data tiers

A _data tier_ is a collection of nodes with the same data role that
typically share the same hardware profile:

* <<content-tier, Content tier>> nodes handle the indexing and query load for content such as a product catalog.
* <<hot-tier, Hot tier>> nodes handle the indexing load for time series data such as logs or metrics
and hold your most recent, most-frequently-accessed data.
* <<warm-tier, Warm tier>> nodes hold time series data that is accessed less-frequently
A _data tier_ is a collection of <<modules-node,nodes>> within a cluster which share the same
<<node-roles,data node role>>. Elastic recommends this collection of nodes also shares the same
hardware profile to avoid <<hotspotting,hot spotting>>. Data tiers' usage generally splits along
<<data-management,data categories>> for _content_ and _time series_ data. {es} available
data tiers:

* <<content-tier,Content tier>> nodes handle the indexing and query load for content
indices, such as a <<system-indices,system index>> or a product catalog.
* <<hot-tier,Hot tier>> nodes handle the indexing load for time series,
such as logs or metrics. They hold your most recent, most-frequently-accessed data.
* <<warm-tier,Warm tier>> nodes hold time series data that is accessed less-frequently
and rarely needs to be updated.
* <<cold-tier,Cold tier>> nodes hold time series data that is accessed
infrequently and not normally updated. To save space, you can keep
<<fully-mounted,fully mounted indices>> of
<<ilm-searchable-snapshot,{search-snaps}>> on the cold tier. These fully mounted
indices eliminate the need for replicas, reducing required disk space by
approximately 50% compared to the regular indices.
* <<frozen-tier, Frozen tier>> nodes hold time series data that is accessed
* <<frozen-tier,Frozen tier>> nodes hold time series data that is accessed
rarely and never updated. The frozen tier stores <<partially-mounted,partially
mounted indices>> of <<ilm-searchable-snapshot,{search-snaps}>> exclusively.
This extends the storage capacity even further β€” by up to 20 times compared to
the warm tier.

IMPORTANT: {es} generally expects nodes within a data tier to share the same
hardware profile. Variations not following this recommendation should be
carefully architected to avoid <<hotspotting,hot spotting>>.

When you index documents directly to a specific index, they remain on content tier nodes indefinitely.
Content data will remain on the <<content-tier,content tier>> for its entire
data lifecycle. You can configure your time series data to progress through the
descending temperature data tiers hot, warm, cold, and frozen according to your
performance, resiliency, and data retention requirements. Elastic recommends
automating these lifecycle transitions via <<index-lifecycle-management,{ilm}>>,
specifically also using <<data-streams,Data Streams>>.

[TIP]
====
A data tiers' performance is highly subjective to its backing hardware profile.
See {cloud}/ec-configure-deployment-settings.html#ec-hardware-profiles[{ecloud}'s
hardware profiles] for example {cloud}/ec-reference-hardware.html[hardware configurations].
{es} itself does not require but Elastic generally assumes, for example in {ecloud}
Deployment configurations, that descending temperature data tiers have an increasing
multiplier of cpu and/or heap resources to their data storage ratio, so that later data
tiers can gain more space for data storage at the cost of slower response times.
Under this assumption for a general architecture baseline, the above outline of
descending temperature data tier access proportionalities would reflect as searches
hitting 85% hot, 10% warm, 5% cold, and 1% frozen and ingest targeting
95% hot, 4% warm, 1% cold, and 0% frozen as checked via
<<cat-thread-pool,CAT Threadpools>>. These proportions are not required by {es}
although they encourage stable and highly responsive clusters. They're only intended
to serve as a general architecture baseline to then be applied to your specific
use case, hardware profiles, and architecture per Elastic's
https://www.elastic.co/blog/it-depends[It Depends] philosphy.
====

When you index documents to a data stream, they initially reside on hot tier nodes.
You can configure <<index-lifecycle-management, {ilm}>> ({ilm-init}) policies
to automatically transition your time series data through the hot, warm, and cold tiers
according to your performance, resiliency and data retention requirements.
[discrete]
[[available-tier]]
=== Available data tiers

[discrete]
[[content-tier]]
=== Content tier
==== Content tier

// tag::content-tier[]
Data stored in the content tier is generally a collection of items such as a product catalog or article archive.
Expand All @@ -50,13 +76,14 @@ While they are also responsible for indexing, content data is generally not inge
as time series data such as logs and metrics. From a resiliency perspective the indices in this
tier should be configured to use one or more replicas.

The content tier is required. System indices and other indices that aren't part
of a data stream are automatically allocated to the content tier.
The content tier is required and is frequently seen deployed within the same node
grouping as the hot tier. System indices and other indices that aren't part
of a data stream are automatically allocated to the content tier.
// end::content-tier[]

[discrete]
[[hot-tier]]
=== Hot tier
==== Hot tier

// tag::hot-tier[]
The hot tier is the {es} entry point for time series data and holds your most-recent,
Expand All @@ -71,7 +98,7 @@ data stream>> are automatically allocated to the hot tier.

[discrete]
[[warm-tier]]
=== Warm tier
==== Warm tier

// tag::warm-tier[]
Time series data can move to the warm tier once it is being queried less frequently
Expand All @@ -84,7 +111,7 @@ For resiliency, indices in the warm tier should be configured to use one or more

[discrete]
[[cold-tier]]
=== Cold tier
==== Cold tier

// tag::cold-tier[]
When you no longer need to search time series data regularly, it can move from
Expand All @@ -106,7 +133,7 @@ but doesn't reduce required disk space compared to the warm tier.

[discrete]
[[frozen-tier]]
=== Frozen tier
==== Frozen tier

// tag::frozen-tier[]
Once data is no longer being queried, or being queried rarely, it may move from
Expand All @@ -120,9 +147,13 @@ sometimes fetch frozen data from the snapshot repository, searches on the frozen
tier are typically slower than on the cold tier.
// end::frozen-tier[]

[discrete]
[[configure-data-tiers]]
=== Configure data tiers

[discrete]
[[configure-data-tiers-cloud]]
=== Configure data tiers on {ess} or {ece}
==== On {ess} or {ece}

The default configuration for an {ecloud} deployment includes a shared tier for
hot and content data. This tier is required and can't be removed.
Expand Down Expand Up @@ -156,7 +187,7 @@ tier].

[discrete]
[[configure-data-tiers-on-premise]]
=== Configure data tiers for self-managed deployments
==== On self-managed deployments

For self-managed deployments, each node's <<data-node,data role>> is configured
in `elasticsearch.yml`. For example, the highest-performance nodes in a cluster
Expand All @@ -174,25 +205,49 @@ tier.
[[data-tier-allocation]]
=== Data tier index allocation

When you create an index, by default {es} sets
<<tier-preference-allocation-filter, `index.routing.allocation.include._tier_preference`>>
to `data_content` to automatically allocate the index shards to the content tier.

When {es} creates an index as part of a <<data-streams, data stream>>,
by default {es} sets
<<tier-preference-allocation-filter, `index.routing.allocation.include._tier_preference`>>
to `data_hot` to automatically allocate the index shards to the hot tier.

You can explicitly set `index.routing.allocation.include._tier_preference`
to opt out of the default tier-based allocation.
You can check an existing index's data tier by <<indices-get-settings,polling its
settings>> for <<tier-preference-allocation-filter,`index.routing.allocation.include._tier_preference`>>:

[source,console]
--------------------------------------------------
GET /my-index-000001/_settings?filter_path=*.settings.index.routing.allocation.include._tier_preference
--------------------------------------------------

This `_tier_preference` setting may include a descending preference list for later data tier
temperatures, for example <<cold-tier,cold tier>> would state `data_cold,data_warm,data_hot`.
See <<ilm-migrate,ILM Migrate>> for more context.

{es} will attempt to <<index-modules-allocation,allocate>> the index's shards
according to this setting. This setting will not overpower and may conflict with
other allocation settings preventing the shard from allocating. This historically
has occurred when a cluster has not yet been or has been insufficiently <<troubleshoot-migrate-to-tiers,migrated
to data tiers>>. This setting will not unallocate a currently allocated shard, but
may for example prevent it from migrating from its current location to its designated
data tier. To troubleshoot, run <<cluster-allocation-explain,Allocation Explain>>
against the suspected problematic shard.

A created index will default the `_tier_preference` setting to `data_content` which
will allocate the index' shards to the content tier. A <<data-streams,Data Stream>>
will override its backing created index to `data_hot` to instead default allocate to the
hot tier. You can override these default actions upon index creation by explicitly setting
the preferred value either via an <<index-templates,Index Template>>, see
<<getting-started-index-lifecycle-management,bootstrapping ILM>>, or from within the
<<indices-create-index,created index>> request body itself. You may also override this
setting at any time by <<indices-update-settings,updating index settings>> to the preferred
value.

You may set the `_tier_preference` value to `null` to remove the data tier preference
setting which will allow it to allocate to any data node within the cluster and will not
reset the index's setting back to its respective upon-creation default. Forewarning if you
do that an <<ilm-migrate,ILM Migrate>> may apply a value at a later point if the index is managed.

[discrete]
[[data-tier-migration]]
=== Automatic data tier migration
==== Automatic data tier migration

{ilm-init} automatically transitions managed
indices through the available data tiers using the <<ilm-migrate, migrate>> action.
By default, this action is automatically injected in every phase.
You can explicitly specify the migrate action with `"enabled": false` to disable automatic migration,
You can explicitly specify the migrate action with `"enabled": false` to <<ilm-disable-migrate-ex,disable automatic migration>>,
for example, if you're using the <<ilm-allocate, allocate action>> to manually
specify allocation rules.

0 comments on commit 94d72a8

Please sign in to comment.