Action "allocation" finished executing (may or may not have been successful)

[karpe-sushant]

To submit a bug or report an issue

Exception saying 'Action "allocation" finished executing (may or may not have been successful)'

Expected Behavior

I have not worked on allocation action before so I do not know the message for successful allocation but it should say 'index conlog-2020-12-28-11-25 moved to warm state'

Actual Behavior

2020-12-28 12:18:11,245 DEBUG                curator.cli         process_action:101  Doing the action here.
2020-12-28 12:18:11,245 DEBUG     curator.actions.allocation              do_action:232  Cannot get change shard routing allocation of closed indices.  Omitting any closed indices.
2020-12-28 12:18:11,245 DEBUG          curator.indexlist          filter_closed:717  Filtering closed indices
2020-12-28 12:18:11,245 DEBUG          curator.indexlist       empty_list_check:226  Checking for empty list
2020-12-28 12:18:11,245 DEBUG          curator.indexlist           working_list:237  Generating working list of indices
2020-12-28 12:18:11,245 DEBUG          curator.indexlist          filter_closed:722  Index conlog-2020-12-24-04-45 state: open
2020-12-28 12:18:11,245 DEBUG          curator.indexlist           __actionable:35   Index conlog-2020-12-24-04-45 is actionable and remains in the list.
2020-12-28 12:18:11,245 DEBUG          curator.indexlist       empty_list_check:226  Checking for empty list
2020-12-28 12:18:11,245 INFO      curator.actions.allocation              do_action:237  Updating 1 selected indices: ['conlog-2020-12-24-04-45']
2020-12-28 12:18:11,245 INFO      curator.actions.allocation              do_action:238  Updating index setting {'index.routing.allocation.require.box_type': 'warm'}
2020-12-28 12:18:11,318 DEBUG     curator.actions.allocation              do_action:248  Waiting for shards to complete relocation for indices: conlog-2020-12-24-04-45
2020-12-28 12:18:11,318 DEBUG              curator.utils            wait_for_it:1808 Elapsed time: 0 seconds
2020-12-28 12:18:11,318 DEBUG              curator.utils           health_check:1542 KWARGS= "{'relocating_shards': 0}"
2020-12-28 12:18:11,327 DEBUG              curator.utils           health_check:1562 MATCH: Value for key "0", health check data: 0
2020-12-28 12:18:11,327 INFO               curator.utils           health_check:1565 Health Check for all provided keys passed.
2020-12-28 12:18:11,327 DEBUG              curator.utils            wait_for_it:1811 Response: True
2020-12-28 12:18:11,327 DEBUG              curator.utils            wait_for_it:1816 Action "allocation" finished executing (may or may not have been successful)
2020-12-28 12:18:11,328 DEBUG              curator.utils            wait_for_it:1834 Result: True
2020-12-28 12:18:11,328 INFO                 curator.cli                    run:223  Action ID: 1, "allocation" completed.
2020-12-28 12:18:11,328 INFO                 curator.cli                    run:224  Job completed.

Steps to Reproduce the Problem

My config file

---
client:
  hosts:
    - https://search-mycluster-foobar.us-east-1.es.amazonaws.com
  url_prefix:
  use_ssl: False
  certificate:
  client_cert:
  client_key:
  ssl_no_validate: False
  http_auth:
  timeout: 30
  master_only: False

logging:
  loglevel: DEBUG
  logfile: /foo/bar/mylog.log
  logformat: default
  blacklist: ['elasticsearch', 'urllib3']

My action file:

---
actions:
  1:
    action: allocation
    description: "Apply shard allocation filtering rules to the specified indices"
    options:
      key: box_type
      value: warm
      allocation_type: require
      wait_for_completion: true
      timeout_override:
      continue_if_exception: false
      disable_action: false
    filters:
    - filtertype: pattern
      kind: prefix
#      value: conlog-
      value: conlog-2020-12-24-04-45
    - filtertype: age
      source: creation_date
      direction: older
      timestring: '%Y-%m-%d-%H-%M'
      unit: days
      unit_count: 1

Result is shared in the Actual Behavior section.

Specifications

• Version: curator, version 5.8.1, AWS Elasticsearch version 7.7.0
• Platform: CentOS7

Context (Environment)

I am in the process of moving ultrawarm policy execution from RunDeck to curator. This is proving a lot harder than I thought, specially when there is very less material regarding issues on the net. I am trying to use curator for rollover, allocation and deletion of ES indices. Rollover is working beautifully via curator. If the indexes wont move to warm state I wont be able to move to next operation 'delete'.

Curator not allocating the indexes form hot-to-warm state.

Detailed Description

I am a newbee for working with curator. Recently I used rollover action successfully for my ES cluster. Please note that I am using AWS ES so i cannot work with ILM. I started working on allocation action to move the indexes from hot to warm state. To begin with I just hardcoded the one index name in action file. If its had been successful then I wanted to use it for the whole cluster. The action file does work in the sense I do see result:true and job completed but with an exception saying 'Action "allocation" finished executing (may or may not have been successful)'. I am not sure what is causing this. Since even the debug logs dont throw any error or exception more than this line. I am unable to find a workaround for this issue.

Please guide me on this or let me know if additional details are required to analyze this issue.

[untergeek]

So, here's what's actually happening with that log message. If you see Action "allocation" finished executing (may or may not have been successful), what it really means is that Curator successfully applied and verified that the allocation setting has been applied to all indicated indices. Whether your shards have re-allocated, however, is not guaranteed. Curator does not wait until all of your shards have migrated before moving to the next action. There are cases where—because of node issues—shards have the allocation tagging properly applied but did not or will not migrate. That's what this message means. It means Curator applied the allocation and verified it. It doesn't mean your shards actually moved. I agree, it's not a comforting or confident message, but it was changed to this format in order to be accurate.

[karpe-sushant]

Hi @untergeek ,

Thank you for quick reply. Your response suggests that the issue is not at curator end then. I must add few more lines of code as those show that filtertype is using ilm to do this task. I am using amazon ES on an ec2 instance which is not using / does not support ILM but works with ISM instead. I should have provided the 5 lines of log which comes before what I shared as curator logs earlier. Rest of the logs are same as I mentioned in my last post in Actual Behavior section. This shows that Curator is using ilm. Is there a work around for this?

---

2020-12-28 12:18:11,239 DEBUG     curator.validators.SchemaCheck               __init__:27   "filter" config: {'filtertype': 'ilm'}
2020-12-28 12:18:11,240 DEBUG          curator.indexlist        iterate_filters:1234 Parsed filter args: {'filtertype': 'ilm'}
2020-12-28 12:18:11,240 DEBUG          curator.indexlist             filter_ilm:1180 Filtering indices with index.lifecycle.name
2020-12-28 12:18:11,245 DEBUG          curator.indexlist           __actionable:35   Index conlog-2020-12-24-04-45 is actionable and remains in the list.
2020-12-28 12:18:11,245 DEBUG          curator.indexlist            __excludify:58   Remains in actionable list: index.lifecycle.name is not set for index conlog-2020-12-24-04-45

---

[untergeek]

Please see the documentation here for how to get around this.