Jinja is not rendered in action default values prior to schema checking #3820

nmaludy · 2017-10-30T20:07:03Z

Problem

When trying to call an action with Jinja for its default value an error is raised complaining about a type mismatch.

Action metadata

---
description: Run a local linux command
enabled: true
runner_type: mistral-v2
entry_point: workflows/mistral-basic.yaml
name: mistral-basic
pack: examples
parameters:
  cmd:
    required: true
    type: string
  timeout:
    type: integer
    default: 60
  kv_test:
    type: integer
    default: "{{ st2kv.system.nick_test | int }}"

Error - CLI

[root@stackstorm st2]# st2 key set nick_test 123
+------------------+-----------+
| Property         | Value     |
+------------------+-----------+
| name             | nick_test |
| value            | 123       |
| expire_timestamp |           |
+------------------+-----------+

[root@stackstorm st2]# st2 run examples.mistral-basic cmd="ls"
ERROR: 400 Client Error: Bad Request
MESSAGE: '{{ st2kv.system.nick_test | int }}' is not of type 'integer' for url: http://127.0.0.1:9101/v1/executions

Error - Logs

2017-10-30 16:03:59,072 107609136 ERROR actionexecutions [-] Unable to execute action. Parameter validation failed.
Traceback (most recent call last):
  File "/opt/stackstorm/st2/lib/python2.7/site-packages/st2api/controllers/v1/actionexecutions.py", line 128, in _handle_schedule_execution
    pack=action_db.pack)
  File "/opt/stackstorm/st2/lib/python2.7/site-packages/st2api/controllers/v1/actionexecutions.py", line 177, in _schedule_execution
    liveaction_db, actionexecution_db = action_service.create_request(liveaction_db)
  File "/opt/stackstorm/st2/lib/python2.7/site-packages/st2common/services/action.py", line 87, in create_request
    allow_default_none=True)
  File "/opt/stackstorm/st2/lib/python2.7/site-packages/st2common/util/schema/__init__.py", line 293, in validate
    jsonschema.validate(instance=instance, schema=schema, cls=cls, *args, **kwargs)
  File "/opt/stackstorm/st2/lib/python2.7/site-packages/jsonschema/validators.py", line 541, in validate
    cls(schema, *args, **kwargs).validate(instance)
  File "/opt/stackstorm/st2/lib/python2.7/site-packages/jsonschema/validators.py", line 130, in validate
    raise error
ValidationError: u'{{ st2kv.system.nick_test | int }}' is not of type u'integer'

Failed validating u'type' in schema['properties'][u'kv_test']:
    {u'default': u'{{ st2kv.system.nick_test | int }}',
     u'type': u'integer'}

On instance[u'kv_test']:
    u'{{ st2kv.system.nick_test | int }}'
2017-10-30 16:03:59,072 107609136 ERROR router [-] Failed to call controller function "post" for operation "st2api.controllers.v1.actionexecutions:action_executions_controller.post": '{{ st2kv.system.nick_test | int }}' is not of type 'integer'
Traceback (most recent call last):
  File "/opt/stackstorm/st2/lib/python2.7/site-packages/st2common/router.py", line 434, in __call__
    resp = func(**kw)
  File "/opt/stackstorm/st2/lib/python2.7/site-packages/st2api/controllers/v1/actionexecutions.py", line 575, in post
    show_secrets=show_secrets)
  File "/opt/stackstorm/st2/lib/python2.7/site-packages/st2api/controllers/v1/actionexecutions.py", line 134, in _handle_schedule_execution
    abort(http_client.BAD_REQUEST, re.sub("u'([^']*)'", r"'\1'", e.message))
  File "/opt/stackstorm/st2/lib/python2.7/site-packages/st2common/router.py", line 53, in abort
    raise exc.status_map[status_code](message)
HTTPBadRequest: '{{ st2kv.system.nick_test | int }}' is not of type 'integer'
2017-10-30 16:03:59,073 107609136 INFO logging [-] 2c636a0c-d2ff-4405-9632-8268bd09d76b - 400 86 19.061ms (content_length=86,request_id='2c636a0c-d2ff-4405-9632-8268bd09d76b',runtime=19.061,remote_addr='127.0.0.1',status=400,method='POST',path='/v1/executions')

humblearner · 2017-11-07T02:35:31Z

@nmaludy: The problem doesn't seem to be jinja filters. It is the quotes around the integer. The default value for integer type should be a number instead of string (this is expected behavior).
Example:

$ st2  run examples.mistral-basic cmd="ls"
ERROR: 400 Client Error: Bad Request
MESSAGE: '2020' is not of type 'integer' for url: http://127.0.0.1:9101/v1/executions

Action metadata

---
description: Run a local linux command
enabled: true
runner_type: mistral-v2
entry_point: workflows/mistral-basic.yaml
name: mistral-basic
pack: examples
parameters:
  cmd:
    required: true
    type: string
  timeout:
    type: integer
    default: 60
  kv_test:
    type: integer
    default: "2020"

and you are interested in providing default value for params. in action meta from datastore. For that you can do something like this:

Action metadata (does't need jinja filter)

---
description: Run a local linux command
enabled: true
runner_type: mistral-v2
entry_point: workflows/mistral-basic.yaml
name: mistral-basic
pack: examples
parameters:
  cmd:
    required: true
    type: string
  timeout:
    type: integer
    default: 60
  kv_test:
    type: string   #----> Use string and datastore will automatically take care of datatype for you.
    default: "{{ st2kv.system.nick_test}}"

Workflow

version: '2.0'

examples.mistral-basic:
    description: A basic workflow that runs an arbitrary linux command.
    type: direct
    input:
        - cmd
        - timeout
        - kv_test
    output:
        stdout: <% $.stdout %>
    tasks:
        task1:
            action: core.local cmd="if echo <% $.kv_test %> | egrep -q '^[0-9]+$'; then echo 'yes'; else echo 'no';fi"  #cmd=<% $.cmd %> timeout=<% $.timeout %>
            publish:
                stdout: <% task(task1).result.stdout %>
                stderr: <% task(task1).result.stderr %>

String in datastore

$ st2 key set nick_test "123isnotnumber"
+------------------+----------------+
| Property         | Value          |
+------------------+----------------+
| name             | nick_test      |
| value            | 123isnotnumber |
| expire_timestamp |                |
+------------------+----------------+

$ st2  run examples.mistral-basic cmd=""
....
id: 5a0124a3de59a42542829c21
action.ref: examples.mistral-basic
parameters:
  cmd: ''
status: succeeded
result_task: task1
result:
  failed: false
  return_code: 0
  stderr: ''
  stdout: 'no'  #------> not a number
  succeeded: true
start_timestamp: 2017-11-07T03:12:35.552628Z
end_timestamp: 2017-11-07T03:12:42.013246Z
+--------------------------+------------------------+-------+------------+-------------------------------+
| id                       | status                 | task  | action     | start_timestamp               |
+--------------------------+------------------------+-------+------------+-------------------------------+
| 5a0124a3de59a42542829c24 | succeeded (1s elapsed) | task1 | core.local | Tue, 07 Nov 2017 03:12:35 UTC |
+--------------------------+------------------------+-------+------------+-------------------------------+

Integer in datastore

$ st2 key set nick_test "123"
+------------------+-----------+
| Property         | Value     |
+------------------+-----------+
| name             | nick_test |
| value            | 123       |
| expire_timestamp |           |
+------------------+-----------+

$ st2  run examples.mistral-basic cmd=""
...
id: 5a0124bbde59a42542829c26
action.ref: examples.mistral-basic
parameters:
  cmd: ''
status: succeeded
result_task: task1
result:
  failed: false
  return_code: 0
  stderr: ''
  stdout: 'yes'  #----> number
  succeeded: true
start_timestamp: 2017-11-07T03:12:59.067428Z
end_timestamp: 2017-11-07T03:13:05.018635Z
+--------------------------+------------------------+-------+------------+-------------------------------+
| id                       | status                 | task  | action     | start_timestamp               |
+--------------------------+------------------------+-------+------------+-------------------------------+
| 5a0124bbde59a42542829c29 | succeeded (0s elapsed) | task1 | core.local | Tue, 07 Nov 2017 03:12:59 UTC |
+--------------------------+------------------------+-------+------------+-------------------------------+

Hope this helps!

nmaludy · 2017-11-07T10:53:17Z

@humblearner The example i gave was simplified to demonstrate the problem.

This also breaks when doing something like:

Objects

---
description: Run a local linux command
enabled: true
runner_type: mistral-v2
entry_point: workflows/mistral-basic.yaml
name: mistral-basic
pack: examples
parameters:
  cmd:
    required: true
    type: string
  timeout:
    type: integer
    default: 60
  kv_test:
    type: object
    default: "{{ st2kv.system.nick_test | from_json_string }}"

st2 key set nick_test '{"test_key": "test_value}'

In this case, in the workflow i want to utilize data within the object natively using <% $.kv_test.test_key %>

Arrays

---
description: Run a local linux command
enabled: true
runner_type: mistral-v2
entry_point: workflows/mistral-basic.yaml
name: mistral-basic
pack: examples
parameters:
  cmd:
    required: true
    type: string
  timeout:
    type: integer
    default: 60
  kv_test:
    type: array
    default: "{{ st2kv.system.nick_test | from_json_string }}"

st2 key set nick_test '["a", "b", "c"]'

In this case, in the workflow i want to utilize data within the arra natively using <% $.kv_test[1] %>

Mierdin · 2017-11-30T22:07:17Z

This is happening because of two reasons. First, the actual execution request takes place before any rendering takes place. Second, even if this wasn't true, the ActionExecutions API controller is calling out to param_utils.render_live_params which only renders "live" params (meaning params being explicitly passed in, which doesn't include default values. So, when the controller requests an execution, the schema is checked, and this value is still an unrendered template, which is of course a string. So with this behavior, the value you're intending to get better also be a string. This behavior is more coincidental than intentional, so we should fix this.

Note in the docs, each example of this is a string value - this is a bit misleading because the actual call to render final params doesn't happen until actual execution of the action.

I'm looking at alternative approaches now, but figured I'd update the discussion with a root cause. I also see the reference to #3828 but as @Kami suggests, that's a much bigger change, so I'm looking to see if there's a less dangerous, temporary solution to this problem.

Bottom line, we at least need to update docs to say only string values are supported for this, but ideally we can render these values before sending the request.

nmaludy mentioned this issue Nov 9, 2017

Feature: Use Jinja's new NativeEnvironment to get native types instead of strings! #3828

Open

Mierdin self-assigned this Nov 15, 2017

Mierdin mentioned this issue Dec 5, 2017

Retrieve live params first, include template-driven defaults #3892

Merged

Mierdin closed this as completed in #3892 Dec 15, 2017

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Jinja is not rendered in action default values prior to schema checking #3820

Jinja is not rendered in action default values prior to schema checking #3820

nmaludy commented Oct 30, 2017

humblearner commented Nov 7, 2017 •

edited

nmaludy commented Nov 7, 2017 •

edited

Mierdin commented Nov 30, 2017 •

edited

Jinja is not rendered in action default values prior to schema checking #3820

Jinja is not rendered in action default values prior to schema checking #3820

Comments

nmaludy commented Oct 30, 2017

Problem

Action metadata

Error - CLI

Error - Logs

humblearner commented Nov 7, 2017 • edited

Action metadata

Action metadata (does't need jinja filter)

Workflow

String in datastore

Integer in datastore

nmaludy commented Nov 7, 2017 • edited

Objects

Arrays

Mierdin commented Nov 30, 2017 • edited

humblearner commented Nov 7, 2017 •

edited

nmaludy commented Nov 7, 2017 •

edited

Mierdin commented Nov 30, 2017 •

edited