Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Can we increase the Oozie WorkflowDefinitionMaxLength? #33

Open
cfournie opened this issue Mar 28, 2017 · 5 comments
Open

Can we increase the Oozie WorkflowDefinitionMaxLength? #33

cfournie opened this issue Mar 28, 2017 · 5 comments

Comments

@cfournie
Copy link
Contributor

cfournie commented Mar 28, 2017 •
edited

What happens if we increase the size of oozie.service.WorkflowAppService.WorkflowDefinitionMaxLength in oozie-default.xml? Can we construct workflwos greater than 100kb in size?

Currently, when creating a large workflow we get the following error (from /usr/local/oozie-4.1.0/logs/oozie.log):

org.apache.oozie.command.CommandException: E0736: Workflow definition length [109,985] exceeded maximum allowed length [100,000]
        at org.apache.oozie.command.wf.SubmitXCommand.execute(SubmitXCommand.java:272)
        at org.apache.oozie.command.wf.SubmitXCommand.execute(SubmitXCommand.java:75)
        at org.apache.oozie.command.XCommand.call(XCommand.java:281)
        at org.apache.oozie.DagEngine.submitJobFromCoordinator(DagEngine.java:139)
        at org.apache.oozie.command.coord.CoordActionStartXCommand.execute(CoordActionStartXCommand.java:185)
        at org.apache.oozie.command.coord.CoordActionStartXCommand.execute(CoordActionStartXCommand.java:63)
        at org.apache.oozie.command.XCommand.call(XCommand.java:281)
        at org.apache.oozie.command.XCommand.call(XCommand.java:351)
        at org.apache.oozie.command.coord.CoordActionReadyXCommand.execute(CoordActionReadyXCommand.java:121)
        at org.apache.oozie.command.coord.CoordActionReadyXCommand.execute(CoordActionReadyXCommand.java:41)
        at org.apache.oozie.command.XCommand.call(XCommand.java:281)
        at org.apache.oozie.command.XCommand.call(XCommand.java:351)
        at org.apache.oozie.command.coord.CoordActionInputCheckXCommand.execute(CoordActionInputCheckXCommand.java:230)
        at org.apache.oozie.command.coord.CoordActionInputCheckXCommand.execute(CoordActionInputCheckXCommand.java:68)
        at org.apache.oozie.command.XCommand.call(XCommand.java:281)
        at org.apache.oozie.service.CallableQueueService$CompositeCallable.call(CallableQueueService.java:323)
        at org.apache.oozie.service.CallableQueueService$CompositeCallable.call(CallableQueueService.java:252)
        at org.apache.oozie.service.CallableQueueService$CallableWrapper.run(CallableQueueService.java:174)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
        at java.lang.Thread.run(Thread.java:745)
Caused by: org.apache.oozie.workflow.WorkflowException: E0736: Workflow definition length [109,985] exceeded maximum allowed length [100,000]
        at org.apache.oozie.service.WorkflowAppService.readDefinition(WorkflowAppService.java:129)
        at org.apache.oozie.service.LiteWorkflowAppService.parseDef(LiteWorkflowAppService.java:45)
        at org.apache.oozie.command.wf.SubmitXCommand.execute(SubmitXCommand.java:164)
        ... 20 more
@cfournie
Copy link
Contributor Author

cfournie commented Mar 28, 2017 •
edited

Changed /usr/local/oozie-4.1.0/conf/oozie-default.xml from 100000 to 200000 for WorkflowDefinitionMaxLength as follows:

    <property>
        <name>oozie.service.WorkflowAppService.WorkflowDefinitionMaxLength</name>
        <value>200000</value>
        <description>
            The maximum length of the workflow definition in bytes
            An error will be reported if the length exceeds the given maximum
        </description>
    </property>

When submitting another large workflow, the same error arises:

org.apache.oozie.command.CommandException: E0736: Workflow definition length [109,985] exceeded maximum allowed length [100,000]

Changing WorkflowDefinitionMaxLength in oozie-default.xml appear to be ineffectual.

@cfournie
Copy link
Contributor Author

Added the following to /usr/local/oozie-4.1.0/conf/oozie-site.xml:

    <property>
        <name>oozie.service.WorkflowAppService.WorkflowDefinitionMaxLength</name>
        <value>200000</value>
        <description>
            The maximum length of the workflow definition in bytes
            An error will be reported if the length exceeds the given maximum
        </description>
    </property>

When submitting another large workflow, a new error arises:

2017-03-28 17:11:55,150 ERROR SubmitXCommand:538 - SERVER[oozie] USER[-] GROUP[-] TOKEN[-] APP[-] JOB[0000000-170328171057119-oozie-root-C] ACTION[0000000-170328171057119-oozie-root-C@49] XException,
org.apache.oozie.command.CommandException: E0803: IO error, java.io.UTFDataFormatException: encoded string too long: 165335 bytes
        at org.apache.oozie.command.wf.SubmitXCommand.execute(SubmitXCommand.java:278)
        at org.apache.oozie.command.wf.SubmitXCommand.execute(SubmitXCommand.java:75)
        at org.apache.oozie.command.XCommand.call(XCommand.java:281)
        at org.apache.oozie.DagEngine.submitJobFromCoordinator(DagEngine.java:139)
        at org.apache.oozie.command.coord.CoordActionStartXCommand.execute(CoordActionStartXCommand.java:185)
        at org.apache.oozie.command.coord.CoordActionStartXCommand.execute(CoordActionStartXCommand.java:63)
        at org.apache.oozie.command.XCommand.call(XCommand.java:281)
        at org.apache.oozie.command.XCommand.call(XCommand.java:351)
        at org.apache.oozie.command.coord.CoordActionReadyXCommand.execute(CoordActionReadyXCommand.java:121)
        at org.apache.oozie.command.coord.CoordActionReadyXCommand.execute(CoordActionReadyXCommand.java:41)
        at org.apache.oozie.command.XCommand.call(XCommand.java:281)
        at org.apache.oozie.command.XCommand.call(XCommand.java:351)
        at org.apache.oozie.command.coord.CoordActionInputCheckXCommand.execute(CoordActionInputCheckXCommand.java:230)
        at org.apache.oozie.command.coord.CoordActionInputCheckXCommand.execute(CoordActionInputCheckXCommand.java:68)
        at org.apache.oozie.command.XCommand.call(XCommand.java:281)
        at org.apache.oozie.service.CallableQueueService$CompositeCallable.call(CallableQueueService.java:323)
        at org.apache.oozie.service.CallableQueueService$CompositeCallable.call(CallableQueueService.java:252)
        at org.apache.oozie.service.CallableQueueService$CallableWrapper.run(CallableQueueService.java:174)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
        at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.RuntimeException: java.io.UTFDataFormatException: encoded string too long: 165335 bytes
        at org.apache.oozie.util.WritableUtils.toByteArray(WritableUtils.java:51)
        at org.apache.oozie.WorkflowJobBean.setWorkflowInstance(WorkflowJobBean.java:351)
        at org.apache.oozie.command.wf.SubmitXCommand.execute(SubmitXCommand.java:203)
        ... 20 more
Caused by: java.io.UTFDataFormatException: encoded string too long: 165335 bytes
        at java.io.DataOutputStream.writeUTF(DataOutputStream.java:364)
        at java.io.DataOutputStream.writeUTF(DataOutputStream.java:323)
        at org.apache.oozie.workflow.lite.NodeDef.writeVersionOne(NodeDef.java:231)
        at org.apache.oozie.workflow.lite.NodeDef.write(NodeDef.java:263)
        at org.apache.oozie.workflow.lite.LiteWorkflowApp.write(LiteWorkflowApp.java:123)
        at org.apache.oozie.workflow.lite.LiteWorkflowInstance.write(LiteWorkflowInstance.java:597)
        at org.apache.oozie.util.WritableUtils.toByteArray(WritableUtils.java:46)
        ... 22 more

Changing WorkflowDefinitionMaxLength in oozie-site.xml is effectual, but a new issue arises.

@kmtaylor-github
Copy link
Contributor

kmtaylor-github commented Mar 28, 2017 •
edited

UTFDataFormatException: encoded string too long: 165335 bytes

Java's conception of UTF-8 has a 2-byte length field, thus capping encoded string length at 65535.
https://docs.oracle.com/javase/7/docs/api/java/io/DataOutput.html#writeUTF(java.lang.String)

This raises another question: why can we get close to 100,000 (which is >64K)

@kmtaylor-github
Copy link
Contributor

How did you create your testcase? The 64k limitation might be an undocumented restriction on any single element of a workflow.xml file, rather than the entire text.

@cfournie
Copy link
Contributor Author

How did you create your testcase?

Poorly. I added a massive configuration block to a workflow.

This raises another question: why can we get close to 100,000 (which is >64K)

Oozie will break the definition up into smaller chunks. Still following the code to see why it wouldn't break up my workflow, but I suspect that my massive config block might be at fault. A large workflow with multiple actions may fare better.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@cfournie @kmtaylor-github