Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Major optimization in parser code #307

Closed
wants to merge 2 commits into from

Conversation

bertfrees
Copy link

This is a long-standing issue that I finally had to tackle because due to our increasingly complex pipelines it had become impossible to execute our code without throwing a huge amount of memory at it.

The problem was that when parsing a XProc step declaration, every single import would result in parsing that file plus all imports of that file, leading to the same files possibly being parsed multiple times. And when you, like us, work with libraries that collect a lot of steps that in turn depend on other libraries, you are soon dealing with tens of thousands of files being parsed. This results in a huge memory consumption before the execution of the pipeline has even started.

Actually I'm a bit surprised that no one else has noticed the issue before. Nevertheless I think my fix will be interesting for others too.

In the end the solution that I came up with is pretty simple: I cache the PipelineLibrary objects. But in order to be able to do this I had to move some things around. I hope you like it.

bertfrees added a commit to daisy/xmlcalabash1 that referenced this pull request Aug 19, 2020
@ndw
Copy link
Owner

ndw commented Oct 11, 2020

Applied to version 1.2.5 for Saxon 9.9 and Saxon 10.x

@ndw ndw closed this Oct 11, 2020
@bertfrees
Copy link
Author

Thanks!

@bertfrees bertfrees deleted the parser-optimization branch March 4, 2021 17:31
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

2 participants