Performance improvement on str(tokenlist)

[righthandabacus]

No description provided.

[vmuriart]

Thanks for the pr. The change you're suggesting has had the effect of changing the behavior of how the output is generated and as a consequence it changed the parsed output.
Multiple tests are failing on this commit.

[vmuriart]

I'm currently going over the changes that would be needed to make the pr work, and its overall effects would net on a slower sqlparse as we would need to add multiple sections to update the value of the tokenlists since they are not updated after they are created.

[righthandabacus]

Hi,

Thanks. I have a terribly long SQL (but simple, no nested queries, just a
thousand result columns and many joins) and found it takes many seconds to
parse it. Upon investigating by cProfile, I believe that is the hot spot.
My change works on my example and cut the time spent to only 1/4.

Please let me know if I can help on something. I am new to sqlparse, just
used a few minutes on reading the source code.

Sincerely,
Adrian (righthandabacus)

On Tue, Aug 9, 2016 at 9:18 PM, Vik notifications@github.com wrote:

I'm currently going over the changes that would be needed to make the pr
work, and its overall effects would net on a slower sqlparse as we would
need to add multiple sections to update the value of the tokenlists since
they are not updated after they are created.

—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub
#281 (comment),
or mute the thread
https://github.com/notifications/unsubscribe-auth/ALyScsZ3_9NuoWjeOXuWH5X29VG7XBvEks5qeSbigaJpZM4JgSrO
.

[vmuriart]

Sorry for the very late follow-up.
Can you provide this example (or a similar one) to use as a benchmark.

Did you notice the delay during the parsing or when using any of the filters? (formatting, etc).

If its during the parsing phase, I think the issue is actually with this line that calls the function you listed multiple times. AFAIK TokenList.value is barely used, so we can probably replace it with a lazy evaluation and that may provide the speed-up you saw on your pull request.

[living180]

I suspect the issue that this PR was attempting to be address was the same as reported in #621, and should be fully addressed by PR #710.