Tokenizer::scan performance improvements

[sam-osborne]

This PR provides a 77.6% reduction in wall time of the Tokenizer::scan method and outlines each of the changes. The profile below shows the overall difference before and after the changes to the Tokenizer::scan method. Profilers were generated based on an execution count of 10,000 and a Handlebars string of 5KB.

1: The majority of the wall time was spent in trying to detect a tag change. The internals of this method use an index to get the get a pair of characters out of the entire message to determine whether it is an opening or closing tag. Evaluating a large message this way was the main culprit of the runtime bottleneck. To avoid checking characters that we not an opening or closing tag, the first change was putting a check for an opening or closing character before executing the tagChange method. This change cut down a majority of the wall time from 144 seconds to 35 seconds given a 10,000 execution count cycle:

2: The second change was made to limit computing the length of the opening and closing tag length which avoids unnecessarily recalculating every time the tag change method was executed. This optimization brought the wall time down 1.3 seconds given a 10,000 execution count cycle.

3: The third change was to replace string lookups by index with references. The first update was to create a reference to the character in evaluation iteration and replace the text lookup by index with the reference. The second update was to create a reference for the first character in the opening and closing tag to avoid a lookup by index in the references created in the first optimization. This optimization brought the wall time down 1.4 seconds given a 10,000 execution count cycle.

Handlebars message utilized:

{{Recipient.FirstName}},

Lorem ipsum dolor sit amet, consectetur adipiscing elit. Aliquam nulla sem, tristique eu nisi sagittis, sodales pulvinar est. Suspendisse purus diam, dapibus vitae suscipit nec, volutpat non est. Sed eget magna a justo pellentesque volutpat vitae id ex. Sed ultricies risus sed vehicula egestas. Ut placerat tellus id vestibulum convallis. Vivamus vel metus vitae sapien fermentum mollis. Phasellus vehicula tortor eu nulla fermentum maximus.

Lorem ipsum dolor sit amet, consectetur adipiscing elit. Aliquam nulla sem, tristique eu nisi sagittis, sodales pulvinar est. Suspendisse purus diam, dapibus vitae suscipit nec, volutpat non est. Sed eget magna a justo pellentesque volutpat vitae id ex. Sed ultricies risus sed vehicula egestas. Ut placerat tellus id vestibulum convallis. Vivamus vel metus vitae sapien fermentum mollis. Phasellus vehicula tortor eu nulla fermentum maximus.

Lorem ipsum dolor sit amet, consectetur adipiscing elit. Aliquam nulla sem, tristique eu nisi sagittis, sodales pulvinar est. Suspendisse purus diam, dapibus vitae suscipit nec, volutpat non est. Sed eget magna a justo pellentesque volutpat vitae id ex. Sed ultricies risus sed vehicula egestas. Ut placerat tellus id vestibulum convallis. Vivamus vel metus vitae sapien fermentum mollis. Phasellus vehicula tortor eu nulla fermentum maximus.

Lorem ipsum dolor sit amet, consectetur adipiscing elit. Aliquam nulla sem, tristique eu nisi sagittis, sodales pulvinar est. Suspendisse purus diam, dapibus vitae suscipit nec, volutpat non est. Sed eget magna a justo pellentesque volutpat vitae id ex. Sed ultricies risus sed vehicula egestas. Ut placerat tellus id vestibulum convallis. Vivamus vel metus vitae sapien fermentum mollis. Phasellus vehicula tortor eu nulla fermentum maximus.

Lorem ipsum dolor sit amet, consectetur adipiscing elit. Aliquam nulla sem, tristique eu nisi sagittis, sodales pulvinar est. Suspendisse purus diam, dapibus vitae suscipit nec, volutpat non est. Sed eget magna a justo pellentesque volutpat vitae id ex. Sed ultricies risus sed vehicula egestas. Ut placerat tellus id vestibulum convallis. Vivamus vel metus vitae sapien fermentum mollis. Phasellus vehicula tortor eu nulla fermentum maximus.

Lorem ipsum dolor sit amet, consectetur adipiscing elit. Aliquam nulla sem, tristique eu nisi sagittis, sodales pulvinar est. Suspendisse purus diam, dapibus vitae suscipit nec, volutpat non est. Sed eget magna a justo pellentesque volutpat vitae id ex. Sed ultricies risus sed vehicula egestas. Ut placerat tellus id vestibulum convallis. Vivamus vel metus vitae sapien fermentum mollis. Phasellus vehicula tortor eu nulla fermentum maximus.

Lorem ipsum dolor sit amet, consectetur adipiscing elit. Aliquam nulla sem, tristique eu nisi sagittis, sodales pulvinar est. Suspendisse purus diam, dapibus vitae suscipit nec, volutpat non est. Sed eget magna a justo pellentesque volutpat vitae id ex. Sed ultricies risus sed vehicula egestas. Ut placerat tellus id vestibulum convallis. Vivamus vel metus vitae sapien fermentum mollis. Phasellus vehicula tortor eu nulla fermentum maximus.

Lorem ipsum dolor sit amet, consectetur adipiscing elit. Aliquam nulla sem, tristique eu nisi sagittis, sodales pulvinar est. Suspendisse purus diam, dapibus vitae suscipit nec, volutpat non est. Sed eget magna a justo pellentesque volutpat vitae id ex. Sed ultricies risus sed vehicula egestas. Ut placerat tellus id vestibulum convallis. Vivamus vel metus vitae sapien fermentum mollis. Phasellus vehicula tortor eu nulla fermentum maximus.

Lorem ipsum dolor sit amet, consectetur adipiscing elit. Aliquam nulla sem, tristique eu nisi sagittis, sodales pulvinar est. Suspendisse purus diam, dapibus vitae suscipit nec, volutpat non est. Sed eget magna a justo pellentesque volutpat vitae id ex. Sed ultricies risus sed vehicula egestas. Ut placerat tellus id vestibulum convallis. Vivamus vel metus vitae sapien fermentum mollis. Phasellus vehicula tortor eu nulla fermentum maximus.

Lorem ipsum dolor sit amet, consectetur adipiscing elit. Aliquam nulla sem, tristique eu nisi sagittis, sodales pulvinar est. Suspendisse purus diam, dapibus vitae suscipit nec, volutpat non est. Sed eget magna a justo pellentesque volutpat vitae id ex. Sed ultricies risus sed vehicula egestas. Ut placerat tellus id vestibulum convallis. Vivamus vel metus vitae sapien fermentum mollis. Phasellus vehicula tortor eu nulla fermentum maximus.


{{Unsubscribe}}

{{{Organization.Address}}}

Happy to contribute some performance tests as well if necessary.

[bobthecow]

I made the second change in mustache.php a while back and it made a huge difference: bobthecow/mustache.php@6215e6b

From my testing, it looks like the "tag change" logic does best when completely inlined, and it's not any harder to follow, so that's what I went with: bobthecow/mustache.php@72d0752