remove tokenstream vec alloc

[PSeitz]

remove unnecessary intermediate vec

[codecov-commenter]

Codecov Report

Merging #1611 (2efebdb) into main (5f565e7) will increase coverage by 0.02%.
The diff coverage is 60.00%.

@@            Coverage Diff             @@
##             main    #1611      +/-   ##
==========================================
+ Coverage   93.78%   93.81%   +0.02%     
==========================================
  Files         252      252              
  Lines       46380    46414      +34     
==========================================
+ Hits        43499    43541      +42     
+ Misses       2881     2873       -8     

Impacted Files	Coverage Δ
src/indexer/segment_writer.rs	96.73% <60.00%> (+0.32%)	⬆️
src/fastfield/multivalued/mod.rs	98.45% <0.00%> (-0.78%)	⬇️
src/directory/mmap_directory.rs	90.40% <0.00%> (+0.53%)	⬆️
src/reader/mod.rs	94.26% <0.00%> (+0.63%)	⬆️
src/directory/error.rs	3.27% <0.00%> (+1.63%)	⬆️
src/schema/document.rs	94.77% <0.00%> (+1.96%)	⬆️
src/schema/value.rs	67.39% <0.00%> (+5.86%)	⬆️

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.