async support v2 #417

999eagle · 2022-07-11T08:40:31Z

As discussed in #314, this is a new attempt to add support for async reading.

This PR is still WIP and open for discussion on implementation details.

Mingun · 2022-07-11T09:12:53Z

Actually, I'll plan to merge similar traits XmlRead and XmlSource into one and use approach with a special intermediate struct SliceReader / IoReader instead of directly implementing XmlSource for BufRead and &[u8]. With that approach it should be possible to have an AsyncSliceReader / AsyncIoReader (which would wrap only a inner reader) for which new traits will be implemented.

So it is good time for you to investigate that surface and how it can be integrated with async support

999eagle · 2022-07-11T17:27:44Z

Okay, I've moved a lot of stuff around to hopefully separate some things better. I've also implemented async support on the existing Reader. The following things have been changed as of now:

The Reader is now generic over the parser in use. This separates namespaced access and non-namespaced access through type checking at compile time.
To implement this, there is now a Parser-trait, implemented by a DefaultParser and a NamespacedParser.
The Reader is now built via a builder-pattern so no changes to the parser settings (like whether whitespace is trimmed or comments/end tags are checked) are possible after being built. This does change the public API and thus is a breaking change but allows the Parser separation and thus eliminates a lot of duplicate code.
There is now an AsyncXmlSource trait that's automatically implemented for everything that implements tokio's AsyncBufRead trait.
The Reader now has some read_..._async methods. Using the same names for sync and async methods sadly won't work as that would require to tell Rust that no type can ever implement both XmlSource and AsyncXmlSource (or BufRead and AsyncBufRead) at the same time which is (for now) impossible.
There is a new test_async test which is rather bare-bones as of now but I'll work on that.
Currently, tests with the serialize feature are broken because I haven't changed the deserialization to work with the new Reader yet.

Mingun · 2022-07-12T05:05:22Z

1852c70 is a pretty big change which is hard to review. Could you split this commit into several? At least I can suggest extract introducing builder pattern into a separate commit. Also, can you rebase your work and squash up all fixup changes as much as possible before the finishing work (for example, you've changed AsyncXmlSource trait in the commit where you are actually implement it. Actually, it would be better to introduce this trait in the same commit where you implement it).

Also, some changes seems to a little complicated to me. Actually I was thinking about something like (signatures approximate, just to illustrate the idea):

struct SliceReader<'i>(&'i [u8]);
struct IoReader<R: BufRead>(R);
struct AsyncIoReader<R: AsyncBufRead>(R);

/// The existing Reader struct
pub struct Reader<R> {
  reader: R,
  // ...
}

impl<R> Reader<R> {
  // common parsing code, setup code
}

impl<'i> Reader<SliceReader<'i>> {
  pub fn read_event(&mut self) { /* ... */ }
  pub fn read_event_async(&mut self) { /* ... */ }
}

impl<R: BufRead> Reader<IoReader<R>> {
  pub fn read_event_into(&mut self, buf: &mut Vec<u8>) { /* ... */ }
}

impl<R: AsyncBufRead> Reader<AsyncIoReader<R>> {
  pub fn read_event_into_async(&mut self, buf: &mut Vec<u8>) { /* ... */ }
}

////////////////////////////////////////////////////////////////////////////

/// The struct that I would introduce in my namespaces PR
pub struct NsReader<R> {
  reader: Reader<R>,
  // ...
}

impl<R> NsReader<R> {
  // common parsing code, setup code
}

impl<'i> NsReader<SliceReader<'i>> {
  pub fn read_event(&mut self) { /* ... */ }
  pub fn read_event_async(&mut self) { /* ... */ }
}

impl<R: BufRead> NsReader<IoReader<R>> {
  pub fn read_event_into(&mut self, buf: &mut Vec<u8>) { /* ... */ }
}

impl<R: AsyncBufRead> NsReader<AsyncIoReader<R>> {
  pub fn read_event_into_async(&mut self, buf: &mut Vec<u8>) { /* ... */ }
}

999eagle · 2022-07-13T09:10:34Z

Okay, so you'd prefer having the NsReader<R> wrap the "standard" Reader<R> and then implement the basic parsing functions like that. I think that would work too for async support. Your example code also greatly simplifies reading as the generic bound for the buffer in XmlSource doesn't exist anymore and the different wrapper structs allow for strictly separating sync/async access.
With that approach, I think it's better to abandon my current code entirely and start anew. I'll try again from the current master using your approach.

999eagle · 2022-07-13T15:17:01Z

@Mingun I think I've reimplemented this now the way you suggested and with minimal unrelated changes. The builder pattern can still be added again in a later PR. I also didn't separate the namespaced read methods in this PR but it should be relatively easy to add.

codecov-commenter · 2022-07-13T17:08:35Z

Codecov Report

Merging #417 (6b35382) into master (a6588c2) will increase coverage by 0.90%.
The diff coverage is 73.44%.

@@            Coverage Diff             @@
##           master     #417      +/-   ##
==========================================
+ Coverage   49.58%   50.48%   +0.90%     
==========================================
  Files          22       26       +4     
  Lines       13935    14429     +494     
==========================================
+ Hits         6909     7284     +375     
- Misses       7026     7145     +119

Flag	Coverage Δ
unittests	`50.48% <73.44%> (+0.90%)`	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files	Coverage Δ
benches/macrobenches.rs	`0.00% <0.00%> (ø)`
benches/microbenches.rs	`0.00% <0.00%> (ø)`
examples/custom_entities.rs	`0.00% <0.00%> (ø)`
examples/read_buffered.rs	`0.00% <0.00%> (ø)`
examples/read_texts.rs	`0.00% <0.00%> (ø)`
src/writer.rs	`49.01% <0.00%> (-0.28%)`	⬇️
src/lib.rs	`12.74% <16.66%> (+0.41%)`	⬆️
src/de/mod.rs	`77.38% <50.00%> (+0.09%)`	⬆️
src/reader/async_reader.rs	`65.66% <65.66%> (ø)`
src/reader/io_reader.rs	`77.89% <77.89%> (ø)`
... and 6 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update a6588c2...6b35382. Read the comment docs.

Mingun

This is a big change, so now I made only a quick overview. I'll plan to made a second review, probably after my remarks will be addressed.

Fist of all, this variant looks much more simpler and better that a previous attempt. For the maintenance purposes I prefer that each commit was as smaller as possible and focusing on the one thing. Your f9cdf27 commit actually does the much more that just splitting reader. I suggest to split it for a several commits:

the one that changes usage of buffered methods in tests and examples into borrowing ones
the one that introduces a helper macros which conditionally includes await and async keywords
the one that moves code to the separate files
the one that actually implements async reader. The adding tests commit then probably can be included in that commit
any other that you'll decide to made to keep the review burden low

Please ensure, that after each commit code in the buildable state by running commands that CI run and additionally cargo doc --all-features. If some tests will temporary fail, please include the list of failed tests to the commit message (just copy output of cargo test after "failures:")

Try to keep copies of code low. Use helper functions / traits / macros if needed. For example, the original read_bang_element probably contains a bug, because read are always >=1 but there is read == 0 check, I do not want to copy it over the whole project.

Also add an entry to the Changelog.md.

It also will be a great to run benchmarks for the async code and compare results with the sync code, so it will be a plus if you add these benchmarks.

@dralley, I also would to hear your opinion

Mingun · 2022-07-13T17:44:15Z

src/reader.rs

+
+#[cfg(feature = "async")]
+pub use self::async_reader::AsyncReader;
+pub use self::{io_reader::IoReader, slice_reader::SliceReader};


I prefer to not group imports

Suggested change

pub use self::{io_reader::IoReader, slice_reader::SliceReader};

pub use self::io_reader::IoReader;

pub use self::slice_reader::SliceReader;

Mingun · 2022-07-13T17:47:41Z

src/reader.rs

 /// loop {
-///     match reader.read_event_into(&mut buf) {
+///     match reader.read_event() {


I like the idea to use borrowing reading in tests and examples, but I also want to show the buffering variant somewhere, so the users know, how to use it.

Besides, this is a quite isolated change and I prefer to have it as a separate commit. This will make the history much more clean and allow to focus on really important things when doing commit-by-commit review

Agreed with this, the README does have an example, but it would be helpful for the documentation to have one readily available as well.

Mingun · 2022-07-13T17:52:37Z

src/reader.rs

                    use crate::errors::Error;
-                    use crate::reader::{BangType, XmlSource};
+                    use crate::reader::{BangType};


Suggested change

use crate::reader::{BangType};

use crate::reader::BangType;

and in other similar places

Mingun · 2022-07-13T18:02:51Z

src/reader/async_reader.rs

+use tokio::{
+    fs::File,
+    io::{self, AsyncBufRead, AsyncBufReadExt, AsyncRead, BufReader},
+};
+
+use crate::{
+    events::{BytesText, Event},
+    name::{QName, ResolveResult},
+    Error, Result,
+};


I prefer to not group imports

Suggested change

use tokio::{

fs::File,

io::{self, AsyncBufRead, AsyncBufReadExt, AsyncRead, BufReader},

};

use crate::{

events::{BytesText, Event},

name::{QName, ResolveResult},

Error, Result,

};

use tokio::fs::File;

use tokio::io::{self, AsyncBufRead, AsyncBufReadExt, AsyncRead, BufReader};

use crate::events::{BytesText, Event};

use crate::name::{QName, ResolveResult};

use crate::{Error, Result};

Also, could you add a brief documentation for a module?

Mingun · 2022-07-13T18:04:45Z

src/reader/async_reader.rs

+                let available = match self.fill_buf().await {
+                    Ok(n) if n.is_empty() => break,
+                    Ok(n) => n,
+                    Err(ref e) if e.kind() == io::ErrorKind::Interrupted => continue,


If this ref required?

Suggested change

Err(ref e) if e.kind() == io::ErrorKind::Interrupted => continue,

Err(e) if e.kind() == io::ErrorKind::Interrupted => continue,

Mingun · 2022-07-13T18:35:52Z

src/reader/io_reader.rs

+use std::{
+    fs::File,
+    io::{self, BufRead, BufReader, Read},
+    ops::{Deref, DerefMut},
+    path::Path,
+};
+
+use crate::{
+    events::{BytesText, Event},
+    name::{QName, ResolveResult},
+    Error, Result,
+};


I prefer to not group imports

Suggested change

use std::{

fs::File,

io::{self, BufRead, BufReader, Read},

ops::{Deref, DerefMut},

path::Path,

};

use crate::{

events::{BytesText, Event},

name::{QName, ResolveResult},

Error, Result,

};

use std::fs::File;

use std::io::{self, BufRead, BufReader, Read};

use std::ops::{Deref, DerefMut};

use std::path::Path;

use crate::events::{BytesText, Event};

use crate::name::{QName, ResolveResult};

use crate::{Error, Result};

Also, could you add a brief documentation for a module?

Mingun · 2022-07-13T18:39:03Z

src/reader/slice_reader.rs

+use crate::{
+    events::{BytesText, Event},
+    name::{QName, ResolveResult},
+    Error, Result,
+};


I prefer to not group imports

Suggested change

use crate::{

events::{BytesText, Event},

name::{QName, ResolveResult},

Error, Result,

};

use crate::events::{BytesText, Event};

use crate::name::{QName, ResolveResult};

use crate::{Error, Result};

Also, could you add a brief documentation for a module?

Mingun · 2022-07-13T18:42:45Z

tests/async_test.rs

+    let mut buf = Vec::new();
+    let mut count = 0;
+    loop {
+        match reader.read_event_into(&mut buf).await.unwrap() {


I prefer that function was read_event_into_async, because a reader usually will not see a reader type

Mingun · 2022-07-13T18:44:10Z

tests/async_test.rs

+#[tokio::test]
+async fn test_sample() {
+    let path = PathBuf::from(env!("CARGO_MANIFEST_DIR"));
+    let mut reader = Reader::from_file_async(path.join("tests/documents/sample_rss.xml"))


Please use include_bytes! or include_str! unless this test tests the from_file_async method.

Mingun · 2022-07-13T18:47:36Z

Cargo.toml

@@ -94,6 +97,9 @@ serialize = ["serde"]
 ## Enables support for recognizing all [HTML 5 entities](https://dev.w3.org/html5/html-author/charref)
 escape-html = []

+## Enable support for asynchronous reading


Could you add more details here? References to corresponding builder methods would be enough

999eagle · 2022-07-14T10:59:43Z

Okay, I think I have addressed all your concerns now. Feel free to mark your discussions as resolved where it applies and keep them open otherwise. I've added some docs, updated examples and split the commits as best as I could.

999eagle · 2022-07-15T11:15:17Z

Rebased onto current master to resolve conflicts.

dralley · 2022-07-16T21:27:18Z

@999eagle Apologies for all the breakages!

Thank you for splitting apart the commits - I actually suggest to go a step further and break off anything in this PR which is independent of the fundamental changes (e.g. maybe commits #1 and #4) into a separate PR that can be merged easily. Likewise if there are fundamental changes which aren't tied directly to the async-ones, we can possibly review and merge that separately.

The motivation is just to reduce the amount of the unmerged work and prevent merge conflicts. There are other fundamental changes being worked on simultaneously right now (see #158) and reconciling the massive patch sets together will not be a pleasant experience unless it is done in smaller pieces.

999eagle · 2022-07-17T10:35:05Z

Hi @dralley, while those two commits are somewhat independent of the other changes, they don't really do anything useful on their own. The first commit is nothing but a requirement for testing the changes of the third commit, which in turn requires an example for explicitly buffered access for files as most examples just read strings or byte slices and thus don't have buffered reading anymore.
In my opinion, it'd make sense to split this PR into two separate PRs at most: Splitting the Reader into the explicit SliceReader and IoReader (that is, commits 1 through 4 from this current PR) for the first PR and all the async-specific implementation in a second PR. If that's what you want to do, tell me and I'll open a new PR for the non-async changes.

dralley · 2022-07-17T15:26:00Z

@999eagle Yes, that makes perfect sense, and it would be helpful.

dralley · 2022-07-17T19:27:10Z

src/reader/slice_reader.rs

+    }
+}
+
+/// Private functions for a [`Reader`] based on an [`IoReader`].


Typo I believe?

Suggested change

/// Private functions for a [`Reader`] based on an [`IoReader`].

/// Private functions for a [`Reader`] based on a [`SliceReader`].

dralley · 2022-07-17T22:45:19Z

Almost done reviewing the first 4 commits (which as you said will be split into the new PR), and it is looking fine so far. I really appreciate all the work you've put in to get this cleaned up.

This also changes the test cases in the `reader::test::check` macro to allow for reader-specific tests.

…he input slice

999eagle · 2022-07-18T06:37:00Z

@dralley I've opened #425 for the first few changes. I've also rebased onto the current master. As this PR is now based on #425, it must be merged afterwards.

dralley · 2022-07-27T16:27:25Z

This is relevant: https://blog.rust-lang.org/inside-rust/2022/07/27/keyword-generics.html

Anything that comes of that initiative would be months or years down the road, but it might be worthwhile to provide feedback and examples early in the process.

999eagle mentioned this pull request Jul 11, 2022

Allow using tokio's AsyncBufRead [Rebased] #314

Closed

999eagle force-pushed the feature/async branch 6 times, most recently from ad259c9 to 208d801 Compare July 13, 2022 15:04

Mingun requested changes Jul 13, 2022

View reviewed changes

Mingun requested a review from dralley July 13, 2022 19:15

999eagle force-pushed the feature/async branch 2 times, most recently from 3c2b0a6 to 6b35382 Compare July 14, 2022 10:57

999eagle marked this pull request as ready for review July 14, 2022 12:29

999eagle force-pushed the feature/async branch 2 times, most recently from 9df6f20 to 2af8199 Compare July 15, 2022 11:14

999eagle changed the title ~~WIP: async support v2~~ async support v2 Jul 15, 2022

999eagle force-pushed the feature/async branch from 2af8199 to d0b908e Compare July 15, 2022 11:22

dralley mentioned this pull request Jul 16, 2022

Allow using tokio's AsyncBufRead #233

Closed

999eagle force-pushed the feature/async branch from d0b908e to b514275 Compare July 17, 2022 10:52

dralley reviewed Jul 17, 2022

View reviewed changes

999eagle added 4 commits July 18, 2022 08:22

Change the check! macro to more flexibly define buffers

0fa2a8b

Split reader into IoReader and SliceReader

01ff58d

This also changes the test cases in the `reader::test::check` macro to allow for reader-specific tests.

Remove buffered access for SliceReader as events always borrow from t…

b09495a

…he input slice

Add example for buffered access when reading from a file

51fd38f

999eagle mentioned this pull request Jul 18, 2022

Split Reader into SliceReader and BufferedReader #425

Closed

999eagle and others added 3 commits July 18, 2022 08:34

Change the check! macro to allow for async tests

7d49259

Implement async reader

c936535

Add test for async

a1c9a0d

999eagle force-pushed the feature/async branch from b514275 to a1c9a0d Compare July 18, 2022 06:35

Mingun mentioned this pull request Jul 31, 2022

Async support #450

Merged

Mingun closed this in #450 Aug 14, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

async support v2 #417

async support v2 #417

999eagle commented Jul 11, 2022

Mingun commented Jul 11, 2022

999eagle commented Jul 11, 2022

Mingun commented Jul 12, 2022

999eagle commented Jul 13, 2022

999eagle commented Jul 13, 2022

codecov-commenter commented Jul 13, 2022 •

edited

Mingun left a comment

Mingun Jul 13, 2022

Mingun Jul 13, 2022

dralley Jul 17, 2022

Mingun Jul 13, 2022

Mingun Jul 13, 2022

Mingun Jul 13, 2022

Mingun Jul 13, 2022

Mingun Jul 13, 2022

Mingun Jul 13, 2022

Mingun Jul 13, 2022

Mingun Jul 13, 2022

999eagle commented Jul 14, 2022

999eagle commented Jul 15, 2022

dralley commented Jul 16, 2022 •

edited

999eagle commented Jul 17, 2022

dralley commented Jul 17, 2022

dralley Jul 17, 2022 •

edited

dralley commented Jul 17, 2022

999eagle commented Jul 18, 2022

dralley commented Jul 27, 2022

	pub use self::{io_reader::IoReader, slice_reader::SliceReader};
	pub use self::io_reader::IoReader;
	pub use self::slice_reader::SliceReader;

	Err(ref e) if e.kind() == io::ErrorKind::Interrupted => continue,
	Err(e) if e.kind() == io::ErrorKind::Interrupted => continue,

	/// Private functions for a [`Reader`] based on an [`IoReader`].
	/// Private functions for a [`Reader`] based on a [`SliceReader`].

async support v2 #417

async support v2 #417

Conversation

999eagle commented Jul 11, 2022

Mingun commented Jul 11, 2022

999eagle commented Jul 11, 2022

Mingun commented Jul 12, 2022

999eagle commented Jul 13, 2022

999eagle commented Jul 13, 2022

codecov-commenter commented Jul 13, 2022 • edited

Codecov Report

Mingun left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

999eagle commented Jul 14, 2022

999eagle commented Jul 15, 2022

dralley commented Jul 16, 2022 • edited

999eagle commented Jul 17, 2022

dralley commented Jul 17, 2022

dralley Jul 17, 2022 • edited

Choose a reason for hiding this comment

dralley commented Jul 17, 2022

999eagle commented Jul 18, 2022

dralley commented Jul 27, 2022

codecov-commenter commented Jul 13, 2022 •

edited

dralley commented Jul 16, 2022 •

edited

dralley Jul 17, 2022 •

edited