Allowed properties overloading for CoreNLPParser tag, Authors.md #2789
Conversation
|
For those interested in the reasoning behind the changes in this PR, feel free to have a look at #2112 itself, where all the changes are already described and laid out. |
| @@ -339,9 +339,11 @@ def tag_sents(self, sentences): | |||
| """ | |||
| # Converting list(list(str)) -> list(str) | |||
| sentences = (" ".join(words) for words in sentences) | |||
| return [sentences[0] for sentences in self.raw_tag_sents(sentences)] | |||
| if properties is None: | |||
There was a problem hiding this comment.
Please add a description for the properties argument in the docstring, also mention that "tokenize.whitespace" is set to True by default (but it won't if properties are specified.
would it be possible to make a test out of this example? |
Sure , bit busy, will add over weekend |
@tomaarsen @dimazest |
|
you should be able to manually use CoreNLP Rest API and intercept returned result |
|
Keep in mind that the CoreNLP tests are being skipped, because our GitHub Actions CI doesn't currently support downloading third party tools such as the Stanford models. Preferably we would still have tests for this PR though, as we likely want to support third party tools eventually. |
|
The actual call to corenlp should be mock, then tests won't need corenlp running. |
|
Since #2820 there are automated tests for CoreNLP. We would like to see some tests to back up these changes, is possible. |
| @@ -339,9 +339,11 @@ def tag_sents(self, sentences): | |||
| """ | |||
| # Converting list(list(str)) -> list(str) | |||
| sentences = (" ".join(words) for words in sentences) | |||
| return [sentences[0] for sentences in self.raw_tag_sents(sentences)] | |||
| if properties is None: | |||
| properties = {"tokenize.whitespace": "true"} | |||
There was a problem hiding this comment.
I've merged develop into this PR. Because the CoreNLP tests now run on the CI, we can see that some tests fail due to this change.
There was a problem hiding this comment.
@tomaarsen apart from automated tests for CoreNLP is there any other test that needs to be added
There was a problem hiding this comment.
The PR should have some tests that show that the new behaviour works like intended, so you would only need to add tests for the functionality that you added. So, just for these functions.
|
@BatMrE: just wondering if there's interest in wrapping this up. I think it's just a question of adding a test or two. |
This is a fix to #2112
Added the overloading properties in code as per the standards
before fix : [('my', 'O'), ('phone', 'O'), ('number', 'O'), ('is', 'O'), ('1111\xa01111\xa01111', 'O')]
after fix : [('my', 'O'), ('phone', 'O'), ('number', 'O'), ('is', 'O'), ('1111', 'DATE'), ('1111', 'DATE'), ('1111', 'DATE')]