add ensure_binary/str/text helper functions #204

jz1371 · 2017-07-09T22:03:04Z

Summary
add ensure_binary, ensure_str, ensure_text helper functions to solve the encoding compatibility
between py2 and py3 more easily.
-- ensure_binary, input --> six.binary_type
-- ensure_str, input -> six.string_type
-- ensure_text, input -> six.text_type
This is mentioned in Lisa Guo's keynote in Pycon 2017 (from 31:21 in https://www.youtube.com/watch?v=66XoCk79kjM), which helped Instagram to upgrade to Python 3 and I hope this can help other developers, too
Test Plan
add new unit tests to cover added code

 195 passed, 2 skipped in 0.48 seconds

benjaminp

Thanks for the PR. A few things:

documentation/index.rst needs to be updated.
There need to be tests for the second and third parameters to each function.
The tests should actually check the type of what they're getting back from the functions under test, since converting between types is the main point of these functions.

benjaminp · 2017-07-26T04:45:11Z

six.py

+
+
+def ensure_str(s, encoding='utf-8', errors='strict'):
+    """ A helper function to ensure output is six.str_type.


six.str_type doesn't exist.

good catch, will update the doc to six.string_types and rename function to ensure_string

benjaminp · 2017-07-26T04:46:34Z

six.py

+        raise TypeError("not expecting type '%s'" % type(s))
+    if isinstance(s, text_type):
+        return s.encode(encoding, errors)
+    else:


This can be elif isinstance(s, binary_type): and then the first if statement in this function may be removed.

@benjaminp do you mean something like?

def ensure_binary(s, encoding='utf-8', errors='strict'): if isinstance(s, text_type): return s.encode(encoding, errors) elif isinstance(s, binary_type): return s else: raise TypeError("not expecting type '%s'" % type(s))

I'd like to keep the raise TypeError part to fail the error loudly; be consistent with the raise behavior from ensure_string.

benjaminp · 2017-07-26T04:48:46Z

six.py

+        raise TypeError("not expecting type '%s'" % type(s))
+    if isinstance(s, binary_type):
+        return s.decode(encoding, errors)
+    else:


The isinstance checks can be rewritten in a way similar to what I suggested in ensure_binary.

sure, will update to the following if we agree to keep raise exception part

if isinstance(s, binary_type): return s.decode(encoding, errors) elif isinstance(s, text_type): return s else: raise TypeError("not expecting type '%s'" % type(s))

jz1371 · 2017-08-06T19:37:21Z

Thanks for the review, @benjaminp

for doc, I plan to add that after function six.iterbytes(buf) in section https://pythonhosted.org/six/#binary-and-text-data, i.e.

please let me know if you have a better suggestion

sure, will add tests to cover arguments errors, encoding
and test for verifying the return type

- add checking for type in tests; - fix six.str_types typo; - simplify logic in ensure_;

benjaminp

Almost there. There still could be some more testing of errors and encoding. e.g., what happens when invalid utf-8 is given with error="strict|replace|igonre"?

benjaminp · 2017-08-27T20:44:05Z

documentation/index.rst

+.. function:: ensure_binary(s, encoding='utf-8', errors='strict')
+
+  A helper function to ensure output is :data:`binary_type`. ``encoding``, ``errors``
+  are the same as https://docs.python.org/3/library/stdtypes.html#str.encode


Write

:meth:`py3:str.encode`

here and below

benjaminp · 2017-08-27T20:45:06Z

test_six.py

@@ -903,3 +903,56 @@ def __bytes__(self):
        assert str(my_test) == six.u("hello")

    assert getattr(six.moves.builtins, 'bytes', str)(my_test) == six.b("hello")
+
+
+class EnsureTests(unittest.TestCase):


Don't need to inherit from TestCase here.

mariocj89 · 2017-10-12T18:35:31Z

Hi! @benjaminp, @jz1371 is it moving forward? @jz1371 if you have no time I am happy to finish it.

You can see there is a need for these functions in the community.

They even made it to a keynote at PyCon 🤣 https://youtu.be/66XoCk79kjM?t=31m15s

jz1371 · 2018-01-19T05:34:57Z

@benjaminp, any other changes you'd like to have in this PR?

cclauss · 2018-03-14T23:31:06Z

THIS IS SO AWESOME!! I was watching the Pycon 2017 Instagram video and thinking "This needs to be in six!" and it is already. Any sense on when this will be in a release?

mariocj89 · 2018-03-15T00:08:41Z

Indeed. I am looking forward to remove this from m codebase :)
@benjaminp do you plan to cut a release or is there any way we can help you do it?

python-autopxd already requires six, but the new `ensure_binary` function is only in the git repo and hasn't been released to PyPI yet, so I temporarily copied it here. The source comes from: benjaminp/six#204 It's a trivial function (every py2.py3 project has its own way to do the same thing). Fixes tarruda#12 Supersedes tarruda#3

anthrotype · 2018-03-25T00:17:12Z

It would be great if we could have this released to pypi some time soon, thanks!

ofek · 2018-10-26T03:59:16Z

@benjaminp Hello there! Any idea when this may be released?

JinmingZhao · 2020-03-31T14:07:57Z

six==1.14 will work with the error 'module 'six' has no attribute 'ensure_str''

cclauss · 2020-03-31T15:13:40Z

six==1.14.0 (with the .0) as in https://pypi.org/project/six/

https://github.com/benjaminp/six/releases

python3 -c "import six ; print(six.__version__)"

add ensure_binary/str/text helper functions

c6a4033

jz1371 force-pushed the ensure-helper branch from 5dab39a to c6a4033 Compare July 9, 2017 23:21

benjaminp reviewed Jul 26, 2017

View reviewed changes

- update doc;

429d7e2

- add checking for type in tests; - fix six.str_types typo; - simplify logic in ensure_;

benjaminp reviewed Aug 27, 2017

View reviewed changes

update doc; update tests

90e32b8

benjaminp merged commit db3d0d6 into benjaminp:master Jan 26, 2018

benjaminp added a commit that referenced this pull request Jan 26, 2018

changelog updates for #204

877dbe8

anthrotype mentioned this pull request Mar 24, 2018

[py2.py3] add function to coerce strings to six.binary_type tarruda/python-autopxd#16

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

add ensure_binary/str/text helper functions #204

add ensure_binary/str/text helper functions #204

jz1371 commented Jul 9, 2017

benjaminp left a comment

benjaminp Jul 26, 2017

jz1371 Aug 6, 2017

benjaminp Jul 26, 2017

jz1371 Aug 6, 2017

benjaminp Jul 26, 2017

jz1371 Aug 6, 2017

jz1371 commented Aug 6, 2017

benjaminp left a comment

benjaminp Aug 27, 2017

benjaminp Aug 27, 2017

mariocj89 commented Oct 12, 2017

jz1371 commented Jan 19, 2018

cclauss commented Mar 14, 2018

mariocj89 commented Mar 15, 2018

anthrotype commented Mar 25, 2018

ofek commented Oct 26, 2018

JinmingZhao commented Mar 31, 2020

cclauss commented Mar 31, 2020 •

edited



		def ensure_str(s, encoding='utf-8', errors='strict'):
		""" A helper function to ensure output is six.str_type.

add ensure_binary/str/text helper functions #204

add ensure_binary/str/text helper functions #204

Conversation

jz1371 commented Jul 9, 2017

benjaminp left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jz1371 commented Aug 6, 2017

benjaminp left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

mariocj89 commented Oct 12, 2017

jz1371 commented Jan 19, 2018

cclauss commented Mar 14, 2018

mariocj89 commented Mar 15, 2018

anthrotype commented Mar 25, 2018

ofek commented Oct 26, 2018

JinmingZhao commented Mar 31, 2020

cclauss commented Mar 31, 2020 • edited

cclauss commented Mar 31, 2020 •

edited