pep-0237.txt

PEP: 237
Title: Unifying Long Integers and Integers
Version: $Revision$
Last-Modified: $Date$
Author: Moshe Zadka, Guido van Rossum
Status: Final
Type: Standards Track
Content-Type: text/x-rst
Created: 11-Mar-2001
Python-Version: 2.2
Post-History: 16-Mar-2001, 14-Aug-2001, 23-Aug-2001


Abstract
========

Python currently distinguishes between two kinds of integers (ints): regular
or short ints, limited by the size of a C long (typically 32 or 64 bits), and
long ints, which are limited only by available memory.  When operations on
short ints yield results that don't fit in a C long, they raise an error.
There are some other distinctions too.  This PEP proposes to do away with most
of the differences in semantics, unifying the two types from the perspective
of the Python user.


Rationale
=========

Many programs find a need to deal with larger numbers after the fact, and
changing the algorithms later is bothersome.  It can hinder performance in the
normal case, when all arithmetic is performed using long ints whether or not
they are needed.

Having the machine word size exposed to the language hinders portability.  For
examples Python source files and .pyc's are not portable between 32-bit and
64-bit machines because of this.

There is also the general desire to hide unnecessary details from the Python
user when they are irrelevant for most applications. An example is memory
allocation, which is explicit in C but automatic in Python, giving us the
convenience of unlimited sizes on strings, lists, etc.  It makes sense to
extend this convenience to numbers.

It will give new Python programmers (whether they are new to programming in
general or not) one less thing to learn before they can start using the
language.


Implementation
==============

Initially, two alternative implementations were proposed (one by each author):

1. The ``PyInt`` type's slot for a C long will be turned into a::

       union {
           long i;
           struct {
               unsigned long length;
               digit digits[1];
           } bignum;
       };

   Only the ``n-1`` lower bits of the ``long`` have any meaning; the top bit
   is always set.  This distinguishes the ``union``.  All ``PyInt`` functions
   will check this bit before deciding which types of operations to use.

2. The existing short and long int types remain, but operations return
   a long int instead of raising ``OverflowError`` when a result cannot be
   represented as a short int.  A new type, ``integer``, may be introduced
   that is an abstract base type of which both the ``int`` and ``long``
   implementation types are subclassed.  This is useful so that programs can
   check integer-ness with a single test::

       if isinstance(i, integer): ...

After some consideration, the second implementation plan was selected, since
it is far easier to implement, is backwards compatible at the C API level, and
in addition can be implemented partially as a transitional measure.


Incompatibilities
=================

The following operations have (usually subtly) different semantics for short
and for long integers, and one or the other will have to be changed somehow.
This is intended to be an exhaustive list. If you know of any other operation
that differ in outcome depending on whether a short or a long int with the same
value is passed, please write the second author.

- Currently, all arithmetic operators on short ints except ``<<`` raise
  ``OverflowError`` if the result cannot be represented as a short int.  This
  will be changed to return a long int instead. The following operators can
  currently raise ``OverflowError``: ``x+y``, ``x-y``, ``x*y``, ``x**y``,
  ``divmod(x, y)``, ``x/y``, ``x%y``, and ``-x``.  (The last four can only
  overflow when the value ``-sys.maxint-1`` is involved.)

- Currently, ``x<<n`` can lose bits for short ints.  This will be changed to
  return a long int containing all the shifted-out bits, if returning a short
  int would lose bits (where changing sign is considered a special case of
  losing bits).

- Currently, hex and oct literals for short ints may specify negative values;
  for example ``0xffffffff == -1`` on a 32-bit machine.  This will be changed
  to equal ``0xffffffffL`` (``2**32-1``).

- Currently, the ``%u``, ``%x``, ``%X`` and ``%o`` string formatting operators
  and the ``hex()`` and ``oct()`` built-in functions behave differently for
  negative numbers: negative short ints are formatted as unsigned C long,
  while negative long ints are formatted with a minus sign.  This will be
  changed to use the long int semantics in all cases (but without the trailing
  *L* that currently distinguishes the output of ``hex()`` and ``oct()`` for
  long ints).  Note that this means that ``%u`` becomes an alias for ``%d``.
  It will eventually be removed.

- Currently, ``repr()`` of a long int returns a string ending in *L* while
  ``repr()`` of a short int doesn't.  The *L* will be dropped; but not before
  Python 3.0.

- Currently, an operation with long operands will never return a short int.
  This *may* change, since it allows some optimization.  (No changes have been
  made in this area yet, and none are planned.)

- The expression ``type(x).__name__`` depends on whether *x* is a short or a
  long int.  Since implementation alternative 2 is chosen, this difference
  will remain.  (In Python 3.0, we *may* be able to deploy a trick to hide the
  difference, because it *is* annoying to reveal the difference to user code,
  and more so as the difference between the two types is less visible.)

- Long and short ints are handled different by the ``marshal`` module, and by
  the ``pickle`` and ``cPickle`` modules.  This difference will remain (at
  least until Python 3.0).

- Short ints with small values (typically between -1 and 99 inclusive) are
  *interned* -- whenever a result has such a value, an existing short int with
  the same value is returned.  This is not done for long ints with the same
  values.  This difference will remain.  (Since there is no guarantee of this
  interning, it is debatable whether this is a semantic difference -- but code
  may exist that uses ``is`` for comparisons of short ints and happens to work
  because of this interning.  Such code may fail if used with long ints.)


Literals
========

A trailing *L* at the end of an integer literal will stop having any
meaning, and will be eventually become illegal.  The compiler will choose the
appropriate type solely based on the value. (Until Python 3.0, it will force
the literal to be a long; but literals without a trailing *L* may also be
long, if they are not representable as short ints.)


Built-in Functions
==================

The function ``int()`` will return a short or a long int depending on the
argument value.  In Python 3.0, the function ``long()`` will call the function
``int()``; before then, it will continue to force the result to be a long int,
but otherwise work the same way as ``int()``. The built-in name ``long`` will
remain in the language to represent the long implementation type (unless it is
completely eradicated in Python 3.0), but using the ``int()`` function is
still recommended, since it will automatically return a long when needed.


C API
=====

The C API remains unchanged; C code will still need to be aware of the
difference between short and long ints.  (The Python 3.0 C API will probably
be completely incompatible.)

The ``PyArg_Parse*()`` APIs already accept long ints, as long as they are
within the range representable by C ints or longs, so that functions taking C
int or long argument won't have to worry about dealing with Python longs.


Transition
==========

There are three major phases to the transition:

1. Short int operations that currently raise ``OverflowError`` return a long
   int value instead.  This is the only change in this phase.  Literals will
   still distinguish between short and long ints.  The other semantic
   differences listed above (including the behavior of ``<<``) will remain.
   Because this phase only changes situations that currently raise
   ``OverflowError``, it is assumed that this won't break existing code.
   (Code that depends on this exception would have to be too convoluted to be
   concerned about it.)  For those concerned about extreme backwards
   compatibility, a command line option (or a call to the warnings module)
   will allow a warning or an error to be issued at this point, but this is
   off by default.

2. The remaining semantic differences are addressed.  In all cases the long
   int semantics will prevail.  Since this will introduce backwards
   incompatibilities which will break some old code, this phase may require a
   future statement and/or warnings, and a prolonged transition phase.  The
   trailing *L* will continue to be used for longs as input and by
   ``repr()``.

   A. Warnings are enabled about operations that will change their numeric
      outcome in stage 2B, in particular ``hex()`` and ``oct()``, ``%u``,
      ``%x``, ``%X`` and ``%o``, ``hex`` and ``oct`` literals in the
      (inclusive) range ``[sys.maxint+1, sys.maxint*2+1]``, and left shifts
      losing bits.
   B. The new semantic for these operations are implemented. Operations that
      give different results than before will *not* issue a warning.

3. The trailing *L* is dropped from ``repr()``, and made illegal on input.
   (If possible, the ``long`` type completely disappears.) The trailing *L*
   is also dropped from ``hex()`` and ``oct()``.

Phase 1 will be implemented in Python 2.2.

Phase 2 will be implemented gradually, with 2A in Python 2.3 and 2B in
Python 2.4.

Phase 3 will be implemented in Python 3.0 (at least two years after Python 2.4
is released).


OverflowWarning
===============

Here are the rules that guide warnings generated in situations that currently
raise ``OverflowError``.  This applies to transition phase 1.  Historical
note: despite that phase 1 was completed in Python 2.2, and phase 2A in Python
2.3, nobody noticed that OverflowWarning was still generated in Python 2.3.
It was finally disabled in Python 2.4.  The Python builtin
``OverflowWarning``, and the corresponding C API ``PyExc_OverflowWarning``,
are no longer generated or used in Python 2.4, but will remain for the
(unlikely) case of user code until Python 2.5.

- A new warning category is introduced, ``OverflowWarning``.  This is a
  built-in name.

- If an int result overflows, an ``OverflowWarning`` warning is issued, with a
  message argument indicating the operation, e.g. "integer addition".  This
  may or may not cause a warning message to be displayed on ``sys.stderr``, or
  may cause an exception to be raised, all under control of the ``-W`` command
  line and the warnings module.

- The ``OverflowWarning`` warning is ignored by default.

- The ``OverflowWarning`` warning can be controlled like all warnings, via the
  ``-W`` command line option or via the ``warnings.filterwarnings()`` call.
  For example::

      python -Wdefault::OverflowWarning

  cause the ``OverflowWarning`` to be displayed the first time it occurs at a
  particular source line, and::

      python -Werror::OverflowWarning

  cause the ``OverflowWarning`` to be turned into an exception whenever it
  happens.  The following code enables the warning from inside the program::

      import warnings
      warnings.filterwarnings("default", "", OverflowWarning)

  See the python ``man`` page for the ``-W`` option and the ``warnings``
  module documentation for ``filterwarnings()``.

- If the ``OverflowWarning`` warning is turned into an error,
  ``OverflowError`` is substituted.  This is needed for backwards
  compatibility.

- Unless the warning is turned into an exceptions, the result of the operation
  (e.g., ``x+y``) is recomputed after converting the arguments to long ints.


Example
=======

If you pass a long int to a C function or built-in operation that takes an
integer, it will be treated the same as a short int as long as the value fits
(by virtue of how ``PyArg_ParseTuple()`` is implemented).  If the long value
doesn't fit, it will still raise an ``OverflowError``.  For example::

    def fact(n):
        if n <= 1:
        return 1
    return n*fact(n-1)

    A = "ABCDEFGHIJKLMNOPQ"
    n = input("Gimme an int: ")
    print A[fact(n)%17]

For ``n >= 13``, this currently raises ``OverflowError`` (unless the user
enters a trailing *L* as part of their input), even though the calculated
index would always be in ``range(17)``.  With the new approach this code will
do the right thing: the index will be calculated as a long int, but its value
will be in range.


Resolved Issues
===============

These issues, previously open, have been resolved.

- ``hex()`` and ``oct()`` applied to longs will continue to produce a trailing
  *L* until Python 3000.  The original text above wasn't clear about this,
  but since it didn't happen in Python 2.4 it was thought better to leave it
  alone.  BDFL pronouncement here:

  https://mail.python.org/pipermail/python-dev/2006-June/065918.html

- What to do about ``sys.maxint``?  Leave it in, since it is still relevant
  whenever the distinction between short and long ints is still relevant (e.g.
  when inspecting the type of a value).

- Should we remove ``%u`` completely?  Remove it.

- Should we warn about ``<<`` not truncating integers?  Yes.

- Should the overflow warning be on a portable maximum size?  No.


Implementation
==============

The implementation work for the Python 2.x line is completed; phase 1 was
released with Python 2.2, phase 2A with Python 2.3, and phase 2B will be
released with Python 2.4 (and is already in CVS).


Copyright
=========

This document has been placed in the public domain.



..
  Local Variables:
  mode: indented-text
  indent-tabs-mode: nil
  End: