Merge pull request #1035 from google/google_sync

Google sync

docs/developers/abstract_values.md

@@ -12,7 +12,7 @@ freshness: { owner: 'mdemello' reviewed: '2021-08-09' }
       * [Matching](#matching)
       * [Construction](#construction)
 
-<!-- Added by: rechen, at: 2021-10-05T20:10-07:00 -->
+<!-- Added by: rechen, at: 2021-10-19T16:13-07:00 -->
 
 <!--te-->
 
@@ -244,7 +244,7 @@ are then converted to abstract values as needed.
 NOTE: Conversely, [`pytype.output`][stub-generation] converts abstract values
 into PyTD nodes.
 
-[abstract_utils]: https://github.com/google/pytype/blob/master/pytype/abstract_utils.py
+[abstract_utils]: https://github.com/google/pytype/blob/master/pytype/abstract/abstract_utils.py
 
 [matcher]: https://github.com/google/pytype/blob/master/pytype/matcher.py
 

docs/developers/annotations.md

@@ -12,7 +12,7 @@
          * [Conversion to abstract types](#conversion-to-abstract-types)
       * [Tracking local operations](#tracking-local-operations)
 
-<!-- Added by: rechen, at: 2021-10-05T20:10-07:00 -->
+<!-- Added by: rechen, at: 2021-10-19T16:13-07:00 -->
 
 <!--te-->
 

docs/developers/attributes.md

@@ -12,7 +12,7 @@
          * [get_special_attribute](#get_special_attribute)
          * [valself](#valself)
 
-<!-- Added by: rechen, at: 2021-10-05T20:10-07:00 -->
+<!-- Added by: rechen, at: 2021-10-19T16:13-07:00 -->
 
 <!--te-->
 

docs/developers/bytecode.md

@@ -10,7 +10,7 @@ freshness: { owner: 'mdemello' reviewed: '2021-08-09' }
       * [Host and Target Versions](#host-and-target-versions)
       * [Opcodes](#opcodes)
 
-<!-- Added by: rechen, at: 2021-10-05T20:10-07:00 -->
+<!-- Added by: rechen, at: 2021-10-19T16:13-07:00 -->
 
 <!--te-->
 

docs/developers/config.md

@@ -16,7 +16,7 @@ freshness: { owner: 'mdemello' reviewed: '2020-12-04' }
       * [Adding a new option](#adding-a-new-option)
       * [Config files](#config-files)
 
-<!-- Added by: rechen, at: 2021-10-05T20:10-07:00 -->
+<!-- Added by: rechen, at: 2021-10-19T16:13-07:00 -->
 
 <!--te-->
 

docs/developers/directives.md

@@ -9,7 +9,7 @@ freshness: { owner: 'mdemello' reviewed: '2021-08-09' }
       * [Overview](#overview)
       * [Director](#director)
 
-<!-- Added by: rechen, at: 2021-10-05T20:10-07:00 -->
+<!-- Added by: rechen, at: 2021-10-19T16:13-07:00 -->
 
 <!--te-->
 

docs/developers/features.md

@@ -18,7 +18,7 @@
          * [PrintVisitor](#printvisitor)
       * [Partial support](#partial-support)
 
-<!-- Added by: rechen, at: 2021-10-05T20:10-07:00 -->
+<!-- Added by: rechen, at: 2021-10-19T16:13-07:00 -->
 
 <!--te-->
 

docs/developers/frames.md

@@ -14,7 +14,7 @@ freshness: { owner: 'mdemello' reviewed: '2021-08-16' }
          * [LOAD and STORE operations](#load-and-store-operations)
       * [The block stack](#the-block-stack)
 
-<!-- Added by: rechen, at: 2021-10-05T20:10-07:00 -->
+<!-- Added by: rechen, at: 2021-10-19T16:13-07:00 -->
 
 <!--te-->
 

docs/developers/index.md

@@ -12,7 +12,7 @@ pytype's core concepts and code layout, as well as tips on suggested workflow.
       * [Basic concepts](#basic-concepts)
       * [Updating the developer guide](#updating-the-developer-guide)
 
-<!-- Added by: rechen, at: 2021-10-05T20:10-07:00 -->
+<!-- Added by: rechen, at: 2021-10-19T16:13-07:00 -->
 
 <!--te-->
 

docs/developers/main_loop.md

@@ -8,7 +8,7 @@
       * [Processing a single opcode](#processing-a-single-opcode)
       * [Two-pass Analysis](#two-pass-analysis)
 
-<!-- Added by: rechen, at: 2021-10-05T20:10-07:00 -->
+<!-- Added by: rechen, at: 2021-10-19T16:13-07:00 -->
 
 <!--te-->
 

docs/developers/overlays.md

@@ -9,7 +9,7 @@
       * [Mechanics](#mechanics)
       * [Adding an overlay](#adding-an-overlay)
 
-<!-- Added by: rechen, at: 2021-10-05T20:10-07:00 -->
+<!-- Added by: rechen, at: 2021-10-19T16:13-07:00 -->
 
 <!--te-->
 

docs/developers/process.md

@@ -10,7 +10,7 @@
       * [Debugging](#debugging)
       * [Profiling](#profiling)
 
-<!-- Added by: rechen, at: 2021-10-05T20:10-07:00 -->
+<!-- Added by: rechen, at: 2021-10-19T16:13-07:00 -->
 
 <!--te-->
 

docs/developers/special_builtins.md

@@ -15,7 +15,7 @@ freshness: { owner: 'mdemello' reviewed: '2020-09-18' }
          * [Instances](#instances)
       * [Variables and data](#variables-and-data)
 
-<!-- Added by: rechen, at: 2021-09-22T20:26-07:00 -->
+<!-- Added by: rechen, at: 2021-10-19T16:13-07:00 -->
 
 <!--te-->
 
@@ -78,15 +78,16 @@ and method accesses.
 
 ## Invoking
 
-`VirtualMachine.__init__` defines a mapping `self.special_builtins` from the
-names of python builtins to instances of the corresponding special builtins.
-Each special builtin then provides its own implementation of `call`, and when
-the VM encounters a call to e.g. `next(arg)` it delegates to
+`Context.__init__` defines a mapping `self.special_builtins` from the names of
+python builtins to instances of the corresponding special builtins. Each special
+builtin then provides its own implementation of `call`, and when the VM
+encounters a call to e.g. `next(arg)` it delegates to
 `special_builtins.Next().call(arg)`.
 
 The VM loads python builtins by calling `load_builtin()` in `byte_LOAD_NAME`.
 `load_builtin()` in turn calls `load_special_builtin()` to check if the name we
-are loading is defined in the `self.special_builtins` mapping mentioned above.
+are loading is defined in the `self.ctx.special_builtins` mapping mentioned
+above.
 
 ## Implementation details
 
@@ -105,7 +106,7 @@ calls
 self.get_underlying_method(node, arg, "__abs__")
 ```
 
-and then just reinvokes the regular `vm.call_function()` but now calling the
+and then just reinvokes the regular `ctx.vm.call_function()` but now calling the
 bound method `x.__abs__` rather than the built in function `abs` (this is a good
 example of something that is fairly straightforward but nevertheless impossible
 to do via type signatures).
@@ -136,7 +137,7 @@ class PropertyInstance(mixin.HasSlots, ...):
     self.set_slot("__get__", self.fget_slot)
 
   def fget_slot(self, ...):
-    return self.vm.call_function(self.fget, ...)
+    return self.ctx.vm.call_function(self.fget, ...)
 ```
 
 NOTE: Slots are implemented via the `get_special_attribute` method in the
@@ -201,7 +202,7 @@ Look for the pattern
 
 ```
 def call(*args):
-  result = self.vm.program.NewVariable()
+  result = self.ctx.program.NewVariable()
 
   # unpack data from args
   ...

docs/developers/tools.md

@@ -14,7 +14,7 @@ freshness: { owner: 'mdemello' reviewed: '2020-11-06' }
       * [xref](#xref)
       * [Utilities](#utilities)
 
-<!-- Added by: rechen, at: 2021-08-10T21:18-07:00 -->
+<!-- Added by: rechen, at: 2021-10-19T16:13-07:00 -->
 
 <!--te-->
 

docs/developers/type_stubs.md

@@ -13,7 +13,7 @@
          * [AST simplification](#ast-simplification)
          * [Pickling](#pickling)
 
-<!-- Added by: rechen, at: 2021-08-10T21:18-07:00 -->
+<!-- Added by: rechen, at: 2021-10-19T16:13-07:00 -->
 
 <!--te-->
 

docs/developers/typegraph.md

@@ -22,7 +22,7 @@ freshness: { owner: 'tsudol' reviewed: '2020-11-20' }
          * [A More Complex Example](#a-more-complex-example)
          * [Shortcircuiting and the solver cache](#shortcircuiting-and-the-solver-cache)
 
-<!-- Added by: rechen, at: 2021-08-10T21:18-07:00 -->
+<!-- Added by: rechen, at: 2021-10-19T16:13-07:00 -->
 
 <!--te-->
 

docs/errors.md

@@ -65,7 +65,7 @@ See [Silencing Errors][silencing-errors] for a more detailed example.
       * [wrong-arg-types](#wrong-arg-types)
       * [wrong-keyword-args](#wrong-keyword-args)
 
-<!-- Added by: tsudol, at: 2021-09-01T15:22-07:00 -->
+<!-- Added by: rechen, at: 2021-10-19T16:13-07:00 -->
 
 <!--te-->
 

docs/faq.md

@@ -19,7 +19,7 @@
       * [How can I automatically generate type annotations for an existing codebase?](#how-can-i-automatically-generate-type-annotations-for-an-existing-codebase)
       * [How do I annotate *args and <code>**kwargs</code>?](#how-do-i-annotate-args-and-kwargs)
 
-<!-- Added by: rechen, at: 2021-09-13T11:20-07:00 -->
+<!-- Added by: rechen, at: 2021-10-19T16:13-07:00 -->
 
 <!--te-->
 

docs/support.md

@@ -13,7 +13,7 @@ of pytype.
          * [Typing](#typing)
          * [Third-Party Libraries](#third-party-libraries)
 
-<!-- Added by: rechen, at: 2021-10-11T13:17-07:00 -->
+<!-- Added by: rechen, at: 2021-10-19T16:14-07:00 -->
 
 <!--te-->
 
@@ -73,7 +73,7 @@ Feature                                                         | Version | Supp
 [PEP 613 -- Explicit Type Aliases][613]                         | 3.11+   | ❌       | [#787][typealias]
 [PEP 647 -- User-Defined Type Guards][647]                      | 3.10    | ❌       | [#916][type-guards]
 Custom Recursive Types                                          | 3.6     | ❌       | [#407][recursive-types]
-Generic Type Aliases                                            | 3.6     | ✅       | Requires `--preserve-union-macros` flag
+Generic Type Aliases                                            | 3.6     | ✅       |
 Type Annotation Inheritance                                     | 3.6     | ❌       | [#81][annotation-inheritance]
 
 ### Third-Party Libraries

docs/typing_faq.md

@@ -11,7 +11,7 @@ freshness: { owner: 'mdemello' reviewed: '2021-07-27' }
       * [Why is Optional[Optional[T]] the same as Optional[T]?](#why-is-optionaloptionalt-the-same-as-optionalt)
       * [Why is pytype not more like $other_language?](#why-is-pytype-not-more-like-other_language)
 
-<!-- Added by: rechen, at: 2021-09-13T11:19-07:00 -->
+<!-- Added by: rechen, at: 2021-10-19T16:14-07:00 -->
 
 <!--te-->
 

docs/user_guide.md

@@ -10,7 +10,7 @@
       * [Pyi stub files](#pyi-stub-files)
       * [Pytype's pyi stub files](#pytypes-pyi-stub-files)
 
-<!-- Added by: rechen, at: 2021-08-10T21:18-07:00 -->
+<!-- Added by: rechen, at: 2021-10-19T16:14-07:00 -->
 
 <!--te-->
 

pytype/abstract/abstract.py

@@ -2840,8 +2840,6 @@ def _convert_member(self, member, subst=None):
   def _new_instance(self, args):  # pylint: disable=arguments-renamed
     if self.full_name == "builtins.tuple" and args.is_empty():
       value = Tuple((), self.ctx)
-    elif self.full_name == "builtins.dict" and args.is_empty():
-      value = Dict(self.ctx)
     else:
       value = Instance(
           self.ctx.convert.constant_to_value(self.pytd_cls), self.ctx)

pytype/abstract/function.py

@@ -346,13 +346,17 @@ def __new__(cls, posargs, namedargs=None, starargs=None, starstarargs=None):
         starargs=starargs,
         starstarargs=starstarargs)
 
-  def is_empty(self):
-    if self.posargs or self.starargs or self.starstarargs:
-      return False
+  def has_namedargs(self):
     if isinstance(self.namedargs, dict):
-      return not self.namedargs
+      return bool(self.namedargs)
     else:
-      return not self.namedargs.pyval
+      return bool(self.namedargs.pyval)
+
+  def has_non_namedargs(self):
+    return bool(self.posargs or self.starargs or self.starstarargs)
+
+  def is_empty(self):
+    return not (self.has_namedargs() or self.has_non_namedargs())
 
   def starargs_as_tuple(self, node, ctx):
     try:
@@ -533,6 +537,19 @@ def get_variables(self):
       variables.append(self.starstarargs)
     return variables
 
+  def replace_posarg(self, pos, val):
+    new_posargs = self.posargs[:pos] + (val,) + self.posargs[pos + 1:]
+    return self._replace(posargs=new_posargs)
+
+  def replace_namedarg(self, name, val):
+    new_namedargs = dict(self.namedargs)
+    new_namedargs[name] = val
+    return self._replace(namedargs=new_namedargs)
+
+  def delete_namedarg(self, name):
+    new_namedargs = {k: v for k, v in self.namedargs.items() if k != name}
+    return self._replace(namedargs=new_namedargs)
+
 
 class ReturnValueMixin:
   """Mixin for exceptions that hold a return node and variable."""
@@ -1068,3 +1085,63 @@ def call_function(ctx,
     assert error
     error.set_return(node, result)
     raise error  # pylint: disable=raising-bad-type
+
+
+def match_all_args(ctx, node, func, args):
+  """Call match_args multiple times to find all type errors.
+
+  Args:
+    ctx: The abstract context.
+    node: The current CFG node.
+    func: An abstract function
+    args: An Args object to match against func
+
+  Returns:
+    A tuple of (new_args, errors)
+      where new_args = args with all incorrectly typed values set to Any
+            errors = a list of [(type mismatch error, arg name, value)]
+
+  Reraises any error that is not function.InvalidParameters
+  """
+  positional_names = func.get_positional_names()
+  needs_checking = True
+  errors = []
+  while needs_checking:
+    try:
+      func.match_args(node, args)
+    except FailedFunctionCall as e:
+      if isinstance(e, WrongKeywordArgs):
+        errors.append((e, e.extra_keywords[0], None))
+        for i in e.extra_keywords:
+          args = args.delete_namedarg(i)
+      elif isinstance(e, DuplicateKeyword):
+        errors.append((e, e.duplicate, None))
+        args = args.delete_namedarg(e.duplicate)
+      elif isinstance(e, MissingParameter):
+        errors.append((e, e.missing_parameter, None))
+        args = args.replace_namedarg(
+            e.missing_parameter, ctx.new_unsolvable(node))
+      elif isinstance(e, WrongArgTypes):
+        arg_name = e.bad_call.bad_param.name
+        for name, value in e.bad_call.passed_args:
+          if name != arg_name:
+            continue
+          errors.append((e, name, value))
+          try:
+            pos = positional_names.index(name)
+          except ValueError:
+            args = args.replace_namedarg(name, ctx.new_unsolvable(node))
+          else:
+            args = args.replace_posarg(pos, ctx.new_unsolvable(node))
+          break
+        else:
+          raise AssertionError(
+              "Mismatched parameter %s not found in passed_args" %
+              arg_name) from e
+      else:
+        # This is not an InvalidParameters error.
+        raise
+    else:
+      needs_checking = False
+
+  return args, errors

pytype/config.py

@@ -159,10 +159,6 @@ def add_basic_options(o):
       dest="bind_properties", default=False,
       help=("Bind @property methods to the classes they're defined on for more "
             "precise type-checking. " + temporary))
-  o.add_argument(
-      "--preserve-union-macros", action="store_true",
-      dest="preserve_union_macros", default=True,
-      help="Preserve Union generic type aliases in pyi files. " + temporary)
   o.add_argument(
       "--use-enum-overlay", action="store_true",
       dest="use_enum_overlay", default=False,
@@ -269,10 +265,6 @@ def add_infrastructure_options(o):
       "-Z", "--quick", action="store_true",
       dest="quick", default=None,
       help=("Only do an approximation."))
-  o.add_argument(
-      "--chex-overlay", action="store_true",
-      dest="chex_overlay", default=False,
-      help="Temporary flag to aid the release of an overlay for chex.dataclass")
 
 
 def add_debug_options(o):