V0.39 (#311)

* RubyVM visibility processing.

* RubyVM syntax error handling.

* RubyVM rescue processing.

* RubyVM alias_method processing.

* Node type fix.

* RubyVM constant rebasing.

* RubyVM include and extend processing.

* RubyVM processes private constants.

* RubyVM module_function processing.

* Check call type of visibility methods.

* Dirty assignment tests.

* RubyVM NodeChainer.

* BaseVariable#probe gets range from node.

* RubyVM string detection.

* RubyVM chains.

* RubyVM recipient node detection.

* RubyVM cursor, chain, and return node fixes.

* RubyVM block_pass links.

* RubyVM node chainer processes opcalls.

* RubyVM detects strings in unsynchronized sources.

* RubyVM chains and probes.

* Parser-independent test.

* Vary specs based on the parser.

* Environment template checks parser type.

* Resbody node checks for exception variable.

* Finding references includes local variables.

* Nil node handling in variable probes.

* Register lambda node processor.

* Temporarily disable exception variables.

* CommentRipper caches buffer lines array.

* RubyVM detects literal booleans.

* RubyVM detects rescued exception variables.

* TypeChecker handles RubyVM VCALL nodes.

* Skip legacy specs when using RubyVM.

* Off by one error.

* Off by one error.

* Fix module reference in legacy NodeMethods spec.

* Register DASGN_CURR node processor.

* Chain ATTRASGN nodes.

* Refactoring TypeChecker to use chains instead of nodes (WIP)

* Handle blank lines when stringifying comments.

* Minor refactoring.

* TypeChecker reports not enough arguments.

* Kernel method handling.

* Remove Chain::Call#head?

* SendNode handles mapped methods without arguments.

* Stale code.

* DRY node methods.

* CommentRipper ignores frozen_string_literal.

* CommentRipper handles invalid byte sequences.

* Formatting.

* Remove BLOCK from foldable node types.

* Predicate method specs.

* Add STR to foldable node types.

* Method pin checks for nil type return nodes.

* RubyVM AST changes in Ruby 2.7.0

* TypeChecker rules and checks (WIP).

* Typechecker#internal?

* TypeChecker requires type tags at strong level.

* Tag type checks.

* TypeChecker normal level specs.

* Typed and strict TypeChecker specs.

* Strict call validation specs.

* Ignore type inference for method macros.

* Argument type checks.

* Stale comments.

* Remove duplicate test.

* Normal type check ignores undefined argument types.

* Source method pins with empty bodies infer nil.

* Nil method inferences from clips.

* Refactored typecheck reporter.

* Multi-level parameter checks.

* Unresolved call handling.

* Typecheck kwargs in calls.

* Type checker validates duck type parameters.

* TypeChecker validates kwrestargs.

* TypeChecker treats optional hashes as kwrestargs.

* Remove stale TypeChecker code.

* TypeChecker qualifies param types in declaration context.

* Do not check inference on attributes.

* CLI executable uses new type checker.

* TypeChecker attribute spec.

* New method node stub.

* Documentation.

* TypeChecker optional parameter spec.

* Self handling in type checks.

* TypeChecker resolves self on declared and inferred types.

* Reduce normal type checking rules.

* Strong type checking rules.

* TypeChecker validates calls at strict or higher.

* Reduce strictness of typed level.

* Cached core pins.

* Serialize stdlib pins (WIP)

* Serialize gem pins.

* Stale code.

* Moved chainer calls.

* Log YardMap loads.

* RubyVM detects method body nodes.

* RubyVM call node detection (WIP).

* Method parameter types.

* Unused node methods.

* Refactoring node methods (WIP).

* Move location of methods defined in macros.

* Documentation.

* CoreDocs creates gems directory.

* CoreDocs.require_minimum always creates cache directories.

* RubyVM handles args in for constructs.

* Chains infer String from interpolated strings.

* Stale code.

* Return tags.

* Virtual new method spec.

* Infer undefined from methods with nil nodes.

* Rearranging type checker specs by level (WIP).

* Stale typecheck diagnostics test.

* Validate attribute writer arguments.

* Strong type checks inherit param tags.

* Simple matches in any_types_match?

* Fix argument chains.

* Return nodes include AND.

* Return AND nodes inside RETURN nodes.

* Type tags.

* Object#dup core fill deletes overloads.

* String#each_line core fill.

* Implicit rest and block parameter types.

* Typed level uses loose return tag matches.

* Return tag validation specs.

* Documentation.

* Type tag.

* Check variable assignments for external calls.

* Validate kwoptargs without arguments.

* Parameter traversal and display.

* Validate optional hash args as kwrestargs.

* Rearranging node methods.

* Legacy node chainer attaches nodes.

* Legacy node methods and specs.

* Add OPCALL to call nodes.

* KW_ARG node changed in Ruby 2.7.

* Stale code.

* Add asgn_code to kwoptargs.

* Prefer @return to @type in constant tags.

* All BaseMethod pins have node attributes.

* Constant links resolve nested constants.

* Constant link iterates over all pins for resolution.

* TypeChecker handles `@abstract` tags.

* Cursor recipient detection (WIP).

* Clip#signify edge cases (WIP).

* Cursor#recipient_node edge cases (WIP).

* Cursor#recipient_node handles unsynchronized sources with parentheses.

* Cursor#recipient_node handles synchronized sources with trailing commas.

* Redundant code and related specs.

* ApiMap#cursor_at raises FileNotFoundError.

* Cursor#recipient_node with unsynchronized nested symbols.

* Cursor#recipient_node nesting and fencepost issues.

* Host#diagnose rescues FileNotFoundError.

* Refactored find_recipient_node.

* Legacy find_recipient_node (WIP).

* Legacy find_recipient_node conforms to specs.

* Numeric core fills.

* public_suffix dependency for Ruby < 2.3.

* Fuzzier inheritance checks.

* Unused method.

* RubyVM NodeChainer recognizes SUPER.

* Superclass namespace detection.

* Documentation and minor refactoring.

* Fix super/zsuper bug in legacy parser.

* Refactored uri queue synchronization. (#289)

`Solargraph::LanguageServer::Host::Sources` currently utilizes a queue
of uris that is polled every `0.25s` to see if there's anything there.

This commit refactors that to use a `ConditionVariable` that
automatically blocks until a uri is available. Two new methods -
`add_uri` and `next_uri` wrap this logic to avoid manual waiting and
signaling.

* Constant resolution finds nearest names (#287)

* Unused method Cursor#first_char_offset.

* Minor typecheck fixes.

* Arity checks.

* Integer#+ override.

* Numeric#+ mapping and checking.

* Verify child node in convert_hash

* Broaden rest argument check.

* Validate optional argument arity.

* Exception for restargs in core methods.

* BundlerMethods documentation.

* BaseMethod#explicit? attribute.

* TypeChecker does not check arity of non-explicit methods.

* Core == method hack.

* Arity checks accept valid block passes.

* Arity checks verify extra block passes.

* Detect return nodes in case statements (#283)

* Update argument array nodes for Ruby 2.7.

* Chains can be splats.

* Verify arity of calls with splats.

* Spec for return nodes from case statements.

* Handle ARGSCAT in chains and typechecker.

* Handle BLOCK_PASS with other arguments.

* Add Errno classes to core pins.

* Add @return text to documentaiton.

* Unused convention method.

* Infer literal DOT3 ranges.

* TypeChecker allows undeclared block args.

* Kernel#puts returns nil.

* Shell cache handling.

* Legacy case node handling.

* Legacy splat handling.

* Plugins configuration.

* Rescue specific errors in gemspec evaluation.

* Full parameters in signature help.

* feat: treat prepend module as include module (#302)

* Set prepended module ancestry.

* Legacy SendNode typo.

* and -> &&

* Array#uniq CoreFill.

* Validate constants.

* Legacy const_nodes_from method.

* Update YARD dependency.

* Rescue ENOENT errors when loading sources (#308)

Co-authored-by: SolaWing <316786359@qq.com>

lib/solargraph.rb

@@ -38,6 +38,7 @@ class BundleNotFoundError <     StandardError; end
   autoload :Environ,          'solargraph/environ'
   autoload :Convention,       'solargraph/convention'
   autoload :Documentor,       'solargraph/documentor'
+  autoload :Parser,           'solargraph/parser'
 
   dir = File.dirname(__FILE__)
   YARDOC_PATH = File.realpath(File.join(dir, '..', 'yardoc'))

lib/solargraph/api_map.rb

@@ -149,7 +149,7 @@ def local_path_hash
     # @return [Source::Cursor]
     def cursor_at filename, position
       position = Position.normalize(position)
-      raise "File not found: #{filename}" unless source_map_hash.has_key?(filename)
+      raise FileNotFoundError, "File not found: #{filename}" unless source_map_hash.has_key?(filename)
       source_map_hash[filename].cursor_at(position)
     end
 
@@ -308,6 +308,7 @@ def get_methods fqns, scope: :instance, visibility: [:public], deep: true
         result.concat inner_get_methods('Kernel', :instance, visibility, deep, skip)
       else
         result.concat inner_get_methods(fqns, scope, visibility, deep, skip)
+        result.concat inner_get_methods('Kernel', :instance, [:public], deep, skip) if visibility.include?(:private)
       end
       resolved = resolve_method_aliases(result, visibility)
       cache.set_methods(fqns, scope, visibility, deep, resolved)
@@ -560,6 +561,12 @@ def inner_get_methods fqns, scope, visibility, deep, skip, no_core = false
       return [] if skip.include?(reqstr)
       skip.add reqstr
       result = []
+      if deep && scope == :instance
+        store.get_prepends(fqns).reverse.each do |im|
+          fqim = qualify(im, fqns)
+          result.concat inner_get_methods(fqim, scope, visibility, deep, skip, true) unless fqim.nil?
+        end
+      end
       result.concat store.get_methods(fqns, scope: scope, visibility: visibility).sort{ |a, b| a.name <=> b.name }
       if deep
         if scope == :instance
@@ -601,7 +608,11 @@ def inner_get_methods fqns, scope, visibility, deep, skip, no_core = false
     def inner_get_constants fqns, visibility, skip
       return [] if fqns.nil? || skip.include?(fqns)
       skip.add fqns
-      result = store.get_constants(fqns, visibility)
+      result = []
+      store.get_prepends(fqns).each do |is|
+        result.concat inner_get_constants(qualify(is, fqns), [:public], skip)
+      end
+      result.concat store.get_constants(fqns, visibility)
                     .sort { |a, b| a.name <=> b.name }
       store.get_includes(fqns).each do |is|
         result.concat inner_get_constants(qualify(is, fqns), [:public], skip)
@@ -727,7 +738,7 @@ def resolve_method_alias pin
         comments: origin.comments,
         scope: origin.scope,
         visibility: origin.visibility,
-        args: origin.parameters
+        parameters: origin.parameters
       )
     end
   end

lib/solargraph/api_map/bundler_methods.rb

@@ -6,6 +6,8 @@ class ApiMap
     module BundlerMethods
       module_function
 
+      # @param directory [String]
+      # @return [Hash]
       def require_from_bundle directory
         @require_from_bundle ||= begin
           Solargraph.logger.info "Loading gems for bundler/require"
@@ -16,6 +18,7 @@ def require_from_bundle directory
         end
       end
 
+      # @return [void]
       def reset_require_from_bundle
         @require_from_bundle = nil
       end

lib/solargraph/api_map/source_to_yard.rb

@@ -60,12 +60,7 @@ def rake_yard store
           code_object_map[pin.path].docstring = pin.docstring
           code_object_map[pin.path].visibility = pin.visibility || :public
           code_object_map[pin.path].parameters = pin.parameters.map do |p|
-            n = p.match(/^[a-z0-9_]*:?/i)[0]
-            v = nil
-            if p.length > n.length
-              v = p[n.length..-1].gsub(/^ = /, '')
-            end
-            [n, v]
+            [p.name, p.asgn_code]
           end
         end
       end

lib/solargraph/api_map/store.rb

@@ -19,7 +19,7 @@ def initialize pins = []
       # @return [Array<Solargraph::Pin::Base>]
       def get_constants fqns, visibility = [:public]
         namespace_children(fqns).select { |pin|
-          !pin.name.empty? and (pin.is_a?(Pin::Namespace) || pin.is_a?(Pin::Constant)) && visibility.include?(pin.visibility)
+          !pin.name.empty? && (pin.is_a?(Pin::Namespace) || pin.is_a?(Pin::Constant)) && visibility.include?(pin.visibility)
         }
       end
 
@@ -34,7 +34,7 @@ def get_methods fqns, scope: :instance, visibility: [:public]
       end
 
       # @param fqns [String]
-      # @return [String]
+      # @return [String, nil]
       def get_superclass fqns
         return superclass_references[fqns].first if superclass_references.key?(fqns)
         return 'Object' if fqns != 'BasicObject' && namespace_exists?(fqns)
@@ -48,6 +48,12 @@ def get_includes fqns
         include_references[fqns] || []
       end
 
+      # @param fqns [String]
+      # @return [Array<String>]
+      def get_prepends fqns
+        prepend_references[fqns] || []
+      end
+
       # @param fqns [String]
       # @return [Array<String>]
       def get_extends fqns
@@ -172,6 +178,10 @@ def include_references
         @include_references ||= {}
       end
 
+      def prepend_references
+        @prepend_references ||= {}
+      end
+
       def extend_references
         @extend_references ||= {}
       end
@@ -210,13 +220,16 @@ def index
         pins.each do |pin|
           namespace_map[pin.namespace] ||= []
           namespace_map[pin.namespace].push pin
-          namespaces.add pin.path if pin.is_a?(Pin::Namespace) and !pin.path.empty?
+          namespaces.add pin.path if pin.is_a?(Pin::Namespace) && !pin.path.empty?
           namespace_pins.push pin if pin.is_a?(Pin::Namespace)
           method_pins.push pin if pin.is_a?(Pin::BaseMethod)
           symbols.push pin if pin.is_a?(Pin::Symbol)
           if pin.is_a?(Pin::Reference::Include)
             include_references[pin.namespace] ||= []
             include_references[pin.namespace].push pin.name
+          elsif pin.is_a?(Pin::Reference::Prepend)
+            prepend_references[pin.namespace] ||= []
+            prepend_references[pin.namespace].push pin.name
           elsif pin.is_a?(Pin::Reference::Extend)
             extend_references[pin.namespace] ||= []
             extend_references[pin.namespace].push pin.name
@@ -245,6 +258,9 @@ def index
             pin.docstring.add_tag(tag)
           end
         end
+        # @todo This is probably not the best place for these overrides
+        superclass_references['Integer'] = ['Numeric']
+        superclass_references['Float'] = ['Numeric']
       end
     end
   end

lib/solargraph/complex_type/type_methods.rb

@@ -113,13 +113,13 @@ def qualify api_map, context = ''
           t.qualify api_map, context
         end
         if list_parameters?
-          Solargraph::ComplexType.parse("#{fqns}<#{rtypes.map(&:tag).join(', ')}>").first
+          Solargraph::ComplexType.parse("#{fqns}<#{rtypes.map(&:tag).join(', ')}>")
         elsif fixed_parameters?
-          Solargraph::ComplexType.parse("#{fqns}(#{rtypes.map(&:tag).join(', ')})").first
+          Solargraph::ComplexType.parse("#{fqns}(#{rtypes.map(&:tag).join(', ')})")
         elsif hash_parameters?
-          Solargraph::ComplexType.parse("#{fqns}{#{ltypes.map(&:tag).join(', ')} => #{rtypes.map(&:tag).join(', ')}}").first
+          Solargraph::ComplexType.parse("#{fqns}{#{ltypes.map(&:tag).join(', ')} => #{rtypes.map(&:tag).join(', ')}}")
         else
-          Solargraph::ComplexType.parse(fqns).first
+          Solargraph::ComplexType.parse(fqns)
         end
       end
     end

lib/solargraph/convention/base.rb

@@ -13,11 +13,6 @@ def match? source
         false
       end
 
-      # @return [Environ]
-      def process
-        match? ? EMPTY_ENVIRON : environ
-      end
-
       # The Environ for this convention.
       # Subclasses should override this method.
       #

lib/solargraph/core_fills.rb

@@ -65,11 +65,13 @@ module CoreFills
   @return [self]
 @return_single_parameter
       )),
+      Override.method_return('Array#uniq', 'self'),
 
       Override.from_comment('BasicObject#==', %(
 @param other [BasicObject]
 @return [Boolean]
       )),
+      Override.method_return('BasicObject#initialize', 'void'),
 
       Override.method_return('Class#new', 'self'),
       Override.method_return('Class.new', 'Class<Object>'),
@@ -84,6 +86,11 @@ module CoreFills
       Override.method_return('File.extname', 'String'),
       Override.method_return('File.join', 'String'),
 
+      Override.from_comment('Float#+', %(
+@param y [Numeric]
+@return [Numeric]
+      )),
+
       Override.from_comment('Hash#[]', %(
 @return_value_parameter
       )),
@@ -94,9 +101,25 @@ module CoreFills
 @param_tuple
       )),
 
+      Override.from_comment('Integer#+', %(
+@param y [Numeric]
+@return [Numeric]
+      )),
+
+      Override.method_return('Kernel#puts', 'nil'),
+
+      # Override.method_return('Module#attr_reader', 'void'),
+      # Override.method_return('Module#attr_writer', 'void'),
+      # Override.method_return('Module#attr_accessor', 'void'),
+
+      Override.from_comment('Numeric#+', %(
+@param y [Numeric]
+@return [Numeric]
+      )),
+
       Override.method_return('Object#!', 'Boolean'),
       Override.method_return('Object#clone', 'self', delete: [:overload]),
-      Override.method_return('Object#dup', 'self'),
+      Override.method_return('Object#dup', 'self', delete: [:overload]),
       Override.method_return('Object#freeze', 'self', delete: [:overload]),
       Override.method_return('Object#inspect', 'String'),
       Override.method_return('Object#taint', 'self'),
@@ -121,7 +144,10 @@ module CoreFills
 
       Override.method_return('String#freeze', 'self'),
       Override.method_return('String#split', 'Array<String>'),
-      Override.method_return('String#lines', 'Array<String>')
+      Override.method_return('String#lines', 'Array<String>'),
+      Override.from_comment('String#each_line', %(
+@yieldparam [String]
+      ))
     ].concat(
       methods_with_yieldparam_subtypes.map do |path|
         Override.from_comment(path, %(

lib/solargraph/diagnostics/base.rb

@@ -5,6 +5,7 @@ module Diagnostics
     # The base class for diagnostics reporters.
     #
     class Base
+      # @return [Array<String>]
       attr_reader :args
 
       def initialize *args

lib/solargraph/diagnostics/type_check.rb

@@ -8,19 +8,19 @@ module Diagnostics
     class TypeCheck < Base
       def diagnose source, api_map
         return [] unless args.include?('always') || api_map.workspaced?(source.filename)
-        severity = (args.include?('strict') ? Diagnostics::Severities::ERROR : Diagnostics::Severities::WARNING)
-        checker = Solargraph::TypeChecker.new(source.filename, api_map: api_map)
-        result = checker.return_type_problems + checker.param_type_problems
-        result.concat checker.strict_type_problems if args.include?('strict')
-        result.sort! { |a, b| a.location.range.start.line <=> b.location.range.start.line }
-        result.map do |problem|
-          {
-            range: extract_first_line(problem.location, source),
-            severity: severity,
-            source: 'Typecheck',
-            message: problem.message
-          }
-        end
+        severity = Diagnostics::Severities::ERROR
+        level = (args.reverse.find { |a| ['normal', 'typed', 'strict', 'strong'].include?(a) }) || :normal
+        checker = Solargraph::TypeChecker.new(source.filename, api_map: api_map, level: level.to_sym)
+        checker.problems
+          .sort { |a, b| a.location.range.start.line <=> b.location.range.start.line }
+          .map do |problem|
+            {
+              range: extract_first_line(problem.location, source),
+              severity: severity,
+              source: 'Typecheck',
+              message: problem.message
+            }
+          end
       end
 
       private

lib/solargraph/language_server/host.rb

@@ -207,6 +207,15 @@ def diagnose uri
                 type: LanguageServer::MessageTypes::ERROR,
                 message: "Error in diagnostics: #{e.message}"
               }
+            rescue FileNotFoundError => e
+              # @todo This appears to happen when an external file is open and
+              #   scheduled for diagnosis, but the file was closed (i.e., the
+              #   editor moved to a different file) before diagnosis started
+              logger.warn "Unable to diagnose #{uri} : #{e.message}"
+              send_notification 'textDocument/publishDiagnostics', {
+                uri: uri,
+                diagnostics: []
+              }
             end
           else
             logger.info "Deferring diagnosis of #{uri}"
@@ -245,7 +254,7 @@ def queue message
       #
       # @return [String] The most recent data or an empty string.
       def flush
-        tmp = nil
+        tmp = ''
         @buffer_semaphore.synchronize do
           tmp = @buffer.clone
           @buffer.clear
@@ -539,7 +548,7 @@ def search query
       end
 
       # @param query [String]
-      # @return [String]
+      # @return [Array]
       def document query
         result = []
         libraries.each { |lib| result.concat lib.document(query) }

lib/solargraph/language_server/host/sources.rb

@@ -14,6 +14,7 @@ class Sources
         def initialize
           @mutex = Mutex.new
           @stopped = true
+          @has_uri = ConditionVariable.new
         end
 
         def stopped?
@@ -25,17 +26,14 @@ def start
           return unless @stopped
           @stopped = false
           Thread.new do
-            until stopped?
-              tick
-              sleep 0.25 if queue.empty?
-            end
+            tick until stopped?
           end
         end
 
         # @return [void]
         def tick
-          return if queue.empty?
-          uri = mutex.synchronize { queue.shift }
+          uri = mutex.synchronize { next_uri }
+
           return if queue.include?(uri)
           mutex.synchronize do
             nxt = open_source_hash[uri].finish_synchronize
@@ -45,6 +43,18 @@ def tick
           end
         end
 
+        # @return [void]
+        def add_uri(uri)
+          queue.push(uri)
+          @has_uri.signal
+        end
+
+        # @return [String]
+        def next_uri
+          @has_uri.wait(mutex) if queue.empty?
+          queue.shift
+        end
+
         # @return [void]
         def stop
           @stopped = true
@@ -88,7 +98,7 @@ def async_update uri, updater
           src = find(uri)
           mutex.synchronize do
             open_source_hash[uri] = src.start_synchronize(updater)
-            queue.push uri
+            add_uri(uri)
           end
           changed
           notify_observers uri
@@ -126,7 +136,7 @@ def clear
 
         private
 
-        # @return [Array<Source>]
+        # @return [Hash]
         def open_source_hash
           @open_source_hash ||= {}
         end