Fix serving files that clash with directories

Jekyll extends WEBrick to serve file.html if file is requested but
doesn't exist (see jekyll#3452). This doesn't work if file exists as a
directory though.

This commit extends WEBrick further so that if file exists as a
directory, we first try and serve file/index.html (as normal), but
fallback to file.html if no index can be found within the file
directory (reported as jekyll#6222).

It is slightly hacky, to avoid having to re-implement the entire
set_filename method, we instead extend search_index_file and temporarily
mutate res.filename. This is all clearly commented.

See base class search_index_file implementation:

https://github.com/ruby/webrick/blob/3fb6a011/lib/webrick/httpservlet/filehandler.rb#L356-L363

lib/jekyll/commands/serve/servlet.rb

@@ -21,10 +21,43 @@ def initialize(server, root, callbacks)
         # up with a different preference on which comes first.
 
         def search_file(req, res, basename)
-          # /file.* > /file/index.html > /file.html
+          # /file.* -> /file.html
           super || super(req, res, "#{basename}.html")
         end
 
+        # rubocop:disable Lint/AssignmentInCondition
+        def search_index_file(req, res)
+          # /file/index.html -> /file.html
+
+          # First, let's see if the default implementation can figure it out.
+          # (Check for index files in the res.filename directory)
+          if file = super
+            return file
+          end
+
+          # Ok, I guess that didn't work, I guess there's no basename/index.html
+          # Let's look for basename.html instead...
+
+          # We need to extract the final part of the path
+          path_arr = res.filename.scan(%r!/[^/]*!)
+          while basename = path_arr.pop
+            break unless basename == "/"
+          end
+
+          # We need to change res.filename to the parent directory for
+          #  search_file to work, so make a backup incase it doesn't work
+          old_filename = res.filename
+          res.filename = path_arr.join
+
+          # Try and find a file named dirname.html in the parent directory
+          unless file = search_file(req, res, basename + ".html")
+            # Don't modify filename unless we actually found a file to serve
+            res.filename = old_filename
+          end
+          return file
+        end
+        # rubocop:enable Lint/AssignmentInCondition
+
         # rubocop:disable Style/MethodName
         def do_GET(req, res)
           rtn = super