Hidden Resource attributes are shown in scan results when running scancode on Python 3.7

[JonoYang]

After getting the latest version of scancode dependencies that work with Python 3.7, I was able to run scancode toolkit. However, there is an issue with the results where all the attributes of a Resource is shown in the scan results.

  "files": [
    {
      "path": "README.rst",
      "type": "file",
      "name": "README.rst",
      "base_name": "README",
      "extension": ".rst",
      "size": 9754,
      "location": "/home/jonathanyang/scancode-toolkit-develop/README.rst",
      "rid": 0,
      "pid": null,
      "cache_location": null,
      "is_file": true,
      "is_filtered": false,
      "children_rids": [],
      "size_count": 0,
      "files_count": 0,
      "dirs_count": 0,
      "scan_errors": [],
      "scan_time": 0,
      "scan_timings": {
        "info": 0.10853147506713867,
        "licenses": 2.458117961883545,
        "copyrights": 0.24155974388122559,
        "packages": 0.005839824676513672
      },
      "extra_data": {},
      ...

The fields location, rid, pid, cache_location, is_file, is_filtered, children_rids, size_count, files_count, dirs_count, scan_errors, scan_time, scan_timings, and extra_data should not show up in the scan results.

[tdruez]

Same issue on my side with Python3.6

[pombredanne]

This happens when you install scancode and do not use the bundled dependencies. attrs==20.1.0 is the source of the issue (we were using a pinned attrs==19.3.0 before
I cannot see in python-attrs/attrs@19.3.0...20.1.0 something obvious that would cause this to change and @hynek changelog does not list a backward incompatible change there either

The place where this likely comes from is here

scancode-toolkit/src/scancode/resource.py

Line 1276 in 31b9d58

def to_dict(self, with_timing=False, with_info=False, skinny=False):

We use an attr exclude filter to

scancode-toolkit/src/scancode/resource.py

Line 1293 in 31b9d58

self_fields_filter = attr.filters.exclude(*attr.fields(Resource))

before we invoke asdict and this seems to be broken.
I will need to investigate more ASAP as this block the release at this stage.

[hynek]

Could this be related to python-attrs/attrs#684?

[pombredanne]

@hynek Thank you++ for the hint. Let me check that right away!

[pombredanne]

@hynek I confirm that python-attrs/attrs#684 solves the issue!
Thank you ++
There is nothing best than a bug fixed before it is even reported! 🙇

IMHO this warrants a new PyPI release ASAP as this is a breaking change without this fix.

[hynek]

Yes, I'll try to get out a bugfix release this week.

[pombredanne]

@hynek Thank you: you are awesome (I truly mean this)
In the meantime, we will test things out in more contexts.

[pombredanne]

I thought for a sec that I could get away with a temp use of an attrs2 https://github.com/nexB/attrs/pull/1 but I would also need to rename the packages as attrs is a dependency for other deps of ours too. So instead, I will apply a local workaround until there is a new release upstream

[hynek]

Just attrs!=20.1.0?

[pombredanne]

@hynek good point, but I had recently relaxed the version constraints in 30a7070 and there was a reason that I am trying to track down

[pombredanne]

@hynek Thank you! That's so much simpler 🤦

[pombredanne]

Fixed with #2209
Thank you all!