Adjusted Mutual Information #402

robertlayton · 2011-10-19T10:29:48Z

Mutual Information, adjusted for chance. See [1] for details (specifically, look at the references for detail)

I have tested this against the Matlab code, and it works! Took me a while, as I had log for entropy and log2 for Expected Mutual Information (should be the other way around). I think that the _expected_mutual_information can be optimised, but I went with "get it right" first.

[1] http://en.wikipedia.org/wiki/Adjusted_Mutual_Information

…later.

…d on this to come!)

…m in the morning.

ogrisel · 2011-10-19T11:48:06Z

Thanks for this contrib!
Can you please merge the current master to your branch to make it easier for reviewers to test your code?

ogrisel · 2011-10-19T11:52:11Z

sklearn/metrics/cluster/supervised.py

+
+
+def expected_mutual_information(contingency, n_samples):
+    """ Calculate the expected mutual information for two labellings. """


Typo (the U.S. spelling with a single "l" is more common) and PEP257 style:

"""Calculate the expected mutual information for two labelings."""

ogrisel · 2011-10-19T12:05:29Z

I have added new items to the TODO list.

Conflicts: sklearn/metrics/cluster/__init__.py

- See Also sections updated - mutual_information -> mutual_information_score (and updating subsequent imports)

…sted_for_chance_measures.py example Commiting to get help on error

robertlayton · 2011-10-21T00:01:06Z

There is an overflow problem, which happens when running examples/cluster/plot_adjusted_for_chance_measures.py. I'm not quite sure how to fix this. Any suggestions? My thinking is to rearrange the third term of the equation, or to skip when a[i]== 0 etc.
edit: I understand why the matlab code is how it is - reducing factorials etc. I just worked out the equations to reduce the number of factorials needed and I'll do that for now.

edit #2: I think I've fixed it. I need to test against the matlab code with larger arrays, but I have to do that on my other computer.

…come)

This fixes the tests (which needed a bit of updating)

robertlayton · 2011-10-21T09:49:52Z

I'm not familiar with the style of plots used in the plot_adjusted_for_chance_measures.py example. Can someone show me how to make the colours of the legend match with the actual plots? They are just showing up as a broken line on my machine.

robertlayton · 2011-10-21T10:14:38Z

Apart from the issue with plot_adjusted_for_chance_measures.py, the rest of this code is ready for a more thorough review.

ogrisel · 2011-10-21T22:50:51Z

Sorry for the late reply, I will try to have a look at the plot_adjusted_for_chance_measures stuff tomorrow.

ogrisel · 2011-10-22T22:52:47Z

Here are the pictures I get:

The legends are good. However the AMI score should always be zero (as the ARI) which is not the case. This might be caused by the following warnings I get when computing the AMI scores:

Warning: divide by zero encountered in log
Warning: invalid value encountered in multiply

Also it seems that the AMI scores are much slower to compute than ARI and V-Measure. Would be great to try and optimize the runtime once the incorrect value issues are fixed.

ogrisel · 2011-10-22T23:12:01Z

Also here is the outcome of the doctests:

nosetests -s --with-doctest --doctest-tests --doctest-extension=rst \
    --doctest-fixtures=_fixture doc/ doc/modules/
.Warning: divide by zero encountered in log
Warning: invalid value encountered in multiply
Warning: divide by zero encountered in log
Warning: invalid value encountered in multiply
Warning: divide by zero encountered in log
Warning: invalid value encountered in multiply
Warning: divide by zero encountered in log
Warning: invalid value encountered in multiply
Warning: divide by zero encountered in log
Warning: invalid value encountered in multiply
F..........
======================================================================
FAIL: Doctest: clustering.rst
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/usr/lib/python2.7/doctest.py", line 2166, in runTest
    raise self.failureException(self.format_failure(new.getvalue()))
AssertionError: Failed doctest test for clustering.rst
  File "/home/ogrisel/coding/scikit-learn/doc/modules/clustering.rst", line 0

----------------------------------------------------------------------
File "/home/ogrisel/coding/scikit-learn/doc/modules/clustering.rst", line 491, in clustering.rst
Failed example:
    metrics.ami_score(labels_true, labels_pred)  # doctest: +ELLIPSIS
Expected:
    0.24...
Got:
    0.22504228319830885
----------------------------------------------------------------------
File "/home/ogrisel/coding/scikit-learn/doc/modules/clustering.rst", line 498, in clustering.rst
Failed example:
    metrics.ami_score(labels_true, labels_pred)  # doctest: +ELLIPSIS
Expected:
    0.24...
Got:
    0.22504228319830885
----------------------------------------------------------------------
File "/home/ogrisel/coding/scikit-learn/doc/modules/clustering.rst", line 505, in clustering.rst
Failed example:
    metrics.ami_score(labels_pred, labels_true)  # doctest: +ELLIPSIS
Expected:
    0.24...
Got:
    0.22504228319830885
----------------------------------------------------------------------
File "/home/ogrisel/coding/scikit-learn/doc/modules/clustering.rst", line 518, in clustering.rst
Failed example:
    metrics.ami_score(labels_true, labels_pred)  # doctest: +ELLIPSIS
Expected:
    0.0...
Got:
    -0.10526315789473678

>>  raise self.failureException(self.format_failure(<StringIO.StringIO instance at 0x241bd40>.getvalue()))


----------------------------------------------------------------------
Ran 12 tests in 1.360s

…e example

robertlayton · 2011-11-08T02:41:31Z

Don't check this yet - I didn't commit one lot from home and need to do that first (it updated the examples, which don't work now!)

…to ami Conflicts: doc/whats_new.rst

…into ami

robertlayton · 2011-11-10T05:38:44Z

Ready to be checked. I added a note in the docstring of adjusted_mutual_info_score. If you have a better spot, let me know.

ogrisel · 2011-11-10T06:07:56Z

doc/whats_new.rst

@@ -19,7 +19,7 @@ Changelog
   - Faster tests by `Fabian Pedregosa`_.

   - Silhouette Coefficient cluster analysis evaluation metric added as
-     ``sklearn.metrics.silhouette_score`` by Robert Layton.
+     ``sklearn.metrics.silhouette_score`` by `Robert Layton`.


If you to make your name a link, you need to add a trailing _ to it.

ogrisel · 2011-11-10T06:20:10Z

Looks good. +1 for merge.

robertlayton · 2011-11-10T11:02:42Z

Merge when ready.

GaelVaroquaux · 2011-11-10T13:03:57Z

Merge when ready.

I am doing changes on this pull request right now :)

Adjusted Mutual Information: Mutual information adjusted for chance

GaelVaroquaux · 2011-11-10T22:00:47Z

I made a mistake sending my mail, and it didn't get to the pull request.
So I am resending. In brief, you merged code that is failing a lot of
tests on my box (I tried again after your merge) :(.

--Earlier mail--

I was about to merge, after integrating the changes that I made on my
'ami' branch, but I found that I get quite a few test failures.
@robertlayton, could you please merge my ami branch, and check if you can
reproduce the following failures:

nosetests -s --with-doctest --doctest-tests --doctest-extension=rst \
    --doctest-fixtures=_fixture doc/ doc/modules/
  DeprecationWarning)
.F......F...
======================================================================
FAIL: Doctest: clustering.rst
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/usr/lib/python2.6/doctest.py", line 2163, in runTest
    raise self.failureException(self.format_failure(new.getvalue()))
AssertionError: Failed doctest test for clustering.rst
  File "/home/varoquau/dev/scikit-learn/doc/modules/clustering.rst", line
0

----------------------------------------------------------------------
File "/home/varoquau/dev/scikit-learn/doc/modules/clustering.rst", line
493, in clustering.rst
Failed example:
    metrics.adjusted_mutual_info_score(labels_true, labels_pred)  #
doctest: +ELLIPSIS
Expected:
    0.24...
Got:
    0.22504228319830874
----------------------------------------------------------------------
File "/home/varoquau/dev/scikit-learn/doc/modules/clustering.rst", line
500, in clustering.rst
Failed example:
    metrics.adjusted_mutual_info_score(labels_true, labels_pred)  #
doctest: +ELLIPSIS
Expected:
    0.24...
Got:
    0.22504228319830874
----------------------------------------------------------------------
File "/home/varoquau/dev/scikit-learn/doc/modules/clustering.rst", line
507, in clustering.rst
Failed example:
    metrics.adjusted_mutual_info_score(labels_pred, labels_true)  #
doctest: +ELLIPSIS
Expected:
    0.24...
Got:
    0.22504228319830874
----------------------------------------------------------------------
File "/home/varoquau/dev/scikit-learn/doc/modules/clustering.rst", line
520, in clustering.rst
Failed example:
    metrics.adjusted_mutual_info_score(labels_true, labels_pred)  #
doctest: +ELLIPSIS
Expected:
    0.0...
Got:
Got:
    -0.10526315789473642

>>  raise self.failureException(self.format_failure(>  instance at 0x9c86b8c>.getvalue()))

robertlayton · 2011-11-10T22:03:08Z

Checking now

(I re-read your comment - I thought you meant "merge when ready, I'm already working off a branch of this"! Sorry about that).

GaelVaroquaux · 2011-11-10T22:05:25Z

Checking now

Thanks. I've merged the changes that I had made in my fork of your branch
in master.

robertlayton · 2011-11-10T22:12:11Z

Issue is confirmed, which is weird - it only happened after I merged the branches :S

I think -- this may be a problem with some of my libraries. I had an issue previously where doctests weren't showing up, despite everything else working. I'll fix that problem independently.

I have the matlab code for AMI, so I'll double check the doctests versus that code.

GaelVaroquaux · 2011-11-10T22:15:10Z

Issue is confirmed, which is weird - it only happened after I merged the branches :S

The good news is that you can reproduce it. :$. Maybe you can do a
bisection to find out what commit exactly is to blame.

robertlayton · 2011-11-10T22:31:14Z

After checking with the matlab code, these values are wrong, but the code is right!

Checking the build now. Do I start a new PR?

GaelVaroquaux · 2011-11-10T22:33:59Z

After checking with the matlab code, these values are wrong, but the
code is right!

So, you are saying that the tests are wrong?

Checking the build now. Do I start a new PR?

Don't start a new PR: this is bug fixing, and I don't think that it needs
a new PR. If you want, you can push a fix to your fork, and ask for
informal feedback, but if you have a fix that you a confident with, you
should push it to master. I don't like having failing tests in master: I
am worried about the broken window effect.

robertlayton · 2011-11-10T22:37:01Z

Yup, tests were wrong. I'll look back over the code to work out why I did that, but its more important just to get the fix in, so that the build is good (as you said).

The fix works (just the values are changed, and I took them straight from matlab). I'll push to master in a second (I'll double check it won't break anything else!).

GaelVaroquaux · 2011-11-10T22:38:50Z

Yup, tests were wrong.

Fair enough. Looks like we don't run the tests enough when reviewing pull
requests :)

GaelVaroquaux · 2011-11-10T22:54:41Z

Fair enough. Looks like we don't run the tests enough when reviewing pull
requests :)

If it gives you any comfort, I pretty much did the same thing in my pull
request, and forgot to rerun the tests after implementing a change. I was
about to pull to master, and routinely ran the tests... and found the bug
:)

robertlayton · 2011-11-10T23:29:53Z

That does give me comfort. Thanks for checking though -- broken tests are bad!

@ogrisel It was my understanding that the AMI is bounded between 0 and 1. However one of the examples in doctests gives a negative score -- and it does with the matlab code as well! Any thoughts?

ogrisel · 2011-11-10T23:57:53Z

Negative score can happen for "random-like" labelings because with are doing a diff with the expected value of random labelings. It is just never occurring in practice if your are evaluating a cluster algorithm that does its job of finding slightly above average clustering.

robertlayton · 2011-11-11T00:00:12Z

Fair enough. That had me a little worried (but it was what the matlab code had...)

robertlayton added 5 commits October 14, 2011 14:34

Trying to fix NaN errors, but its not working. Pushing to work on it …

2eb9833

…later.

Mutual information now works (tested!)

c3d3906

AMI now works, and has been tested against the matlab code (test base…

9f05c11

…d on this to come!)

Remove phantom double v-measure !?

e473f70

Added tests. There are two errors, but I'm going to bed. I'll fix the…

2d8677f

…m in the morning.

ogrisel reviewed Oct 19, 2011
View reviewed changes

robertlayton added 5 commits October 21, 2011 09:28

Merge branch 'master' into ami

af91582

Conflicts: sklearn/metrics/cluster/__init__.py

Merge branch 'ami' of github.com:robertlayton/scikit-learn into ami

770020f

- AMI in the cluster examples

193ac02

- See Also sections updated - mutual_information -> mutual_information_score (and updating subsequent imports)

Higher level import for ami_score

867ec2f

There is an overflow problem. It can be reproduced with the plot_adju…

18b9a78

…sted_for_chance_measures.py example Commiting to get help on error

robertlayton added 4 commits October 21, 2011 13:26

Narrative doc, and I think I fixed the overflow issue (more tests to …

aaf5c23

…come)

Fixed logs to match the matlab code results.

fb0c3f8

This fixes the tests (which needed a bit of updating)

Test now tests a much larger array

db845fa

Test actually does what I meant it to do, and works sufficiently

77ec530

robertlayton added 2 commits October 21, 2011 21:03

Fixed this example. Tested the others (they worked!)

52fc4c5

pep8 and pyflakes

c021b59

Merge remote-tracking branch 'robertlayton/ami' into robertlayton-ami

3052291

ogrisel added 2 commits October 23, 2011 10:43

measure runtimes for various clustering metrics in adjusted for chanc…

b7d7642

…e example

FIX warnings by avoiding 0.0 values in the log + cosmit

3845fd8

"What's new?" AMI!

66f7a0b

robertlayton added 4 commits November 10, 2011 15:31

Merge branch 'ami' of https://github.com/robertlayton/scikit-learn in…

b9685ff

…to ami Conflicts: doc/whats_new.rst

mutual_information_score -> mutual_info_score

f26ed76

and in plot_adjusted example (mutual_info_score)

17ee6c4

Merge branch 'master' of https://github.com/scikit-learn/scikit-learn …

ff0dbf9

…into ami

ogrisel reviewed Nov 10, 2011
View reviewed changes

cosmit

118e8bd

robertlayton added a commit that referenced this pull request Nov 10, 2011

Merge pull request #402 from robertlayton/ami

6263124

Adjusted Mutual Information: Mutual information adjusted for chance

robertlayton merged commit 6263124 into scikit-learn:master Nov 10, 2011

amueller mentioned this pull request Dec 14, 2017

NMI and AMI use inconsistent definitions of mutual information #10308

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Adjusted Mutual Information #402

Adjusted Mutual Information #402

robertlayton commented Oct 19, 2011

ogrisel commented Oct 19, 2011

ogrisel Oct 19, 2011

ogrisel commented Oct 19, 2011

robertlayton commented Oct 21, 2011

robertlayton commented Oct 21, 2011

robertlayton commented Oct 21, 2011

ogrisel commented Oct 21, 2011

ogrisel commented Oct 22, 2011

ogrisel commented Oct 22, 2011

robertlayton commented Nov 8, 2011

robertlayton commented Nov 10, 2011

ogrisel Nov 10, 2011

ogrisel commented Nov 10, 2011

robertlayton commented Nov 10, 2011

GaelVaroquaux commented Nov 10, 2011

GaelVaroquaux commented Nov 10, 2011

robertlayton commented Nov 10, 2011

GaelVaroquaux commented Nov 10, 2011

robertlayton commented Nov 10, 2011

GaelVaroquaux commented Nov 10, 2011

robertlayton commented Nov 10, 2011

GaelVaroquaux commented Nov 10, 2011

robertlayton commented Nov 10, 2011

GaelVaroquaux commented Nov 10, 2011

GaelVaroquaux commented Nov 10, 2011

robertlayton commented Nov 10, 2011

ogrisel commented Nov 10, 2011

robertlayton commented Nov 11, 2011



		def expected_mutual_information(contingency, n_samples):
		""" Calculate the expected mutual information for two labellings. """

Adjusted Mutual Information #402

Adjusted Mutual Information #402

Conversation

robertlayton commented Oct 19, 2011

ogrisel commented Oct 19, 2011

ogrisel Oct 19, 2011

Choose a reason for hiding this comment

ogrisel commented Oct 19, 2011

robertlayton commented Oct 21, 2011

robertlayton commented Oct 21, 2011

robertlayton commented Oct 21, 2011

ogrisel commented Oct 21, 2011

ogrisel commented Oct 22, 2011

ogrisel commented Oct 22, 2011

robertlayton commented Nov 8, 2011

robertlayton commented Nov 10, 2011

ogrisel Nov 10, 2011

Choose a reason for hiding this comment

ogrisel commented Nov 10, 2011

robertlayton commented Nov 10, 2011

GaelVaroquaux commented Nov 10, 2011

GaelVaroquaux commented Nov 10, 2011

robertlayton commented Nov 10, 2011

GaelVaroquaux commented Nov 10, 2011

robertlayton commented Nov 10, 2011

GaelVaroquaux commented Nov 10, 2011

robertlayton commented Nov 10, 2011

GaelVaroquaux commented Nov 10, 2011

robertlayton commented Nov 10, 2011

GaelVaroquaux commented Nov 10, 2011

GaelVaroquaux commented Nov 10, 2011

robertlayton commented Nov 10, 2011

ogrisel commented Nov 10, 2011

robertlayton commented Nov 11, 2011