Revamp producers and consumers to be used as pointers #80

sonalmahajan15 · 2023-10-12T21:18:08Z

Currently, NilAway initializes and passes producers and consumers as struct objects. This PR revamps that design to make producers and consumers as pointers. For this, the PR makes the following changes:

change receivers of producers and consumers in produce_trigger.go and consume_trigger.go, respectively, to pointers
initialize producer and consumer annotations as pointers in struct initialization of ProduceTrigger and ConsumeTrigger
implement an equals method to compare producer/consumer
update full trigger merging logic to quadratic merge using equals

[includes changes from the PR stack: #86 , #87 , #153 ]
[closes #79 ]

codecov · 2023-10-12T21:21:07Z

Codecov Report

Attention: 133 lines in your changes are missing coverage. Please review.

Comparison is base (fe96ce9) 88.92% compared to head (b2995c6) 89.06%.

Files	Patch %	Lines
annotation/consume_trigger.go	88.46%	79 Missing ⚠️
annotation/produce_trigger.go	93.81%	18 Missing ⚠️
assertion/function/assertiontree/backprop.go	75.92%	9 Missing and 4 partials ⚠️
util/asthelper/asthelper.go	80.32%	7 Missing and 5 partials ⚠️
annotation/key.go	95.80%	6 Missing ⚠️
annotation/util.go	96.22%	2 Missing ⚠️
assertion/function/assertiontree/backprop_util.go	95.83%	2 Missing ⚠️
...tion/function/assertiontree/root_assertion_node.go	94.11%	1 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main      #80      +/-   ##
==========================================
+ Coverage   88.92%   89.06%   +0.14%     
==========================================
  Files          54       55       +1     
  Lines        8260     9156     +896     
==========================================
+ Hits         7345     8155     +810     
- Misses        758      836      +78     
- Partials      157      165       +8

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

yuxincs

Overall this LGTM, but I have a few nits and one change request for a type assertion typo along with requests for unit tests to catch such typos 😃

yuxincs · 2023-10-23T15:48:29Z

annotation/consume_trigger.go

+
+// equals returns true if the passed ConsumingAnnotationTrigger is equal to this one
+func (i *MapAccess) equals(other ConsumingAnnotationTrigger) bool {
+	if other, ok := other.(*PtrLoad); ok {


Suggested change

if other, ok := other.(*PtrLoad); ok {

if other, ok := other.(*MapAccess); ok {

Right?

I would suggest we add some unit tests to guard against these (we can definitely use reflections in test code since performance usually isn't a concern)

That is, we can use reflect to find all structs that satisfies the ConsumingAnnotationTrigger interface and then (1) create two structs for each type and call equals on them (which should return true), and (2) create two structs of different types and equals should return false. (3) for even better coverage, we can use reflection to assign different values to the fields and call equals, but I think having (1) and (2) is good enough and is relatively easy to implement.

Good catch! I agree that we should add unit tests for catching such inadvertent mistakes. Added unit tests for checking implementations of equals method of ConsumingAnnotationTrigger, ProducingAnnotationTrigger, and Key interface. The unit tests check for (1) and (2), since (3) would prove to be quite tedious to write the test cases.

Also, as discussed offline, reflection could not be used for this purpose. Hence, for ensuring that we have unit tests for all the structs implementing the given interface, I resorted to the following:

expected: extract all structs implementing the interface using types package

actual: extract all structs initialized in the test case

compare expected and actual to find and report missing structs, if any

yuxincs · 2023-10-23T16:20:55Z

annotation/consume_trigger.go

@@ -50,13 +50,16 @@ type ConsumingAnnotationTrigger interface {
 	UnderlyingSite() Key


(how do you add comments on collapsed content?)

This is actually meant for L34-L36: since we are kinda doing refactors already, could we remove the expired comments there? not a must, obviously 😃

yuxincs · 2023-10-23T16:23:40Z

annotation/consume_trigger.go

+func (*ConsumeTriggerTautology) CheckConsume(Map) bool {
 	return true
 }


minor nit: put this in a single line for consistency with the methods above?

yuxincs · 2023-10-23T16:33:28Z

annotation/consume_trigger.go

-	TriggerIfNonNil
-	AffiliationPair
+	*TriggerIfNonNil
+	*AffiliationPair


Do we really need to make AffiliationPair a pointer, i.e., do we really have the need to modify AffiliationPair?

I'm OK with making it a pointer, but I have a feeling that it is actually immutable.

That's a good point. It is actually OK if we don't make it a pointer. I just followed the same convention for consistency, but you make a good point about immutability. Removed pointer for AffiliationPair

yuxincs · 2023-10-23T16:37:29Z

annotation/consume_trigger.go

@@ -961,15 +1272,15 @@ type UseAsNonErrorRetDependentOnErrorRetNilabilityPrestring struct {
 func (u UseAsNonErrorRetDependentOnErrorRetNilabilityPrestring) String() string {
 	via := ""
 	if u.IsNamedReturn {
-		via = fmt.Sprintf(" via the named return value `%s`", u.RetName)
+		via = fmt.Sprintf(" via named return `%s`", u.RetName)


Small refactor included?

Totally onboarded with the change, but just want to make sure this isn't actually meant for another PR ;)

No, I just came across it while testing this PR, and hence included the error message refinement in this PR itself :)

yuxincs · 2023-10-23T16:38:02Z

annotation/consume_trigger.go

@@ -1074,8 +1395,8 @@ type ConsumeTrigger struct {
 }

 // Eq compares two ConsumeTrigger pointers for equality


Suggested change

// Eq compares two ConsumeTrigger pointers for equality

// equals compares two ConsumeTrigger pointers for equality

Side note: as we discussed, we should probably re-architect the trigger structs, but of course let's do that in future PRs

Done

+1 about re-architecting the trigger structs.

yuxincs · 2023-10-23T16:52:39Z

annotation/produce_trigger.go

 }

 // TriggerIfNilable is a general trigger indicating that the bad case occurs when a certain Annotation
 // key is nilable
 type TriggerIfNilable struct {
-	Ann Key
+	Ann        Key
+	NeedsGuard bool


Since we have exposed method to access it, should we un-export this field?

In principle, we should. However, it would require refactors where this field is being directly set (e.g., annotation/util.go#L189). I feel such refactors are out of scope for this PR. I'll create an issue to track this and refactor in follow-up PRs.

yuxincs · 2023-10-23T16:53:46Z

annotation/produce_trigger.go

 // SetNeedsGuard for a `TriggerIfNilable` is, by default, a noop, as guarding
 // applies mostly to deep reads, but this behavior is overriden
 // for `VariadicFuncParamDeep`s, which have the semantics of
 // deep reads despite consulting shallow annotations


We should update the comments here, right? ;)

(and a few places below)

Oh yes! :) Done

yuxincs · 2023-10-23T16:55:09Z

annotation/produce_trigger.go


 // ProduceTriggerTautology is used for trigger producers that will always result in nil
-type ProduceTriggerTautology struct{}
+type ProduceTriggerTautology struct {
+	NeedsGuard bool


Here as well, do we need to export this field?

(same as discussed above)

yuxincs · 2023-10-23T16:58:28Z

annotation/produce_trigger.go

-	TriggerIfNilable
-	AffiliationPair
+	*TriggerIfNilable
+	*AffiliationPair


same here: do we want to make this a ptr?

(as discussed above, removed the pointer)

yuxincs

Overall this LGTM, thanks!

I just have a few suggestions and a question, but no real blockers.

yuxincs · 2023-11-14T17:51:42Z

annotation/equals_test.go

+func getImplementedMethods(t *types.Named) []*types.Func {
+	visitedMethods := make(map[string]*types.Func) // helps in only storing the latest overridden implementation of a method
+	visitedStructs := make(map[*types.Struct]bool) // helps in avoiding infinite recursion if there is a cycle in the struct embedding
+	collectMethods(t, visitedMethods, visitedStructs)
+	return maps.Values(visitedMethods)
+}


Sorry it's been a while 😅 can I get a quick reminder on why the corresponding functions in types doesn't work here?

In this case, likely because we are parsing ASTs for files in a given package, it was not able to extract complete type information, particularly, methods of embedded fields.

yuxincs · 2023-11-27T15:34:07Z

go.mod

@@ -15,6 +15,7 @@ require (
 	github.com/davecgh/go-spew v1.1.1 // indirect
 	github.com/kr/text v0.2.0 // indirect
 	github.com/pmezard/go-difflib v1.0.0 // indirect
+	github.com/stretchr/objx v0.5.0 // indirect


Question: is this used anywhere? or does testify/mock requires it?

We don't use it, but testify/mock requires it

yuxincs · 2023-11-27T15:45:27Z

annotation/equals_test.go

+}
+
+// EqualsTestSuite defines the test suite for the `equals` method.
+type EqualsTestSuite struct {


I'm a bit tempted to move this to a new package internal/nilawaytest (internal prevents other people from depending on it, and nilawaytest follows the convention in Go that test utility packages are named pkgtest).

This might make this more general to be shared for checking "a particular method for all structs defined in a package"

Obviously this can be a separate PR :)

Agreed :) Created issue #172 to perform this refactoring in a follow-up PR.

yuxincs · 2023-11-27T15:45:59Z

annotation/equals_test.go

+// EqualsTestSuite defines the test suite for the `equals` method.
+type EqualsTestSuite struct {
+	suite.Suite
+	initStructs   []any


Can we make this a generic struct instead of using any (with a constraint on having equals)?

type interface equaler { equals() } type EqualsTestSuite[T equaler] struct { ... initStructs[T] ... }

Created issue #172 to perform this refactoring in a follow-up PR.

yuxincs · 2023-11-27T15:49:10Z

annotation/equals_test.go

+		switch t := initStruct.(type) {
+		case ConsumingAnnotationTrigger:
+			s.Truef(t.equals(t), msg, t)
+		case ProducingAnnotationTrigger:
+			s.Truef(t.equals(t), msg, t)
+		case Key:
+			s.Truef(t.equals(t), msg, t)
+		default:
+			s.Failf("unknown type", "unknown type `%T`", t)


If we define a generic struct, then here we probably won't need a type switch anymore (we can just call .equals due to our type constraint).

Created issue #172 to perform this refactoring in a follow-up PR.

yuxincs · 2023-11-27T15:49:26Z

annotation/equals_test.go

+				case ConsumingAnnotationTrigger:
+					if t2, ok := s2.(ConsumingAnnotationTrigger); ok {
+						s.Falsef(t1.equals(t2), msg, t1, t2)
+					}
+				case ProducingAnnotationTrigger:
+					if t2, ok := s2.(ProducingAnnotationTrigger); ok {
+						s.Falsef(t1.equals(t2), msg, t1, t2)
+					}
+				case Key:
+					if t2, ok := s2.(Key); ok {
+						s.Falsef(t1.equals(t2), msg, t1, t2)
+					}


Created issue #172 to perform this refactoring in a follow-up PR.

yuxincs · 2023-11-27T15:55:14Z

annotation/consume_trigger_test.go

+type ConsumingAnnotationTriggerTestSuite struct {
+	EqualsTestSuite
+}


Maybe we can just have a NewEqualsTestSuite that takes the interfaceName and initStructs parameters, instead of having to inherit EqualsTestSuite?

Inheritance is second-class in Go, so I would try to avoid it when we can (plus having a constructor makes the code here simpler) :)

No strong preferences, of course.

(If we do change it here, we should also update the tests for produce trigger and key.

Created issue #172 to perform this refactoring in a follow-up PR.

This PR modifies the consumers to add support for deep copy when the consumers are being copied in backpropagation. [depends on #80 ]

sonalmahajan15 requested review from lazaroclapp and yuxincs October 12, 2023 21:18

sonalmahajan15 changed the base branch from main to sonalmahajan15/delete-string October 12, 2023 21:27

sonalmahajan15 force-pushed the sonalmahajan15/revamp-producer-consumer branch 3 times, most recently from 79ec162 to 4cf1263 Compare October 13, 2023 22:52

Base automatically changed from sonalmahajan15/delete-string to main October 17, 2023 18:28

sonalmahajan15 force-pushed the sonalmahajan15/revamp-producer-consumer branch from bc95a95 to 9b0b5d3 Compare October 18, 2023 17:57

uber-go deleted a comment from ketkarameya Oct 18, 2023

sonalmahajan15 mentioned this pull request Oct 19, 2023

Add deep copy for consumers #86

Merged

sonalmahajan15 force-pushed the sonalmahajan15/revamp-producer-consumer branch from 9b0b5d3 to daae825 Compare October 20, 2023 17:14

yuxincs reviewed Oct 23, 2023

View reviewed changes

sonalmahajan15 force-pushed the sonalmahajan15/revamp-producer-consumer branch from 337055f to 019ced5 Compare November 6, 2023 03:54

yuxincs approved these changes Nov 27, 2023

View reviewed changes

yuxincs reviewed Nov 27, 2023

View reviewed changes

sonalmahajan15 force-pushed the sonalmahajan15/revamp-producer-consumer branch 3 times, most recently from 9534584 to c7b4d03 Compare November 30, 2023 20:02

sonalmahajan15 force-pushed the sonalmahajan15/revamp-producer-consumer branch from c7b4d03 to 35156bd Compare December 12, 2023 00:33

sonalmahajan15 added 7 commits January 17, 2024 13:12

change producers and consumers to pointer types; add equals

b05890c

fix condition

4122ea0

updated comment

389bc99

unit test for equals()

6d319c1

address code review comments

0befa61

rebase

74c9c89

add testify/mock dependency

6814686

sonalmahajan15 force-pushed the sonalmahajan15/revamp-producer-consumer branch from 35156bd to 6814686 Compare January 17, 2024 21:14

sonalmahajan15 mentioned this pull request Jan 17, 2024

Refactor tests in annotation package #172

Open

Add deep copy for consumers (#86)

b2995c6

This PR modifies the consumers to add support for deep copy when the consumers are being copied in backpropagation. [depends on #80 ]

sonalmahajan15 merged commit c0cacdd into main Jan 17, 2024
5 checks passed

sonalmahajan15 deleted the sonalmahajan15/revamp-producer-consumer branch January 17, 2024 21:41

	if other, ok := other.(*PtrLoad); ok {
	if other, ok := other.(*MapAccess); ok {

		@@ -50,13 +50,16 @@ type ConsumingAnnotationTrigger interface {
		UnderlyingSite() Key

		@@ -1074,8 +1395,8 @@ type ConsumeTrigger struct {
		}

		// Eq compares two ConsumeTrigger pointers for equality

	// Eq compares two ConsumeTrigger pointers for equality
	// equals compares two ConsumeTrigger pointers for equality

Revamp producers and consumers to be used as pointers #80

Revamp producers and consumers to be used as pointers #80

Conversation

sonalmahajan15 commented Oct 12, 2023 • edited

codecov bot commented Oct 12, 2023 • edited

Codecov Report

yuxincs left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

sonalmahajan15 Oct 30, 2023 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

sonalmahajan15 Oct 30, 2023 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

yuxincs left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

sonalmahajan15 Jan 17, 2024 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

sonalmahajan15 commented Oct 12, 2023 •

edited

codecov bot commented Oct 12, 2023 •

edited

sonalmahajan15 Oct 30, 2023 •

edited

sonalmahajan15 Oct 30, 2023 •

edited

sonalmahajan15 Jan 17, 2024 •

edited