itxn_next to give us inner groups

Questions remain around how TxID and GroupID should be defined for inners.

… branch. (#3017)

## Summary

 Disable the participation key refresh on the TestPartitionHalfOffline test. The test was failing with the following threads running:

```
algod(20739) : github.com/algorand/go-algorand/agreement.(*asyncPseudonode).loadRoundParticipationKeys(0xc0033dad20, 0x2, 0xc0039c8180, 0xc000530000, 0xc000dcb0d0)
```
```
algod(20739) : github.com/algorand/go-algorand/data/account.PersistedParticipation.DeleteOldKeys(0x92c6b21bbd8fefac, 0xed2d99ce769768ea, 0xac85a3b9466cead1, 0xc32937d7e4c532f9, 0xc000043260, 0xc0002b04b0, 0x0, 0x2dc6c0, 0x2710, 0xc0001892c0, ...)
```
```
algod(20739) : github.com/algorand/go-algorand/data.(*AccountManager).AddParticipation(0xc0001191a0, 0xd175b5d042fa1c39, 0x2140fe15c6a19653, 0xc66f7b9c58cf5ed1, 0x107d657815b44d49, 0xc0037a4d20, 0xc0002b0690, 0x0, 0x2dc6c0, 0x2710, ...)
```

By avoiding the participation key reload ( which isn't needed for this test ), I'm hoping to avoid the issue.

## Test Plan

This is a test.

## Summary

Resolves #2738

Adds the following:

- consensus.go: Adds the default MaxExpiredAccountsToProcess value for
  the new consensus protocol
- block.go: Adds necessary block header entries
- eval.go: Scans for expired accounts, modifies them to be offline and adds
  validation for this use case
- eval_test.go: Basic unit tests
- participationExpiration_test.go: Added e2e tests that verify that
  different consensus protocols behave differently

## Test Plan

Unit test and e2e tests added

## Summary

rebase Tolik's PR on master; this PR add the ability for the indexer to preload account data so that it would be used by the evaluator.

## Test Plan

Existing unit tests updated.

## Summary

* Moved code from **acctupdates.go** to **trackerdb.go**.
* Extracted  db creation and init from`accountsInitialize` to `trackerDBInitialize`
* `reloadLedger` now calls `trackerDBInitialize` right after block db init
* `reloadLedger` also has post-creation actions like vacuuming

Limitations:
1. Tables creation still calls `accountsInit` for initial tables creation and inserting genesis accounts.
2. Migration code still have some business logic. 

## Test Plan

Use existing test suite

…cleci nightly testing. (#3016)

For CircleCI testing, increase machine size from medium to large and set parallelism to 4.

In testing.T, Error, Errorf, and Fail do not terminate the test.
Error and Errorf call Fail.
There will be another call to FailNow to terminate the test. (FailNow also calls Fail.)

When `dontReportFailures` is set in Error, Errorf, and Fail, the test will not be terminated.
Moreover, the subsequent call to FailNow will be ignored because of the set dontReportFailures flag.

In this change, dontReportFailures will not be set for Error, Errorf, and Fail so that the subsequent FailNow will terminate the test.

## Summary

This PR reorganize the files inside the ledger package by move the evaluator related files into its own `internal` package.
The files in the internal package cannot access the root ledger files, and therefore using the shared `ledgercore` as a place to share interfaces.

```
ledger/
├── apply/
├── internal/
├── ledgercore/
└── testing/
```

## Test Plan

use existing test, and update existing tests.

Summary
This test has a special handling is case the protocol upgrade completes before we can validate that without an upgrade the rekey feature won't work. However, when that does happen, AccountA is already rekeyed, and would require a different handling.

To avoid that while retaining the execution of the test, I've moved the second part to use a different set of accounts.

Test Plan
This is a test

## Summary

This PR addresses a regression introduced in #2983. 
The culprit is that an interface might contain a nil pointer, which makes it insufficient to test the interface pointer itself.

Fail cases: https://app.circleci.com/pipelines/github/algorand/go-algorand/2451/workflows/8807ced7-89ae-4b6b-96b0-1bc5bdf9d84c/jobs/27259/tests#failed-test-0

## Test Plan

Use existing unit testing. In particular TestCatchupOverGossip.

## Summary

The TestMultiThreaded was running too slow on Travis machines, causing it to fail due to deadlock detection false-positive.

The change in this PR modify the execution time of the test, ensuring that it won't run for more than 5 second ( which should be enough for data racing detection testing )

## Test Plan

This is a test.

## Summary

During fast catchup, we restart the transaction sync service very quickly.
This can cause a network message being sent, and the response would be returned to the "restarted" txnsync.

Since we don't want to disconnect the network connection itself ( which could have some messages enqueued ), the transaction sync would need to store the "returned" channel before sending the message. That would avoid the data race ( and safely ignore the incoming message ).

## Test Plan

Use existing testing, and confirm against that.

## Summary

move StartEvaluator parameters into an object. Given that most of the time the parameters are using a "default" state, it reduce the number of passed parameters, improving the caller readability.

## Test Plan

Unit tests updated.

Fix the test TestReproducibleCatchpointLabels, by provide proper calculation of the account totals.

Due to Travis not being able to handle our builds anymore, we are removing travis integration tests.

…n values (#3041)

## Summary

The e2e test `TestPartitionHalfOffline` was failing due to the deadlock detection failing while the old keys are being deleted.
This change would reduce the key dilution, and should shorten the duration it takes to generate a new batch.

## Test Plan

This is a test.

…rationFuture (#3043)

## Summary

Reduce unneeded tests from TestParticipationAccountsExpirationFuture.

Resolves #3042

## Test Plan

This is a test

## Summary

The database handle is not closed in this code path.
In normal situations, this passes unnoticed, since it will be closed once the test terminates. However, when running the test multiple times, the OS will complain about too many open files. 

## Test Plan
This is a test infrastructure fix.

Fix global-state-schema key name in REST API spec

## Summary

1. Add  a new `OnlineAccountData` data type
2. Implement `LookupAgreement`

## Test Plan

Fixed existed tests: added `LookupAgreement` to mocked ledgers

## Summary

Use LookupAgreement in AlgorandFullNode.VotingKeys()

Post #3046 fixes.

## Summary

The test had a conceptual buggy pattern. It was using 
```golang
round, err = client.CurrentRound()
...
_, err = client.BroadcastTransaction(signedTxn)
...
_, err = client.WaitForRound(round + 3)
```
which is doomed to fail if the test process goes to sleep for a minute between the `client.CurrentRound()` call and the `client.BroadcastTransaction` call. The trivial solution is to reorder the calls so that 
```golang
_, err = client.BroadcastTransaction(signedTxn)
...
round, err = client.CurrentRound()
...
_, err = client.WaitForRound(round + 3)
```


## Test Plan

This is a test.

Summary
Adds additional resilience to participation expiration test TestParticipationAccountsExpirationFuture.

Test Plan
This is a test.

… test (#3056)

## Summary

Historically only `account updates` tracker was committing into DB. The PR changes this:
1. `dbRound` management and `commitSyncer` business moved to trackerRegistry.
2.  `account updates` is used as a driver for committing tasks scheduling - by given `commttedRound` and `dbRound` it figures out offset in detlas for saving to the disk. This is basically old `au.committedUpTo` method`.
3. `commitRound` replaced by `prepareCommit` that returns commit and post commit callbacks
4. trackerRegistry has own `commitRound` that calls  trackers' `prepareCommit` and commit/post commit procs if any.
5. `account updates` still have a cached dbRound to be used in Lookup methods.

## Test Plan

It is refactoring, use existing tests

Our travisci keeps breaking for different reasons so we decided to move our deploy to circleci.

## Summary

This PR eliminate the `totalsNewRounds`, which perform the exact same calculation previously performed by the block evaluator.

## Test Plan

- [x] Unit tests updates.
- [x] Catchpoint label testing : starting a node catching up to 16,800,000 and the correct catchpoint label was observed on round 16,810,000.

Noticed our environment passing variable is not working as intended, implemented a different way of passing it.

## Summary

When a 2.1 (i.e. no txsync) client connects to a 3.0 relay (i.e. txsync), the relay needs to request the client to keep sending it a TX messages - otherwise, these transactions would not get propagated.

The 2.1 and 3.0 above are network protocol versions, not a algod release version.

## Test Plan

e2e test added.

Fixing naming due to new way of displaying CircleCI workflow.

…#3087)

)

Summary
avoid requesting blocks that aren't needed by the ledger.

This isn't done for the sake of optimization - it's done so we won't have misleading warning messages in the log file due to frequently canceled contexts.

Test Plan
tested manually.

Since we increased a lot of things last week to pass beta tests quickly, we needed to go back and update machine sizes and parallelism for all but integration tests.

## Summary

This PR adds the node ( both client and server ), the ability to measure the time it takes to establish an outgoing connection ( excluding the TCP connection time ).
This duration is captured as the initial latency, which would need to get updated via a pingpong style logic.

## Test Plan

- [x] Extend existing unit tests
- [x] mainnet-model testing is needed as well to confirm correctness

…tion messages (#3102)

Summary
This PR adds the missing bridge between the txnsync and the classic transaction relaying:
when a transaction message arrive and being added to the transaction pool, we need to attempt to
relay these messages right away using the classic transaction messages. That would allow relays to
be compatible with both 2.1 and 3.0 nodes.

Test Plan
e2e test was added.

Removed data/abi/abi_value.go
Encode/Decode method take golang values, the behavior should be based on ABI type schemes

## Summary

This change wait for a given broadcasted transaction to be included instead of expecting it to be included after a predetermined number of rounds. This would increase the testing platform flexibility in case we're testing on slow machines.

## Test Plan

This is a test.

setting group index to be the one associated with the run

In tealdbg, the txn obj has a number of fields that are only available after the transaction has been evaluated in ApplyData. At the moment we show "Unable to obtain effects from top-level transactions" from https://github.com/algorand/go-algorand/blob/master/data/transactions/logic/eval.go#L1939 . This should be hidden for now.

This adds buildpulse integration using their orb.

## Summary

I noticed log messages like this:
```
{"file":"txHandler.go","function":"github.com/algorand/go-algorand/data.(*solicitedAsyncTxHandler).loop","level":"info","line":541,"msg":"solicitedAsyncTxHandler was unable to relay transaction message : %!v(MISSING)","time":"2021-10-20T23:18:40.220089Z"}
{"file":"txHandler.go","function":"github.com/algorand/go-algorand/data.(*solicitedAsyncTxHandler).loop","level":"info","line":541,"msg":"solicitedAsyncTxHandler was unable to relay transaction message : %!v(MISSING)","time":"2021-10-20T23:18:40.220226Z"}
{"file":"txHandler.go","function":"github.com/algorand/go-algorand/data.(*solicitedAsyncTxHandler).loop","level":"info","line":541,"msg":"solicitedAsyncTxHandler was unable to relay transaction message : %!v(MISSING)","time":"2021-10-20T23:18:40.220300Z"}
{"file":"txHandler.go","function":"github.com/algorand/go-algorand/data.(*solicitedAsyncTxHandler).loop","level":"info","line":541,"msg":"solicitedAsyncTxHandler was unable to relay transaction message : %!v(MISSING)","time":"2021-10-20T23:18:40.228731Z"}
{"file":"txHandler.go","function":"github.com/algorand/go-algorand/data.(*solicitedAsyncTxHandler).loop","level":"info","line":541,"msg":"solicitedAsyncTxHandler was unable to relay transaction message : %!v(MISSING)","time":"2021-10-20T23:18:40.228828Z"}
{"file":"txHandler.go","function":"github.com/algorand/go-algorand/data.(*solicitedAsyncTxHandler).loop","level":"info","line":541,"msg":"solicitedAsyncTxHandler was unable to relay transaction message : %!v(MISSING)","time":"2021-10-20T23:18:40.228893Z"}
{"file":"txHandler.go","function":"github.com/algorand/go-algorand/data.(*solicitedAsyncTxHandler).loop","level":"info","line":541,"msg":"solicitedAsyncTxHandler was unable to relay transaction message : %!v(MISSING)","time":"2021-10-20T23:18:40.228946Z"}
{"file":"txHandler.go","function":"github.com/algorand/go-algorand/data.(*solicitedAsyncTxHandler).loop","level":"info","line":541,"msg":"solicitedAsyncTxHandler was unable to relay transaction message : %!v(MISSING)","time":"2021-10-20T23:18:40.229012Z"}
{"file":"txHandler.go","function":"github.com/algorand/go-algorand/data.(*solicitedAsyncTxHandler).loop","level":"info","line":541,"msg":"solicitedAsyncTxHandler was unable to relay transaction message : %!v(MISSING)","time":"2021-10-20T23:18:40.229080Z"}
```

## Test Plan

No test changes.

Summary
Indexer needs to be able to set those fields.

Enable totals calculation in eval for indexer

## Summary

The previous web socket version was not flushing the write buffer when writing a Close control message.
As a result, we ended up disconnection the connection on one side correctly, while having it in a zombie state on the other side.

## Test Plan

Use existing tests. The `TestSlowPeerDisconnection` was already observing the issue being resolved.

## Summary


Improve the bandwidth estimation within the transaction sync by having the estimation account for latency, transaction compression time, and time spent waiting in the incoming queue.

## Test Plan


Wrote unit tests for correctness, ran network on mainnet model and observed measured bandwidths. Before the bandwidth would converge to the minimum over time as well have erratic inaccuracies. Now the numbers look much more in range, at most a factor of 2 off.

## Summary

The catchup was occasionally reporting
```
(1): fetchAndWrite(13932148): ledger write failed: block evaluation for round 13932148 requires sequential evaluation while the latest round is 13932148
```

This issue indicates that the catchup was attempting to validate a block which is not the latest+1, but rather newer.
In this case, we can safely ignore this error, and skip applying this block, since the block was already added to the ledger.

## Test Plan

Tested manually.

Trimming the whitespaces before and after each line in TEAL code.

In #2168, white space before #pragma version XXX leads to a failure in inferring the TEAL version. Relaxing assembler rules resolves the issue.

Closes #2168.

## Summary

Running betanet, we've seen errors such as
```
unable to enqueue incoming message into peer incoming message backlog. disconnecting from peer.
```
as well as multiple
```
Peer 162.202.32.72:56543 disconnected: SlowConnection
```

The goal of this PR is to remove from the pending incoming message queue entries that are associated with network peers that have been disconnected ( by the network package ), or have been scheduled for disconnection ( by the transaction synch package ).

Removing these would increase the "available" space in the incoming message queue and would prevent redundant disconnections.

## Test Plan

- [x] unit tests updated.
- [x] ran s1 network with and without load, successfully.

## Summary

Add an option to support version checking for arbitrary packages. This would allow us to deploy indexer releases at `algorand-releases/<channel>/indexer-*`

## Test Plan

Ran the command and see the correct regex:
```
~$ updater ver check -c nightly -p indexer
Checking for files matching: 'channel/nightly/indexer_nightly_linux-amd64_' in bucket algorand-releases
no updates found for channel 'nightly'
```

)

Summary
Refactor internal ledger evaluator testing to use the real ledger instead of using a dummy ledger.
In order to do this, I moved the related tests into an "internal_test" package. That allowed me to import both "ledger/internal" as well as "ledger", and conduct the test against a real ledger.

Test Plan
These are tests.

Summary

@barnjamin mentioned in #2899 that labstack/gommon/issues/46 has a similar issue.

Should be fixed by up version of sys. Closing #2899.

This makes it possible to configure which port algod is running on when using heapWatch.py

…tilization. (#3153)

## Summary

In order to reduce risk on the migration to the new transaction sync, the network protocols order was flipped.
Nodes would default to use the old transaction propagation, and would use the new transaction propagation only when configured via the config.json "NetworkProtocol" flag.

## Test Plan

Need to be tested on betanet.

We moved everything but windows to CircleCI, this is just cleaning up.

Add c5d.4xlarge and c5d.18xlarge us-east-2 to hosttemplates.json so that our team can use them moving forward for performance tests.

For performance testing, we've needed to add these to each codebase we're working with to use these machines. This is part of our effort to automate performance testing internally.

## Summary

Move Local structure definition to localTemplate, so that changes to localTemplate are more obvious to be a change of template than a change of data.

## Test Plan

Tested manually.

## Summary

This refactoring would eliminate the need to change the maxPaysetHint while we modify the block size.

## Test Plan

Use existing tests.

## Summary

1. It doesn't make sense to calculate new totals in `finalValidation()`
2. `endOfBlock()` already does some validation

Moving the content of `finalValidation()` to `endOfBlock()` makes things more reasonable.

## Summary

Change constants in the txnsync around message size limits into variables based on maxTxPoolSize. This will allow for support of larger message sizes when dealing with larger load.

## Test Plan

Reran existing tests

* Add unit test to check for string fields

* Fix seen set and add exceptions for state deltas

* Also check AccountData

* Minor rename

* Add license

* Accidentally pushed to origin, need new hash

## Summary

This PR is doing the following:
1. Moves the proposal deadline computation outside of the agreement package and onto the node package. The agreement wasn't doing anything with the deadline anyway.
2. Make the proposal deadline be configured using a configuration parameter rather then a global hard-coded value in the config package.

## Test Plan

Existing unit test provides sufficient coverage for this change.

## Summary

FillDBWithParticipationKeys will return empty PersistedParticipation when lastValid is less than firstValid.
When this happens, newPart will not have partdb in newPart.Store, instead, will have an empty object.
Calling Close on the empty object will call close on nil Accessor pointer and panic.

This change avoids using the returned object with non-nil error, and properly closes the db with the valid object.
## Test Plan

Added test to verify the proper handling of the different errors returned from FillDBWithParticipationKeys and handled by GenParticipationKeysTo.

fix typo: "on" -> "one" in ErrBatchVerificationFailed

## Summary

This PR rolls back the txsync, retaining only the config file changes

## Summary

Going to ledger directly prevents caching of accounts.

I'm looking into poor indexer import performance on testnet at rounds > 17M, and I noticed that one account is looked up a lot of times in same round.

## Summary

Extract catchpoint logic out of the accounts update tracker.

## Test Plan

Refactor existing unit tests.

* Add error strings to v2 TEAL compile endpoint

* Move number of errors to end so it is consistent with goal clerk compile

* Refactor ReportProblems() to take io.Writer

* Don't report number of errors on the API call

* add abi encoding

* accord with go-algorand code format

* minor modification

* add partition test

* resolve review, need more testcase rewrite

* move from cmd/goal to data, resolve review

* rewrite is-dynamic, dynamic array

* update dynamic array 0 element support, test needed

* minor

* minor

* minor

* add more testcase for abi encode/decode

* update comments in encoding test

* minor

* attempt to split abi.go into 2 files

* separate abi files to smaller files

* resolve reviews, work on random gen tuple value encode/decode

* add random tuple test

* remove math-rand, use crypto-rand

* minor

* minor

* some change requested from community

* fix for 1 corner case

* resolve review comments

* resolve review comments

* minor

* minor

* update encode slot capacity

* minor

* resolve reviews

* minor update on bool bytelen calculate

* update encode/decode from types

* random test remain to be modified

* testing variable renaming, encode int support (u)int types

* update test scripts and remove value struct

* follow golint

* partly resolving comments

* whoops uint encoding update

* update int decode to primitive types method

* go fmt

* update parseAppArg to accept abi input (attempt)

* need to check cmdline arg validity

* update unmarshal from JSON in ABI type

* unmarshal from json for ABI type

* update ABI type unmarshal values from JSON bytes

* update ABI methods for string/array/address

* update unmarshal from JSON in abi

* fix for error in ufixed json unmarshal

* fix

* update on method sub command

* minor

* probably better separate abi json to a single file

* i just want to add a required flag plz...

* minor fix on interface from json

* consider some rough test cases

* minor

* add partition test

* update static uint test

* update marshal/unmarshal json methods for abi

* marshal byte array to b64 string

* abi json polish

* update golangci lint rules

* revert golangci config

* update method impl

* update method signature return type check

* minor

* copy-paste code from call app cmd

* minor

* add method flag to txn flags

* minor

* update changes

* minor

* moving helper functions to abi

* update comments

* update method app call

* resolve part in abi impl

* add oncomplete support

* minor

* try to use stringarrayvar

* minor

* update goal return log handing process

* go simple

* add a line of e2e test for now

* update

* minor

* minor

* minor

* go fmt

* approval/clear prog nil

* discard all changes to e2d-app-cross-round, going to write separately e2e test

* update e2d tests

* check ret valu

* use constant

* resolve review partly

* resolve review on code reformatting

* resolve review on code reformatting, use code chunk for datadir and client

* go fmt

* export tuple type maker

* update comments in e2e test

* update filter empty string

* resolve issues with JSON abi

* minor

* adding missing  to opcode docs for external gets

* update TEAL_opcodes.md

## Summary

* new `ParticipationRegistry` interface designed to manage keys and collect usage metrics.
* new REST endpoints for interacting with keys.
* improved `goal account partkeyinfo` and `goal account listpartkeys` formatting.

## Test Plan

New unit and integration tests.

Integration tests (specifically, e2e-go tests) seem to be having resource issues lately, so this returns integration tests to using large instances (4 cores, ~16GB RAM) after they were downsized to medium (2 cores ~8GB RAM) in #3095. Nightly integration tests are already using large VMs.

## Summary

When implementing the catchpoint tracker, I missed the re-initilization location for some of the local variables.
This would generate incorrect catchpoint labels after a node completes a fast-catchup.

#3085

## Test Plan

- [x] Add unit tests
- [x] Perform manual testing

## Summary

This PR updates the websocket library to version 1.4.4, which includes a fix for deadlocked writers.

## Summary

This PR addresses two separate issues in the `pseudonodeNode` implementation:
1. `pseudonodeVotesTask.execute` and `pseudonodeProposalsTask.execute ` do not handle vote verification enqueueing failures ( to the execution pool ). This could lead the the pseudonode processing go-routine being stuck.
2. The `pseudonodeVotesTask.execute` could block forever in case the output channel is not being read from. 

## Trigger
This issue was detected by the telemetry:
```
 ({num-occurances}): {telemetry error string}
 (67): pseudonode.MakeVotes call failed(attest) pseudonode input channel is full
```

## Test Plan

Few unit tests added. More to come.

Check if the outFilename argument is passed and write to the path specified as other commands

Re-merge go-algorand 3.1.3-beta