Based on the matrix job @algojack added in #2749, this turns the other per-platform jobs into matrix jobs too. The most significant change in this PR is switching from building in the home directory (/home/circleci on Linux and /Users/distiller on Mac) to instead build out of /opt/cibuild to make it easier for the matrix jobs to share configuration.

Add a new ledger interface for Indexer that is significantly simpler and allows implementing batching in a more straight-forward way. As a result, the Indexer code doesn't need to know the specifics of go-algorand and doesn't need to implement its own accounts cache for batching.

TestWaitAndAddConnectionTimeLongtWindow is randomly failing when the execution of time sensitive statements is separated by more than the expected time gap.

This issue is fixed by replacing the sleep with simulated time passing by changing the recorded time, thus giving the impression that the event happened timeout time ago.

The unit test TestLatestSigsFromThisNode was data racing. The fix is trivial.

## Summary

The account totals are being tested once every round to ensure that the total amount of money in the system doesn't change. Prior to this PR, the accounts update newBlock method was responsible for testing that : this would ensure that we're not writing a new round to disk that violates the totals predicate.

While this was working correctly, conducting this test at the time we're writing this information to the disk is too late. By that time, we've already agreed upon applying this (problematic) block to disk. Ideally, we could detect this situation ahead of time, and avoid agreeing on a block which would violate the totals predicate.

In this PR, we've moving the totals calculation as the last step of the delta state calculation. Few thoughts:
- From security perspective, it's just as secured, since the StateDeltas are always in-process and never shared across the network.
- From agreement perspective - in case we have a bug in the evaluator that would cause it to generate a state deltas that would violate the totals predicate, the evaluator is going to fail assembling the state deltas on the generator, validator and applicator ( i.e. "consumer" ). Ideally, in the future, we would be able to propose the empty block in such case, allowing the problematic transactions to time-out.
- From code-flow perspective - this seem to be a much better approach: a failure to store the block to the disk has only one potential outcome - panicking. Having the ability to detect that the state delta is invalid would give us a way out of that.

## Test Plan

1. Unit tests updated.
2. Tested against mainnet : catchup from scratch to force validation of the entire blockchain.

## Summary

This PR is to take advantage of arm64's ability to run arm32 binaries. Using this, we are able to build for arm32 on arm64 hosts by running containers with arm32 OS's on them. This is much faster than our qemu set up because it doesn't rely on emulations. The binaries produced with this process work and the arm64/arm32 build times are as fast as the intel builds, reducing our build time by about an hour.

## Test Plan

I ran our build pipeline against this branch using arm64 hosts and tested the produced artifacts using emulated cpu's to make sure they still work on their native architectures.
Tested manually on raspberry pi 2 model b, and it's working as expected.

Rotated slack token in environment variable and removed the codecov token from Travis file. codecov token is removed because it's not needed for public repos.

## Summary

<!-- Explain the goal of this change and what problem it is solving. Format this cleanly so that it may be used for a commit message, as your changes will be squash-merged. -->

The test TestRequestBlockBytesErrors has failed, and the reason is unidentifiable.
Various fixes are here, including more verbose error reporting to identify the issue when it occurs again.

- Add waitgrop to ledger to wait before closing the ledger and blockQ when goroutines are using them.
- In universalFetcher, return the error immediately in case of an error.
- Proper closing of objects in TestRequestBlockBytesErrors, and more verbose error reporting.


The waitGroup is needed for the following situation:

in blockService.go listenForCatchupReq

When, immediately after calling `bq.handleCatchupReq`, `BlockService.Stop()` is called, `blockQueue` will be `nil` by the time `blockQueue.getEncodedBlockCert` is called. 
```

0  0x0000000004e16846 in github.com/algorand/go-algorand/ledger.(*blockQueue).checkEntry
   at /Users/shantkarakashian/go/src/github.com/algorand/go-algorand/ledger/blockqueue.go:215
1  0x0000000004e174a9 in github.com/algorand/go-algorand/ledger.(*blockQueue).getEncodedBlockCert
   at /Users/shantkarakashian/go/src/github.com/algorand/go-algorand/ledger/blockqueue.go:295
2  0x0000000004e3a2a7 in github.com/algorand/go-algorand/ledger.(*Ledger).EncodedBlockCert
   at /Users/shantkarakashian/go/src/github.com/algorand/go-algorand/ledger/ledger.go:516
3  0x0000000004f46eed in github.com/algorand/go-algorand/rpcs.topicBlockBytes
   at /Users/shantkarakashian/go/src/github.com/algorand/go-algorand/rpcs/blockService.go:355
4  0x0000000004f45e1b in github.com/algorand/go-algorand/rpcs.(*BlockService).handleCatchupReq
   at /Users/shantkarakashian/go/src/github.com/algorand/go-algorand/rpcs/blockService.go:299
5  0x0000000004f45318 in github.com/algorand/go-algorand/rpcs.(*BlockService).listenForCatchupReq
   at /Users/shantkarakashian/go/src/github.com/algorand/go-algorand/rpcs/blockService.go:245

```

## Test Plan
Test is enhanced. 
<!-- How did you test these changes? Please provide the exact scenarios you tested in as much detail as possible including commands, output and rationale. -->

Remove time ledger tracker as not used.

fix a typo.

Summary
In the EvalForIndexer, we don't want to call finalValidation(). This would
skip the calculation of the account totals in the state delta, which is a serious
issue if it were to be used by algod, but it's perfectly fine for the indexer since
it doesn't track any totals and therefore cannot calculate the new totals.

Test Plan
Test against the indexer.

The evaluator shouldn't ask the ledger who is the creator of an asset multiple times. This PR adds a cache for ledger's responses in cow base.

Timing issues between writing to and reading from the database cause the test to occasionally fail.
Make the test more tolerant to such timing issues by polling.

Encoding/Decoding method for ABI Values.
Serialization/Deserialization of ABI Types.
Make/Get methods for ABI Types/Values.
Basic testcases for Encoding/Decoding.

* moving logs e2e to scripts

AVM 1.0 introduces application accounts
goal clerk needs to consider this account when creates dryrun request object
Added a new option --dryrun-accounts for adding accounts that were rekeyed to an app account (or any other accounts).

Similar to app-accounts test, but tests all sorts of asset txns instead of pays.

Made it nice to write e2e test in python by creating a "goal" module
that emulates the convenience of goal, but in python so that
structured results come out for easy asserts.

* Enable cost and fee pooling in tealdbg

* CR fixes

Fixes found during recent cluster bandwidth testing

## Summary

The plan to change the tracker database to maintain the latest 320 rounds for the online accounts only have some other required modification; one of them is the semantics of the `Totals` method:
- At this time, it supports retrieving `AccountTotals` for any of the recent 320 rounds.
- We need the method to support the `AccountTotals` for the latest round only, as well as provide the circulating supply for latest-320 rounds ( i.e. the circulating supply is a subset of the `AccountTotals` ).

Once the database implementation is complete, the database would contain the `AccountTotals` information for the latest round only, plus the online circulation for the past 320 rounds.

In order to support that, I've broken up the interface into:
- `LatestTotals`, which would return the latest totals as well as the latest round associated with these totals.
- `OnlineTotals`, which receive a round number and return the online totals associated with that round.

This change is focused around the Ledger interface. Further changes would be required in the accounts update before the high-level goals could be achieved.

## Test Plan

Unit tests updated.

Release3.0.1 mergeback manual

lower acct bal so it receives no reward

Return a bit more details on eval failure for app calls.

Start work on Teal 6 (AVM 1.1)

 vFuture, LogicVersion changed to 6
 Support RekeyTo field, and Keyreg transactions

## Summary

This PR implements the transaction sync 2.0.

More details can be found in the design document and the feature presentation .

## Test Plan

Unit tests, e2e tests and performance tests were executed against this branch successfully.

## Reviewer notes

This PR is pretty large. Consider reviewing it in browsers other than Safari.

itxn_next to give us inner groups

Questions remain around how TxID and GroupID should be defined for inners.

… branch. (#3017)

## Summary

 Disable the participation key refresh on the TestPartitionHalfOffline test. The test was failing with the following threads running:

```
algod(20739) : github.com/algorand/go-algorand/agreement.(*asyncPseudonode).loadRoundParticipationKeys(0xc0033dad20, 0x2, 0xc0039c8180, 0xc000530000, 0xc000dcb0d0)
```
```
algod(20739) : github.com/algorand/go-algorand/data/account.PersistedParticipation.DeleteOldKeys(0x92c6b21bbd8fefac, 0xed2d99ce769768ea, 0xac85a3b9466cead1, 0xc32937d7e4c532f9, 0xc000043260, 0xc0002b04b0, 0x0, 0x2dc6c0, 0x2710, 0xc0001892c0, ...)
```
```
algod(20739) : github.com/algorand/go-algorand/data.(*AccountManager).AddParticipation(0xc0001191a0, 0xd175b5d042fa1c39, 0x2140fe15c6a19653, 0xc66f7b9c58cf5ed1, 0x107d657815b44d49, 0xc0037a4d20, 0xc0002b0690, 0x0, 0x2dc6c0, 0x2710, ...)
```

By avoiding the participation key reload ( which isn't needed for this test ), I'm hoping to avoid the issue.

## Test Plan

This is a test.

## Summary

Resolves #2738

Adds the following:

- consensus.go: Adds the default MaxExpiredAccountsToProcess value for
  the new consensus protocol
- block.go: Adds necessary block header entries
- eval.go: Scans for expired accounts, modifies them to be offline and adds
  validation for this use case
- eval_test.go: Basic unit tests
- participationExpiration_test.go: Added e2e tests that verify that
  different consensus protocols behave differently

## Test Plan

Unit test and e2e tests added

## Summary

rebase Tolik's PR on master; this PR add the ability for the indexer to preload account data so that it would be used by the evaluator.

## Test Plan

Existing unit tests updated.

## Summary

* Moved code from **acctupdates.go** to **trackerdb.go**.
* Extracted  db creation and init from`accountsInitialize` to `trackerDBInitialize`
* `reloadLedger` now calls `trackerDBInitialize` right after block db init
* `reloadLedger` also has post-creation actions like vacuuming

Limitations:
1. Tables creation still calls `accountsInit` for initial tables creation and inserting genesis accounts.
2. Migration code still have some business logic. 

## Test Plan

Use existing test suite

…cleci nightly testing. (#3016)

For CircleCI testing, increase machine size from medium to large and set parallelism to 4.

In testing.T, Error, Errorf, and Fail do not terminate the test.
Error and Errorf call Fail.
There will be another call to FailNow to terminate the test. (FailNow also calls Fail.)

When `dontReportFailures` is set in Error, Errorf, and Fail, the test will not be terminated.
Moreover, the subsequent call to FailNow will be ignored because of the set dontReportFailures flag.

In this change, dontReportFailures will not be set for Error, Errorf, and Fail so that the subsequent FailNow will terminate the test.

…cleci nightly testing. (#3016)

For CircleCI testing, increase machine size from medium to large and set parallelism to 4.

## Summary

This PR reorganize the files inside the ledger package by move the evaluator related files into its own `internal` package.
The files in the internal package cannot access the root ledger files, and therefore using the shared `ledgercore` as a place to share interfaces.

```
ledger/
├── apply/
├── internal/
├── ledgercore/
└── testing/
```

## Test Plan

use existing test, and update existing tests.

Summary
This test has a special handling is case the protocol upgrade completes before we can validate that without an upgrade the rekey feature won't work. However, when that does happen, AccountA is already rekeyed, and would require a different handling.

To avoid that while retaining the execution of the test, I've moved the second part to use a different set of accounts.

Test Plan
This is a test

## Summary

This PR addresses a regression introduced in #2983. 
The culprit is that an interface might contain a nil pointer, which makes it insufficient to test the interface pointer itself.

Fail cases: https://app.circleci.com/pipelines/github/algorand/go-algorand/2451/workflows/8807ced7-89ae-4b6b-96b0-1bc5bdf9d84c/jobs/27259/tests#failed-test-0

## Test Plan

Use existing unit testing. In particular TestCatchupOverGossip.

## Summary

The TestMultiThreaded was running too slow on Travis machines, causing it to fail due to deadlock detection false-positive.

The change in this PR modify the execution time of the test, ensuring that it won't run for more than 5 second ( which should be enough for data racing detection testing )

## Test Plan

This is a test.

## Summary

During fast catchup, we restart the transaction sync service very quickly.
This can cause a network message being sent, and the response would be returned to the "restarted" txnsync.

Since we don't want to disconnect the network connection itself ( which could have some messages enqueued ), the transaction sync would need to store the "returned" channel before sending the message. That would avoid the data race ( and safely ignore the incoming message ).

## Test Plan

Use existing testing, and confirm against that.

## Summary

move StartEvaluator parameters into an object. Given that most of the time the parameters are using a "default" state, it reduce the number of passed parameters, improving the caller readability.

## Test Plan

Unit tests updated.

Fix the test TestReproducibleCatchpointLabels, by provide proper calculation of the account totals.

Due to Travis not being able to handle our builds anymore, we are removing travis integration tests.

…n values (#3041)

## Summary

The e2e test `TestPartitionHalfOffline` was failing due to the deadlock detection failing while the old keys are being deleted.
This change would reduce the key dilution, and should shorten the duration it takes to generate a new batch.

## Test Plan

This is a test.

…rationFuture (#3043)

## Summary

Reduce unneeded tests from TestParticipationAccountsExpirationFuture.

Resolves #3042

## Test Plan

This is a test

## Summary

The database handle is not closed in this code path.
In normal situations, this passes unnoticed, since it will be closed once the test terminates. However, when running the test multiple times, the OS will complain about too many open files. 

## Test Plan
This is a test infrastructure fix.

Fix global-state-schema key name in REST API spec

## Summary

1. Add  a new `OnlineAccountData` data type
2. Implement `LookupAgreement`

## Test Plan

Fixed existed tests: added `LookupAgreement` to mocked ledgers

## Summary

Use LookupAgreement in AlgorandFullNode.VotingKeys()

Post #3046 fixes.

## Summary

The test had a conceptual buggy pattern. It was using 
```golang
round, err = client.CurrentRound()
...
_, err = client.BroadcastTransaction(signedTxn)
...
_, err = client.WaitForRound(round + 3)
```
which is doomed to fail if the test process goes to sleep for a minute between the `client.CurrentRound()` call and the `client.BroadcastTransaction` call. The trivial solution is to reorder the calls so that 
```golang
_, err = client.BroadcastTransaction(signedTxn)
...
round, err = client.CurrentRound()
...
_, err = client.WaitForRound(round + 3)
```


## Test Plan

This is a test.

Summary
Adds additional resilience to participation expiration test TestParticipationAccountsExpirationFuture.

Test Plan
This is a test.

… test (#3056)

## Summary

Historically only `account updates` tracker was committing into DB. The PR changes this:
1. `dbRound` management and `commitSyncer` business moved to trackerRegistry.
2.  `account updates` is used as a driver for committing tasks scheduling - by given `commttedRound` and `dbRound` it figures out offset in detlas for saving to the disk. This is basically old `au.committedUpTo` method`.
3. `commitRound` replaced by `prepareCommit` that returns commit and post commit callbacks
4. trackerRegistry has own `commitRound` that calls  trackers' `prepareCommit` and commit/post commit procs if any.
5. `account updates` still have a cached dbRound to be used in Lookup methods.

## Test Plan

It is refactoring, use existing tests

Our travisci keeps breaking for different reasons so we decided to move our deploy to circleci.

Our travisci keeps breaking for different reasons so we decided to move our deploy to circleci.

## Summary

This PR eliminate the `totalsNewRounds`, which perform the exact same calculation previously performed by the block evaluator.

## Test Plan

- [x] Unit tests updates.
- [x] Catchpoint label testing : starting a node catching up to 16,800,000 and the correct catchpoint label was observed on round 16,810,000.

go-algorand 3.1.0-beta

Noticed our environment passing variable is not working as intended, implemented a different way of passing it.

Noticed our environment passing variable is not working as intended, implemented a different way of passing it.

Fixing CircleCI deploy for Betanet and Stablenet (#3077)

## Summary

When a 2.1 (i.e. no txsync) client connects to a 3.0 relay (i.e. txsync), the relay needs to request the client to keep sending it a TX messages - otherwise, these transactions would not get propagated.

The 2.1 and 3.0 above are network protocol versions, not a algod release version.

## Test Plan

e2e test added.

Fixing naming due to new way of displaying CircleCI workflow.

…#3087)

## Summary

When a 2.1 (i.e. no txsync) client connects to a 3.0 relay (i.e. txsync), the relay needs to request the client to keep sending it a TX messages - otherwise, these transactions would not get propagated.

The 2.1 and 3.0 above are network protocol versions, not a algod release version.

## Test Plan

e2e test added.

3.1.1 hotfix

)

Summary
avoid requesting blocks that aren't needed by the ledger.

This isn't done for the sake of optimization - it's done so we won't have misleading warning messages in the log file due to frequently canceled contexts.

Test Plan
tested manually.

Since we increased a lot of things last week to pass beta tests quickly, we needed to go back and update machine sizes and parallelism for all but integration tests.

## Summary

This PR adds the node ( both client and server ), the ability to measure the time it takes to establish an outgoing connection ( excluding the TCP connection time ).
This duration is captured as the initial latency, which would need to get updated via a pingpong style logic.

## Test Plan

- [x] Extend existing unit tests
- [x] mainnet-model testing is needed as well to confirm correctness

…tion messages (#3102)

Summary
This PR adds the missing bridge between the txnsync and the classic transaction relaying:
when a transaction message arrive and being added to the transaction pool, we need to attempt to
relay these messages right away using the classic transaction messages. That would allow relays to
be compatible with both 2.1 and 3.0 nodes.

Test Plan
e2e test was added.

…tion messages (#3102)

Summary
This PR adds the missing bridge between the txnsync and the classic transaction relaying:
when a transaction message arrive and being added to the transaction pool, we need to attempt to
relay these messages right away using the classic transaction messages. That would allow relays to
be compatible with both 2.1 and 3.0 nodes.

Test Plan
e2e test was added.

Removed data/abi/abi_value.go
Encode/Decode method take golang values, the behavior should be based on ABI type schemes

## Summary

This change wait for a given broadcasted transaction to be included instead of expecting it to be included after a predetermined number of rounds. This would increase the testing platform flexibility in case we're testing on slow machines.

## Test Plan

This is a test.

go-algorand v3.1.2-beta

setting group index to be the one associated with the run

In tealdbg, the txn obj has a number of fields that are only available after the transaction has been evaluated in ApplyData. At the moment we show "Unable to obtain effects from top-level transactions" from https://github.com/algorand/go-algorand/blob/master/data/transactions/logic/eval.go#L1939 . This should be hidden for now.

This adds buildpulse integration using their orb.

## Summary

I noticed log messages like this:
```
{"file":"txHandler.go","function":"github.com/algorand/go-algorand/data.(*solicitedAsyncTxHandler).loop","level":"info","line":541,"msg":"solicitedAsyncTxHandler was unable to relay transaction message : %!v(MISSING)","time":"2021-10-20T23:18:40.220089Z"}
{"file":"txHandler.go","function":"github.com/algorand/go-algorand/data.(*solicitedAsyncTxHandler).loop","level":"info","line":541,"msg":"solicitedAsyncTxHandler was unable to relay transaction message : %!v(MISSING)","time":"2021-10-20T23:18:40.220226Z"}
{"file":"txHandler.go","function":"github.com/algorand/go-algorand/data.(*solicitedAsyncTxHandler).loop","level":"info","line":541,"msg":"solicitedAsyncTxHandler was unable to relay transaction message : %!v(MISSING)","time":"2021-10-20T23:18:40.220300Z"}
{"file":"txHandler.go","function":"github.com/algorand/go-algorand/data.(*solicitedAsyncTxHandler).loop","level":"info","line":541,"msg":"solicitedAsyncTxHandler was unable to relay transaction message : %!v(MISSING)","time":"2021-10-20T23:18:40.228731Z"}
{"file":"txHandler.go","function":"github.com/algorand/go-algorand/data.(*solicitedAsyncTxHandler).loop","level":"info","line":541,"msg":"solicitedAsyncTxHandler was unable to relay transaction message : %!v(MISSING)","time":"2021-10-20T23:18:40.228828Z"}
{"file":"txHandler.go","function":"github.com/algorand/go-algorand/data.(*solicitedAsyncTxHandler).loop","level":"info","line":541,"msg":"solicitedAsyncTxHandler was unable to relay transaction message : %!v(MISSING)","time":"2021-10-20T23:18:40.228893Z"}
{"file":"txHandler.go","function":"github.com/algorand/go-algorand/data.(*solicitedAsyncTxHandler).loop","level":"info","line":541,"msg":"solicitedAsyncTxHandler was unable to relay transaction message : %!v(MISSING)","time":"2021-10-20T23:18:40.228946Z"}
{"file":"txHandler.go","function":"github.com/algorand/go-algorand/data.(*solicitedAsyncTxHandler).loop","level":"info","line":541,"msg":"solicitedAsyncTxHandler was unable to relay transaction message : %!v(MISSING)","time":"2021-10-20T23:18:40.229012Z"}
{"file":"txHandler.go","function":"github.com/algorand/go-algorand/data.(*solicitedAsyncTxHandler).loop","level":"info","line":541,"msg":"solicitedAsyncTxHandler was unable to relay transaction message : %!v(MISSING)","time":"2021-10-20T23:18:40.229080Z"}
```

## Test Plan

No test changes.

Summary
Indexer needs to be able to set those fields.

Enable totals calculation in eval for indexer

## Summary

The previous web socket version was not flushing the write buffer when writing a Close control message.
As a result, we ended up disconnection the connection on one side correctly, while having it in a zombie state on the other side.

## Test Plan

Use existing tests. The `TestSlowPeerDisconnection` was already observing the issue being resolved.

## Summary


Improve the bandwidth estimation within the transaction sync by having the estimation account for latency, transaction compression time, and time spent waiting in the incoming queue.

## Test Plan


Wrote unit tests for correctness, ran network on mainnet model and observed measured bandwidths. Before the bandwidth would converge to the minimum over time as well have erratic inaccuracies. Now the numbers look much more in range, at most a factor of 2 off.

## Summary

During fast catchup, we restart the transaction sync service very quickly.
This can cause a network message being sent, and the response would be returned to the "restarted" txnsync.

Since we don't want to disconnect the network connection itself ( which could have some messages enqueued ), the transaction sync would need to store the "returned" channel before sending the message. That would avoid the data race ( and safely ignore the incoming message ).

## Test Plan

Use existing testing, and confirm against that.

## Summary

This PR adds the node ( both client and server ), the ability to measure the time it takes to establish an outgoing connection ( excluding the TCP connection time ).
This duration is captured as the initial latency, which would need to get updated via a pingpong style logic.

## Test Plan

- [x] Extend existing unit tests
- [x] mainnet-model testing is needed as well to confirm correctness

## Summary

I noticed log messages like this:
```
{"file":"txHandler.go","function":"github.com/algorand/go-algorand/data.(*solicitedAsyncTxHandler).loop","level":"info","line":541,"msg":"solicitedAsyncTxHandler was unable to relay transaction message : %!v(MISSING)","time":"2021-10-20T23:18:40.220089Z"}
{"file":"txHandler.go","function":"github.com/algorand/go-algorand/data.(*solicitedAsyncTxHandler).loop","level":"info","line":541,"msg":"solicitedAsyncTxHandler was unable to relay transaction message : %!v(MISSING)","time":"2021-10-20T23:18:40.220226Z"}
{"file":"txHandler.go","function":"github.com/algorand/go-algorand/data.(*solicitedAsyncTxHandler).loop","level":"info","line":541,"msg":"solicitedAsyncTxHandler was unable to relay transaction message : %!v(MISSING)","time":"2021-10-20T23:18:40.220300Z"}
{"file":"txHandler.go","function":"github.com/algorand/go-algorand/data.(*solicitedAsyncTxHandler).loop","level":"info","line":541,"msg":"solicitedAsyncTxHandler was unable to relay transaction message : %!v(MISSING)","time":"2021-10-20T23:18:40.228731Z"}
{"file":"txHandler.go","function":"github.com/algorand/go-algorand/data.(*solicitedAsyncTxHandler).loop","level":"info","line":541,"msg":"solicitedAsyncTxHandler was unable to relay transaction message : %!v(MISSING)","time":"2021-10-20T23:18:40.228828Z"}
{"file":"txHandler.go","function":"github.com/algorand/go-algorand/data.(*solicitedAsyncTxHandler).loop","level":"info","line":541,"msg":"solicitedAsyncTxHandler was unable to relay transaction message : %!v(MISSING)","time":"2021-10-20T23:18:40.228893Z"}
{"file":"txHandler.go","function":"github.com/algorand/go-algorand/data.(*solicitedAsyncTxHandler).loop","level":"info","line":541,"msg":"solicitedAsyncTxHandler was unable to relay transaction message : %!v(MISSING)","time":"2021-10-20T23:18:40.228946Z"}
{"file":"txHandler.go","function":"github.com/algorand/go-algorand/data.(*solicitedAsyncTxHandler).loop","level":"info","line":541,"msg":"solicitedAsyncTxHandler was unable to relay transaction message : %!v(MISSING)","time":"2021-10-20T23:18:40.229012Z"}
{"file":"txHandler.go","function":"github.com/algorand/go-algorand/data.(*solicitedAsyncTxHandler).loop","level":"info","line":541,"msg":"solicitedAsyncTxHandler was unable to relay transaction message : %!v(MISSING)","time":"2021-10-20T23:18:40.229080Z"}
```

## Test Plan

No test changes.

## Summary

The previous web socket version was not flushing the write buffer when writing a Close control message.
As a result, we ended up disconnection the connection on one side correctly, while having it in a zombie state on the other side.

## Test Plan

Use existing tests. The `TestSlowPeerDisconnection` was already observing the issue being resolved.

## Summary


Improve the bandwidth estimation within the transaction sync by having the estimation account for latency, transaction compression time, and time spent waiting in the incoming queue.

## Test Plan


Wrote unit tests for correctness, ran network on mainnet model and observed measured bandwidths. Before the bandwidth would converge to the minimum over time as well have erratic inaccuracies. Now the numbers look much more in range, at most a factor of 2 off.

## Summary

The catchup was occasionally reporting
```
(1): fetchAndWrite(13932148): ledger write failed: block evaluation for round 13932148 requires sequential evaluation while the latest round is 13932148
```

This issue indicates that the catchup was attempting to validate a block which is not the latest+1, but rather newer.
In this case, we can safely ignore this error, and skip applying this block, since the block was already added to the ledger.

## Test Plan

Tested manually.