go-algorand 3.2.1-stable #3279
Merged
go-algorand 3.2.1-stable #3279
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Based on the matrix job @algojack added in #2749, this turns the other per-platform jobs into matrix jobs too. The most significant change in this PR is switching from building in the home directory (/home/circleci on Linux and /Users/distiller on Mac) to instead build out of /opt/cibuild to make it easier for the matrix jobs to share configuration.
Add a new ledger interface for Indexer that is significantly simpler and allows implementing batching in a more straight-forward way. As a result, the Indexer code doesn't need to know the specifics of go-algorand and doesn't need to implement its own accounts cache for batching.
TestWaitAndAddConnectionTimeLongtWindow is randomly failing when the execution of time sensitive statements is separated by more than the expected time gap. This issue is fixed by replacing the sleep with simulated time passing by changing the recorded time, thus giving the impression that the event happened timeout time ago.
The unit test TestLatestSigsFromThisNode was data racing. The fix is trivial.
## Summary The account totals are being tested once every round to ensure that the total amount of money in the system doesn't change. Prior to this PR, the accounts update newBlock method was responsible for testing that : this would ensure that we're not writing a new round to disk that violates the totals predicate. While this was working correctly, conducting this test at the time we're writing this information to the disk is too late. By that time, we've already agreed upon applying this (problematic) block to disk. Ideally, we could detect this situation ahead of time, and avoid agreeing on a block which would violate the totals predicate. In this PR, we've moving the totals calculation as the last step of the delta state calculation. Few thoughts: - From security perspective, it's just as secured, since the StateDeltas are always in-process and never shared across the network. - From agreement perspective - in case we have a bug in the evaluator that would cause it to generate a state deltas that would violate the totals predicate, the evaluator is going to fail assembling the state deltas on the generator, validator and applicator ( i.e. "consumer" ). Ideally, in the future, we would be able to propose the empty block in such case, allowing the problematic transactions to time-out. - From code-flow perspective - this seem to be a much better approach: a failure to store the block to the disk has only one potential outcome - panicking. Having the ability to detect that the state delta is invalid would give us a way out of that. ## Test Plan 1. Unit tests updated. 2. Tested against mainnet : catchup from scratch to force validation of the entire blockchain.
## Summary This PR is to take advantage of arm64's ability to run arm32 binaries. Using this, we are able to build for arm32 on arm64 hosts by running containers with arm32 OS's on them. This is much faster than our qemu set up because it doesn't rely on emulations. The binaries produced with this process work and the arm64/arm32 build times are as fast as the intel builds, reducing our build time by about an hour. ## Test Plan I ran our build pipeline against this branch using arm64 hosts and tested the produced artifacts using emulated cpu's to make sure they still work on their native architectures. Tested manually on raspberry pi 2 model b, and it's working as expected.
Rotated slack token in environment variable and removed the codecov token from Travis file. codecov token is removed because it's not needed for public repos.
## Summary <!-- Explain the goal of this change and what problem it is solving. Format this cleanly so that it may be used for a commit message, as your changes will be squash-merged. --> The test TestRequestBlockBytesErrors has failed, and the reason is unidentifiable. Various fixes are here, including more verbose error reporting to identify the issue when it occurs again. - Add waitgrop to ledger to wait before closing the ledger and blockQ when goroutines are using them. - In universalFetcher, return the error immediately in case of an error. - Proper closing of objects in TestRequestBlockBytesErrors, and more verbose error reporting. The waitGroup is needed for the following situation: in blockService.go listenForCatchupReq When, immediately after calling `bq.handleCatchupReq`, `BlockService.Stop()` is called, `blockQueue` will be `nil` by the time `blockQueue.getEncodedBlockCert` is called. ``` 0 0x0000000004e16846 in github.com/algorand/go-algorand/ledger.(*blockQueue).checkEntry at /Users/shantkarakashian/go/src/github.com/algorand/go-algorand/ledger/blockqueue.go:215 1 0x0000000004e174a9 in github.com/algorand/go-algorand/ledger.(*blockQueue).getEncodedBlockCert at /Users/shantkarakashian/go/src/github.com/algorand/go-algorand/ledger/blockqueue.go:295 2 0x0000000004e3a2a7 in github.com/algorand/go-algorand/ledger.(*Ledger).EncodedBlockCert at /Users/shantkarakashian/go/src/github.com/algorand/go-algorand/ledger/ledger.go:516 3 0x0000000004f46eed in github.com/algorand/go-algorand/rpcs.topicBlockBytes at /Users/shantkarakashian/go/src/github.com/algorand/go-algorand/rpcs/blockService.go:355 4 0x0000000004f45e1b in github.com/algorand/go-algorand/rpcs.(*BlockService).handleCatchupReq at /Users/shantkarakashian/go/src/github.com/algorand/go-algorand/rpcs/blockService.go:299 5 0x0000000004f45318 in github.com/algorand/go-algorand/rpcs.(*BlockService).listenForCatchupReq at /Users/shantkarakashian/go/src/github.com/algorand/go-algorand/rpcs/blockService.go:245 ``` ## Test Plan Test is enhanced. <!-- How did you test these changes? Please provide the exact scenarios you tested in as much detail as possible including commands, output and rationale. -->
Remove time ledger tracker as not used.
fix a typo.
Summary In the EvalForIndexer, we don't want to call finalValidation(). This would skip the calculation of the account totals in the state delta, which is a serious issue if it were to be used by algod, but it's perfectly fine for the indexer since it doesn't track any totals and therefore cannot calculate the new totals. Test Plan Test against the indexer.
The evaluator shouldn't ask the ledger who is the creator of an asset multiple times. This PR adds a cache for ledger's responses in cow base.
Timing issues between writing to and reading from the database cause the test to occasionally fail. Make the test more tolerant to such timing issues by polling.
Encoding/Decoding method for ABI Values. Serialization/Deserialization of ABI Types. Make/Get methods for ABI Types/Values. Basic testcases for Encoding/Decoding.
* moving logs e2e to scripts
AVM 1.0 introduces application accounts goal clerk needs to consider this account when creates dryrun request object Added a new option --dryrun-accounts for adding accounts that were rekeyed to an app account (or any other accounts).
Similar to app-accounts test, but tests all sorts of asset txns instead of pays.
Made it nice to write e2e test in python by creating a "goal" module that emulates the convenience of goal, but in python so that structured results come out for easy asserts.
* Enable cost and fee pooling in tealdbg * CR fixes
Fixes found during recent cluster bandwidth testing
## Summary The plan to change the tracker database to maintain the latest 320 rounds for the online accounts only have some other required modification; one of them is the semantics of the `Totals` method: - At this time, it supports retrieving `AccountTotals` for any of the recent 320 rounds. - We need the method to support the `AccountTotals` for the latest round only, as well as provide the circulating supply for latest-320 rounds ( i.e. the circulating supply is a subset of the `AccountTotals` ). Once the database implementation is complete, the database would contain the `AccountTotals` information for the latest round only, plus the online circulation for the past 320 rounds. In order to support that, I've broken up the interface into: - `LatestTotals`, which would return the latest totals as well as the latest round associated with these totals. - `OnlineTotals`, which receive a round number and return the online totals associated with that round. This change is focused around the Ledger interface. Further changes would be required in the accounts update before the high-level goals could be achieved. ## Test Plan Unit tests updated.
Release3.0.1 mergeback manual
lower acct bal so it receives no reward
Return a bit more details on eval failure for app calls.
Start work on Teal 6 (AVM 1.1) vFuture, LogicVersion changed to 6 Support RekeyTo field, and Keyreg transactions
## Summary This PR implements the transaction sync 2.0. More details can be found in the design document and the feature presentation . ## Test Plan Unit tests, e2e tests and performance tests were executed against this branch successfully. ## Reviewer notes This PR is pretty large. Consider reviewing it in browsers other than Safari.
Re-merge go-algorand 3.1.3-beta
* Implement transactions as arguments * Fix indexing and dryrun issue * Add docstring * Satisfy review dog * Fix pointer issue * Fix group command * Rename e2e test * Fix filename variable * Add e2e test * Use tab
## Summary Small change: libboost-math-dev requires just 4 packages to install, while libboost-all-dev requires > 100. Only Debian/Ubuntu distributions provide fine-grained boost packages like this, but should shave a little time off the CI builds. (Our only boost include is boost/math/distributions/binomial.hpp.) ## Test Plan Builds should pass as before. Now that we are no longer using Travis for Linux builds, the side effect of libboost-all-dev installing make and other missing build tools on Travis encountered in #2717 is no longer a concern.
## Summary - Test to make sure RES has the right input before counting line numbers for result size. - Rest RES to empty so that the same output is not recycled in case of an error. - exit 1 in case of an error - Reduce LAST_ROUND from 1200000 to 120 - "Get List of Keys" before getting NUM_IDS_3 otherwise it will recycle old RES value.
## Summary Some e2e tests require a python environment for testing. Unfortunately, setting up that environment adequately similar to the testing environment may not be trivial. This change introduces an interactive mode to the e2e.sh script which stops at the point of running the tests, and allows the user to run the tests from the same testing environment. ## Test Plan No tests needed. Tested the script locally.
go-algorand 3.2.0-beta
## Summary Fix a couple flaws in the new go-e2e tests built ontop of DevMode: * Shutdown the fixture when finished. * Don't run in parallel. * Longer delays / better algorithms to wait for data flushing to complete. * Check for "out of order" keys. ## Test Plan N/A, this is a test.
## Summary The sandbox is not building with dev config using master branch algorand/sandbox#85, complains about libtool not being installed Guessing from this change #3223 Adding libtool to UBUNTU_DEPS in install scripts ## Test Plan Set config in sandbox to my branch `sandbox up dev` It built
## Summary Error from account preloading was shadowed by returning a wrong err variable. This caused subsequent problems in account updates and masked the original failure. ## Test Plan Use existing tests
## Summary This test doesn't work properly, disable it until #3255 addresses any underlying problems.
go-algorand 3.2.1-beta
algobarb
requested review from
tsachiherman,
algojack,
bricerisingalgorand,
egieseke and
onetechnical
Codecov Report
@@ Coverage Diff @@
## rel/stable #3279 +/- ##
==============================================
+ Coverage 47.32% 47.41% +0.08%
==============================================
Files 355 369 +14
Lines 57182 59620 +2438
==============================================
+ Hits 27059 28266 +1207
- Misses 27066 28072 +1006
- Partials 3057 3282 +225
Continue to review full report at Codecov.
|
algojack
approved these changes
Dec 6, 2021
onetechnical
approved these changes
Dec 7, 2021
egieseke
approved these changes
Dec 7, 2021
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Changelog
Goal
goal app method#3228)goal app method#3233)Algod
systo resolve golang 1.17 local build crash (Updatesysto resolve golang 1.17 local build crash #3157)Ledger
indexerLedgerForEvalinterface #2897)ledger.Creatablepublic (Make fields inledger.Creatablepublic #3125)endOfBlock()(ledger: do final validation inendOfBlock()#3132)roundCowBase: uselookup()instead of going to ledger directly (roundCowBase: uselookup()instead of going to ledger directly #3173)Agreement
Catchup
Consensus
DevOps
Indexer
Network
REST API
Teal and tealdbg
Testing
stringfields (Add unit test to avoid newstringfields #3101)ABI
Misc
Protocol Upgrade
This release does not contain a protocol upgrade.
Additional Resources