Releases · ethereum/tests

06 May 08:42

winsvega

faf33b4

Latest

The recent tests changes:

Fix the issue in blockchain tests with beacon root precompile, cosmetic updates
EIP7610 legacy tests fix
EIP tests for BLS

Announcement

New docker instructions for all t8n supported clients: https://github.com/ethereum/retesteth/tree/develop#docker-instructions
The test process diagram: https://tinyurl.com/y525ezs7
Reminder of a brand new web interface for test debugging: http://retesteth.ethdevops.io/web/
Tests stats on latest builds: http://retesteth.ethdevops.io/
(implement test RPC support or t8ntool protocol to apply for the page. https://ethereum-tests.readthedocs.io/en/latest/t8ntool-ref.html)
Previous release link: https://github.com/ethereum/tests/releases/tag/v13.2

Test format

Tests Fillers

GeneralStateTests/BlockchainTests Filled

Test Regeneration

Update all:

Documentation
PR #1356 (fix typos)

TESTS

Fix legacy tests to revert on create collision with non empty storage dest (EIP7610)
PR #1367
**Fix the blockchain tests logic with beacon root in genesis **
PR #1357
Add more ecpairing test vectors
PR #1358
Update bad opcode tests for Cancun
PR #1362
Update pyspec tests, rename Merge to Paris
PR #1363

EIPTests/EOFTests

Reintroduce BLS precompile tests
PR #1366
Fix and update EOF tests
PR #1349
More EOF validation tests from evmone
PR #1364

Upcoming changes:

Convert .json .yml yo .py
Pyspecs filling support

Assets 2

13 Mar 14:04

winsvega

v13.2

066a587

Tests update 13.2 Cancun tests

The recent tests changes:

Cosmetic update
Insert beacon precompile into bc tests's pre >= Cancun
Fix genesis excessBlobGasUsed for first block excessBlobGasUsed to be = 0 or x calculation
Remove empty storage records from pre state

Announcement

Test format

Tests Fillers

GeneralStateTests/BlockchainTests Filled

Insert beacon precompile into tests pre if not exist for BlockchainTests >= Cancun

"0x000f3df6d732807ef1319fb7b8bb8522d0beac02" : {
                "balance" : "0x00",
                "code" : "0x3373fffffffffffffffffffffffffffffffffffffffe14604d57602036146024575f5ffd5b5f35801560495762001fff810690815414603c575f5ffd5b62001fff01545f5260205ff35b5f5ffd5b62001fff42064281555f359062001fff015500",
                "nonce" : "0x01",
                "storage" : {
                }
            },

Test Regeneration

Update all:

Pyspec update: remove beacon contract from tests, remove empty storage records
PR #1345
Refill all blockchain tests, add beacon root on cancun
PR #1351
Documentation

TESTS

--
PR --

EIPTests/EOFTests

update for relaxed stack validation
PR #1334
**Move EOF creation tests into EOF test format **
PR #1337

Cosmetic

Move random blockhash calling state tests to bc format
PR #1354
Change exception name in PYSPEC
PR #1342
Remove empty storage keys (0x00) in pre state
PR #1347

Upcoming changes:

Convert .json .yml yo .py
Pyspecs filling support

Assets 2

14 Feb 14:07

winsvega

v13.1

853b1e0

Tests update 13.1 Cancun tests

The recent tests changes:

Rename Merge to Paris (note: Pyspecs tests still have Merge keyword)
Remove empty accounts from post Paris
Update cancun rules, state test format, update pyspecs vectors
Update filled state test format

Announcement

Test format

Tests Fillers

Pyspec tests folder moved from /src/BlockchainTestsFiller/Pyspecs to /src/GeneralStateTestsFiller/Pyspecs
and the generated tests moved accordingly. as .py tests now could have state and blockchain tests.
--filltests flag will generate all state tests out of .py files
--filltests --fillchain flag will generate all blockchain tests out of .py files.

GeneralStateTests/BlockchainTests Filled

rename forkname in all of the tests: "Merge" => "Paris"
empty accounts (zero balance, no code, zero nonce, with or without storage) in pre state are not allowed post Paris.

In state tests remove blockchain related fields:

 "currentBeaconRoot" : "0x0000000000000000000000000000000000000000000000000000000000000000",
 "currentWithdrawalsRoot" : "0x56e81f171bcc55a6ff8345e692c0f86e5b48e01b996cadc001622fb5e363b421",`
 "previousHash" : "0x5e20a0453cecd065ea59c37ac63e079ee08998b6045136a8ce6635c7912ec0b6"

currentBeaconRoot although not set, the value could still be found in pre state in beacon root contract.
(ethereum/execution-spec-tests#413)

In blockchain tests if invalid block rlp is unreadable, or supposed to fail during rlp parsing, rlp_decoded field no longer provided

Test Regeneration

Update all:

All tests regeneration with Cancun rules
PR #1332
PR #1333 (PYSPEC UPDATE)
Adapt tests and remove empty accounts from state/bc tests post Paris/Merge
PR #1335
PR #1336
Documentation
PR #1323

TESTS

Update the TestEachOpcode template for Cancun
PR #1320
Fix genesis blobgasused params
PR #1328

EIPTests/EOFTests

Update EOF tests to not contain unreachable code sections
PR #1319

Cosmetic

---
PR ---

Upcoming changes:

Convert .json .yml yo .py
Pyspecs filling support

Assets 2

02 Nov 09:44

winsvega

v13

428f218

Tests update 13 Cancun tests

The recent tests changes:

Regenerate all tests using cancun rules
Move Cancun eip tests from EIPTests folder into normal test folders
Test diagram updated: https://tinyurl.com/y525ezs7

Announcement

Test format

New test folder GeneralStateTests/Cancun
New test folder /src/GeneralStateTestsFiller/Cancun

Tests Fillers

defining solidity yul now allows to switch optimisations and select evm version:
--evm-version shanghai, use default solc optimisations

      code: |
        :yul shanghai optimise {
          sstore(0, 1)
      }

--evm-version london, no optimisations (default)

      code: |
        :yul london {
          sstore(0, 1)
      }

This is because each new version of solc introduce default optimisations that change the compiled code bytes. to avoid this affecting the tests by default solc optimisations in tests are disabled

GeneralStateTests/BlockchainTests Filled

Invalid blocks are now transcripted. new format of invalid bc block:

{
                "blocknumber" : "3",
                "chainname" : "default",
                "expectException" : "InvalidGasLimit2",
                "rlp" : "0x...",
                "rlp_decoded" : {
                    "blockHeader" : {
                             ...
                    },
                    "transactions" : [
                        {
                               ...
                        }
                    ],
                    "uncleHeaders" : [
                        {
                               ...
                        }
                    ],
                    "withdrawals" : [
                    ]
                }
}

Keep in mind that this are debug info fields: blocknumber, chainname, expectException

Test Regeneration

Update all:

All tests regeneration with Cancun rules
PR #1297
PR #1312
Documentation
PR ---

TESTS

Tests for contract creation
PR #1306
Regenerate transaction sequence into invalid blocks
PR #1316
Move Cancun tests from EIP Tests
PR #1313

EIPTests/EOFTests

Pyspecs Cancun tests
PR #1299 Update Pyspecs to 1.0.5

Cosmetic

Fix fillers json with missing commas
PR #1310

Upcoming changes:

Support pyspec .py test filling with different clients using retesteth
Nimbus --vmtrace support

Assets 2

19 Sep 08:12

winsvega

v12.4

6613563

Tests update 12.4 Preparations to fill Cancun

The recent tests changes:

Add Cancun blob fields and transactions to json tests
Update pyspec tests

Announcement

New docker instructions for all t8n supported clients: https://github.com/ethereum/retesteth/tree/develop#docker-instructions
The test process diagram: http://tinyurl.com/2ctak8yn
Reminder of a brand new web interface for test debugging: http://retesteth.ethdevops.io/web/
Tests stats on latest builds: http://retesteth.ethdevops.io/
(implement test RPC support or t8ntool protocol to apply for the page. https://ethereum-tests.readthedocs.io/en/latest/t8ntool-ref.html)
Previous release link: https://github.com/ethereum/tests/releases/tag/v12.3

Test format

Tests Fillers

new cancun fields in env section

    parentExcessBlobGas: 0x1000
    parentBlobGasUsed: 0x2000
    currentBeaconRoot: 0x45a915e4d060149eb4365960e6a7a45f334393093061116b197e3240065ff2d8

withdrawals root explicitly exported in state tests

    "currentWithdrawalsRoot" : "0x56e81f171bcc55a6ff8345e692c0f86e5b48e01b996cadc001622fb5e363b421",

new lllc instruction

             [[0]] (BLOBHASH 0)

new transaction type (blob transaction)

  transaction:
    data:
        - data: :label declaredKeyWrite :raw 0x00
          accessList:
          - address: 0x095e7baea6a6c7c4c2dfeb977efac326af552d87
            storageKeys:
            - 0x00
            - 0x01

    maxFeePerGas: '0x12A05F200'
    maxPriorityFeePerGas: '2'
    gasLimit:
    - '4000000'
    nonce: '0'
    to: 095e7baea6a6c7c4c2dfeb977efac326af552d87
    value:
    - '100000'
    secretKey: "45a915e4d060149eb4365960e6a7a45f334393093061116b197e3240065ff2d8"
    maxFeePerBlobGas: '10'
    blobVersionedHashes:
        - "0x01a915e4d060149eb4365960e6a7a45f334393093061116b197e3240065ff2d8"

GeneralStateTests/BlockchainTests Filled

new block header fields

                    "blobGasUsed" : "0x00",
                    "excessBlobGas" : "0x00",
                    "parentBeaconBlockRoot" : "0x01a915e4d060149eb4365960e6a7a45f334393093061116b197e3240065ff2d8"

Test Regeneration

Update all:

Refill solc/yul tests using solc evm version flag
PR #1280 PR #1279 PR #1278 PR #1277
PR #1276 PR #1275 PR #1274 PR #1273
PR #1272 PR #1271 PR #1270 PR #1269
PR #1268 PR #1267 PR #1266 PR #1265
PR #1264 PR #1263 PR #1262 PR #1261
PR #1260 PR #1259 PR #1258 PR #1257
PR #1256 PR #1255 PR #1254 PR #1253
PR #1252 PR #1251 PR #1250
Documentation
PR #1283
Update Pyspecs tests
PR #1288
PR #1292
State tests transactions from non existing account
PR #1290
PR #1291

EIPTests/EOFTests

Pyspecs Cancun tests
PR #1282
PR #1288
Refill EIPTests with new blob names
PR #1224
Update tests according to Mega-EOF spec
PR #1238
PR #1285

Cosmetic
PR #1247
PR #1248
PR #1281
PR #1289

Upcoming changes:

Fill all tests with cancun rules
Support pyspec .py test filling with different clients using retesteth
Nimbus --vmtrace support

Assets 2

30 Jul 12:56

winsvega

v12.3

06e2767

Tests update 12.3 EIP tests and Pyspecs support

The recent tests changes:

Update .py tests to the latest version (but not Cancun)
Update EIPTests to new specs and add more test cases

Announcement

Test format

Tests Fillers

GeneralStateTests/BlockchainTests Filled

Test Regeneration

Update all:

Documentation
PR #1216
PR #1223
PR #1241
Refill invalid tx state tests into invalid block bc tests
PR #1225
Update Pyspecs tests
PR #1240
PR #1245

EIPTests/EOFTests

More EIP1153 (Transient storage) tests
PR #1220
PR #1235
PR #1237
CALLF Stack Overflow test
PR #1214
Update EOF tests
PR #1219 (Forbid DELEGATECALL EOF → legacy)
MCOPY tests from evmone
PR #1229
PR #1234
BLOBHASH tests from nimbus
PR #1226
PR #1243

Cosmetic

Fix nimbus test generation exception
PR #1242
Mark bigint in bc withdrawals tests
PR #1213
PR #1232
More test exception names fix
PR #1230
PR #1225
Nimbus version string to valid json
PR #1231

Upcoming changes:

Support pyspec .py test filling with different clients using retesteth
Nimbus --vmtrace support

Assets 2

27 Apr 19:52

winsvega

v12.2

b25623d

Tests update 12.2: EOF Tests

The recent tests changes:

Update EOFTests (EIPTests)

Announcement

Test format

Tests Fillers

GeneralStateTests/BlockchainTests Filled

Test Regeneration

Update all:

Documentation

EIPTests/EOFTests

EIP-5450: State Tests And EOF Tests
PR #1163 (EOF Stack validation)
EIP-4750: State Tests And EOF Tests
PR #1115 (EOF Functions)
EIP4200: EOF - Static relative jumps
(3540-3670-4200
PR #1103 (EOF Jumps)
EIP3540: EOF - Ori's tests
PR #1206 (change to Cancun frokword)

Cosmetic

Change ttWrongRLP tx exception
PR #1211

Upcoming changes:

SSZ tests
Implement malicious rlp tests in StateTest or BlockchainTest format (need more debugging with t8n)
Besu retesteth RPC support (web vmtracing support)
Random state test script

Assets 2

13 Apr 18:16

winsvega

v12.1

04f338a

Tests update 12.1: EIPTests separate, Pyspecs

The recent tests changes:

Move EIPTests into a separate folder so not to confuse
Introduce Pyspecs test vectors and support .py fillers

Announcement

Test format

Tests Fillers

new folder /src/BlockchainTestsFiller/Pyspecs
new folder /src/EIPTestsFiller/StateTests
new folder /src/EIPTestsFiller/BlockchainTests

GeneralStateTests/BlockchainTests Filled

new folder /BlockchainTests/Pyspecs
new folder /EIPTests/StateTests
new folder /EIPTests/BlockchainTests/StateTests (EIP state tests generated in blockchain form)

Test Regeneration

Update all:
PR #1197 (Update Transaction and Difficulty tests)
PR #1203 (Update Legacy Tests)

Documentation
Pyspec test vectors
PR #1209
Move EIPTests into separate folder from all of the common tests
PR #1192
EIP1153 - Transient Storage Tests
PR #1196
More withdrawals tests
PR #1198
PR #1199 (Merge era comes block with withdrawals body)
Fix typos
PR #1201
PR #1202

EIPTests

Update EIP-3540 and EIP-3670 Tests to Cancun
PR #1193 (EIPTests)
EOFTests add exception checks
PR #1208

Upcoming changes:

Implement malicious rlp tests in StateTest or BlockchainTest format (need more debugging with t8n)
Besu retesteth RPC support (web vmtracing support)
Random state test script

Assets 2

15 Mar 12:14

winsvega

v12

291118c

Tests update 12: Shanghai tests

The recent tests changes:

Regenerate all test vectors with Shanghai rules
Introducing a shortened vectors for EOF code validation
Added nimbus to web interface http://retesteth.ethdevops.io/web

Announcement

The test process diagram: http://tinyurl.com/2ctak8yn
Reminder of a brand new web interface for test debugging: http://retesteth.ethdevops.io/web/
Tests stats on latest builds: http://retesteth.ethdevops.io/
(implement test RPC support or t8ntool protocol to apply for the page. https://ethereum-tests.readthedocs.io/en/latest/t8ntool-ref.html)
Previous release link: https://github.com/ethereum/tests/releases/tag/v11.3

Test format

Tests Fillers

new folder /src/EOFTestsFiller
new folder /src/GeneralStateTestsFiller/Shanghai

GeneralStateTests/BlockchainTests Filled

new folder EOFTests
new folder GeneralStateTests/Shanghai

Test Regeneration

Update all:
PR #1164 (generate all test vectors with Shanghai rules)
PR #1180

Documentation
PR #1156 (fix typos)
PR #1149 (warning about test validity)
PR #1150 (EOFTests suite docs)
PR #1167
Fix test bugs/typos
PR #1155 (bcShanghaiExample)
PR #1158 (0 withdrawals hash)
PR #1161 (transaction tests src path)
PR #1168
PR #1191 (lower gas limit in underflow test)
Move Shanghai tests from experimental folder to Shanghai tests
PR #1180
EOF Test suite
PR #1159
EIP-3651 tests (warm coinbase)
PR #1082
update ORI'S badOpcode and diffPlaces tests
PR #1166
Fill all remaining tests to Shanghai
PR #1188
PR #1187
PR #1186
PR #1190
Double call contract that has suicide after initial suicide is called
PR #1169
PR #1181
PR #1182 (upgrade the test)

Upcoming changes:

Support pyspec .py fillers
Organize EIPTests
Implement malicious rlp tests in StateTest or BlockchainTest format (need more debugging with t8n)
Besu retesteth RPC support (web vmtracing support)
Random state test script

Assets 2

24 Jan 10:16

winsvega

v11.3

69c4c2a

Tests update 11.3: EIP tests, prepare to Shanghai

The recent tests changes:

Introduces new folder EIPTests in GeneralStateTests that have tests for potential EIPS to be included in Shanghai
Tests in EIPTests folders are not mandatory to pass and can contain errors or outdated specs. Those are early development stage tests that will be transformed into proper tests once the specs are finalized.
This allows us to have coverage for future EIPs that not necesseraly implemented by all clients just yet.

Announcement

Test format

New test ID tracker system: /src/README.MD contains id tags that can be found in tests that cover the provided test vectors

Tests Fillers

new folder /src/GeneralStateTestsFiller/EIPTests
new folder /src/GeneralStateTestsFiller/EIPTests/stEOF
new folder /src/BlockchainTestsFiller/EIPTests

GeneralStateTests/BlockchainTests Filled

new folder GeneralStateTests/EIPTests
new folder GeneralStateTests/EIPTests/stEOF
new folder BlockchainTests/EIPTests
new folder BlockchainTests/GeneralStateTests/EIPTests
new folder BlockchainTests/GeneralStateTests/EIPTests/stEOF

Test Regeneration

Update all:
Regenerate basefee bug in EIPTests
PR #1141
Update EIPTests to latests specs
PR #1147
PR #1152

Documentation
PR #1121 (Example of new debug options with retesteth)
PR #1128
Shanghai transition tests
PR #1123
PR #1134 (switch to timestamp based)
**Shanghai Withdrawals tests **
PR #1118 (example)
PR #1124
PR #1129
PR #1137 (basefee fix)
Rare bug with refund on Create OOG during deployment
PR #1111
Move fork+xxx EIP tests to a separate folder
PR #1110
EIP-3855 PUSH0 instruction tests
PR #1033
EIP-3860 Update tests to new specs
PR #1105 (create/2 init code size)
PR #1012
PR #1127
PR #1125
EIP-3860 Transaction tests
PR #1131
Remove JUMP tests from EIP3540
PR #1107
PR #1132 (fix 3540 tests to updated specs)
PR #1126 (EOF code deployment)
EOF contract deployment tests from ORI
PR #1136
EIP3670 tests
PR #1135

Upcoming changes:

EOFTest suite
Implement malicious rlp tests in StateTest or BlockchainTest format (need more debugging with t8n)
Besu retesteth RPC support (web vmtracing support)
Random state test script

Assets 2

Releases: ethereum/tests

Tests update 13.3 Cancun tests

The recent tests changes:

Announcement

Test format

Tests Fillers

GeneralStateTests/BlockchainTests Filled

Test Regeneration

Upcoming changes:

Tests update 13.2 Cancun tests

The recent tests changes:

Announcement

Test format

Tests Fillers

GeneralStateTests/BlockchainTests Filled

Test Regeneration

Upcoming changes:

Tests update 13.1 Cancun tests

The recent tests changes:

Announcement

Test format

Tests Fillers

GeneralStateTests/BlockchainTests Filled

Test Regeneration

Upcoming changes:

Tests update 13 Cancun tests

The recent tests changes:

Announcement

Test format

Tests Fillers

GeneralStateTests/BlockchainTests Filled

Test Regeneration

Upcoming changes:

Tests update 12.4 Preparations to fill Cancun

The recent tests changes:

Announcement

Test format

Tests Fillers

GeneralStateTests/BlockchainTests Filled

Test Regeneration

Upcoming changes:

Tests update 12.3 EIP tests and Pyspecs support

The recent tests changes:

Announcement

Test format

Tests Fillers

GeneralStateTests/BlockchainTests Filled

Test Regeneration

Upcoming changes:

Tests update 12.2: EOF Tests

The recent tests changes:

Announcement

Test format

Tests Fillers

GeneralStateTests/BlockchainTests Filled

Test Regeneration

Upcoming changes:

Tests update 12.1: EIPTests separate, Pyspecs

The recent tests changes:

Announcement

Test format

Tests Fillers

GeneralStateTests/BlockchainTests Filled

Test Regeneration

Upcoming changes:

Tests update 12: Shanghai tests

The recent tests changes:

Announcement

Test format

Tests Fillers

GeneralStateTests/BlockchainTests Filled

Test Regeneration

Upcoming changes:

Tests update 11.3: EIP tests, prepare to Shanghai

The recent tests changes:

Announcement

Test format

Tests Fillers

GeneralStateTests/BlockchainTests Filled

Test Regeneration