Yet another case of state reset in State Resolution v2

[wj25czxj47bu6q]

Description

I have encountered yet another room state reset.

I suspect that the reset of m.room.join_rules was the root cause of the 51 m.room.member resets, as the latter would now be rejected events. There is no obvious explanation for the m.room.join_rules reset.

No state events were sent or redacted in the time immediately preceding the state reset. The last state event before the reset was sent approximately 2 hours prior.

Steps to reproduce

1. Use the room as normal.
2. Observe 51 members spontaneously dropped from the room.

Homeserver

arcticfoxes.net

Synapse Version

1.96.0rc1

Installation Method

Debian packages from packages.matrix.org

Database

PostgreSQL 15.4, single server, not ported, restored a very long time ago

Workers

Single process

Platform

• Ubuntu 22.04.3 LTS
• Hetzner EX101
  • Intel Core i9-13900
  • 64 GB DDR5 ECC
  • Dual Samsung PM9A3 1.92 TB NVMe datacenter SSDs in ZFS mirror configuration
  • 1 Gbps symmetric internet connection
• Synapse and PostgreSQL run directly on the host OS, no containerization or virtualization involved.

Configuration

• Presence is enabled.
• The Mjölnir antispam Synapse module is in use.
• Several experimental features are enabled.

Relevant log output

SELECT * FROM current_state_delta_stream WHERE room_id='!frWBWOPyriepLexnsA:arcticfoxes.net' AND stream_id=25794806;

 stream_id |               room_id               |       type        |              state_key              |                   event_id                   |                prev_event_id                 | instance_name 
-----------+-------------------------------------+-------------------+-------------------------------------+----------------------------------------------+----------------------------------------------+---------------
  25794806 | !frWBWOPyriepLexnsA:arcticfoxes.net | m.room.member     | @aldorithms:matrix.org              |                                              | $74_XqgpHw_D18w0Xq0Lrc7_nmBvzDx1UajEn9yKc3Qc | master
  25794806 | !frWBWOPyriepLexnsA:arcticfoxes.net | m.room.member     | @jhs177992f:arcticfoxes.net         |                                              | $rFYpIiFmMMFHYqY_835sCFOEpJVUyLpEDM5Xoqr4uPc | master
  25794806 | !frWBWOPyriepLexnsA:arcticfoxes.net | m.room.member     | @ariyaro:matrix.org                 |                                              | $JqoI5tf1YrjoifsPEdxT67IeLJW6BCAdhIsFgxIJQ9Y | master
  25794806 | !frWBWOPyriepLexnsA:arcticfoxes.net | m.room.member     | @andreus:matrix.org                 |                                              | $CUryaK8LXBZxMsDlVohrahm3LaiBC-uCoHHOot0UWS4 | master
  25794806 | !frWBWOPyriepLexnsA:arcticfoxes.net | m.room.member     | @heliopolis:nitro.chat              |                                              | $utSCExwz0EzMAD5eIpCZ9BeEcinoslMFKTjgLO-H6rU | master
  25794806 | !frWBWOPyriepLexnsA:arcticfoxes.net | m.room.member     | @backpack:matrix.org                |                                              | $cCS3Ry7EFHCXQHTyMrJLratjz0GYRGsyCqU5AkDcnQ4 | master
  25794806 | !frWBWOPyriepLexnsA:arcticfoxes.net | m.room.member     | @tfox42:matrix.org                  |                                              | $93-srvPW852lSXze_r-y5VsDsi32Jf6H0GbXxrI8zFo | master
  25794806 | !frWBWOPyriepLexnsA:arcticfoxes.net | m.room.member     | @houndvoyager:matrix.org            |                                              | $5XMMvwgqdHpX6eyBzamDKpTADycXXvdUhMAIKhtPbTY | master
  25794806 | !frWBWOPyriepLexnsA:arcticfoxes.net | m.room.member     | @mercifulrage:converser.eu          |                                              | $EDFgLxKC6dz2cG7BYk7gO5xY2NNIKKRcMr7bM9XrPyA | master
  25794806 | !frWBWOPyriepLexnsA:arcticfoxes.net | m.room.member     | @crypticangel:pragma-messenger.ch   |                                              | $QG5tHgkzolxZAS0E3auilaPTjZqssp5fFr3NPgiwex0 | master
  25794806 | !frWBWOPyriepLexnsA:arcticfoxes.net | m.room.member     | @coalminecanary:arcticfoxes.net     |                                              | $SG_cN3tHXhg9_7_mRHe5Qt4wFlQO7o8hvoV1XPepkTA | master
  25794806 | !frWBWOPyriepLexnsA:arcticfoxes.net | m.room.member     | @vovafreeman:matrix.org             |                                              | $r9YGaq9q9dbye19fLlO09TO-IC9k6bfOWl-unkJRNNw | master
  25794806 | !frWBWOPyriepLexnsA:arcticfoxes.net | m.room.member     | @qoijjj:arcticfoxes.net             |                                              | $XbGI2ynAn3WC5hGRrmsBmRbW6f3l5mzzyXoDsqp_C00 | master
  25794806 | !frWBWOPyriepLexnsA:arcticfoxes.net | m.room.member     | @mebious:tchncs.de                  |                                              | $zYy_smuirJ0TDeYdC5vgbyQQRxHqxTTF9OTdQ1-cjhs | master
  25794806 | !frWBWOPyriepLexnsA:arcticfoxes.net | m.room.member     | @windystone:matrix.org              |                                              | $Qa-UYRrnev1cof2voq4VrjlmnDmvEn7cF6QgIYomxFA | master
  25794806 | !frWBWOPyriepLexnsA:arcticfoxes.net | m.room.member     | @user0761:matrix.org                |                                              | $4w8S40ugloh8ixEufM0tj-_QvnR3oaiXXqdBIPicZDg | master
  25794806 | !frWBWOPyriepLexnsA:arcticfoxes.net | m.room.member     | @matrixruinedmylife:arcticfoxes.net |                                              | $Q0mg0Le1GBwXh_H2tFLvwCM97Kf-njC0tW5GcqyFVPU | master
  25794806 | !frWBWOPyriepLexnsA:arcticfoxes.net | m.room.member     | @suicidalprincess:nitro.chat        |                                              | $Jv4B2FBIDgxkMR93mc9CNE88aTJqG8NrB79tMeVqhJY | master
  25794806 | !frWBWOPyriepLexnsA:arcticfoxes.net | m.room.member     | @hat:arcticfoxes.net                |                                              | $IDqAum_6ytZX_u0tz-8JglUBjwD39edl329bmExXrzo | master
  25794806 | !frWBWOPyriepLexnsA:arcticfoxes.net | m.room.member     | @iustitia:4d2.org                   |                                              | $86lLB5SWm9xB93vZroSDU4hfjGQ-ZYrJr9fbnE2DZRE | master
  25794806 | !frWBWOPyriepLexnsA:arcticfoxes.net | m.room.member     | @beyiyaf450:matrix.org              |                                              | $DgRnGVumPcveQGtNWGLKrMzBwowHRJwtrRggmA_ndFo | master
  25794806 | !frWBWOPyriepLexnsA:arcticfoxes.net | m.room.member     | @fpmksoba:matrix.org                |                                              | $2GtIIACc89vvMrYb1AKJeF0ky4IgDtsZpsFCy_trBTM | master
  25794806 | !frWBWOPyriepLexnsA:arcticfoxes.net | m.room.member     | @sneber:nope.chat                   |                                              | $9mpOqALVoUK1SQ6BbV4H0wKaiooTXbbHnmAIyDjXlvw | master
  25794806 | !frWBWOPyriepLexnsA:arcticfoxes.net | m.room.member     | @austin:tchncs.de                   |                                              | $zg1FQYQk3pTix5UOY7PRxSCmE6Q-94lbXnpgMbByoys | master
  25794806 | !frWBWOPyriepLexnsA:arcticfoxes.net | m.room.member     | @bhydden:matrix.org                 |                                              | $tp1MSrvb4KOEPtxqtJNQ6bKRMdzOVs7puBihkkI6d_4 | master
  25794806 | !frWBWOPyriepLexnsA:arcticfoxes.net | m.room.member     | @masterrank:matrix.org              |                                              | $GJ4SZsaZuJg0NOS0GauO9DcNX3Qa1ESTSP-C84YbSYQ | master
  25794806 | !frWBWOPyriepLexnsA:arcticfoxes.net | m.room.member     | @w14:matrix.org                     |                                              | $i-ZGHSTUJuRLjw6y7IZVSxAbxYZFi42AgYK_xkzEeOA | master
  25794806 | !frWBWOPyriepLexnsA:arcticfoxes.net | m.room.member     | @cyri:matrix.org                    |                                              | $hxptXTV6_m2O4ZoP0w69HOIc1FQ22GC-UnJUz6d0HwQ | master
  25794806 | !frWBWOPyriepLexnsA:arcticfoxes.net | m.room.member     | @ilove9999:matrix.org               |                                              | $k3C1PKERqlnX3ulqHe5qzpzZKfTVWQrknX1z5avD2Dc | master
  25794806 | !frWBWOPyriepLexnsA:arcticfoxes.net | m.room.member     | @quietlyjaded:matrix.org            |                                              | $NRY6fXAazYFaVa0kLpYWIhy2gfqJW8tVCFyZxVul01E | master
  25794806 | !frWBWOPyriepLexnsA:arcticfoxes.net | m.room.member     | @green_sheep:matrix.org             |                                              | $cGwRLAiQxivw7d3h9uMca5egIkDOhRVgZnD2Gl_zWVk | master
  25794806 | !frWBWOPyriepLexnsA:arcticfoxes.net | m.room.member     | @tisktisk:monero.social             |                                              | $lZ01gtBU50UhZ8sacAtpp6Ia5h6Gd54jON7o6Rxu9mc | master
  25794806 | !frWBWOPyriepLexnsA:arcticfoxes.net | m.room.member     | @loliloli:arcticfoxes.net           |                                              | $iPSlyT37ecv2ZqGH4yOAhkJ6_C51_J-muduqoeSf7PU | master
  25794806 | !frWBWOPyriepLexnsA:arcticfoxes.net | m.room.member     | @chili1803:matrix.org               |                                              | $jUwmF9DuNmdb56KzAP_D364QM7aocojUY-O95j6hx0c | master
  25794806 | !frWBWOPyriepLexnsA:arcticfoxes.net | m.room.member     | @sclanchy:matrix.org                |                                              | $27hLXs_qKe4WIgGqy6GigIV6TuknsBmzDSgBHn4c3TI | master
  25794806 | !frWBWOPyriepLexnsA:arcticfoxes.net | m.room.member     | @faxex:matrix.org                   |                                              | $tz3qjuYJnJTHQBaUotnXXVQOEeoUAyGIQTd-1sdhKXc | master
  25794806 | !frWBWOPyriepLexnsA:arcticfoxes.net | m.room.member     | @ghr89sb:matrix.org                 |                                              | $UgsO3HsBuDnYVpxzpda-7WUH5vh9AujnN7dEcXX7L5A | master
  25794806 | !frWBWOPyriepLexnsA:arcticfoxes.net | m.room.member     | @poisonivy:arcticfoxes.net          |                                              | $pVH8QOPHfNfuv1cILOOSELlOuLAPKbliYYbflMkOg-g | master
  25794806 | !frWBWOPyriepLexnsA:arcticfoxes.net | m.room.member     | @johan942:matrix.org                |                                              | $jjKcZyb7w2DcbYvaKWf5c_052nutBDxcMbM3FJI5V5o | master
  25794806 | !frWBWOPyriepLexnsA:arcticfoxes.net | m.room.member     | @carlfriedrich:tchncs.de            |                                              | $CeMBICUBs6UPB1hpKooo7YjiTsbevuzC7IRjxihiOxU | master
  25794806 | !frWBWOPyriepLexnsA:arcticfoxes.net | m.room.member     | @2nami:matrix.org                   |                                              | $MEz7DkYNjmX1dXW4aJoiGN8kkNnQ1zjmGekNh3fQijU | master
  25794806 | !frWBWOPyriepLexnsA:arcticfoxes.net | m.room.member     | @un32:matrix.org                    |                                              | $JAAIV6FTLNgANKHyi1dhxskos-RngW_0tr44roqbRL4 | master
  25794806 | !frWBWOPyriepLexnsA:arcticfoxes.net | m.room.member     | @ancloadi:matrix.org                |                                              | $Dw7r7MLjd3JqLzzVdHKuXe0eVsO4wXQzNpOUvoCGtAQ | master
  25794806 | !frWBWOPyriepLexnsA:arcticfoxes.net | m.room.member     | @losintos:matrix.org                |                                              | $Viw3Yw_2fyjF6Jnk4-ozYMOq6JWrt2TasbEhagy7jyE | master
  25794806 | !frWBWOPyriepLexnsA:arcticfoxes.net | m.room.member     | @xyhhx:arcticfoxes.net              |                                              | $F6xBiS5WgeyYB7-R4B8eOMKWOD9LA0yNjGj_mDDJEX0 | master
  25794806 | !frWBWOPyriepLexnsA:arcticfoxes.net | m.room.join_rules |                                     | $AxE3gEzTLPT_hy7ML1MJKsX7DI6XkqujWP50LTDBODM | $P2FwR8kpzW1LDIZU2lKRc_wmghbXlIQZpwvE5D82d5Q | master
  25794806 | !frWBWOPyriepLexnsA:arcticfoxes.net | m.room.member     | @dahustlah:matrix.org               | $8GW8hJvAHAGPTQIsPysF_mePLpd-GtZyqYIb4VzfoiU | $aGradtN72do5IOaG1dvRExtW4PAwFfjdlTNZVyOP6sQ | master
  25794806 | !frWBWOPyriepLexnsA:arcticfoxes.net | m.room.member     | @gvl7bcq6dn2xrbl:matrix.org         | $_s6-w6-v0tE3-tYkWYQ1bbukUs844bn_9bUkttYCXF0 | $iNV97F-d1DnI0dGqj7L23a2pTtRmvSCNPpfDfGbGbg4 | master
  25794806 | !frWBWOPyriepLexnsA:arcticfoxes.net | m.room.member     | @masayuki:wired.rehab               | $yMVmIFtT_hpeMh-9GLSdY84NrNYSRhn8rUX2Z2AD5_g | $egJIEy0PhxdUfdDCAPAVJDSrIYTLYtdAIIWCQ3zvsNA | master
  25794806 | !frWBWOPyriepLexnsA:arcticfoxes.net | m.room.member     | @matchboxbananasynergy:matrix.org   | $A4R_VVl2TaN53EWiLBoipPyigkEZodgWOiceeieshaQ | $s-1rvX3ldLsL5Azxj7mSjkhWxboqaEuQiJlMCglMIYk | master
  25794806 | !frWBWOPyriepLexnsA:arcticfoxes.net | m.room.member     | @skull_gap:archoslinux.cz           | $MX2EhPV0TYzMxegHKJX1U1KKvqTrAp8vwS5DwqqDlNw | $_0SGO3HGR2RgYWYAxczIkwXzCl7e3PLuZgrO_HTgj6Y | master
  25794806 | !frWBWOPyriepLexnsA:arcticfoxes.net | m.room.member     | @slate-module:matrix.org            | $xM6Y-mUyvW9yOZf_lODJOdqLI0Ao1nYuaYQYonysi84 | $2IUW5LONVOx_k1I5nxRv9AuF8BFAp5NL8I_KxFa16Zk | master
(52 rows)

Anything else that would be useful to know?

In addition to arcticfoxes.net documented above, this state reset was confirmed via the C‑S API on matrix.org (Synapse 1.96.0rc1) and a third server (Synapse 1.95.1). I do not know the exact details of their deployments.

[thestinger]

This keeps happening for the GrapheneOS rooms, which were among the largest Matrix communities until this happened. It's always the same symptom of join rules, power levels, etc. resetting and then all users who joined since the join rules it resets to being dropped out of the room with the room bricked in Element where they can't send, receive or leave without clearing cache.