New survey page based on Next.js v13 using the app router and server actions

[henrycatalinismith]

Combining all our work from #1636, #1719 and #1738. Targeting the branch for #1719 as the code changes here depend on merging the Next v13 upgrade to main. Once #1719 ships we can update the base branch of this PR to main.

[henrycatalinismith]

Nice. The playwright failures here make sense. It's the result of combining the new playwright tests we wrote at code camp with the server action version of the survey form.

[henrycatalinismith]

First little burst of activity since the hackathon produced 5459f24. The context here is I'm trying to get the survey page playwright tests green, and the executive summary is that this commit gets 2/5 of them green and then skips the remaining three. More details below.

1. ✅ submits responses
   This one was failing because it was trying to sign as the logged-in user. That's a detail we agreed to cut from the initial scope of the app router migration of this feature, so the reason it doesn't work is simply that it's not implemented. Switching to anonymous signing is fine here as a compromise because the test isn't about the signature.

2. ✅ submits email signature
   This one was failing because I had moved the survey data fetching to <SurveyForm />, which gets its API client instance for the data fetch from <ClientContext />, where it's hardcoded as BrowserApiClient. I think playwright noticed the error when the server render attempted to use the client-side API client and failed the test for it or something like that. Whatever the specifics, fetching this in the server render fixes the test and was what I originally intended to do anyway.

3. ❌ submits user signature
   Not possible to make this pass until the authentication code is migrated to the app router. Have left the test in with .skip() on it because it's not a helpful reason for the build to fail.

4. ❌ submits anonymous signature
   Actually not sure why this one is failing. Suspect there's something subtly different about the server action requests & responses. Next on the list. Skipping for now for no good reason really, just want the peace of mind of a nice green build.

5. ❌ preserves inputs on error
   The final boss. Skipping for the same reason as 4.

[henrycatalinismith]

92e0d10 gets the submits anonymous signature test enabled and passing.

My first iteration of the app router thing here was actually mega sloppy about the server component / client component boundary. Now there's a clear & coherent boundary where the <Page /> components in the src/app directory are server components that handle data fetching server side and then hand off to client components from the src/features/survey directory.

Tidying up that boundary line was enough to get this test passing.

[henrycatalinismith]

6964e23 has done two things: gotten the preserves inputs on error playwright test passing, and broken a bunch of others. The key changes here that made the test pass are as follows.

• Tweak the server action in features/surveys/actions/submit.ts so that it returns a value denoting the outcome of the submit attempt.
• Remove the submitted page and instead render the success message on the same URL as the form when the submit attempt's outcome is success. I tried to avoid this because I prefer it when people can refresh the success page. But the redirect necessary to enable it doesn't seem to fit within the new server actions philosophy.
• Install canary versions of react and react-dom in order to be able to use useFormState. Seems to be simply impossible to do error handling under this new paradigm without this hook, and it's not shipped yet.
• Update to version 14 of Next.js. Didn't love that this was a requirement, but it just doesn't work without it. You get a Converting circular structure to JSON error during yarn tsc --noEmit without it. Seems like some aspect of the canary versions of react and react-dom upsets TypeScript otherwise.

Bit of a two steps forward one step back commit given the number of new playwright test failures it introduces and the worrying dependency on canary versions of key dependencies. But! BUT! We now finally have a decently clear view here of how it'll probably look to write this kind of code at Zetkin a little while in the future once all this bleeding edge stuff goes mainstream. And god help me I think I actually like it!

[richardolsson]

I'm just having a look at the preview deployment of this, and I'm missing the "What is this question?" question that should have two options. Are multi-choice questions not implemented? Or maybe it's because there is no response_config (should default to radio buttons)?

What's being rendered on the preview

The API response for this survey

{
   "data" : {
      "access" : "open",
      "allow_anonymous" : true,
      "callers_only" : false,
      "campaign" : null,
      "elements" : [
         {
            "hidden" : false,
            "id" : 1,
            "question" : {
               "description" : "Answer whatever",
               "options" : [
                  {
                     "id" : 1,
                     "text" : "Option 1"
                  },
                  {
                     "id" : 2,
                     "text" : "Option 2"
                  }
               ],
               "question" : "What is this question?",
               "required" : false,
               "response_config" : {},
               "response_type" : "options"
            },
            "type" : "question"
         },
         {
            "hidden" : false,
            "id" : 2,
            "text_block" : {
               "content" : "With some instructions",
               "header" : "This is a text block"
            },
            "type" : "text"
         },
         {
            "hidden" : false,
            "id" : 3,
            "question" : {
               "description" : null,
               "question" : "Is this a free text question?",
               "required" : false,
               "response_config" : {
                  "multiline" : true
               },
               "response_type" : "text"
            },
            "type" : "question"
         }
      ],
      "expires" : null,
      "id" : 1,
      "info_text" : "This is a survey",
      "org_access" : "sameorg",
      "organization" : {
         "id" : 1,
         "title" : "My Organization"
      },
      "published" : "2024-02-22T06:16:19.016317",
      "signature" : "allow_anonymous",
      "title" : "A very open survey"
   }
}

[henrycatalinismith]

Good spot, I’ll get that sorted this evening. Will also have another look at the playwright thing. The canary version of react-dom is what’s behind all the failing tests so I’m keeping an eye open for new versions in case one fixes that.

[henrycatalinismith]

Found myself a moment to test a full survey containing all the possible options. Here's a screen recording of filling out an example submission.

submission.mov

Pay particular attention to the checkbox input values I gave at 00:20. I selected both.

Now here's me browsing the result. Look how many checkbox values are listed: only one!

result.mov

So yeah there's a data-loss bug here! Here's the full JSON dump of the submission data that's stored in the database.

{
	"data": {
		"id": 51,
		"respondent": {
			"id": 2,
			"email": "testadmin@example.com",
			"first_name": "Angela",
			"last_name": "Davis"
		},
		"organization": {
			"id": 1,
			"title": "My Organization"
		},
		"survey": {
			"id": 7,
			"title": "Kitchen Sink Survey"
		},
		"submitted": "2024-03-29T15:09:00.530122+00:00",
		"responses": [
			{
				"question_id": 22,
				"response": "This is the answer to the first question."
			},
			{
				"question_id": 23,
				"response": "And here's an answer to question two.\r\n\r\nIt's very long."
			},
			{
				"question_id": 24,
				"options": [
					179
				]
			},
			{
				"question_id": 25,
				"options": [
					181
				]
			},
			{
				"question_id": 26,
				"options": [
					182
				]
			}
		]
	}
}

[henrycatalinismith]

Fixed in 3bc0f5a.

fixed.mov

It was a mistake not to pass the original FormData object through to prepareSurveyApiSubmission(), I think. That Object.fromEntries([...formData.entries()]) conversion we were doing on the data before passing it to the function was where the data loss occurred. If you do e.g. Object.fromEntries([["a", 1], ["a", 2]]), you get { a: 2 } as the outcome. So formdata is too funky to distill neatly down to a plain JS object, cos checkbox values are represented as duplicate keys with different values.

My fix in 3bc0f5a passes the actual FormData object instead. This enables the function to use the built-in getAll() method to handle multiple values with the same key.

[richardolsson]

Great work finding and fixing this @henrycatalinismith! It would be great to have playwright tests testing this too.

[henrycatalinismith]

Done in dfc47a6!

I also created a separate throwaway branch without the fix from 3bc0f5a and then tried these new playwright assertions there just for maximum certainty. Here's how they fail when the bug is actually present.

[henrycatalinismith]

Quick side-by-side visual comparison of the old survey vs the new one.

Old	New

[henrycatalinismith]

Here's a full completion of the members' survey followed by a visual review of the submission in the organizer UI. Did this while halfway out the door with the kids so may have missed something, but I thought it seemed 100% okay.

test.mov

[richardolsson]

Awesome @henrycatalinismith! Is there currently anything testing the hidden: false flag?

[henrycatalinismith]

Fixed one of these just now. Presumably related to the TypeScript upgrade!

[richardolsson]

The orgId in both of these places should be survey.organization.id (which may or may not be the organization that we're currently working within). You should be able to get it using useSurvey().

See #1941 for related changes (but we refrained from making changes to the URL there to avoid merge conflicts).

[henrycatalinismith]

This looks familiar!!! Did #1936 turn out to be the tip of the iceberg? I'll get this sorted right away.

[henrycatalinismith]

0265ec7