Improving the moderation workflow in large conversations

[ebarry]

📬 I'm posting this discussion on behalf of a powerful facilitation partner who runs national scale conversations.

What we mean when we say moderation

At the outset, this facilitator describes themselves as a "strong moderator" , which they define as:

1. Pre-pre-launch: invest in a framing process to observe and document distinct position statements that together span the breadth of the topic including minority positions.
2. Pre-launch: load well-crafted seed statements associated with each position into the Polis conversation
3. Post-launch: do not accept low quality or duplicate statements.

This facilitator does not ascribe to the permissive moderation mode of "let the system handle the noise" which is often recommended for large conversations, because that approach does not protect the experience of the "next participant" in terms of:

1. the damage to the next participant in receiving the burden of reading other participant's unmoderated expressions
2. the damage to the quality and reliability of the outcome if the Polis conversation itself failed to disrupt the dynamics of the "tyranny of the majority" in the form of overwhelming communication from well-funded lobbies, or failed to hold a space where all can participate with their dignity intact.

Current manual workflow for moderating large conversations

Here is my understanding of the manual workflow that this national-scale facilitator uses for moderation:

• The lead moderator manually copies out each participant-submitted statement into a Google Spreadsheet
• Manually add thematic coding
• Highlight the statements that are accepted into the Polis, then go back to the Polis and accept these statements
• As submitted statements balloon, orient other team members to the thematic coding process and to assessing similarity in statements to decide what is a "duplicate" and so doesn't need to be added
• Continue to grow the team size to meet the moderation burden

Problem:

This workflow is maxxed out. It's expensive in terms of human time and inefficient.

Suggested solution:

Not sure. Improve Polis? Use a third party site for processing the similarity of submitted statements?

Alternative suggestions:

Um...

Additional context:
Please add any other context or screenshots about the feature request here.

I will add a link to their existing Google Spreadsheet if i can get one

[ebarry]

Update from the same facilitator:

The moderation policy I use is usually based on the following:

• On topic
• Makes [grammatical] sense, the moderators can understand it/know what is meant, the statement is not too ambiguous (different moderators read it in different ways)
• Not too similar to statements already moderated into the Polis. Opposite statements are not usually needed.
• Not abusive, vulgar, etc. (these are extremely rare to date - in fact I can’t remember rejecting a statement on these grounds)
• My moderation teams do sometimes rewrite poorly written statements that raise new issues.

Request from the facilitator:

Ways of reducing the load on moderators involved making any of these assessments or carrying out these tasks (e.g. the ability to edit a statement) would help. We are about to start trying some specialist qualitative analysis tools to see if they are useful for moderation and then also for analysis when we combine metadata and statements.

More thoughts from the facilitator

Strong moderation is important in order to not place too much burden on participants. There's been plenty of feedback that people get bored assessing randomly assigned ’tweets’ or that assessing even high quality statements is really taxing when they are not part of a narrative.

I’ve wondered about moderation transparency and accountability: should people be able to see what has been submitted, which statements have been accepted and reject, and the reasons for rejection. Deliberative democracy places a premium on reason-giving. On the other hand, no one has ever contacted me about whether their statement was accepted in all the Polis discussions that I’ve moderated so this wouldn’t be on my list of priorities.

[metasoarous]

We may wish to discuss "Opposite statements are not usually needed" more; I think it's actually really valuable for interpretability of reporting of representative comments that it be allowable for framing to be presented from both sides. Otherwise, you can end up with a situation where a representative comment is the same, but have to look at the pie chart to see whether the group agrees or disagrees. It's nice to be able to just say "this group agreed with these statements; the other group agreed with this other set of statements".

[ebarry]

Response from the facilitator:

Re opposite statements, I agree with Chris … but I also disagree! I want each participant to consider as many statements as possible (all would be amazing) - this maximises cognitive diversity and the possibility of new framings/amazing statements. The greater the number of statements, the less chance of achieving this goal. If you need to be economical with statements, that knocks out opposites most of the time.

[ebarry]

Hi everyone,
Two specific requests just came in from this facilitator:

1. Remove the 100 statement display limit from the unmoderated list - we tend to moderate statements in the order they are received but it’s impossible to moderate a statement that isn’t displayed.
2. Include the statement ID number - this makes referencing statements easier.

[metasoarous]

The 100 statement display limit is very important to maintaining a responsive UI. With some of the larger conversations we ran without this limit, the UI become unresponsive, making it impossible to moderate when there are many hundreds or thousands of comments pending. This could either be solved by

• (easiest solution) reversing the order comments are shown in (oldest first instead of newest first) in the interface
• adding pagination, which is a little more work, but would offer more control

[patcon]

@metasoarous third option might be a "load more" button at the bottom? :)

[ebarry]

Response from the facilitator:

I’d go for pagination - different moderators will have different practices but pagination would allow moderators to review all unmoderated statements.

[ebarry]

Idea from another facilitator of Very Large Conversation:

• automatic screening out of nonsense statements
• way to categorize incoming statements by topic, for easier de-duplication within Polis (rather than going to a spreadsheet to categorize and then sort which statements are unique)

[nishio]

I have another suggestion:
If there are about 10-20 users who do not feel burdened by voting, we can make them to vote first.
If the vector of their voting results has low independence from the vector of existing opinions (i.e., if it is unlikely to provide new information), it could be ignored.

This might be related to the existing "Priority" mechanism.