Skip to content

Spam detection in the biography of Zeste de Savoir members

License

Notifications You must be signed in to change notification settings

zestedesavoir/zds-antispam

Folders and files

NameName
Last commit message
Last commit date

Latest commit

 

History

22 Commits
 
 
 
 
 
 
 
 
 
 
 
 
 
 

Repository files navigation

Antispam for Zeste de Savoir

The goal of this project is to detect spam in the biography of Zeste de Savoir member accounts. We have frequently a lot of fake member accounts created only to promote online website with paid services in their biography, it would be nice if those could be flagged automatically as potential spam.

This antispam is a machine learning algorithm trained with data from the actual database of Zeste de Savoir. A bot retrieve the last member accounts and process their biography through the antispam. If the bot thinks the biography is a spam, it sends a private message to alert the moderators.

Getting training data

We need to open Django's shell with /opt/zds/wrapper shell and import those objects:

from django.db.models import Q, F
from zds.member.models import Profile
import json

Then, we want for each user to save their username, their biography and the flag indicating if they are banned or not:

raw_data = list(Profile.objects.values('biography', 'can_read', username=F('user__username')))

Most biographies are empty, so we remove the users if it's the case:

data = []
for elem in raw_data:
    if elem['biography'] != "":
        data.append(elem)

After that, we write the data into a JSON file placed in the webroot so we can access it at the relative URL /static/antispam-data.json:

with open('/opt/zds/webroot/static/antispam-data.json', 'w') as f:
    json.dump(data, f)

Finally, we retrieve the file and do not forget to remove it with sudo rm /opt/zds/webroot/static/antispam-data.json!

In the end, we have antispam-data.json with a list of objects with username, biography and can_read.

Installing the dependencies

We need to create a virtual environment, activate it and install the dependencies:

python3 -m venv venv
source venv/bin/activate
pip install -r requirements.txt

Running the antispam

We need to put the secrets in a JSON file nammed secrets.json with this structure:

{
    "client_id": "foo",
    "client_secret": "bar",
    "username": "something",
    "password": "else",
    "topic_id": "42"
}

Then, we just have to run antispam.py!

It can be automated using a cronjob because it keeps track of already reported potential spammers in a text file named reported_users.txt.

About

Spam detection in the biography of Zeste de Savoir members

Resources

License

Stars

Watchers

Forks

Releases

No releases published

Packages

No packages published

Languages