Source code: GitHub
Credits: Trust & Safety Teaching Consortium and Cornell Tech

For the course project, you and your group will be the Trust and Safety team at a major social media or consumer cloud platform. You will be assigned to a group at the end of April. The team will focus on a particular type of abuse, proposing policies to the executives at your company as well as researching and implementing relevant technological solutions within a content moderation bot in Discord.

The project is split into three milestones, which will be completed over the course of the quarter. The first milestone will be completed individually. The second and third milestones will be completed with your group. The final milestone will culminate in a presentation that your group will give to the teaching team and guest judges from industry, the evening of Wednesday, June 5th.

Please do not generate the text you submit using ChatGPT or any other LLM. You will have plenty of opportunities to use these systems to generate test data or perform abuse detection in future milestones, but anything submitted by students in this assignment should be written by humans.

Percent of Final Grade: 20%

Deliverables:

Submission*:* This first milestone will be completed independently. CS152 students should upload the document to their Canvas site and POLISCI143 students should upload the document to their Canvas site.

Note: The abuse type you choose to focus on for this milestone may or may not be the abuse type your group focuses on for milestones 2 and 3.

Description: The execs recently came to your team and tasked you with looking into a significant type of online abuse, researching the current best options available for dealing with such abuse, and making specific recommendations to the company of how to detect and mitigate it.

In writing this paper, please use citations (choose your preferred style) for factual information and feel free to add your own original interpretations and suggestions. Please follow the structure below for the paper overall, but you are welcome to add additional information in whatever sections you see fit. You are also welcome to use graphics, charts, or diagrams as long as you cite the original source.

Example Abuse Types: You are welcome to write about any of these abuse types or one of equivalent importance. If you want to move away from topics that are covered in the syllabus, please check in with the teaching team first.

A note on choosing abuse type: You may end up working with this abuse type for the entire quarter, so we encourage you to pick a topic that you care about. Note that when you later work to implement technological solutions we will use stand-ins for illegal and/or very harmful material (e.g. pictures of kittens instead of CSAM). Don’t let the potential technical challenge of a topic scare you away from tackling it; your milestones will be graded on effort and thoughtfulness, not whether or not you’re able to effectively solve these problems. They’re still problems in the real world because they’re quite hard, after all!

Required Sections:

Length: 2000-4000 words. We will penalize papers that exceed this word limit.

Create a policy table outlining what platform policies are currently in effect that relate to this kind of abuse. Compile language, pulled from the policies of other platforms, that you think is relevant or appropriate. Please do this for three platforms. An example table for an investigation into coordinated harassment of journalists on Twitter is below, but you may adjust as appropriate for different abuse types. Think critically about what types of columns you think are theoretically important. Please make sure all table cells that reference policy hyperlink to the exact website. Note that there may be cases where you need to write “unclear.” That’s fine, just justify the reasoning. You can see other examples in Figure 1 here, Table 1 here, and here.

In addition to the table, in two or three paragraphs, please explore any research on the effects of these policies, including whether they help mitigate the abuse or enable it.

Coordinated harassment of journalists on Twitter (Note: this table includes made up data) Note to instructors: Including this example table will lead many students to focus on this exact abuse type.

Percent of Final Grade: 20%

This milestone has 3 components and 4 deliverables:

Submission*:* Please have one group member from CS152 upload all documents to the Canvas CS152 site before the deadline. Make sure to include your team number in the name of each document.

For milestone 3 you will be asked to make your bot “smart,” e.g. train a classifier on your abuse type. For this milestone, milestone 2, it is fine if the back end moderator flow is manual.

Select an abuse type to focus on for the group project. This should probably be an abuse type investigated by a member of your group for milestone 1, so that you are starting with a good understanding of the abuse and potential mitigations. While choosing an abuse type to focus on, please keep in mind that milestone 3 will ask you to create some kind of automated detection mechanism for that abuse type, so choose something for which automated detection is at least tractable.

You are allowed to choose an abuse type where realistic testing is illegal or would be extremely difficult for the team, such as the trading of child sexual abuse material. In these cases, you will use a type of proxy content to train and test your detection system. For example, we have used “naked” photos of kittens (versus adult cats) as stand-ins for CSAM detection in the past. If you want to scan for extremely violent videos, you can use cartoon violence instead of content such as beheading videos. For topics where testing might require the use of upsetting content, such as racist language or misogynistic threats, make sure that the entire team is ok with the topic before proceeding with milestone 2.

Your group will design two reporting flows: one based on user reports and the other a behind-the-scenes flow for content moderators. You will implement both these flows (within Discord’s constraints) in your bot for this milestone. The best way to represent these flows is using a flowchart that includes both the user action and the system’s response. The reporting flow should have a variety of abuse types at the highest level, but the more detailed flow (after the first or second prompt) only needs to be built out for your specific abuse type.

An example of a User Reporting Flow focused on hate and harassment that received a high score last year is included here, however as noted above you only need to show a variety of abuse types at the highest level.

Some additional examples from industry:

Your user reporting flow should outline the process that a user is taken through when they attempt to report an instance of your abuse type on your platform. It should do the following:

Your manual review flow should outline the process that a content reviewer goes through when they review a piece of content submitted by a user as abusive using the flow you just created. It should do the following:

Some questions you should think about when designing your flows:

Your company’s T&S-minded CEO has approved your reporting flows and asked that you implement them for this abuse type, commending you on the thought that went into them. Success! She gives you the green light to build out a skeleton of the system for some A/B testing with real users.

You will design and implement a reporting flow within the context of Discord. We are using Discord for the class project because their bot framework is very powerful; many communities build the functionality they want on top of Discord by using bots to greet users as they join servers, auto-respond to messages, moderate chat/ban swear words, and much more. Discord bots are written in asynchronous Python - don’t worry if you haven’t worked with it before, it shouldn’t be too hard to pick up (and the TAs are here to help!). Please consult the Discord Bot Setup Guide at the bottom of this doc.

Your group will be given two pre-configured channels: group-# for general chatting and group-#-mod to serve as a back-end for human moderators (from now on we’ll call them the “main” channel and the “mod” channel). User generated content will go in the main channel, and that content will either be manually reported or automatically flagged by your bot and potentially be sent to the mod channel for human review.

Although we have given you the setup of these two channels, you are not necessarily restricted to the context of a group chat! It is up to you to specify what context your moderation tools exist in, and style your main group-# channel accordingly. For example, you could say it is a feed of content that one specific user is scrolling through (e.g. Instagram/Twitter style), or say the channel is actually a DM between two users. There’s a lot of flexibility here, and as long as you’re clear about what you imagine and how you are simulating that within Discord, you’re welcome to adapt in whatever way makes sense for the abuse type you have chosen. Please reach out to the TAs if you have ideas that you aren’t sure how to adapt; they can also potentially give you additional channels if it would be helpful.

For this part, your bot should be able to do the following:

A note on sensitive content here; the TAs can see all messages sent in your group channels, but we won’t be actively monitoring them. In order to properly do this milestone, you may have to engage with perverse and hateful content; that is the nature of this kind of abuse-fighting. It is important that you test your bots, but this is not an excuse to behave inappropriately to other students, so make sure that it is clear to your team when you are testing and don’t actually target individuals in a way that might be emotionally harmful. Please take care of yourselves and each other, and reach out to the teaching team if you’re finding that this work places any undue emotional burden.

For this milestone we have given you the skeleton of a moderation bot with the following capabilities:

Please set up your bot within the first week of the assignment going out so we can catch any potential problems. The TAs will be ready to answer any questions and help debug during section!

For this part of the milestone you will be submitting a short (up to 6 minute) video demoing all of the functionality of your Discord bot as well as talking through your edge cases. Be sure to begin with a clear description of the context you’ve chosen for your channels.

Below are some resources we think might be useful to you for this part of the milestone.

Here is the documentation for discord.py, Discord’s python package for writing Discord bots. It’s very thorough and fairly readable; this plus google (in addition to the TAs) should be able to answer all of your functionality questions!

Discord bots frequently use emoji reactions as a quick way to offer users a few choices - this is especially convenient in a setting like moderation when mods may have to make potentially many consecutive choices. Check out on_raw_reaction_add() for documentation about how to do this with your bot. You also might want to look into on_raw_message_edit() to notice users editing old messages.

Discord offers “embeds” as a way of getting a little more control over message formatting. Read more about them in this article or in the docs.

unidecode and uni2ascii-janin are two packages which can help with translating unicode characters to their ascii equivalents.

In approximately 500 words, summarize:

For this milestone your group will be making your very own Discord bot. Discord bots are implemented in Python (or Javascript) - don’t stress if you haven’t written Python before! It’s a pretty readable language, so you should be able to pick it up as you go, and the TAs are always here to help.

If you’re not familiar with Discord, that’s totally okay! Check out this short video which overviews Discord’s features and quirks.

First, every member of the team (both CS and POLISCI) should join the Discord server using this invite link:

[insert link here]

Discord can be used in your web browser, although most people prefer the thick client apps.

For the next two milestones, you and your group will have two channels to test and develop your bot in: group-#, and group-#-mod, where # is your group’s number. We will give you and your bot a special role such that only you and the staff can see those channels; that way, everyone will have their own small workspace.

To get the role for your group, click on the TA Bot user to bring up this window.
Type in: .join # where # is replaced by your group number.

If all goes according to plan, you should receive a message back saying that you have been given a role corresponding to your group number and you should see a new role on your user in the server.

Additionally, you should be able to see two new channels under one of the “Group Channels” categories:

If you accidentally join the wrong group, just message the TA Bot .leave # to have the role removed and leave those channels.

Please let [TA] know if something goes awry in this process!

Note: only ONE student per group should follow the rest of these steps.

Fork and clone the GitHub repository here. For instructions on how to fork a github repo, see this article. In order for your group to be able to collaborate effectively on this project, we recommend you create a shared GitHub repository; when you do, make sure you use the .gitignore file included in the starter code so that you don’t accidentally upload your tokens to GitHub. Our GitHub repository already has tokens.json in its .gitignore file. When you clone your project from there, you will have to create your own tokens.json file in the same folder as your bot.py file. The tokens.json file should look like this, replacing the “your key here” with your key. In the below sections, we explain how to obtain Discord keys.

The first thing you’ll want to do is make the bot. To do that, log in to https://discord.com/developers and click “New Application” in the top right corner.

Name your application Group # Bot, where # is replaced with your group number. So, for instance, Group 0 would name their bot like so:

It is very important that you name your bot exactly following this scheme; some parts of the bot’s code rely on this format.

An aside: It’s unsafe to embed API keys in your code directly. If you put that code on GitHub, then anyone could find and use that key! (GitHub actually tries to detect code like this and forbids programmers from uploading it.) That’s why we’re storing them in a separate file which can be ignored by version control software.
Next, we’ll add the bot to the 152 Discord server! You’ll need to generate a link that the teaching team can use to invite your bot.

Note that these permissions are just a starting point for your bot. We think they’ll cover most cases, but it’s entirely possible you’ll run into cases where you want to be able to do more. If you do, you’re welcome to send updated links to the teaching team to re-invite your bot with new permissions.

First things first, the starter code is written in Python. You’ll want to make sure that you have Python 3 installed on your machine. Alternatively, you can use a text editor of your choice.

Once you’ve done that, open a terminal in the same folder as your bot.py file. (If you haven’t used your terminal before, check out this guide!)

You’ll need to install some libraries if you don’t have them already, namely:

Next up, let’s take a look at what bot.py already does. To do this, run bot.py and leave it running in your terminal. Next, go into your team’s private group-# channel and try typing any message. You should see something like this pop up in the group-#-mod channel:

The default behavior of the bot is, any time it sees a message (from a user), it sends that message to the moderator channel with no possible actions. This is obviously not the final behavior you’ll want for your bot - you should update this to match your report flow. However, the infrastructure is there for your bot to automatically flag messages and (potentially) moderate them somehow.

Next up, click on your app in the right sidebar under “Online” to begin direct messaging it (or click on its name). First of all, try sending “help”.

Try following its instructions from there by reporting a message from one of the channels to get a sense for the reporting flow that’s already built out for you. (Make sure to only report messages from channels that the bot is also in.)

If you look through the starter code, you’ll see the beginnings of the reporting flow that are already there. It will be up to you to build that out in whatever way your group decides is best. You’re welcome to edit any part of the starter code you’d like if you want to change what’s already there - we encourage it! This is just meant to be a starting point that you can pattern match off of.

If you’re not familiar with Python and asynchronous programming, please come to a section for an introduction. The TAs are happy to walk you through the starter code and explain anything that’s unclear.

If you’re seeing this error, it probably means that your terminal is not open in the right folder. Make sure that it is open inside the folder that contains bot.py and tokens.json. You can check this by typing in ls and verifying that the output looks something like this:

Discord has a slight incompatibility with Python3 on Mac. To solve this, navigate to your /Applications/Python 3.6/ folder and double click the Install Certificates.command. Try running the bot again; it should be able to connect now.

If you’re still having trouble, try running a different version of Python (i.e. use the command python3.7 or python3.8) instead. If that doesn’t work, come to section and we’ll be happy to help!

This is an issue with the version of Discord API that is installed. Try the following steps:

Percent of Final Grade: 30%

Deliverables and due date:

Submission*:* Please have one group member upload the poster PDF and video to the CS 152 Canvas site. Your code files should be in the Github repository submitted for Milestone 2.

Please fill out the work distribution survey to assess how equally different members of each group contributed to the project. We reserve the right to change grades based on the results.

Your bot will be responsible for handling both manual reports from users (which you implemented in Milestone 2) as well as automatically detecting and flagging abusive content (the primary goal for this Milestone). This will include finding or collecting a dataset of examples to use to evaluate the efficacy of your solutions.

Here are some examples of what we envision you accomplishing for this milestone. Please note that this is not a checklist of things you must accomplish, just ideas.

To handle multiple languages, you can use packages like google_trans_new to automatically translate everything to English or langdetect for language detection (make sure to rate limit yourself where necessary).

You should develop and implement a strategy for evaluating the efficacy of your back-end solution. If you trained a classifier or utilized external APIs, this might look like utilizing your dataset to generate a confusion matrix and figuring out whether your model is over or under sensitive and what kinds of problems this might cause at scale. If your solution is more user or design focused, you could conduct further user studies; for instance, you could invite friends to interact with the bot to assess the additional functionality and identify cases in which your design is clunky or might not scale well. What scenarios does your bot handle effectively? What scenarios does your bot not handle as effectively? What explains your bot’s strengths and weaknesses? With more time and resources, what would be some of your next steps?

A key final deliverable for this milestone will be a poster that you’ll display and discuss with your platform’s “executives” (the teaching team as well as guest industry judges who will stop by your poster during the poster session). You will likely want to explain how your platform currently handles your specific type of abuse (your back-end solution), and address its strengths and shortcomings, leading to clear and specific recommendations for how the platform should move forward. You will also want to answer questions from the “executives.” We encourage you all to think creatively about how to communicate your work! We encourage your group to have a prepared 5 minute pitch, and to make sure all group members who are present have the chance to participate in the pitch.

Your poster should include:

More details on these components are below.

Give a short description of your group’s abuse type and victim profile. You can assume that people viewing your poster have a general awareness of your abuse type.

Create a written policy in the kind of language you have seen from the community standards and terms of service you have seen that is specifically targeted at your abuse type. The policy should be less than 400 words and understandable by a normal user.

Discuss the original goals and final state of your back-end technology in more detail, explaining the work you did to build it and what its current capabilities are. If there are things you tried which didn’t make it into the final product, be sure to mention them here, along with the reasoning behind not including them.

Provide a clear analysis of how well your group’s back-end technology accomplishes what it originally set out to do. Make sure to address both the successes and the shortcomings of your current solution. Discuss any negative unintended consequences you foresee and which users may be more affected by them. Try to think about what critics/stakeholders would say about your technology.

Discuss what impact you believe implementation would have on platform safety. Discuss other engineering approaches that your group didn’t pursue but that you’d want to propose going forward.

You can print your poster however you’d like. The poster session will have corkboards where you can pin up your poster without a rigid backing. We recommend that your poster be 24”x36”. It should not be larger than that, as we have limited space.

Stanford Undergraduate Research has tips for creating good posters here. We encourage you to avoid having a lot of text on your poster; hit the points you need to make, but keep the poster readable.

You are going to want to show off your excellent reporting system to our judges, friends and family! Please record a short demo video that you can show to judges and narrate.

We encourage the political science student to provide strong support for the evaluation component of this milestone, for example by leading a rigorous qualitative evaluation of the bot to supplement a quantitative evaluation. The political science student could generate a typology of abuse manifestations, and evaluate how the bot performs on them. The political science student could recruit testers and do a deep dive into why the bot did or did not respond appropriately to the testers. These are just ideas, think creatively about this! We also encourage the political science student to lead the “policy language” section. The political science student could also assist with dataset creation.