Defining abuse types is the first step in policy creation. The next is to determine what enforcement action moderators should take in response to a given situation. There are a wide variety of possible actions, each with different tradeoffs. Finding the right balance decision depends on a variety of factors. These include the nature and severity of the violation, the type of content (i.e., sponsored or organic content), the potential harm involved, and the likely response and nature of the responsible party. The following are types of enforcement regularly used throughout the industry, roughly ordered from simple and common actions to more complex and rarely used actions.
Content deletion is the removal of a piece of content that violates policy from the platform. Content deletion is the most common action taken by platforms for a wide range of abuse types. Not all violating content is deliberate and malicious, and it is common for content to be removed from platforms with no further consequences to the user.
Banning is the permanent removal or blocking of a user or account from the platform. This can also extend to banning any new accounts that the user attempts to create or uses to access the platform. Content from before an account is banned may still be visible or may be removed as part of the ban, depending on the nature of both the platform and the policy violation
Banning can be the result of a single serious violation or multiple smaller violations. Industrial models in particular often use strike-based systems to ban repeat violators.
Banning and content deletion are the simplest and most intuitive forms of penalty, and as such are usually the first enforcement options implemented on nearly all platforms, across all enforcement models, and at all scales.
Temporary suspension is identical to banning, but lasts either for a specified period of time or until the user completes certain specified actions, after which the account is automatically reinstated.
Timed temporary suspensions of escalating length are frequently used as a precursor to banning a user permanently, in an attempt to both change user behavior and to convey the compounding seriousness of repeated policy violations.
Contingent temporary suspensions are most frequently used in three situations: (1) when the company suspects a user’s account is compromised; (2) when the company wants the user to read and acknowledge a set of policies before returning to the site; or (3) when the company needs the user to contact support teams to answer questions as part of investigating an incident.
Feature blocking encompasess any restriction of access to certain features of a platform based on previous actions of a user, either temporarily or permanently. This might involve specifically removing access to features that have been misused in the past, or to features that would generally be considered higher risk or are more difficult to moderate such as live streaming. Feature blocking has the advantage of allowing users to remain active on the platform, while minimizing potential harm from their actions. Historically, feature blocking has been mostly used on large, established platforms with diverse features that might justify a specific feature blocking action.
Reducing visibility refers to steps that reduce how often and how prominently a piece of content is viewed. These steps are most often used on platforms in which the product itself guides and curates a user’s experience with algorithms. This can take the form of directly imposed visibility reductions by platforms practicing the Centralized / Industrial model of T&S or as a result of user downvotes in companies using Community Reliant models
The form visibility reduction takes varies significantly based on the nature of the product itself. Common examples include removing the user / content from features such as recommendations or trending stories; downranking the user’s or content’s position in search results or feeds; and auto-collapsing comments on threaded posts.
Reducing visibility is distinct from the more general field of search / feed quality, which evaluates whether content is relevant, interesting, or useful for ranking purposes based on views and clicks. Reducing visibility is a deliberate action to minimize viewing because of the nature of the content. However, there is occasionally crossover between these fields such as when a Community Reliant platform hides content with large numbers of downvotes.
Labeling involves attaching a message to a user or piece of content to provide information to the viewer. These labels can be used to inform the viewer of any concerns or of important information relevant to the content or the topic discussed.
Labels can be placed either next to the content or placed over the content so that it is not visible unless a viewer chooses to click through to see it. This is often used to hide content that is allowed on a platform but is potentially offensive or disturbing. The most common example of this is age-restricted content that involves violence or nudity, requiring the user to click through and / or to sign in.
Occasionally, labels or other messaging can be presented to the original user who posted the content, rather than those who view the content. The most common example of this is support and helpline information for users at risk of suicide.
Demonetization is an action that prevents users from earning income and specifically applies to platforms where users can earn money from their content, usually through advertising. Demonetization is often applied to content that is allowed on the platform, but which is controversial or which advertisers may not wish to sponsor or be directly associated with.
Withholding payments is similar to demonetization but applies specifically to marketplaces where the company serves as a middleman between a buyer and a seller. It is most often imposed in response to suspected compromised accounts and fraudulent sales, but may also be used in conjunction with banning and referral to law enforcement as a penalty for serious criminal behavior.
Referral to Law Enforcement
Referral to law enforcement is the most serious action that platforms can take and involves forwarding details of the user and their situation to law enforcement or other related third parties such as NCMEC (the National Center for Missing and Exploited Children). Situations where law enforcement may be contacted include:
- People at imminent risk of serious self harm or suicide
- People planning, threatening, or committing crimes involving serious physical harm such as murder
- Terrorists and terrorist organizations
- Abuse of children, including the sharing of video and images
- Human trafficking
Violations of this type are generally rare compared to other forms of online abuse. However, because these situations are extremely serious and often time sensitive, having clear policy guidelines and action plans for who to contact and how are essential to handling them effectively.
In addition to proactive reports by a company to law enforcement, law enforcement may also make legal requests of platforms for information on users for criminal, civil, regulatory, or other legal reasons. These requests can be made with or without a court order or similar legal ruling. Some of these situations may involve imminent and time sensitive threats, so such requests are generally very high priority.