Offensive Content Detection in Online Social Platforms

Author

Ebuka OkpalaFollow

Date of Award

8-2024

Document Type

Dissertation

Degree Name

Doctor of Philosophy (PhD)

Department

Computer Science

Committee Chair/Advisor

Dr. Long Cheng

Committee Member

Dr. Feng Luo

Committee Member

Dr. Nianyi Li

Committee Member

Dr. Matthew Costello

Abstract

Online social platforms enable users to connect with large, diverse audiences and the ability for a message or content to flow from one user to another user, user to followers, followers to user, and followers to followers. Of course, the advantages of this are apparent, and the dangers are also clearly obvious. The user-generated content could be abusive, offensive, or hateful to other users, possibly leading to adverse health effects or offline harm. As more of society's public discourse and interaction move online and these platforms grow and increase their reach, it is inherently important to protect the safety of the users of these platforms. Platforms ensure safety by enforcing rules on the type of content allowed that, when violated, could lead to a warning, user suspension or the removal of the user-generated content before or after the content is published. Monitoring and removing policy-violating content is labor and resource-intensive. Recently, the growth of machine learning, specifically deep learning-based natural language processing, has made it possible to detect offensive content and flag it for review automatically. The automatic detection of offensive content is non-trivial because of its subjectivity, as what is considered offensive in one country is not in another, its nuances, and the constant evolution of public discussions around political or social issues across different cultures. Detecting and understanding offensive content during political or social issues offers an understanding of how platforms can improve safety and the dynamics of offensive content in public discourse. However, of equal importance is the fairness of the deep learning systems used in detecting offensive content, which can propagate bias in their training datasets and could lead to unequal treatment of minority groups by the platform. The study of offensive content intersects social science and is becoming an emerging socially relevant cybersecurity problem. The works investigated in this dissertation are composed of our efforts to enable healthy online discourse by detecting and analyzing offensive content, understanding and mitigating bias in deep learning-based offensive content detection models, and developing teaching materials and an experiential learning lab to engage students in AI-cyberharassment education.

With regards to the detection and analysis of offensive content, the first study conducts a large-scale analysis of the emotions expressed, the extent of offensive discussions, and the role of emotions in offensive discussions; as anger is the emotion of epistemic injustice, the characteristics of users who generated offensive content and users who received offensive content, and the topics discussed in the Black Lives Matter (BLM) related discussions on social media after the death of George Floyd in 2020, and the protests that followed. To examine offensive language and emotion, we first develop a classifier that uses sentiment representation to aid offensive language detection. We then develop an emotion classifier based on deep attention fusion with sentiment features to classify emotions. The offensive and emotion classifiers were used to detect offensive content and classify emotions in over 20 million tweets. Finally, topic modeling was used to analyze the topics of the offensive and no-offensive tweets.

Regarding bias in offensive content detection models, in the second study, we looked at how offensive language datasets contain bias that offensive content detection models propagate. When these models classify tweets written in African American English (AAE), they predict AAE tweets as a negative class at a higher rate than tweets written in Standard American English (SAE). This study assessed bias in language models fine-tuned for offensive content detection and the effectiveness of adversarial learning in reducing such bias. We introduce AAEBERT, a pre-trained language model for African American English obtained by re-training BERT-base on AAE tweets. The representation of tweets from AAEBERT is fused with the representation of tweets from the offensive content classifier and used as input to an adversarial network to perform debiasing. We then compared the effects of adversarial debiasing in language models before and after debiasing.

Artificial intelligence (AI) is becoming increasingly popular and is being used to complete tasks in our daily activities. The third work extends the second work by exploring the implications of using large language models (LLMs), such as the version of the generative pre-training (GPT) model, GPT-4, in annotating offensive language datasets used in fine-tuning downstream models for detecting offensive content. We used different prompting techniques to annotate several offensive language datasets and fine-tuned models on the LLM-annotated datasets. Then, we assess racial bias towards AAE tweets in the models fine-tuned on LLM-annotated datasets compared to models fine-tuned on human-annotated offensive language datasets, and the rate of false positives in the models fine-tuned on LLM-annotated datasets towards AAE tweets. We also explore whether using dialect priming in the prompt techniques explored helps reduce racial bias in LLM annotation of offensive language datasets.

Finally, the popularity of AI calls for creating an AI-ready workforce across academic disciplines and professions. Most AI education research focuses on developing curricula for computing and engineering students while paying little attention to non-computing students. In the fourth work, given the interdisciplinary nature of this emerging social cybersecurity problem, engaging non-computing students without prior knowledge of AI in AI can be challenging. We take the first step to develop educational materials and a hands-on lab that introduces AI to non-computing students and how AI can be used for socially relevant cybersecurity, like offensive content detection.

Recommended Citation

Okpala, Ebuka, "Offensive Content Detection in Online Social Platforms" (2024). All Dissertations. 3714.
https://open.clemson.edu/all_dissertations/3714