| by Kris McGuffie and Alex Newhouse

News Stories

In 2020, OpenAI developed GPT-3, a neural language model that is capable of sophisticated natural language generation and completion of tasks like classification, question-answering, and summarization. While OpenAI has not open-sourced the model’s code or pre-trained weights at the time of writing, it has built an API to experiment with the model’s capacity. When we researched OpenAI’s predecessor model, GPT-2, last year, we found that language models have the potential of being used as potent generators of ideologically consistent extremist content.

In a new report, CTEC evaluated the revolutionary improvements of GPT-3 for the risk of weaponization by extremists who may attempt to use GPT-3 or hypothetical unregulated models to amplify their ideologies and recruit to their communities.

Experimenting with prompts representative of different types of extremist narrative, structures of social interaction, and radical ideologies, we find:

  • GPT-3 demonstrates significant improvement over its predecessor, GPT-2, in generating extremist texts.

  • GPT-3 shows strength in generating text that accurately emulates interactive, informational, and influential content that could be utilized for radicalizing individuals into violent far-right extremist ideologies and behaviors.

  • While OpenAI’s preventative measures are strong, the possibility of unregulated copycat technology represents significant risk for large-scale online radicalization and recruitment. In the absence of safeguards, successful and efficient weaponization that requires little experimentation is likely.

  • AI stakeholders, the policymaking community, and governments should begin investing as soon as possible in building social norms, public policy, and educational initiatives to preempt an influx of machine-generated disinformation and propaganda. Mitigation will require effective policy and partnerships across industry, government, and civil society.

Read and download our full report here.

This project was made possible by the OpenAI API Academic Access Program. 



References and Further Reading

Bastug, Mehmet F., Aziz Douai and Davut Akca. “Exploring the ‘Demand Side’ of Online Radicalization: Evidence from the Canadian Context.” Studies in Conflict & Terrorism 43, no. 7 (July 2, 2020): 616–37. https://doi.org/10.1080/1057610X.2018.1494409.

Brown, Tom Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. “Language Models are Few-Shot Learners.” ArXiv. Submitted on 28 May 2020 (v1), last revised 22 Jul 2020 (this version, v4). https://arxiv.org/abs/2005.14165 

Gerraerts, Sanne. “‘Digital Radicalisation Of Youth,’  3(1) (2012), Pp. 26–27.” Social Cosmos 3, no. 1 (2012): 25–32.

Gill, Paul, Emily Corner, Maura Conway, Amy Thornton, Mia Bloom, and John Horgan. “Terrorist Use of the Internet by the Numbers: Quantifying Behaviors, Patterns, and Processes.” Criminology & Public Policy 16, no. 1 (February 2017): 99–117. https://doi.org/10.1111/1745-9133.12249.

Horgan, John. “From Profiles to Pathways and Roots to Routes : Perspectives from Psychology on Radicalization into Terrorism.” The ANNALS of the American Academy of Political and Social Science 618, no. 1 (July 2008): 80–94. https://doi.org/10.1177/0002716208317539.

King, Michael, and Donald M. Taylor. “The Radicalization of Homegrown Jihadists: A Review of Theoretical Models and Social Psychological Evidence.” Terrorism and Political Violence 23, no. 4 (September 2011): 602–22. https://doi.org/10.1080/09546553.2011.587064.

Koehler, Daniel. “The Radical Online: Individual Radicalization Processes and the Role of the Internet1.” Journal for Deradicalization Winter 2014/15, no. Nr. 1 (n.d.): 116–34.

Meleagrou-Hitchens, Alexander, and Nick Kaderbhai. “Research Perspectives On Online Radicalisation A Literature Review, 2006–2016.” King’s College London: International Centre for the Study of Radicalisation, 2017.

“Militant Islamist Radicalisation,” The Center for Terror Analysis, Danish Security and Intelligence Service, 28 April 2016, Accessed on August 12, 2020, https://www.pet.dk/English/Center%20for%20Terror%20Analysis/~/media/VTD%202016/20160428MilitantIslamistradicalisationpdf.ashx

Neumann, Peter R. “Options and Strategies for Countering Online Radicalization in the United States.” Studies in Conflict & Terrorism 36, no. 6 (June 2013): 431–59. https://doi.org/10.1080/1057610X.2013.784568.

Newhouse,  Alex, Jason Blazakis and Kris McGuffie. “The Industrialization of

Terrorist Propaganda: Neural Language Models and the Threat of Fake Content Generation.” Center on Terrorism, Extremism, and Counterterrorism. October 2019. https://www.middlebury.edu/institute/sites/www.middlebury.edu.institute/files/2019-11/The%20Industrialization%20of%20Terrorist%20Propaganda%20-%20CTEC.pdf?fv=TzdJnlDw

Sageman, Marc. Leaderless Jihad: Terror Networks in the Twenty-First Century. University of Pennsylvania Press, 2008. https://www.jstor.org/stable/j.ctt3fhbht.

Snyder, Timothy. The Road to Unfreedom. New York: Tim Duggan Books, 2018.

Tanu, Shree and Gupta Sumeet. “Role of Social Media in Online Radicalization : Literature Review and Research Agenda.” Asia Pacific Journal of Information Systems 29, no. 2 (June 30, 2019): 268–82. https://doi.org/10.14329/apjis.2019.29.2.268.

Torok, Robyn. “Developing an Explanatory Model for the Process of Online Radicalisation and Terrorism.” Security Informatics 2, no. 1 (December 2013): 6. https://doi.org/10.1186/2190-8532-2-6.

Our work is made possible by research grants and gifts from supporters. We appreciate your generosity.

Donate Today

Stay up to date on CTEC’s activities!

Join Our Newsletter

Open positions at CTEC are advertised through the Middlebury Institute’s employment opportunities Handshake.

Current Openings