NOTE: This page is in flux! Additional readings are currently being generated by polling our participants. Please help us improve this list! Please submit suggestions for additional resources to “”.

This page is a list of background readings. It includes:

  1. Very Accessible Introductions to the Problem (New: Now Annotated)
  2. Research Framings and Overviews
  3. Algorithm Audits by Researchers
  4. Design Principles and Best Practices
  5. Selected Audit Studies in Other Domains
  6. General Sources about the Social / Cultural / Political / Economic Implications of Algorithmic Curation / Rating / Filtering / Selection


Very Accessible Introductions to the Problem

(Roughly ordered from short, very accessible to longer, more involved.)

Sandvig, C., Hamilton, K., Karahalios, K., & Langbort, C. (2014). “An Algorithm Audit.” In: Seeta Peña Gangadharan (ed.), Data and Discrimination: Collected Essays, pp. 6-10. Washington, DC: New America Foundation.

A very basic three-page overview of the need for audit studies of algorithms.

Walker, B. (2015, October 9). “Enchanting by Numbers.” Theory of Everything. Cambridge, MA: Public Radio Exchange. (audio podcast: 25m)

Introduces algorithm awareness, the socialist calculation debate, folk theories of algorithms, Crandall’s Law, the Difference Engine, Ada Lovelace. Examples: Uber, Facebook, Sabre, the Difference Engine.

Pasquale, F. (2015). The Black Box Society: The Secret Algorithms that Control Money and Information. Cambridge, MA: Harvard University Press.

Discusses the secrecy and complexity of algorithmic systems in finance, media, and information. Argues that an “intelligible society” should assure that key decisions of its most important firms are fair, nondiscriminatory, and open to criticism.

O’Neil, C. (2016). Weapons of Math Destruction: How Big Data Increases Inequality and Threatens Democracy. New York: Crown Random House.

An accessible discussion of the mathematical models that pervade contemporary life. Argues that these models are opaque, unregulated, and uncontestable — and often wrong, “propping up the lucky and punishing the downtrodden.” Hopes that we become more savvy about the models that govern our lives.

Research Framings and Overviews

Christian Sandvig, Kevin Hamilton, Karrie Karahalios and Cedric Langbort. “Auditing Algorithms: Research Methods for Detecting Discrimination on Internet Platforms.” In Data and Discrimination: Converting Critical Concerns into Productive: A preconference at the 64th Annual Meeting of the International Communication Association. Seattle, WA, 2014. (This paper inspired the present workshop. –Ed.)

Diakopoulos, N., “Algorithmic Accountability Reporting: On the Investigation of Black Boxes

Hamilton, Kevin, Christian Sandvig, Karrie Karahalios and Motahhare Eslami. “A Path to Understanding the Effects of Algorithm Awareness.” In CHI 2014. Toronto, ON, 2014.

Barocas, S., Hood, S., & Ziewitz M. (2013). Governing Algorithms: A Provocation Piece. Available at SSRN:

C. Sandvig, K. Hamilton, K. Karahalios, and C. Langbort. (2016). When the Algorithm Itself Is a Racist: Diagnosing Ethical Harm in the Basic Components of Software, Int’l. J. Comm. 10: 4972-4990.


Algorithm Audits by Researchers

Sweeney, L. Discrimination in Online Ad Delivery. CACM 56(5): 44-54.

Eslami, M., Rickman, A., Vaccaro, K., Aleyasen, A., Vuong, A., Karahalios, K., Hamilton, K., and Sandvig, C. “I always assumed that I wasn’t really that close to [her]:” Reasoning about invisible algorithms in the news feed. (CHI ’15)

Eslami, M., Aleyasen, A., Karahalios, K., Hamilton, K., and Sandvig, C. FeedVis: A Path for Exploring News Feed Curation Algorithms. Software demo (CSCW ’15).

M. Eslami, K. Karahalios, C. Sandvig, K. Vaccaro, A. Rickman, K. Hamilton, and A. Kirlik. First I “like” it, then I hide it: Folk Theories of Social Feeds (CHI 2016).

Hannak, A., Soeller, G., Lazer, D., Mislove, A., Wilson, C. (2014). Measuring Price Discrimination and Steering on E-commerce Web Sites. (IMC ’14).

Hannak, A., Sapiezynski, P., Kakhki, A. M., Krishnamurthy, B., Lazer, D., Mislove, A., Wilson, C. (2013). Measuring Personalization of Web Search. (WWW ’13).

Mathias Lecuyer, Guillaume Ducoffe, Francis Lan, Andrei Papancea, Theofilos Petsios, Riley Spahn, Augustin Chaintreau, and Roxana Geambasu. “XRay: Increasing the Web’s Transparency with Differential Correlation.” Technical report, July 2014.

Aniko Hannak, Claudia Wagner, David Garcia, Alan Mislove, Markus Strohmaier, Christo Wilson. Bias in Online Freelance Marketplaces: Evidence from Taskrabbit and Twitter. (CSCW’17)

Soeller, G., Karahalios, K., Sandvig, C., & Wilson, C. MapWatch: Detecting and Monitoring International Border Personalization on Online Maps. (WWW ’16).

Le Chen, Alan Mislove, and Christo Wilson. An Empirical Analysis of Algorithmic Pricing on Amazon Marketplace. (WWW ’16)

Edelman, B. (2011). Bias in Search Results? Diagnosis and ResponseIndian Journal of Law and Technology 7: 16-32.

Le Chen, Alan Mislove, and Christo Wilson. Peeking Beneath the Hood of Uber. (IMC 2015).

Amit Datta, Michael Carl Tschantz, and  Anupam Datta. (2015). Automated Experiments on Ad Privacy Settings: A Tale of Opacity, Choice, and Discrimination. PoPETs 2015: 1.

Muhammad Ahmad Bashir, Sajjad Arshad and William Robertson and Christo Wilson. Tracing Information Flows Between Ad Exchanges Using Retargeted Ads. (USENIX ’16)

M. Eslami, K. Vaccaro, K. Karahalios, and K. Hamilton. “Be careful; things can be worse than they appear”: Understanding Biased Algorithms and Users’ Behavior around Them in Rating Platforms (ICWSM 2017)

J. Kulshrestha, M. Eslami, J. Messias, M. B. Zafar, S. Ghosh, K. Gummadi, and K. Karahalios. Quantifying Search Bias: Investigating Sources of Bias for Political Searches in Social Media (CSCW 2017)


Design Principles and Best Practices

Principles for Accountable Algorithms and a Social Impact Statement for Algorithms (FAT ML)


Audit Methods in Other Domains

Saltman, J. (1975). Implementing Local Housing Laws Through Social Action. Journal of Applied Behavioral Science, 11(1): 39-61.

Ayres, I. & Siegelman, P. (1995). Race and Gender Discrimination in Bargaining for a New Car. American Economic Review 85(3): 304-321.

Schulman KA, Berlin JA, Harless W, Kerner JF, Sistrunk S, et al. The effect of race and sex on physicians’ recommendations for cardiac catheterization. N. Engl. J. Med. 1999;340(8):618–626.

Ridley S, Bayton JA, Outtz JH. Taxi Service in the District of Columbia: Is It Influenced by Patrons’ Race and Destination? Washington, DC: Washington Lawyers’ Comm. Civil Rights Law. Mimeo; 1989. (Can’t find. Anyone have a copy? –Ed.)

Wissoker D, Zimmerman W, Galster G. Testing for Discrimination in Home InsuranceWashington, DC: Urban Inst. Press; 1998.

Pager, D. (2007). The Use of Field Experiments for Studies of Employment Discrimination: Contributions, Critiques, and Directions for the Future. The Annals of the American Academy of Political and Social Science, 609(1): 104-33.

Pager, D. (2009). Field Experiments for Studies of Discrimination. In: E. Hargittai (ed.) Research Confidential: Solutions to Problems Most Social Scientists Pretend They Never Have, pp. 38-60. Ann Arbor, MI: University of Michigan Press.

National Research Council Panel on Measuring Racial Discrimination, The. (2004). Measuring Racial Discrimination. Washington, DC: National Academies Press. (an excellent overview of social science audit methodology appears from p. 103 on. –Ed.)


General Sources about the Social / Cultural / Political / Economic Implications of Algorithmic Curation / Rating / Filtering / Selection

Anderson, C.W. (2012). “Towards a Sociology of Computational and Algorithmic Journalism.” New Media & Society.

Angwin, J. (2014). “Hacked,” In: J. Angwin, Dragnet Nation: A Quest for Privacy, Security, and Freedom in a World of Relentless Surveillance, pp. 1-20. New York: Henry Holt & Co. (Ch. 1 available free online.)

Baker, P. and A. Potts. (2013). “‘Why Do White People Have Thin Lips?’ Google and the Perpetuation of Stereotypes Via Auto-Complete Search Forms.” Critical Discourse Studies 10, no. 2: 187- 204.

Beam, M.A. (2013). “Automating the News: How Personalized News Recommender System Design Choices Impact News Reception.” Communication Research.

Benjamin, S.M. (2013). “Algorithms and Speech.” University of Pennsylvania Law Review 161, no. 6: 1445-1494.

boyd, d. and K. Crawford. “Critical Questions for Big Data.” Information, Communication & Society 15, no. 5: 662-679.

Brunton, F. and H. Nissenbaum. (2011). “Vernacular Resistance to Data Collection and Analysis: A Political Theory of Obfuscation.” First Monday 16, no. 5.

Calo, R. (2011). “Peeping Hals.” Artificial Intelligence 175, no. 5-6 (2011): 940-941.

Clerwall, C. (2014). “Enter the Robot Journalist: Users’ Perceptions of Automated Content.” Journalism Practice.

Crawford, K., “The Hidden Biases in Big Data” (accessed August 27, 2013).

Finn, E. (2017). What Algorithms Want: Imagination in the Age of Computing. Cambridge, MA: MIT Press.

Fuller, M. and A. Goffey (2012). “Algorithms.” In Evil Media, 69-82. Cambridge, MA: MIT Press.

Gillespie, T. (2014). “The Relevance of Algorithms.” In Media Technologies: Essays on Communication, Materiality, and Society, edited by T. Gillespie, P. Boczkowski and K.A. Foot, 167-194. Cambridge, MA: MIT Press.

Granka, L. (2010). “The Politics of Search: A Decade Retrospective.” The Information Society 26, no. 5: 364-374.

Hallinan, Blake and Ted Striphas. (2014). “Recommended for You: The Netflix Prize and the Production of Algorithmic Culture.” New Media & Society.

Hazan, J. G. (2013). “Stop Being Evil: A Proposal for Unbiased Google Search.” Michigan Law Review 111, no. 5: 789-820.

Introna, L. and H. Nissenbaum. (2000). “Shaping the Web: Why the Politics of Search Engines Matters.” The Information Society 16, no. 3: 1-17.

Jiang, Min. (2013). “The Business and Politics of Search Engines: A Comparative Study of Baidu and Google’s Search Results of Internet Events in China ” New Media & Society.

Laidlaw, E. B. (2008). “Private Power, Public Interest: An Examination of Search Engine Accountability.” International Journal of Law & Information Technology 17, no. 1: 113-145.

Mager, A. (2012). “Algorithmic Ideology.” Information, Communication & Society 15, no. 5: 769-787.

Manovich, Lev. (2000). “Database as a Genre of New Media.” AI & Society 14, no. 176-183.

Manovich, Lev, “The Algorithms of Our Lives” /143557/ (accessed August 20, 2014).

Napoli, Philip M. (2014). “Automated Media: An Institutional Theory Perspective on Algorithmic Media Production and Consumption.” Communication Theory 24, no. 3.

Pasquale, F. (2011). “Restoring Transparency to Automated Authority.” Journal on Telecommunications and High Technology Law 9, no. 235: 235-254.

Sandvig, C. (2015). Seeing the Sort: The Aesthetic and Industrial Defense of “The Algorithm.” Media-N 11(1).

Tufekci, Zeynep. (2014). “Engineering the Public: Big Data, Surveillance and Computational Politics.” First Monday 19, no. 7.


Prior Reading List

This is an updated version of the Background Readings List for the previous workshop, “Auditing Algorithms From the Outside: Methods and Implications” at ICWSM.