Social impacts & bias of AI

Update on October 23 2020: After I wrote this post, i was invited to give a talk on this topic of social impacts & bias of AI at the course <Ethics in AI> by Prof. Alice Oh at KAIST. I’m sharing the slide set here: Unreasonably shallow deep learning [slides].

There have been a series of news articles in Korea about AI and its applications that have been worrying me for sometime. I’ve often ranted about them on social media, but I was told that my rant alone is not enough, because it does not tell others why I ranted about those news articles. Indeed that is true. Why would anyone trust my judgement delivered without even a grain of supporting evidence? So, I’ve decided to write a short post on Facebook (shared on Twitter) and perhaps surprisingly in Korean (!) This may have been the first AI/ML-related (though, very casual) post I’ve ever written in Korean, and is definitely not the best written piece from me, although I hope this post would clarify why I’ve been fuming about those news articles.

This post is quite casual and not academic. If I’m missing any important references for general public, that you want me to include here, please drop me a line. As I’m not in any way an expert in this topic, I’m sure I’ve missed many important references, discussions and points.

That said, I realized that it’s not only Korean speakers who engage with this post (via Google Translate, etc.) and that the automatic translation of this post into English is awful (thanks to the hat tip by my colleague Ernest Davis at NYU.) Since it’s a pretty short post, I’ve decided to put its English version along with the original Korean version here in my blog. The version in Korean comes first, and the one in English follows immediately.

Korean version

Twitter와 FB를 비롯한 social media 및 학계에서 많이 논의가 되지만 한국어로 된 논의는 크게 없어 보여서 아주 간단히 Social impact & bias of AI 라는 주제에서 중요하다 생각되는, 밀접히 연관된 point 몇 개를 아래 리스트업 합니다. 아마 있는데 제가 못 찾은 것일 수도 있고, 혹시 관련된 한국어로된 연구 또는 논의가 있으면 답글에 남겨주시기 바랍니다.

[아무래도 한국어로 글을 안 써 버릇해서 영 읽기 불편해 보입니다. 양해 부탁드립니다.]

Amplification

기술은 사회를 반영하는것이 맞습니다. 다만 그렇게 반영된 사회의 특징이 기술을 통해 같은 사회 안에서 증폭이 됩니다. Virginia Eubanks의 또는 Ruha Benjamin의 를 읽어보면 어떻게 이런 증폭이 사람들에게 해를 가하는지 알게 됩니다 (https://www.nytimes.com/2018/05/04/books/review/automating-inequality-virginia-eubanks.html, https://us.macmillan.com/books/9781250074317, https://www.ruhabenjamin.com/race-after-technology) 최근에 제가 AI 인터뷰가 많이 쓰인다는 기사를 보고 열을 냈던 이유 중 하나로, 다들 내 얘기는 아니겠거니 하지만 이런 증폭된 부정적인 면은 궁극적으로 모두를 해하게 됩니다. 혹시 본인의 자녀가 어린 시절 잠깐 강남이 아닌 곳에서 초등학교를 다니는 바람에 AI 인터뷰에서 자동적으로 떨어진 건 아닐까요?

심지어는 완벽한 AI 시스템이 존재해도 amplification 문제는 여전히 존재합니다. 만약 AI 시스템에서 면접 보는 사람이 60%의 확률로 성공적일 것이라고 하고, 실제로 60%가 완벽한 (un)certainty라면 어떻게 할까요? 아마 모두 합격이라고 결정할 것 입니다. AI 시스템이 실전에 사용되면 해당 시스템의 uncertainty를 넘어서는 결정을 내리게 되고 amplification이 더 심해집니다.

Opaqueness of a model

AI/ML 시스템이 현업에서 집중적으로 쓰이기 시작한 것은 꽤 오래된 일지만 이러한 시스템의 complexity가 급격히 높아진 것은 상대적으로 최근입니다. 이런 highly complex한 시스템을 deploy하는 입장과 사용하는 입장 그리고 적용받는 입장에서는 해당 시스템의 특징에 대해 알아야 합니다. 아쉽게도 동작 원리를 알아내는 것은 어렵고 연구 중 또는 기업기밀 이라는 핑계 아래 이런 필요성이 무시 당하곤 합니다. 당연히 어렵고 연구 중인 내용이긴 하지만 실제로 사용자 그리고 적용받는 입장에서는 세세한 과학적 원리를 요구하는게 아니고 해당 시스템의 높은 수준의 동작 원리, 사회적 영향 등 을 필요로 할 뿐 입니다 (환경을 생각해서 자동차 배기량이 얼마나 되는지 알고 싶은데 갑자기 내연기관의 원리 및 해당 차종의 모든 디테일을 알지 못하면 배기량을 아는 것은 의미가 없다면 말이 안 되겠죠.) 이런 내용들이 고지 되지 않으면 앞서 말한 amplification으로 인한 부정적인 영향을 이미 돌이킬 수 없는 상황이 되어서나 알 수 있습니다.

이를 위해서는 model card (https://dl.acm.org/doi/abs/10.1145/3287560.3287596) 및 datasheets for datasets (https://arxiv.org/abs/1803.09010) 등이 이제 겨우 시작이지만 좋은 방향으로 여겨집니다. 과연 자사 AI 시스템을 자랑하는 CEO/CTO 또는 개발자 중 model card와 dataset datasheet에서 추천하는 질문을 자사 시스템에 대해 했을 때 답할 수 있는 사람이 얼마나 될까요? 저 스스로도 잘 못 합니다만 특히나 AI 시스템을 deploy하는 입장에서는 이런 문제에 대한 답을 꼭 할 수 있어야 합니다.

Selection bias of data

위의 내용도 밀접하게 연결되는 내용으로 AI 시스템을 만드는데 사용되는 데이타가 어떻게 만들어지는지가 큰 문제입니다. 다만 이에 대한 논의는 데이타를 많이 사용하는 다른 분야에 비해 (예, survey) 상대적으로 잘 이뤄지지 않습니다. 최근 들어 AI/ML에 대한 관심이 높아지면서 다행히 data에 대한 관심도 많이 높아지고 있고 이에 따라 기존에 눈치 채지 못했던 다양한 문제들이 드러나고 있습니다. 예를 들어 Parbhu & Birhane ( https://arxiv.org/abs/2006.16923) 는 CIFAR-10이란 매우 유명한 데이타셋을 만드는데 사용되었던 TinyImage dataset의 심각한 문제점들을 발견했고 이를 통해 TinyImage dataset이 take-down되었습니다. 지금이야 take-down되었지만 과연 그전까지 해당 데이타를 사용한 AI/ML 시스템들이 데이타의 문제를 고민 하지 않고 만들어진 후 얼마나 현실에 적용되었는지 생각해보지 않을 수 없습니다. Gururangan et al. (https://arxiv.org/abs/1803.02324) 은 자연어처리 분야에서 굉장히 넓게 사용되는 Stanford NLI 데이타 안에 들어있는 문제점을 발견했고, 해당 문제점이 데이타 수집 과정에서 생겼다는 것을 보였습니다. 이런 문제점 발견에는 최신 AI/ML 기술 및 연구자 개개인의 manual한 노력이 필요했습니다.

일반적으로 AI 시스템이 얼마나 잘 동작하는지 자랑하는 기사 및 논문을 보는 것은 어렵지 않습니다. 하지만 사용자 및 AI 시스템의 판단을 받는 사람으로써 더 중요한 것은 과연 해당 시스템이 어떤 특징을 갖고 있는지, 그리고 해당 AI 시스템을 만드는데 사용된 데이타가 얼마나 잘 수집되고 정제되었는지가 더 중요합니다. 이를 위해 더 많은 연구가 필요하고 현업에서는 실제 AI 시스템 개발보다도 더 큰 투자와 노력을 기울여야 합니다.

최근 FB에서 나온 연구 결과를 보면 데이타의 영향이 얼마나 큰지 알 수 있습니다 (https://openaccess.thecvf.com/content_CVPRW_2019/html/cv4gc/de_Vries_Does_Object_Recognition_Work_for_Everyone_CVPRW_2019_paper.html). 이 논문에서는 상용 object recognition 시스템의 정확도가 사진이 찍힌 지역의 소득과 correlate한다는 것을 보였습니다. 혹시 전라남도에 살면 서울에서 모인 데이타가 압도적으로 많이 쓰인 네이버 OCR이 덜 정확한건 아니겠죠? (http://kosis.kr/statHtml/statHtml.do?orgId=101&tblId=DT_1C65, 사실 네이버 OCR이 어떻게 만들어지는지 모릅니다. 다만 서울/경기에서 모인 데이타가 대부분일 것으로 생각되긴 하네요.)

이를 해결하기 위한 방향으로는 human-and-machine-in-the-loop이라는 패러다임이 promising해 보입니다: https://arxiv.org/abs/1909.12434, https://arxiv.org/abs/1910.14599, https://openreview.net/forum?id=H1g8p1BYvS. 다만 이런 패러다임은 어떻게 구현을 하느냐에 따라 결과가 크게 달라질 수 있고, 구현하는 과정에서 피해를 보는 사람들이 생길 수도 있습니다 (에를 들면 https://www.theverge.com/2019/2/25/18229714/cognizant-facebook-content-moderator-interviews-trauma-working-conditions-arizona)

Correlation vs. Causation & systematic generalization

종종 이런 문제는 기술의 문제가 아니라고 주장하는 사람들이 있습니다. 이런 주장은 보통 AI/ML의 근본적인 목표를 이해하지 못해서 하는 것 입니다. 특히나 AI/ML의 목표와 주어진 데이타의 sufficient statistics를 뽑아내는 것을 동일하게 보는 경우가 있는데, 이건 사실이 아닙니다.

AI/ML의 목표는 일반적으로 inductive inference고, Vapnik에 의하면 이것은 “an informal act [with] technical assistance from statisticians” (paraphrase) 입니다. 조금 더 최근에 나온 Arjosvky et al. (2019; invariant risk minimization https://arxiv.org/abs/1907.02893)에서는 좀 더 분명하게 “minimizing training error leads machines into recklessly absorbing all the correlations found in training data” 하여 “machine learning fails to fulfill the promises of artificial intelligence” 라고 합니다. 한 마디로 AI의 목표는 데이타 수집 환경에 구애 받지 않는 mechanism (언제나는 아니지만 많은 경우 causal) 을 찾아내서 out-of-domain (또는 systematic) generalization을 성공적으로 수행하는 것을 목표로 합니다.

안타깝게도 기존에 사용되는 대부분의 ML algorithm들은 이런 면이 부족합니다 (이런 예가 궁금하면 최근 제 발표의 초반을 보면 됩니다: https://drive.google.com/file/d/1CrkxcaQs5sD8K2HL2AWCMnrMRpFoquij/view) 이를 극복하기 위해 meta-learning과 IRM 등의 새로운 paradigm도 제시되고 causal inference from observational data를 ML에 적용시키는 연구도 많이 진행되고 있습니다 (예를 들면 https://arxiv.org/abs/1911.10500, https://arxiv.org/abs/1901.10912, https://arxiv.org/abs/1805.06826.)

단순히 데이타에 있는 correlated feature를 알고리즘이 찾은 것인데 어째서 그것이 문제이냐 묻는다면 일단 AI/ML이 무엇인지에 대한 고민부터 다시 해야 합니다.

English version

Although it’s a topic that’s actively discussed both in academic settings and social media, such as Twitter and FB, I haven’t seen much discussion on the Social Impact & Bias of AI in Korean. To contribute even minimally to addressing this lack of discussion, here’s the list of a few points that are relevant to this topic. It’s possible that I simply have failed to find discussions surrounding this topic in Korean, and if there’s any, please kindly point me to them.

[My apologies for unprofessional writing. It’s not really everyday I write anything in Korean.]

Amplification

It is true that technology reflects the society. It is however also true that such technology that reflects the society is used within the society and that it inevitably amplifies what’s been reflected on the technology. It’s illuminating to read <Automating Inequality> by Virginia Eubanks and <Race after Technology> by Ruha Benjamin to see how such amplification harms people. (https://www.nytimes.com/2018/05/04/books/review/automating-inequality-virginia-eubanks.html, https://us.macmillan.com/books/9781250074317, https://www.ruhabenjamin.com/race-after-technology) This amplification of negative aspects of the society is precisely why I fumed over the recent news articles on wide adoption of AI inteviews in Korea. You may think you’re not the one who’ll suffer from such amplification, but it eventually gets to everyone unless without any intervention. Have you ever considered the possibility that your kid may not have received the job offer because he didn’t attend a primary school in Gangnam when they were small?

Even if one imagines a perfect AI system, the issue of amplification still exists. Consider this hypothetically perfect AI system that has determined a candidate to be 60% fit to the company and that this 60% is perfectly calibrated. As soon as a user of this system simply thresholds at 50% to make a hiring decision, it ends up with the same issue of amplification, because in practice users of such AI system inevitably overrule the supposedly perfect uncertainty estimated by the system.

Opaqueness of a model

Although it has been quite some time since so-called AI/ML systems have been put in practice, it’s relatively recent that their complexity has greatly increased. When a system in practice exhibits such a high level of complexity, it is important for both a provider, user of and those who are influenced by such a system to be aware of the principle behind these systems. Unfortunately there’s a trend that this need and request for awareness are ignored based on a variety of excuses such as that it is difficult to know the full details of the working principles, it is under active research to figure out the working principles and it is a part of corporate secret. Of course it is a difficult scientific issue on its own, but what is needed in terms of transparency is not every single scientific and engineering detail but a high-level description of the working principle behind such systems and understanding of their impacts on the society (think of how ridiculous it would be when a car manufacturer doesn’t tell you the horse power of a car you are considering because there’s no way you can know about all the details of the car such as the minute details of internal combustion engines.) Unless these (even high-level) details are provided together with these AI systems, the negative impact of such systems on the society will only be discovered once the (potentially irreversible) damages have been made.

One promising direction I have observed in recent years is the proposal for model cards and datasheets for datasets: https://dl.acm.org/doi/abs/10.1145/3287560.3287596 and https://arxiv.org/abs/1803.09010. I wonder how many CEO/CTO and developers can answer the questions, suggested for the model cards and datasheets, about their own AI systems they tout as well as data used for those systems. I’m not particularly a good example myself, but I believe the bar is even higher for those who tout and deploy AI systems in the society.

Selection bias of data

It’s quite related to the previous point. It is important to think of how data used for building an AI system was collected and created. Unfortunately and perhaps surprisingly this aspect of data has received relatively little attention compared to other adjacent areas, but the research community has begun to pay more attention to data itself and notice various issues behind widely used datasets. For instance, Parbhu & Birhane (https://arxiv.org/abs/2006.16923) identified serious flaws and issues behind one of the most widely used image datasets, called TinyImages, from which the widely used CIFAR-10 was created. This has led to the removal of the TinyImages dataset after 10 years since the original dataset was created and released. Although it’s now removed, you must wonder how many AI systems have been built using this data and been deployed in practice. Gururangan et al. (https://arxiv.org/abs/1803.02324) found various issues (or artifacts, as they called them) in the Stanford natural language inference (SNLI) data, stemmed from the process of data collection. These findings are the result of the combination of both state-of-the-art AI/ML techniques and individual researchers’ manual efforts.

It’s not difficult to find news articles and academic papers bragging the awesomeness of their AI systems. It is however more important for users and people who are being (either intentionally or unintentionally) judged by such systems to know the properties and characteristics of such systems and to be able to trust the quality of data and its collection process. It is thus imperative to invest more on this aspect of quality assurance than on the actual development of AI systems, in addition to continued research.

A recent work from FB demonstrates well the impact and importance of data and its collection: https://openaccess.thecvf.com/content_CVPRW_2019/html/cv4gc/de_Vries_Does_Object_Recognition_Work_for_Everyone_CVPRW_2019_paper.html. In this paper, the authors demonstrated that the accuracies of commercial object recognition systems correlate with the income levels of the regions in which pictures were taken. Hopefully, it doesn’t mean that the OCR service from Naver is less accurate for those who live in Jeollanam-do (which has the lowest per-capita GDP in Korea according to http://kosis.kr/statHtml/statHtml.do?orgId=101&tblId=DT_1C65) because the OCR system was trained mainly using data from Seoul and its metropolitan area (to be honest, I have no idea how Naver OCR is implemented, but I’m quite sure the majority of data used for building the system were collected from Seoul and its surrounding regions.)

To me, human-and-machine-in-the-loop paradigm looks quite promising: https://arxiv.org/abs/1909.12434, https://arxiv.org/abs/1910.14599 and https://openreview.net/forum?id=H1g8p1BYvS. Although promising, it’s important to keep in our mind that the outcome of such a paradigm heavily depends on how it’s implemented, not to mention that some may suffer from its implementation. See for instance https://www.theverge.com/2019/2/25/18229714/cognizant-facebook-content-moderator-interviews-trauma-working-conditions-arizona.

Correlation vs. Causation & systematic generalization

Often we see people who claim this is not the problem of technology. Such a claim often arises from the lack of understanding the fundamental goal of AI/ML. In particular, some equate the goal of AI/ML to estimating sufficient statistics from given data, which is simply not true.

In general, the goal of AI/ML is inductive inference, and according to Vapnik (https://www.wiley.com/en-us/Statistical+Learning+Theory-p-9780471030034), it’s “an informal act [with] technical assistance from statisticians” (paraphrase). More recently, Arjosvsky et al. (https://arxiv.org/abs/1907.02893) explicitly stated that “minimizing training error leads machines into recklessly absorbing all the correlations found in training data” and this makes “machine learning [fail] to fulfill the promises of artificial intelligence.”In short, the goal of AI is to identify an underlying mechanism that is independent of (or invariant to) changing environments (which are often but not always causal) and successfully generalize to a new environment, which is often referred to as out-of-domain (or systematic) generalization.

Sadly, most of the existing (widely used) ML algorithms fall short in this aspect. See the first part of my recent talk for an example: https://drive.google.com/file/d/1CrkxcaQs5sD8K2HL2AWCMnrMRpFoquij/view. In order to overcome this inability, new paradigms have been proposed, such as meta-learning and invariant risk minimization, and there is an on-going effort in marrying causal inference from observational data with machine learning. See e.g. https://arxiv.org/abs/1911.10500, https://arxiv.org/abs/1901.10912 and https://arxiv.org/abs/1805.06826.

If you still insist that it is not an issue of the algorithm which has faithfully captured correlations that exist in data, I suggest you to think once more what AI/ML is and what its goal is.