The Multiple Personalities of Machine Learning Evaluators: The Case of Sustainable AI

Whilst the International Energy Agency forecasts a doubling of the energy consumption of AI between 2024 and 2026, accurately measuring the environmental footprint of AI has become a priority for tech companies engaged in CO2 emission reductions. In this context, machine learning (ML) developers become evaluators of their own work and are pushed to quantify the energy demands of their models and participate in broader life-cycle analyses. This first section of the paper investigates the tensions that ML evaluators face when trying to estimate the environmental footprint of AI. While some ML evaluators seek ways to measure the objective environmental footprint of ML, others insist on the necessity to look to future, potential harms and reductions enabled by AI systems. As a result, some ML evaluators focus on making AI systems more efficient by reducing the amount of resources needed to develop AI models and infrastructures, whilst others insist on expanding the range of actions to focus not only on AI but on the contexts in which it is used. In their view, regulators should insist on build "frugal", instead of efficient, AI systems. This entails restricting the use of AI to cases where it is necessary and, ideally, in service of the planet. The second section of the paper looks at the strategies ML evaluators develop to deal with these competing approaches. Some attempt to make their organization move beyond efficiency by advocating for more climate action. Others take more radical approaches and leave their companies altogether. Finally, some do not manage to resolve the tensions and struggle to continue working in their current environment. In the last section, we discuss the implication of these tensions for the reliability of the methods used to estimate the environmental footprint of AI.