It's not enough for it to work: you have to justify what is being automated and under what conditions
Ensuring that an AI system functions correctly and fulfills its applicable obligations does not, in itself, answer a prior question: why should it intervene in that decision? Regulations establish requirements regarding purpose, risks, oversight, and responsibilities based on the system and its context of use. However, it is up to each organization to examine whether automating that decision addresses a justifiable need, what consequences it is prepared to accept, and how it will translate its obligations and commitments into effective action. It is in this transition—from compliance to responsible decision-making—that many of the real challenges of deployment arise.
The problem becomes especially apparent when moving from the pilot phase to real-world use. Decisions that seemed purely technical then begin to require criteria for resolving tensions between objectives, interests, and potential impacts, as well as designated individuals with the effective capacity to act. It is at this point that principles cease to be sufficient as mere declarations and must help guide concrete decisions.
Principles guide; real cases force decisions
When GenAiA was born at the end of 2023 (GenAiA, 2023)(1), We began to articulate our work on AI ethics. The first step was to identify shared principles and bring them closer to organizations. Concrete cases soon revealed the limitations of that starting point: invoking autonomy, fairness, transparency, or human oversight helps guide a decision, but it doesn't determine how to act in a specific situation.
Saying that «there will be human oversight» can describe an effective guarantee or simply be a formal provision. Its effectiveness depends on the conditions under which it is exercised: that the supervisor has sufficient information, training, and time; understands the capabilities and limitations of the system; and has real authority to question its outcome or intervene when necessary. For high-risk systems, the AI Regulation expressly incorporates several of these requirements and mandates effective human oversight that is proportionate to the risk, the level of autonomy, and the context of use (Regulation (EU) 2024/1689, art. 14).(2). However, establishing supervision as a requirement does not in itself guarantee that it will work in practice, as research on algorithmic supervision has also pointed out (Green, 2022). This example illustrates a broader problem: the effective application of principles and requirements also depends on the technical, social, and organizational conditions in which the system operates (Mittelstadt, 2019; Selbst et al., 2019).
The frameworks exist; the challenge is to make them operational.
Frameworks already exist that address different dimensions of this problem. At the international level, UNESCO and the OECD have formulated principles and recommendations to guide the development and responsible use of AI, incorporating issues such as human rights, dignity, autonomy, and accountability (OECD, 2024).(3); UNESCO, 2021(4)). In the European context, the Guidelines for Trustworthy AI articulate the relationship between legality, ethics and robustness, while ALTAI translates that approach into a self-assessment tool (European Commission, 2019).(5), 2020(6)). From an organizational perspective, the PIO Model proposes to operationalize principles through indicators and observables, and its 2025 update links them to obligations and rights of the European framework (Sabater Coll & de Manuel, 2022; Lillo Campoy et al., 2025).
Other frameworks extend these guidelines to governance, risk management, and impact assessment. The NIST AI RMF 1.0 structures AI risk management into four functions (govern, map, measure, and manage) that allow for the integration of technical and organizational analysis throughout the system lifecycle (NIST, 2023).(7)). In parallel, ISO/IEC 42001 establishes requirements for an AI management system, while ISO/IEC 42005 provides guidance for assessing the potential effects of a system on people, groups and society (ISO, 2023).(8), 2025(9)).
The challenge is not to replace these instruments, but to integrate them coherently into governance decisions in each case. Model Cards represented a significant advance in documenting the uses, metrics, and limitations of models (Mitchell et al., 2019). However, an analysis of 32,111 cards showed that the sections on limitations, assessment, and environmental impact remained among the least completed (Liang et al., 2024). Even extensive documentation can leave unresolved questions about who should intervene, with what authority, and what response is triggered when harm occurs.
The decision to automate does not belong to a single discipline
Philosophy and ethics help examine whether the need being addressed justifies AI intervention, who might benefit or be excluded, and which decisions should not be delegated. The law defines what is legally permissible and establishes obligations, rights, and guarantees, especially when innovation may affect fundamental rights (Eguíluz et al., 2025). Mathematics and engineering provide evidence on properties that can be technically defined and evaluated, from the representation of uncertainty (Aldomà et al., 2026) to the causal analysis of explanations (Parafita et al., 2025), as well as differences between groups or changes in context. Governance articulates these perspectives in terms of roles, decision-making procedures, and the effective capacity to act.
Not everything that is technically possible deserves to be automated.
Before deploying a system, an organization should be able to answer five questions:
- What decision is intended to be automated and for what purpose?
- Which part of that decision is reasonable to delegate and which should remain under human judgment?
- Who can benefit or be harmed?
- What evidence will allow us to assess whether the system remains adequate for that context?
- And what conditions should lead to reviewing, limiting, or suspending its use?
These questions only acquire value when their answers can be justified and, where appropriate, supported by evidence. But not all of them can be answered in the same way. Some involve normative judgments (what purpose is justifiable, what effects are acceptable, or what part of a decision should not be delegated); others also require empirical and technical evidence about the system's behavior. Technical evidence alone does not determine what deserves to be automated, just as a justifiable purpose is insufficient if the system's performance in practice cannot be evaluated.
This distinction becomes especially important when AI moves from pilot testing to real-world use. A system may deliver good results in trials, yet its integration into a process may not be sufficiently justified if the organization hasn't defined the conditions under which its use remains appropriate and what signals warrant a review. The loss of trust and, in some cases, reputational damage, can then be the visible manifestation of a prior problem: failing to connect purpose, evidence, and the capacity for intervention.
The challenge isn't exclusive to large organizations. While resources, complexity, and applicable obligations may change, our experience with companies of varying sizes and maturity levels reveals a recurring difficulty: translating principles into decision-making criteria, relevant evidence, and review mechanisms that work in practice. At GenAiA's AI Regulation and Ethics Think Tank, we maintain that trust isn't declared in a press release; it's demonstrated in the design of procedures and the effective capacity to respond.
The ultimate question, therefore, is not simply whether the system works. If the context were to change tomorrow, or if someone were to question its effects, could the organization explain why it remains reasonable to use it, what evidence supports that decision, and what conditions would necessitate reconsidering it?
Authors
Axel Brando
Mathematician and computer engineer specializing in Trustworthy AI. Co-leader of the GenAiA AI Regulation and Ethics Think Tank. LinkedIn
Eva Fonseca López
Senior IT specialist. Co-leader of the GenAiA AI Regulation and Ethics Think Tank – LinkedIn
References
(1) GenAiA. (2023, November 28). GenAiA, the first Generative AI Association led by the Spanish entrepreneurial ecosystem, is launched. https://generativeaiassociation.org/articles/genaia-the-first-generative-ai-association-led-by-the-spanish-entrepreneurial-ecosystem-is-launched/
(2) European Parliament and Council of the European Union. (2024). Regulation (EU) 2024/1689 laying down harmonized rules on artificial intelligence. Official Journal of the European Union, L, 2024/1689.
(3) Organization for Economic Co-operation and Development. (2024). OECD AI Principles. https://oecd.ai/en/ai-principles
(4) UNESCO. (2021). Recommendation on the ethics of artificial intelligence. https://www.unesco.org/en/legal-affairs/recommendation-ethics-artificial-intelligence
(5) European Commission, High-Level Expert Group on Artificial Intelligence. (2019). Ethics guidelines for trustworthy AI. European Commission. https://digital-strategy.ec.europa.eu/en/library/ethics-guidelines-trustworthy-ai
(6) European Commission, High-Level Expert Group on Artificial Intelligence. (2020). Assessment list for trustworthy artificial intelligence (ALTAI) for self-assessment. European Commission. https://digital-strategy.ec.europa.eu/en/library/assessment-list-trustworthy-artificial-intelligence-altai-self-assessment
(7) National Institute of Standards and Technology. (2023). Artificial Intelligence Risk Management Framework (AI RMF 1.0) (NIST AI 100-1). US Department of Commerce. https://doi.org/10.6028/NIST.AI.100-1
(8) International Organization for Standardization. (2023). ISO/IEC 42001:2023: Information technology—Artificial intelligence—Management system. https://www.iso.org/standard/42001
(9) International Organization for Standardization. (2025). ISO/IEC 42005:2025: Information technology—Artificial intelligence—AI system impact assessment. https://www.iso.org/standard/42005
Aldomà, A., Gurbindo González, U., & Brando, A. (2026). Position: Predictive uncertainty is not enough—Joint distribution for full uncertainty representation [Position paper accepted at the 43rd International Conference on Machine Learning]. https://icml.cc/virtual/2026/poster/67233
Eguíluz, J., Brando, A., Laukyte, M., & Serra-Vidal, M. (2025). Position: If innovation in AI systematically violates fundamental rights, is it innovation at all? Advances in Neural Information Processing Systems, 38, Position Paper Track. https://proceedings.neurips.cc/paper_files/paper/2025/hash/1ae5c1db7569a6c2f395020765b119a4-Abstract-Position_Paper_Track.html
Green, B. (2022). The flaws of policies requiring human oversight of government algorithms. Computer Law & Security Review, 45, 105681. https://doi.org/10.1016/j.clsr.2022.105681
Liang, W., Rajani, N., Yang, X., Ozoani, E., Wu, E., Chen, Y., Smith, D.S., & Zou, J. (2024). Systematic analysis of 32,111 AI model cards characterizes documentation practice in AI. Nature Machine Intelligence, 6, 744–753. https://doi.org/10.1038/s42256-024-00857-z
Lillo Campoy, A., Brufau i Centelles, A., & Sabater i Coll, A. (2025). The PIO Model Obligations and Rights: A self-assessment approach for compliance with regulations on data and AI systems within the framework of the European Union. Observatori d'Ética en Intel·ligència Artificial de Catalunya, Universitat de Girona.
Mitchell, M., Wu, S., Zaldivar, A., Barnes, P., Vasserman, L., Hutchinson, B., Spitzer, E., Raji, ID, & Gebru, T. (2019). Model cards for model reporting. In Proceedings of the Conference on Fairness, Accountability, and Transparency (pp. 220–229). Association for Computing Machinery. https://doi.org/10.1145/3287560.3287596
Mittelstadt, B. (2019). Principles alone cannot guarantee ethical AI. Nature Machine Intelligence, 1, 501–507. https://doi.org/10.1038/s42256-019-0114-4
Parafita, Á., Garriga, T., Brando, A., & Cazorla, FJ (2025). Practical do-Shapley explanations with estimand-agnostic causal inference. Advances in Neural Information Processing Systems, 38.
Sabater Coll, A., & de Manuel, A. (2022). The PIO model (Principles, Indicators and Observables): A proposal for organizational self-assessment on the ethical use of data and artificial intelligence systems. Observatori d'Ética en Intel·ligència Artificial de Catalunya, Universitat de Girona.
Selbst, A.D., boyd, d., Friedler, S.A., Venkatasubramanian, S., & Vertesi, J. (2019). Fairness and abstraction in sociotechnical systems. In Proceedings of the Conference on Fairness, Accountability, and Transparency (pp. 59–68). Association for Computing Machinery. https://doi.org/10.1145/3287560.3287598