To mark the summit for action on artificial intelligence (AI), ANSSI (the French national cybersecurity agency, Agence nationale de la sécurité des systèmes d’information) published this February a document entitled: “building trust in AI through a cyber risk-based approach”.
This publication, the fruit of work with many international cybersecurity agencies and authorities, aims to encourage the use of AI systems while “taking into account the opportunities and risks of AI and the evolving cyber threat”.
ANSSI’s decision to look at trust in AI through the lens of risks and opportunities is strongly reminiscent of the contract manager’s work! What is more, this publication, which presents itself as a “high-level analysis”, lends itself particularly well to being read and absorbed by a contract manager audience.
In this article we therefore revisit that publication, and more specifically the good contract management practices to adopt when rolling out AI systems.
A. IT risks: a few reminders
An AI system runs on IT infrastructure that carries the same vulnerabilities as a traditional information system. AI, however, presents specific risks arising from the nature of its algorithms and its dependence on data.
ANSSI identifies three broad categories of attack targeting these systems:
- Poisoning of training data (data poisoning): introducing malicious data to corrupt learning and skew results.
- Model extraction (model stealing): unlawfully retrieving the model in order to replicate it or draw sensitive information from it.
- Evasion attacks (adversarial attacks): manipulating inputs to fool the algorithm and obtain erroneous decisions.
These attacks can therefore impair an AI system to a greater or lesser degree, from availability failures (critical for SLAs) to issues affecting the reasoning or decision-making autonomy of these systems, and on to the theft or disclosure of sensitive data.
So far, nothing new compared with the risks facing traditional information systems. AI systems do, however, present one major difference, a source of additional risk: their opacity, due to so-called “black box” models. While these mechanisms are not new, the growing number of parameters handled inside these black boxes (1.8 trillion in the case of GPT-4), together with the use of such models in banking, justice or healthcare, legitimately raises questions about these new risks.
B. AI: the new risks
Beyond attacks directly targeting AI algorithms, ANSSI highlights five major risks specific to integrating AI into an information system:
- Compromise of the hosting and administration infrastructure, through known technical, organisational or human vulnerabilities
- Compromise of the supply chain, whether the datasets or the open source libraries used to develop AI systems, which could be attacked
- Interconnections between AI systems and other components of the information system, since the interfaces between AI and information systems (ERP, CRM and other industrial systems) can be used to extract data, compromise results and so on
- Human and organisational risks, including shadow AI or excessive reliance on AI systems, which can lead to data leaks, an inability to detect anomalies, breaches of laws and regulations or reputational damage.
- Malfunctions in the answers produced by AI systems, in particular through compromised training designed to generate erroneous responses
C. What good practices for the contract manager?
Having reviewed the main families of attack and listed the principal risks specific to AI systems, it is time to look at the good practices ANSSI recommends for handling those risks, and to draw out those that belong in a contract manager’s own good practice.
These good practices come, of course, on top of those that apply to information systems more generally.
The first good practice is to question the objective being pursued, the proportionality of the chosen solution and the existence of safeguards. While the trend is towards massive automation and use of AI, you must make sure that continuous control and supervision mechanisms exist, allowing on the one hand human validation of critical operations and, on the other, assurance that the system works properly, without bias or hallucinations and with reliability levels that match expectations. These mechanisms can then be covered by contractual clauses clearly setting out the arrangements and the supervision terms, but must also be added to the agenda of steering committees and other governance meetings.
After this first round of questioning comes the analysis of the AI system! What is this system made of? A good practice is to map the main components of the AI system in order to identify the elements (hardware, software, data and algorithms) that need close monitoring. This means questioning where those components come from, how reliable and available they are, and so on, so that contractual mechanisms (clauses, monitoring, penalties, etc.) can be put in place to prevent drift.
Once the system’s components have been mapped, you should also confirm the roles and responsibilities of the various players at the interfaces between AI systems and other parts of an information system, in order to avoid compromises and other attacks at those interfaces. This can take the form of RACI matrices inserted into the contracts, among other coordination obligations between cross-functional players.
Finally, the last good practice may sound trivial, but it remains essential. It consists in training yourself and staying informed about AI, and at the very least monitoring technological and regulatory developments. There are plenty of newsletters for this, some written for a general audience, others more technical, which keep any contract manager, whatever their level, afloat!
Conclusion
Introducing AI systems into business processes offers considerable opportunities, but it comes with specific risks that demand a rigorous contractual approach. The contract manager must build these issues into negotiations, drafting and contract management & administration in order to ensure effective and secure AI governance.
By applying the good practices identified – clarifying responsibilities, framing data and algorithms, putting control mechanisms and technology monitoring in place – it becomes possible to harness the potential of AI while minimising its risks, and who knows, perhaps find opportunities in it.