Layer II, Framework for AI Cybersecurity Practices (FAICP), from ENISA
Layer II (AI-specific).
Layer II covers cybersecurity practices needed for addressing the specificities of the AI components, with a view on their life cycle, properties, threats, and security controls, which would be applicable regardless of the industry sector.
AI legislation.
The cybersecurity legislation presented in the first layer is complemented with AI-specific legislative efforts. The most important Commission proposal is the AI Act, which puts forward the proposed regulatory framework on AI with the following specific aims:
• ensure that AI systems placed on the EU market or put into service are safe and respect existing law on fundamental rights and EU values;
• ensure legal certainty to facilitate investment and innovation in AI;
• enhance governance and effective enforcement of existing law on fundamental rights and safety requirements applicable to AI systems;
• facilitate the development of a single market for lawful, safe and trustworthy AI applications and prevent market fragmentation.
In addition to the AI Act proposal, the Commission has published a proposal for an AI liability directive, whose purpose is to ‘improve the functioning of the internal market by laying down uniform rules for certain aspects of non-contractual civil liability for damage caused with the involvement of AI systems’.
The Artificial Intelligence Act
Types of AI.
According to the OECD, ‘An AI system is a machine-based system that can influence the environment by producing an output (predictions, recommendations or decisions) for a given set of objectives. It uses machine and/or human-based data and inputs to:
(i) perceive real and/or virtual environments;
(ii) abstract these perceptions into models through analysis in an automated manner (e.g. with ML) or manually; and
(iii) use model inference to formulate options for outcomes. AI systems are designed to operate with varying levels of autonomy.’
AI is a broad topic which can be further dissected into multiple subfields, which in turn are often mentioned interchangeably. Some of these are described below.
• Computer vision. This is related to the automatic processing of visually rich data such as images or videos. Some of the main tasks under this domain are object detection, facial recognition, action/activity recognition and human pose estimation.
• Expert systems. Expert systems are highly interpretable white-box programs that use a knowledge-based approach, where domain information provided by experts in the field is used by a knowledge engineer to populate a knowledge base (e.g. a set of if–then rules). At the inference phase, the content of the knowledge base is used by an inference engine to derive new conclusions for a given set of observed facts.
• Machine learning. ML is arguably the most disruptive subfield of AI, introducing a new paradigm for the design of intelligent systems. ML algorithms can learn predictive rules from hidden patterns in labelled/unlabelled data on their own, without needing to be explicitly programmed for a specific task. Furthermore, deep learning (DL), which mimics the structure and way of working of the human brain, is currently the most promising branch of ML, benefiting from large amounts of available data.
• Multi-agent systems. These are part of distributed AI and address the interaction between several autonomous entities designated as agents. Agents can perceive their surrounding environment on their own, and collaborate or negotiate with other agents to interact with them in a beneficial manner.
• Natural language processing. This makes use of computational techniques to learn, understand and produce content in human language with respect to several levels of linguistic analysis.
• Robotics. Robotics is related to the development of physical machines with variable degrees of autonomy. These are able to continuously adapt to their ever-changing environments by several loops of actions such as perceiving, planning and executing.
• Speech recognition. The speech recognition domain encompasses methods for processing speech automatically, providing better ways of interfacing with computers.
ML and DL undoubtedly pose the main challenges to security, as grey-box and black-box models dominate the field and imply a dynamic analysis of the threats, not just along the life cycle, but also in the interrelations within other blocks of an ICT infrastructure. The following sections discuss many of the threats related to this subfield.
No-code AI reduces the time to build AI models to minutes, enabling companies to easily adopt ML models in their processes. No-code AI solutions are focused on helping non-technical users build ML models without getting into the details of every step in the process of building the model. This makes them easy to use but harder to customise.
Multiple no-code AI platforms, i.e. software that allows people without specialised skills to build algorithms, are proliferating rapidly. In the future, people will not just want to deploy different models, but potentially thousands of pieces of AI software. They will be able to design and create their own algorithms.
Empowering every employee to build and train AI algorithms will make it impossible to assess the trustworthiness of these algorithms in terms of transparency, ethical, data privacy, non-bias or governance pitfalls. The rise of no-code AI makes it imperative to develop strong auditing tools and policies around the use of AI and have systems in place to ensure that everyone using the no-code software understands and abides by these policies. Advanced tools are needed to audit how these no-code AI models have been trained, in order to secure them by design.
AI assets and procedures
The AI domain is broad and therefore requires a structured and methodical approach to understand its different facets. ENISA has proposed a generic reference model for a functional overview of typical AI systems. However, due to the vast number of technologies, techniques and algorithms involved in these systems, mapping them all in a single life cycle would be too ambitious.
ENISA then proposed a life cycle, illustrated in Figure 7, that is based on ML, as the particularities of the many subfields of AI – namely natural language processing, computer vision, robotics, etc. – make use of ML that has been spearheading the explosion of AI usage in different domains.
In the same report, ENISA identified the most relevant assets, based on the functional description of specific stages and, in order to reflect AI components, also including assets that support the development and deployment of AI systems.
• Data. Raw data, public data sets, training data, testing data, etc.
• Models. Algorithms, models, model parameters, hyper-parameters, etc.
• Artefacts. Data governance policies, descriptive statistical parameters, model frameworks, etc.
• Actors/stakeholders. Data owners, data scientists, data engineers, model providers, etc.
• Processes. Data ingestion, data pre-processing, data collection, data augmentation, feature selection, training, tuning, etc.
• Environment/tools. Algorithm libraries, ML platforms, optimisation techniques, integrated development environments, etc.
AI threat assessment.
AI systems greatly contribute to automate and enhance decision-making in a wide variety of day-to-day tasks, enhancing business processes all over the world. Nonetheless, as with any other ICT system, AI-powered ones can also be victims of cybercriminals and multiple cybersecurity threats with the objective of hijacking their normal functioning for malicious purposes.
The additional required risk assessment efforts that are specific to AI must:
• include not only technical and physical threats, but also threats mentioned in the EU AI Act, such as loss of transparency, loss of interpretability, loss of managing bias and loss of accountability;
• enhance the types of impact factors, such as robustness, resilience, fairness and explainability;
• be dynamic and combined with anomaly detection approaches, as for ICT systems in general.
ETSI has published an AI threat ontology to define what would be considered an AI threat and how it might differ from threats to traditional systems.
As explained in the NIST AI Risk Management Framework, AI systems are socio-technical in nature, meaning that the threats are not only technical, legal or environmental (as in typical ICT systems), but social as well. For example, social threats – such as bias, lack of fairness, lack of interpretability / explainability / equality – are directly connected to societal dynamics and human behaviour in all technical components of an AI system, and they can change during its life cycle.
How these societal threats can impact individuals with different psychological profiles, groups, communities, organisations, democracies and society as a whole need to be analysed and measured before we estimate the risks. Actually, events that can compromise the characteristics of AI systems, as described in Figure 9 in the next section, are specific threats for AI systems which are social, policy and technical AI threats.
For example, bias is a new threat targeting the AI system and the different stages of the AI life cycle (design, development, deploying, monitoring and iteration), as analysed in the BSA framework. The CEPS Artificial Intelligence and Cybersecurity – Technology, governance and policy challenges report also provides an overview of the current threat landscape of AI, ethical implications and recommendations. The ARM framework provides a simple interactive approach to explain the various principles of trustworthy AI.
The AI threats themselves can be of several types and affect all AI subfields. These can be mapped into a high-level categorisation of threats based on ENISA’s threat taxonomy, comprising:
• nefarious activity/abuse
• eavesdropping/intercept/hijacking
• physical attacks
• unintentional damage
• failures or malfunctions
• outages
• disaster
• legal.
On the other hand, ML-related threats can affect different steps of the ML life cycle. The most important high-level ML threats can be described as follows.
• Evasion. Evasion is a type of attack in which the attacker works with the ML algorithm input to find small perturbations which can be used to exploit the algorithm’s output. The generated input perturbations are designated as adversarial examples.
• Poisoning. In a poisoning attack, the attacker alters the data or the model to modify the ML algorithm’s behaviour in a chosen direction (e.g. to sabotage its results or to insert a back door) according to its own motivations.
• Model or data disclosure. This threat is related to the possible leaks of all or partial information about the model, such as its configuration, parameters and training data.
• Compromise of ML application components. This threat refers to the possible compromise of an ML component, for example by exploiting vulnerabilities in the open-source libraries used by the developers to implement the algorithm.
• Failure or malfunction of an ML application. This threat is related to the failure of the ML application. It can be caused by denial of service due to a bad input or by the occurrence of an untreated handling error.
All of these threats can be mapped to multiple vulnerabilities, such as lack of training based on adversarial attacks, poor control over which information is retrieved by the model, lack of sufficient data to withstand poisoning, poor access rights management, usage of vulnerable components and missing integration with the cyber resilience strategy.
In a report of a quantitative study with 139 industrial ML practitioners, despite most attacks being identified as related to the ICT infrastructure, some ML-related attacks were also identified. The number of reported AI threats was marginal, with 2.1 % of evasion attacks and 1.4 % of poisoning attacks recognised by the organisations.
AI security management
The RM conducted for an entire infrastructure will need to be complemented with conducting RM in all AI systems hosted in the ICT infrastructure.
This section introduces AI properties and the security controls that can be employed to minimise the impact of AI threats aimed at compromising AI trustworthiness. The ISO 2700x standards, the NIST AI framework and ENISA’s best practices can all be used for AI RM and it is strongly recommended that they be followed when implementing more general-purpose security controls.
AI trustworthiness
In order to understand the concepts and risks associated with the usage of AI, it is important to start by analysing the level of trustworthiness and the desirable properties to consider. We define AI trustworthiness as the confidence that AI systems will behave within specified norms, as a function of some characteristics such as: accountability, accuracy, explainability, fairness, privacy, reliability, resiliency/security, robustness, safety and transparency.
In this section, an overview of these characteristics is provided, along with their relationships with the risk assessment framework based on NIST.
• Accountability. Ensures responsibility for AI, which in turn implies explanation and justification; humans and organisations should be able to answer and be held accountable for the outcomes of AI systems, particularly adverse impacts stemming from risks.
• Accuracy. Correctness of output compared with reality; RM processes should consider the potential risks that might arise if the underlying causal relationship inferred by an AI model is not valid.
• Explainability. Provides a description of the conclusion/decision made in a way that can be understood by a human; risks due to explainability may arise for many reasons, including for example a lack of fidelity or consistency in explanation methodologies, or if humans incorrectly infer a model’s operation, or the model is not operating as expected.
• Fairness. Neutrality of evidence, not biased by personal preferences, emotions or other limitations introduced by the context, equality (of gender and opportunity). Fairness is a concept that is distinct from but related to bias. According to ISO/IEC TR 24027:2021, bias can influence fairness. Biases can be societal or statistical, can be reflected in or arise from different system components and can be introduced or propagated at different stages of the AI development and deployment life cycle.
• Privacy. Secure management (process, analysis, storage, transport, communication) of personal data and training models; ability to operate without disclosing information (data, model); identifying the impact of risks associated with privacy-related problems is contextual and varies among cultures and individuals.
• Reliability. Ability to maintain a minimum performance level and consistently generate the same results within the bounds of acceptable statistical errors; may give insights about the risks related to decontextualisation.
• Resiliency. Ability to minimise impact, restore safe operating conditions and come out hardened from an adversarial attack.
• Robustness. Ability of an AI system to maintain a previously agreed minimum level of performance under any circumstances; this contributes to sensitivity analysis in the AI RM process.
• Safety. Preventing unintended or harmful behaviour of the system to humans or society; safety is highly correlated to risks.
• Security. Ability to prevent deviations from safe operating conditions when undesirable events occur; ability to resist attacks; ensures confidentiality, integrity, authenticity, non-repudiation, availability of data, processes, services and models.
• Transparency. Ability to foster a general understanding of AI systems, make stakeholders aware of their interactions with AI systems and allow those affected by an AI system to understand the outcome. It also enables those adversely affected by an AI system to challenge its outcome based on plain and easy-to-understand information on the factors, and the logic that served as the basis for the prediction, recommendation or decision.
The NIST AI framework organises these characteristics in three classes (technical, socio-technical and guiding principles) and provides a mapping of the taxonomy to AI policy documents, as can be seen in Figure 9. The technical characteristics in the framework taxonomy refer to factors that are under the direct control of AI system designers and developers and which may be measured using standard evaluation criteria.
At this level, properties like accuracy, reliability, robustness and security are referred to in most of the documents. Socio-technical characteristics in the taxonomy refer to how AI systems are used and perceived in individual, group and societal contexts. At this level, the focus is on safety, explainability and privacy. The guiding principles in the taxonomy refer to broader societal norms and values that indicate societal priorities, where fairness, accountability, transparency and traceability are the most highlighted.
Security controls
On the other hand, specific ML security controls can be mapped for the introduced threats to provide efficient ways of prevention and mitigation. For evasion, tools can be implemented to detect whether a given input is an adversarial example, adversarial training can be used to make the model more robust, and models that are less easily transferable can be used to significantly decrease the ability of a given attacker to properly study the algorithm that works underneath the system.
Similarly, for poisoning attacks, processes that maintain the security levels of ML components over time should be implemented, the exposure level of the used model should be assessed, the training data set should be enlarged as much as possible to reduce its susceptibility to malicious samples, and pre-processing steps that clean the training data from such malicious samples must also be considered.
Model or data disclosure can be protected by applying proper access control and federated learning to minimise the risk of data breaches. Similarly, to reduce the level of compromise of ML application components, these should be compliant with protection policies, fully integrated to existing security operations and asset management processes, and evaluated according to the level of security of their foundation blocks (e.g. libraries that are responsible for the algorithm implementation).
Finally, to prevent failure or malfunction of ML applications, employed algorithms should have their bias reduced, should be properly evaluated to ensure that they are resilient to the environment in which they will operate and should encompass explainability strategies.
New challenges
The security of AI should be considered at all stages of its life cycle, taking into account the following elements.
• AI systems are multi-disciplinary socio-technical systems and their threats are technical, societal, ethical and legal. Collaboration between cybersecurity experts, data scientists, social scientists, psychologists and legal experts is needed in order to identify the continuous evolving AI threat landscape and develop corresponding countermeasures.
• Among the different types of AI, ML and DL undoubtedly pose the main challenges to security and imply a dynamic analysis of the threats, both along the life cycle and in the interrelations within other blocks of an ICT infrastructure.
• AI-specific risk assessment efforts need to consider their unique properties and enhance their robustness, resilience, fairness and explainability, along with preventing loss of transparency, loss of managing bias and loss of accountability.
• Assigning a test verdict is different and more difficult for AI-based systems, since not all of the expected results are known a priori.
Contact us
Cyber Risk GmbH
Dammstrasse 16
8810 Horgen
Tel: +41 79 505 89 60
Email: george.lekatis@cyber-risk-gmbh.com
Web: https://www.cyber-risk-gmbh.com
We process and store data in compliance with both, the Swiss Federal Act on Data Protection (FADP) and the EU General Data Protection Regulation (GDPR). The service provider is Hostpoint. The servers are located in the Interxion data center in Zürich, the data is saved exclusively in Switzerland, and the support, development and administration activities are also based entirely in Switzerland.
Understanding Cybersecurity in the European Union.
2. The European Cyber Resilience Act
3. The Digital Operational Resilience Act (DORA)
4. The Critical Entities Resilience Directive (CER)
5. The Digital Services Act (DSA)
6. The Digital Markets Act (DMA)
7. The European Health Data Space (EHDS)
10. The European Data Governance Act (DGA)
11. The Artificial Intelligence Act
12. The European ePrivacy Regulation
13. The European Digital Identity Regulation
14. The European Cyber Defence Policy
15. The Strategic Compass of the European Union
16. The EU Cyber Solidarity Act
17. The EU Cyber Diplomacy Toolbox
18. The Framework for Artificial Intelligence Cybersecurity Practices (FAICP)