Artificial Intelligence in Safety Systems is Coming
Standards for functional safety and safety instrumented systems have warned against the use of artificial intelligence. Going back to the 1996 version of ISA 84, the precursor to IEC 61508 and those associated standards, artificial intelligence was specifically excluded as an allowable technique to achieve functional safety. Even the most recent version of IEC 61511 contains a requirement related to safety configuration of general-purpose low-variability-language programmable electronic systems that it be configured such that “the system does not use artificial intelligence or dynamic reconfiguration”. The general explanation for the exclusion of AI techniques in safety is that AI techniques are not systematically produced and the method by which the AI makes decisions is not understandable to humans. As a result, verification and validation that an AI based safety function achieves its objectives is if not impossible, is not straight-forward using today’s methods and techniques.
All of this is about to change though. Very complex safety applications of novel technology are not solvable without AI techniques such as machine learning. One of the significant drivers of AI techniques for control and safety is autonomous vehicles as discussed in ISO 26262. Systems that use cameras to detect that a human is in the road and take stopping or evasive action are essentialy. Detection of humans in roadways is simply not feasible using traditional sensing technology, and instead camera images are processed to determine if there are human obstructions in a vehicle’s path. This type of pattern recognition is performed through machine learning techniques that will train a neural net to decide whether or not an image contain a human in it. This determination is obviously a safety critical control functionality, that deserves an appropriate amount of attention throughout its entire safety lifecycle, including hazard and risk analysis to set requirements and verification and validation to ensure that the system operates properly. The obstacle to overcome is verification and validation of a system that not only cannot be reproduced by a human, it often cannot even be understood by a human.
The functional safety community is starting to make forward progress on this front with the development of ISO/IEC TR 5469 – Artificial Intelligence – Functional Safety and AI Systems. This document has been released for comment in an early draft form to members of the technical advisory groups of IEC’s SC65A. Kenexis President Ed Marszal is a member of the US Technical Advisory Group for SC65A which is responsible for functional safety of E/E/PE systems including related software. Mr. Marszal had this to say about the current version of the draft technical report.
The technical report is an excellent starting point for beginning the adoption of AI techniques into functional safety applications across all industries. The report is a very early and very necessary step toward this goal. The report does not have all of the answers of this nascent and developing field, but instead begins laying the groundwork for how we talk about and thing about AI in functional safety applications. It provides some key definitions for types of AI that can be used in functional safety related applications and discusses and categories different use cases. For instance, failure of some AI techniques might be initiators that trigger traditional safety functions, while other AI techniques might be used in the safety functions themselves. The report goes on to classify different techniques and approaches that can be implemented for verification and validation of the objectives of safety instrumented systems that do not rely on a systematic knowledge of how the AI makes its decisions, only the end results of the decisions, a kind of black-box versus white-box approach. While the report is only setting initial frameworks and definitions, it is a critical starting point to let the industrial world know that we are serious about AI in functional safety applications and to allow us to start using the same concepts, classifications, and definitions.