Functional Safety Podcast: episode (S1E56) is now available

Episode Preview and Introduction

During the FAT, you’re gonna find things wrong. You can’t just fix them. You gotta go back to wherever they occur in the safety lifecycle and fix everything from there. Welcome to the Kenexis Functional Safety Podcast. I’m your host, Ed Marszal, President and CEO of Kenexis. Kenexis is a technical safety consultancy that helps chemical process industry companies to analyze risk and design engineered safeguards like safety instrumented systems and fire and gas detection systems.

Kenexis also provides the industry-leading suite of software tools, including our best-in-class Vertigo software for SIS safety lifecycle management.

In this first season of the podcast, we are going to focus on the IEC 61511 standard, doing a deep dive into the standard, including more depth of information on what the standard means and how to apply it, brought to life with personal war stories and behind-the-scenes discussions of the committee members as we develop the standard in ISA 84 and IEC SC 65.

Before we start, a little disclaimer: I will be providing my opinion on technical and engineering topics. This information is provided on a best effort basis and is of a general nature. The information presented in this podcast might not be applicable to your specific application. It is the obligation of every engineer to thoroughly analyze any system that they are designing and not blindly rely on any general advice presented in this podcast.

Recap: Revision Control and the Open Audit Tool

All right, we are picking up and taking a look at the back end of the factory acceptance testing clause in the IEC 61511 standard. We got all the way through 13.2.3 last time, where 13.2.3, if you don’t remember, the FAT shall take place in a defined version of the logic solver. And we started talking about version numbers and revision numbers and approvals and, so on, and how everything in the safety instrumented system needs to have revision numbers and approvals, just basically a revision tab. That actually shows up in clause 19 when we’re at the very, very bitter end of this. We’ll talk about that. Probably talk about it a few times before then, but keeping track of what versions you’re on is something that we’ve got a lot of focus on here at Kenexis because it is important and it is a weakness in a lot of the workflows that are out there. So we’re here to help you with that type of thing.

And, you know, part and parcel of that would be doing your factory acceptance test in a tool like Open Audit. You think, you know, out of the gate, it’s like, well, I want a procedure tool because I’m executing a procedure. Well, an audit is kind of a procedure. You’re stepping through a workflow where you’re checking things and the whole concept in clause 13.2.3 the defined version of the logic solver in open audit, we refer to that as evidence. And we will be able to track the evidence with revision numbers. So the revision number of the cabinet layout diagram, the revision number of the software that we’re running all of our tests on, the revision number of the safety requirement specifications, and so on. So that kind of You know, linkage of what you’re testing to the test results is it’s built in. Open Audit does that really well. Also, the ability to generate and track recommendations, the ability to generate findings, which the findings will generate recommendations. And then you have the ability to export those. You have the ability to use artificial intelligence tools to manipulate that document any way you want because Kenexis makes those AI plugins available for the open audit tool. Also, just straight-up dashboarding and integration with business intelligence systems.

All of that is built into Open Audit out of the box. And did I mention? Did I mention? The desktop version of Open Audit is free. So something I am personally going to be working on is a mechanism to more easily push test plans into open audit. And by test plans, I mean things like FATs and validations, and that would be the SAT for those of you in the UK, or those of you in the US and just other activities.

We already have a lot of functional safety assessment and functional safety audit type templates in there, but doing these FATs commissioning and validation, I think you’re going to find that using a tool like Open Audit is going to help you to track everything to closure. And tracking everything to closure is going to be most of what we’re talking about in this episode, but definitely clause 13.2.7.

You’re going to want to track those modifications, or you’re going to track the things that you had an issue with all the way out to closure. And then 13.2.6 talks about documentation of what you did, which, you know, this makes it very easy to do that. And, you know, what the testing is. Anyway, so we I thought that last episode I’d be able to get all the way through FAT, and obviously I could not. Um, we still have clause 13.2.4 through 13.2.7, before we move on to the next clause, 14 installation and commissioning, where I just mentioned open audit is another great tool and it’s free on the desktop to be able to keep track of that installation and commissioning. But we’re gonna finish clause 13 today, which this episode should be on the shorter side, but you know, I just always find stuff to talk about. Maybe I will come up with some good stories on FAT.

War Stories: Building Custom Controllers at UOP

So, speaking of good stories on FAT, FAT is something that I personally have a boatload of experience in early in my career, where I was working at UOP and I was part of PICS, which was Process Information and Control Systems. That was the name that was developed in the 1990s.

All the way back in the 1970s, that branch of UOP was called Monorex, which, you know, that’s a name straight out of the 70s, but it’s kind of basically a custom systems integration firm that would develop specific controllers that were related to specific technologies at UOP. Specifically, the lock hopper control system for the UOP reforming process, the platforming process, where it’s a sequencing control system that removes catalyst out of the platforming reactor and burns the Coke off it and then puts the catalyst back in. So that’s that whole system was my introduction to safety because, well, if you open the valves in the wrong sequence, you’re going to send the flammable contents of your reactor to the portion of the process where you’re burning coke off and injecting air. So that would be bad. Also, as you’re putting the catalyst back in the process, well, if you do things the wrong way, you can push air into the reactor. That’s not going to be bad. That’s not going to be a good thing either.

So, because of the complexity, UOP basically, instead of sending specifications for the customers to do it themselves, UOP just said, Yeah, we’re just going to build the controller for you and program the logic and test it out.

In addition to that, the other thing over there at Monorax that I spent a lot of time with was the adsorbent chamber control system, which UOP’s Parex process, which would be their paraxylene separation process, is something referred to as a simulated moving bed. So, for those chemical engineers out there, what we’re trying to accomplish is a counter-current extraction between a liquid and a solid. So, how do you get a solid to move counter current with a liquid? That’s quite the trick because solids don’t want to flow like a liquid does. So instead, we create this fancy system where we inject and extract liquid at different points in a vessel to simulate that catalyst bed or the adsorbent bed specifically moving. So simulated moving technology, simulated moving bed technology, very, very cool stuff. I did a lot of work on that back in the 90s when I was in UOP2.

Anyway, I digress a little bit, but not too much because for every one of these controllers that I built and literally built is true-ish. A lot of the construction was done by some of the dedicated electrical guys. Rich McCarsky was one of those guys that I worked with quite a bit. So they would kind of do the panel layout and run all the wires and install stuff. But I would be responsible for putting together the networks that allowed everything to communicate together. That was back in the day when we did Ethernet communications over a CatFi, or no, not Cat5.

We were using coaxial cable. So I would run coaxial cable, strip it out, put the connectors on, hook everything together, load software in, and then ultimately, I was the leader of the factory acceptance test.

War Stories: Leading the FAT as Integrator and Owner’s Rep

So I would write the factory acceptance test plan. Now I cheated a little bit because we had a template that we could start from, so it’s 90% done. But in addition to preparing the procedure, I would also have to, you know, do all the customer interface and then kind of be the master of ceremonies as the test was being executed. So we would set up a little area over in Um, the PICS building there out, you know, pretty close to the O’Hare Airport in Chicago, is where this facility was. And we kind of put a little area together with everything curtained off, because we usually had, I don’t know, 10, 15 controllers in some stage of development. So we want to keep everybody kind of focused on where they’re supposed to be. Plus, as I mentioned in the last episode, there is some job safety assessments, some PPE concerns that we need to be concerned about as we’re doing these types of activities.

So, yeah, I would be the MC and kind of conduct everyone through the process. Sometimes the customer wanted to do all of the diddling with the test equipment. Sometimes they had me do it and they just watched it. Sometimes they had me do it and, well, you know, they just flew in from Asia and they’re kinda a little bit sleepy, and wow, anyway, you know what I’m talking about.

So, I’ve done tons of these FATs from the perspective as of the systems integrator/slash equipment vendor. I’ve also done a lot of these FATs as the representative of an operating company.

So oil, refinery, XYZ would hire me to attend the factory acceptance test on their behalf and make sure that the system does what it is supposed to do.

That’s from the very beginning of Kenexis, we’ve done that type of attend the FAT, attend the SAT.

And even there, we would help in the development of procedures for executing these activities. Back then, we used to use punch lists, so separate documents that contain the exceptions. So yeah, in the new world of relational databases, cloud-based software tools.

Man, things would have gone so much smoother if we had open audit then to generate the reports and do a lot of the work for us.

Clause 13.2.4: FAT Planning and Execution

But all right, well, let’s get into it and start off with clause 13.2.4, which states two little sentences here, the FAT shall be conducted in accordance with the FAT planning.

So there does need to be FAT planning. We already talked about that planning. If you want that plan to reside in an open audit document, that’s a great way where you could basically have one procedure item per line. You can group them together. You have all the spaces available for findings, recommendations, listing of evidence, etc. So having that plan is the starting point, and the next sentence says: these tests shall show that all the logic performs correctly.

So you’ve developed a procedure, and if you go all the way through the procedure, you perform the procedure as it is documented, then you’re going to step through all the logic and confirm that it operates correctly. So execute your plan, and if you execute your plan and your plan is well written, it should demonstrate that all the logic, all the logic, not just a piece of the logic. Every nook and cranny of the logic executes properly.

Clause 13.2.5: Test Documentation Attributes

Clause 13.2.5 is a stem and five bullet points that talks about what the tests need to address. So 13.2.5 states for each test carried out, the following shall be addressed. And then it’s going to give you five bullet points for the five attributes. Number one, the test needs to address the version of the test planning being used. Ah, the version of the test planning being used.

So, this is kind of implying, inferring that there may be multiple revisions of this test plan that were passed around and edited and revised and so on. And it just so happens that the Kenexis Integrated Safety Suite gives you need a great mechanism for not only knowing what the revision number is, making sure that that revision number is the official working revision that you’re working on, but it also maintains a history of all the prior versions of that test plan. So, kind of built in, depending on what software you’re using, if you’re using software a little bit more sophisticated than just a Word document. And heaven forbid, you just use a cause and effect diagram and a yellow and pink highlighter. Shame on you. That’s not enough. All right, so we need to know the version of the test plan being used. Item number two: the SIF and performance characteristic being tested.

So, as we are stepping through our factory acceptance test procedure, we need to know what SIF am I testing in this step. And a lot of steps test all of the SIFs. So if you unplug the plot power and plug it back in and see how the system boots up, that’s going to affect all of the safety instrumented systems. But if you’re exercising a single input, that might only affect a single safety instrumented function. Now, the performance characteristic is: you know, what are we trying to do? What are we trying to confirm is proper. So you might want to confirm that the scaling from 4 to 20 milliamps to engineering units is being done properly. So that’s an example of a characteristic. So, kind of hit it: characteristic by characteristic, and what portion of the system are you testing, and what SIF is that related to?

Third bullet item states that you need to address the detailed procedures and test descriptions. So, um, well yeah, we already said we need a version of the test plan, but when you’re carrying out the test, you need to execute all of the detailed test procedures in accordance with their descriptions of what we are attempting to test.

Okay. Bullet point number four: you need to have a chronological record of all test activities. So, what you tested when you tested it in which sequence. So, chronological record. Well, it just so happens if you’re using open audit, there is a section where you list all of the sessions. So, session one, we tested this, session two, we tested this, session three, we tested this, which includes the date and duration of that session. And then there will also be A a listing of people who attended the activity. So you make a list of all the people and you have the matrix of which persons were attending which session. So another reason that Open Audit makes this just, you know, such a slam dunk for recording and executing these activities. So, chronological record of all test activities, and then the final fifth bullet point for 13.2.5 states that you need to document the tools, the equipment, and the interfaces that are used. So if you’re using a panel to manipulate the inputs, you’re going to want to describe it. If you’re using a tool like a 4 to 20 milliamp signal generator calibration device, etc. You’re going to want to document what the device was and what Um uh possibly even something like calibration sheets and so on, which you can list off as part of the evidence in your open audit document as you go through there.

All right, so 13.2.5, five bullet items describe what needs to be addressed while you’re performing the testing.

Clause 13.2.6: Documenting FAT Results

13.2.6, the next clause talks about the results of the FAT. So the result, all right, 13.2.6 stem, three bullet points and, then another additional clause or another additional sentence after that. It begins with the stem that says, The results of the FAT shall be documented, stating, and then there are three bullet points of things that need to be stated. The first bullet point is that you need to state the test cases. So, what are the tests that we are performing. These are essentially the items, the procedure steps, the questions that need to be answered by the test. What are you trying to what are you testing and what results do you expect from this test? So I am gonna apply 12 milliamps to input number three. I expect that pressure transmitter 205 will read 73 psi.

So that’s your test case. For each test case, there will be test results.

So I expected a certain pressure measurement. What did I get? And you’re going to want to scribble that down. Also, I should note here that a lot of these activities can be automated using automatic tools like Like the software from ProLyTx that will kind of execute the plan, and you can just generally watch what it’s doing as it documents the objectives. Documents any successes and also any failures as you’re executing the tests. So the test case, the expectations, and then the result. The final item is the assessment.

So, the third bullet point states you need to document whether the objectives and test criteria have been met. So, there’s going to be kind of a yes, no. We either achieved our goal or we did not achieve our goal as part of the factory acceptance testing. And then, if you did not achieve the objective, you should have a finding that kind of some text. What did I expect to happen? What actually happened? Possibly even a little bit of assessment of why it happened.

And then every time you have a finding, you should also have a recommendation. So generate in that recommendation list, this is what I recommend to resolve the discrepancy between the test case and the test result. Now, maybe you need to re-engineer the system so that it will generate the expected results. Maybe the system worked properly and your test procedures or your specifications were incorrect, and now you need to go clean up that documentation to make everything match up. That should show up in Recommendations in recommendations, and those recommendations or punch list items are things that you’ll be able to track from that point on.

Okay, so those are the three bullet points of the things that you need to be documented while you’re doing an FAT.

The last part of this clause is a sentence that states: if there is a failure during test, the reasons for the failure shall be documented and analyzed as part documented and analyzed, and the appropriate corrective action should be implemented. So, yeah, there was another sentence here that basically says what I just said in different words, maybe a little bit more wordy. So, yeah, you need to, if you get a failure, document what it was, analyze it, and make the appropriate corrective actions.

Now, you always want to document it first, you don’t want to just go correct it. You sometimes will document it right there on the spot. Oh, we the range for this input value is wrong. I’m just going to go in and change the value. But you do want to document the failure as a finding. You do want to document the recommendation so that you have the full list of recommendations along with their status, that you’ll be able to track 13.2.7 during factory acceptance test during FAT, any modification or change shall be subject to safety analysis to determine.

Clause 13.2.7: Modifications During FAT

So that’s clause 13.2.7. That was the stem. And there are two bullet points for what you’re going to determine with your safety analysis. So, here we go. The first thing that you’re going to want to determine in your safety analysis is the extent of impact on each SIF.

So the change that you’re making, what scope of action is it going to have? Is it going to affect just one input? Is it going to affect an entire SIF? Is it going to affect multiple SIFs? Is it going to affect every SIF in the entire safety instrumented system. So understand what the scope is, and then the second part of it is you need to understand the extent of testing and verification, which shall be defined and implemented. So you don’t just fix stuff and move on. You fix stuff and retest. And that retest might be really simple. So, if I change the range on an input value, I retest that input signal injection to make sure I am reading the correct engineering unit values, that might be enough. But if I change the way the system behaves on startup, then you’re probably going to want to rerun a full battery of startup tests.

If my input value that I messed around with affects multiple different safety instrumented functions, I might need to rerun the testing for all of those safety instrumented functions. So understanding the scope that is impacted by your change is important because while you don’t need to start from the beginning and retest everything, you need to retest the full scope of actions and activities that are impacted by the modification that you just made. All right, so that is the second bullet point. Clause 13.2.7 does have one additional note, which states, commissioning can commence whilst corrective action is undertaken, depending on the results of the FAT.

Note: Commissioning During Outstanding Corrective Action

Alright, so this last note kind of sits with 13.2.7, but actually it sits in general with clause 13.2 as a whole. What we just said in this note is that you did a FAT. That FAT, which if you’re really top-notch, you documented in open audit, and you have a list of findings and you have a list of recommendations. Some might have been done right away. Some might take some time. Some might require you to add another input card and reconfigure that input card. Or maybe I need to buy a different type of logic or CPU for my logic solver because I blew that. Maybe I need to install a new power supply because I noticed that I didn’t have any redundancy in my power supply while I was doing my FAT. So some of these changes could take some time to execute.

Now, if you have all the time in the world, time is not an issue, you don’t care, no big deal, time’s not a problem, leave that logic solver at the equipment vendor until everything is completely fixed and you retest it at the equipment vendor site. Now, in a lot of cases, we don’t have that kind of time.

So the equipment vendor and the end operating company purchaser of the equipment will agree on what the changes need to be. They’ll agree on the program and schedule under which those modifications will be executed, but they’re still gonna ship it.

So you’re gonna take ownership of it with the promise that all of the action items will be resolved as documented, but you don’t have time to wait, so you’re just going to ship it to site and you’re going to start installing it and the resolution of those factory acceptance test items might occur during the installation

process or maybe even all the way through into the decommissioning process. The ultimate deadline for everything that was found during the FAT is that it needs to be resolved Be before you introduce hazards to the process before you put oil in. But even that isn’t specifically true because you can even start the process under an emergency management of change stating that some of the functionality that we need isn’t working yet, but we’re gonna start the plant anyway. Boy, in a perfect world, we would never do that.

But hey, we live in the real world, which sometimes that’s going to be necessary. And if that’s the case, obviously as part of your emergency MOC, that hmm Kenexis MOC might be a good way to document that stuff.

Anyway, that emergency MOC would include as part of that planning workflow your compensating measures to maintain a tolerable level of risk in the presence of that needed safeguard not being available. So, all of that gets handled in the emergency MOC process. See Kenexis MOC on our website.

Episode Wrap-Up and Next Episode Preview

All right, with that, we have a wrap. So, this is a relatively short episode, just a little bit over a half hour, but lots of good stuff here. So, the next episode, we are going to go into installation and commissioning.

Now, I would like to tell you I’m going to do installation and commissioning in one episode, but I don’t know if I’m going to do that because I have so many stories. We’re probably going to break installation and commissioning into two pieces. This is how I started my career. So, even before I worked at PICS or Monorex over at UOP, I was a field instrumentation advisor.

So my job was to do acceptance testing. So part of the commissioning process, part of the buying loops process is I would check the loop out myself to make sure that it was appropriate before I allowed UOP’s customer to purchase it from the engineering customer.

So I’ve got great stories of fist fights in the parking lot. That’s right. We’re going to settle our engineering argument with a fist fight in the parking lot and other, you know, curious PPE practices around the world. Lots of good stuff to talk about, but we’re not going to do that until next episode. We will catch you then.

Kenexis Vertigo Safety Lifecycle Suite

Now that you’ve heard some insights on technical safety, functional safety, and the IEC 61511 standard, let me tell you a little bit more about how to easily and effectively implement the safety lifecycle using the Kenexis integrated safety suite and our SIS safety lifecycle management tool, Vertigo.

Vertigo is a comprehensive tool set for performing assessment calculations, documenting, and maintaining the design of safety instrumented systems. Analysis begins with importing or synchronizing a list of safety instrumented functions with their definitions and associated performance targets from our open PHA tool for HAZOP and LOPA documentation. Each safety function can then be analyzed by performing a SIL verification calculation, complete with a collection of tools for optimizing designs and a database of thousands of potential instruments to define failure rates and diagnostic coverage capabilities.

After the SIL verification calculations are defined, you can build an SRS by automatically generating a cause and effect diagram from the SIF definitions and other defined instruments. Each SIS instrument will include a customizable data sheet and general requirements that are applicable to the SIS as a whole and can be entered individually or even bulk imported from customizable libraries. After the design phase, you can even use Vertigo to track and document testing throughout the entire life of the facility. Kenexis Vertigo is the most integrated, easy-to-use enterprise tool for allowing the development of SIS design basis information more efficiently and effectively than any other software application.