Vendor Validation

[bber]

What are everyones thoughts about a software company performing validation for the end user of that software? Isn't this considered a conflict of interest? My facilily has just purchased a McKesson product and it appears that the software vendor who manufactures the product also has a validation company and now performs validations. This company is automatically written right into the contract and we were not given the opportunity or the information needed to investigate them. I just came back from the AABB and the FDA frowned upon such a type of validation. Any thoughts?

[Lcsmrz]

Computer validation needs to be done on-site. If they want to come to your hospital and do it for you, there is nothing in the regulations I can think of that forbids it.

However ...

There should be a plan that is reviewed and accepted by you as comprehensive enough prior to validation. Simply accepting whatever they give you as a validation would not be acceptable for me -- it could be merely testing they require for installation and not a validation at all.

There should be documentation that they actually performed what they said they would do. That implies completed test plans, screen prints, etc. Everything should be dated and initialed.

There should be a summary of the work they did, any discrepancies that appeared in the testing and their explanation, orginial database configurations, and so on. The documentation shoulld be organized, so you and an average inspector could determine whether the plan was carried out.

The entire document must be reviewed by your facility and pronounced complete before used for patient testing.

One caveat: I learn an awful lot when doing validations. You would be depriving yourself of alot of good knowledge about the system operation and its limitations. I vote for doing it yourself!

[Malcolm Needs ☆]

Computer validation needs to be done on-site. If they want to come to your hospital and do it for you, there is nothing in the regulations I can think of that forbids it.

However ...

There should be a plan that is reviewed and accepted by you as comprehensive enough prior to validation. Simply accepting whatever they give you as a validation would not be acceptable for me -- it could be merely testing they require for installation and not a validation at all.

There should be documentation that they actually performed what they said they would do. That implies completed test plans, screen prints, etc. Everything should be dated and initialed.

There should be a summary of the work they did, any discrepancies that appeared in the testing and their explanation, orginial database configurations, and so on. The documentation shoulld be organized, so you and an average inspector could determine whether the plan was carried out.

The entire document must be reviewed by your facility and pronounced complete before used for patient testing.

One caveat: I learn an awful lot when doing validations. You would be depriving yourself of alot of good knowledge about the system operation and its limitations. I vote for doing it yourself!

I couldn't agree more (and this comes from someone who is totally IT illiterate).

:)

[dferia]

Computer validation needs to be done on-site. If they want to come to your hospital and do it for you, there is nothing in the regulations I can think of that forbids it.

However ...

There should be a plan that is reviewed and accepted by you as comprehensive enough prior to validation. Simply accepting whatever they give you as a validation would not be acceptable for me -- it could be merely testing they require for installation and not a validation at all.

There should be documentation that they actually performed what they said they would do. That implies completed test plans, screen prints, etc. Everything should be dated and initialed.

There should be a summary of the work they did, any discrepancies that appeared in the testing and their explanation, orginial database configurations, and so on. The documentation shoulld be organized, so you and an average inspector could determine whether the plan was carried out.

The entire document must be reviewed by your facility and pronounced complete before used for patient testing.

One caveat: I learn an awful lot when doing validations. You would be depriving yourself of alot of good knowledge about the system operation and its limitations. I vote for doing it yourself!

To expand on Lcsmrz:

Regarding the IQ and OQ, your staff (including quality) would need to reveiw and approve the validation plan prior to the execution and make any additional requests to ensure that the validation is comprehensive enough to identify any possible issues. I would think that the execution of the validation would need to occur on the premises, though with computers, if its on a server, obviously any terminals in other locations would have a wee bit smaller of a validation most likely as they're just connecting to the primary program where ever it is housed. The validating company would need to follow the usual sort of validation processes, such as documenting deviations, getting prior approval (by their quality and yours) prior to making ammendments to the validation or system, etc. After the validations are completed, your organization would need to review the documentation once more and sign off as adequate (or request more changes, etc) prior to the validation being complete.

As for the PQ, really, thats something I think you'd want to do yourself. That's your method of integrating the operation fo the system into your own organization's processes and workflow to ensure it will work out and not start a whole host of unintentional forest fires. Its also one of the better ways you'll elarn what SOPs need to be revised to reflect the new system. I know I always think I have it all covered and find just one more thing that was buiried somewhere that I missed fixing during the PQ.

I couldn't agree more with others, that as much as it frequently sucks, doing your own IT validation can be a blessing in that you learn quirks of a system you might not otherwise get a handle on for years at the cost of saving the time of one or more of your own employees. Can be a tough call depending on current workloads.

Even if you have a vendor do it, I would recommend having someone "tag along" with the vendor to get an idea on what they're doing and pick up a few things, especially if you've never worked with them before and are uncertain as to how closely they follow the validation. I have worked with vendors who wrote a validation plan, which I reviewed and approved in advance, only to find later on down the line that they played a wee bit loose (read as skipped whole steps, lost the concept of rounding values and basic math, etc) with their own validation. The whole thing resulted in weeks of extra work and alot of frustrating back and forth meetings as mroe and more people up the chain in both organizations got involved. Could have happened with our own staff certainly, but adding an external company added that extra touch of insanity to it all.

Your mileage may vary of course. Good luck to you in the new software!

[korchek]

Thanks for all your information. My main concern is that the software manufacturer themselves are now doing the validations and this was written into the contract with the sale of the product. Will the software makes disclose all/ any problems associated their product? Will they test it outside of the scope of what their product does? This is my main concern. Of course I think the best way is to validate it yourself but if you don't have the staffing an 'independant' consulting group would be my next choice and that would not be the company that created the product.

Edited December 8, 2009 by korchek
error

[Dr. Pepper]

Having brought up two LISs myself (Sunquest and Meditech), the only thing I can think of more tedious than doing the validation yourself would be to have to review someone else's! That being said, I echo the comments that it's an invaluable way to familiarize yourself with the quirks of your product, to see what applications are more or less helpful to your unique lab situation, and, since there are often different ways to accomplish the same task within the system, to see what works best for you.

Another problem can be the lab smarts and experience of your vendor staff. Sunquest hired experienced med techs who wanted a career change. They knew exactly what went on in a lab and what the key issues were. Meditech, however, hired new college grads (one trainer told us with some pride that he'd never taken a biology or chemistry course!) who might be computer savvy but were utterly clueless about clinical laboratory science and medicine and hospital workings in general. So, will these validators think of every "what if..." situation that can arise?

[jmolny]

I believe it is a major conflict of interest to have a software manufacturer validate its own software. They will not test the software as the facility will use it. They will test it the way they know it will work.

I know that it is frustrating that a company that you have not had a chance to interview the vendor who will be doing your validation. I find that the reason that it is written into the contract usually is because the facility wants to only pay one bill for the project.

Unfortunately I don’t believe that this will change unless the FDA does more than frown on the practice.

If you cannot do the validation yourself, there are third parties that can do a thorough job. It’s not that you cannot do a good job but with all the tech shortages you probably do not have the time. And I agree that reviewing someone else’s validation is not exciting but it takes less time than the planning, creation of test cases and test case execution and review that needs to be done by the site itself.

[adiescast]

Vendors can only do installation qualification (IQ) and operational qualification (OQ). The lab is always responsible for performance qualification (PQ), which is the part that tests how the software works in your environment. In addition, you as the customer have to approve the plans and sign off on the results of the IQ and OQ.

If you cannot trust them to perform the validation accurately, how can you trust them to provide a decent product? If they are willing to cut corners in one area, why not in others? From your initial post, it sounds like you were not allowed to participate in the vendor selection from the outset, which is a whole 'nother issue.

[ABBWalker]

If your thinking of a independant company to do your validation, this is one area we do cover through our engineering services, in the UK we have already sorted out the Welsh Blood service and talking to the NBS at present, if your looking for help drop me a private message and I will makesure it gets passed to the right person within our organisation.

Cheers

ABBWalker.

[Bill]

According to the New York State Laboratory Standards of Practice, the Laboratory Director and laboratory management must approve any/all changes and validations performed by non-laboratory personnel. This includes IT personnel or outside contractors--anyone not under direct authority of laboratory management. Having installed two LIS systems, I think it would be harder to review outside validation than to perform the validation in-house. I agree with lcsmrz that by performing the validation in-house, you learn alot about the system that you would not learn any other way and therefore would not get all the value from the system.

[Likewine99]

I think it is a conflict of interest.

The experience you gain when you do your own validations, even though it is a tedious process, is worth it.

We have been live on our current system for a little over a year and the fact that I built and validated some of the truth tables and know the table settings, helped me convince a less experenced member of our BB staff understand that yes the TT's were working as designed.

Doing validations yourself also gives you the confidence you need to use the system successfully those first few weeks after go live.

[ABBWalker]

I see valid points from you all in the Validation area, what we do at ABB is work with you and make sure as the process goes through validation, is to make sure you understand everything about Validation and that when complete you have complete faith in your process and it will stand up against any scrutiny.

[Eoin]

Definite conflict of interest and is a no-no.

Cheers

Eoin

[ABBWalker]

Eoin Conflict of interest I think not, just telling the truth, we don't like to see people ripped off by these companies and do a lot work to help them, and makesure they understand everything about what they could potentially buy, and that they actually do what they are supposed to do without being taken in by salesman talk.

And then buying useles system's that don't work ! OFF WHICH I could name quite a few.

[Eoin]

Where did that come from ABBWalker?

I was agreeing that the company who delivered the stuff does not do the final validation. It is only responsible for IQ. You as customer are responsible for PQ and final validation. If you use another company to do that validation, they had better be accredited themselves. I was not advocating the use or non-use of another company and I am not a salesman, I am a Quality Manager with 40+years experience in hospital bloodbanking in hospitals both big and small.

So the question I would have - Are you with the company installing?

[normam]

I guess your best option is to accept the company's validation, review them and then challenge it with your own additional validation. Im' sure you'll find something that did not meet your policy and required validation. That's what happened to me when I did my own validation four years ago and believe me they will work with you to correct them.

[L106]

I guess your best option is to accept the company's validation, review them and then challenge it with your own additional validation. Im' sure you'll find something that did not meet your policy and required validation. That's what happened to me when I did my own validation four years ago and believe me they will work with you to correct them.

I fully agree with everyone's comments that doing the validation yourself truly tests the system with your procedures and how your staff will use the system, and you do really learn the system and develop familiarity/confidence during the validation. However if the personnel and time simply are not there to do all of the validation yourself, I think normam has offered a workable solution.

[M_Allan]

Dear All,

This is my first time on this Forum, and it is interesting to see how

IQ / OQ / and PQ is interpreted.

I wrote plenty of PQ's for different companies in regards to a wide variety of Monitoring systems, and it is very interesting to see, what comes up.

I think one should start from the beginning, very simple and straight forward.

A sales person is trying to sell us his equipment.

If the sales person is able to convince us, that his equipment does exactly that what we expect,

we might consider buying it.

Write down what he states, his system can do. Later, we should be able to find this information in the

IQ/OQ.

Next, it is important to survey, if anybody has some problems with this particular piece

of equipment.

For this, I really think this forum is the right place.

Do not just take the person that advised you to buy that particular piece of equipment

as being honest, for granted.

Millions of Dollars are spend by vendors, to give certain people bonuses, in case they can get

a sales person into your office.

Now, you decide to buy the equipment, and Tech's will come to install it.

It is important, that each component has its own certificate or test-report.

How else can we be sure, the system works within specifications when assembled.

A simple example is:

Device A measures temperature. The output of device A deviates - 0.8ºC

Device B is the interface where device A is connected. Device B has an off-set

of +0.8ºC.

Great, the output of the interface will result in 0.0 ºC deviation.

But…this is wrong. According standards, the deviation is 1.6, and not "Zero"

Most IQ/OQ protocols will have a traceability matrix, in which you will find a lot of information

on how the procedure is performed, but do not just take it for granted.

The vendor wrote it, to make his system look good…. again. If it is correct, that is another question.

Next, the technicians will perform an IQ/OQ on the system they installed.

Somebody in this topic wrote : just let them do it, approve it, and then review it.

I do not agree on that statement.

Usually, by approving the IQ/OQ protocol, you state, that everything is done according

your expectations. I do not know a lot of people, that sign a contract, before reading it.

To my opinion, this should be the same here.

First, let them execute the protocol. Next review it !!, and the last step is, to approve it.

Most vendor IQ/OQ protocols are nothing more then generating a lot of paper that will

show, their system is working according their standards…… not yours.

I have seen enough, where critical tests were not performed, or implemented in the vendors

protocol, since they know their system would fail the test.

Next, we will perform a PQ.

Should this be done internally, or by a third party.

If a company has to work compliant and to certain regulations, I think, that this should be done by a third party specialist. If done internally, one could confront similar problems as with the vendor validation.

The question could arise, if the outcome of the PQ is not in your interest, when you perform it yourselve ??

In many cases, I noticed, that there is not enough knowledge within smaller companies to perform

a profound Performance Qualification.

In regards to monitoring systems (CMS), I have seen a lot of flaws, and nobody questioned them.

How then, is it possible, to perform an internal PQ, if customers don't even now, how a correct installation should be, and should we then not question the knowledge of the vendors technicians.

I think, that an external company, that will perform a PQ for you, and has no personal interest in your system, will be able to give you exactly what you want.

If you want to buy a used car, and your knowledge does not go further than that it has tires and it needs gas, everybody would agree on asking an expert, if that car is sound.

Long story so far, but I hope it is interesting enough to read during the cold days before Christmas, although written in poor English.

All the best for the New-Year, and a merry X-Mas.

[matt14802]

Let me get this straight. You are being forced to use the software vendor to validate their own software. How can that not be a conflict of interest? Validation, whether done by the hospital or by an INDEPENDENT third party, is the opportunity to challenge the software and the software vendor's claims about functionality. What is to stop the software vendor (if they are performing the validation) from overlooking known defects in order to assure a timely go live. Software vendor's get paid when systems go live and I could certainly imagine an instance where validation may be compromised for financial gain.

[M_Allan]

Matt,

You are absolutely right.

I mentioned this before, and I have seen this more often myself.

IQ/OQ done by vendors is only to make their system look good.

I prefer to see it as an installation check, not a validation.

I do not see it as being forced by the vendor to get their software validated. It

is more, that they do a bunch of checks, to show, that their system is working, according

their standards !

For every piece of equipment/software, I sure believe it necessary to perform a PQ.

Since I have a lot to do with PQ's, Hard & Software, on different systems, I can tell you,

that I encountered numerous 21 CFR 11 incompatibilities as well as hardware that is in no way

FDA compliant.

Last but not least, it is also these regulations that are not full-proof.

I cannot, go into detail on systems from different vendors, because there are more then one,

I know inside out, and this would create a very big storm.

Do understand. In this kind of business...... it is all about money, and nothing else.

As I said. Ask the vendor to install a demo unit, and test it........ to the max.

Have fun,

Marc

[smartesoft-biased]

Between vendor self-validation and doing your own validation is the third option of independent third party validation.

There exist independent firms whose whole focus is on this type of validation. They are not tied to the software vendors, and they have deep expertise in this type of validation.

For example, RF Nozick do nothing but blood bank validation, and do not sell any software. They are independent and unbiased.

Cheers,

Gordon

[Disclaimer: I work for SmarteSoft, who provide test automation software that is used for automating testing/validation of blood bank and other software. While I try not to let that influence what I say here, any views I express may be inadvertently biased due to this industry participation, and should be filtered accordingly]

[Malcolm Needs ☆]

How refreshingly honest.

Well done and well said!

:D:D:D:D

[pdameron]

The IT department for my company hired an independant outside company to perform a computer validation for an upgrade a couple years ago since my team (who would normally perform this task) was exceedingly busy with ISBT implementation.

I won't say the name of the company used, but I am assuming that they were the lowest bidders. It took me longer to review their shoddy documentation than it would have taken me to validate in the first place. My 14 year old son could recognize the problems I was seeing. My time spent included writing an 88 (yes, 4 score and 8) page document listing the problems with the documentation provided. The company's response was to submit modified scripts to match the pictures they sent originally or ignore the issues I brought to light.

Needless to say, I personally managed validating our most recent upgrade.

[tbdorgan]

You should do your own validation. It's the best way to learn your system.