validation software

[kate murphy]

Has anyone used any validation software for thier computer validation? Software such as Cyrano, which runs a script on a PC to simulate normal and stress testing.

Was it really a time saver?

Did the cost of the software balance with the time savings?

Were you happy with the results?

And have you gone through any inspections (FDA or AABB) post validation?

[bmarotto]

Kate,

We just ordered the STS Quality Advantage software to help us validate Misys v6.1. I am hoping to have it installed sometime in the next month. So I can't answer any of your questions yet. Once we have it up and running, you are welcome to come check it out.

Bev

[korchek]

Validation guidelines issued by the regulatory agencies state that the software must be validated as it is intended to be used. Unless you plan on using software to enter your day to day results into the software you are not validating the intended use.

Who validated the software that is doing the validation?

[Cliff]

I agree in principle; however, what about other circumstances? Let's say the program was generally a macro tool that first copies my keys strokes then iterated them as many times as I wanted, let's say 100. I did the first one and that reflects the way I would use my software. Also, it could be helpful in stress testing, especially in smaller facilities that might not have many staff available to attempt simultaneous data entry.

I agree that a third party tool can't do the entire project, but I think it can add a lot of value.

Validation guidelines issued by the regulatory agencies state that the software must be validated as it is intended to be used. Unless you plan on using software to enter your day to day results into the software you are not validating the intended use.

Who validated the software that is doing the validation?

[CLIMBER514]

Has anyone utilized automated software scripts as part of a information system validation?

We would be internally developing our scripts per our configuration... but are there third party applications that will specifically test HCLL or Misys?

We will be validating a new Blood Bank information system, and had thoughts of utilizing software scripts to automate a certain percentage of the process. For instance - with a manual method - you might be able to test number and alpha values for a certain field. But with software scripts, you could automate this and test 0-9 & a-z without feeling like you are wasting a real person's time. Plus, it eliminates potential for human error.

As for validation of the validation scripts... this is irrelevant since the product of the script would be evident in the validation itself. The Visual Basic script would drive the same applications, processes and keystrokes that a normal person would. I think it would lead to a more comprehensive validation and be a better attempt to 'break the system'. You could also use these scripts to attempt to max out system capacity. Imagine testing your system to determine the effective threshold of for the max number of patient inqueries, etc.? The advantage of running software scripts is - you can pre-program every conceivable combination of events - hit the record button in an application like Camtasia - document the computer actually running each combination (with preprogrammed pauses, etc. to simulate real workflow)- while you are in the lab attending to your mounds of paperwork.

[Rania]

The blood bank that I work in is working on implementing a new BB system in the near future. Medicore, HCLL.

Have you any experience with this system?

Thanks

[Myella]

Kate,

We just ordered the STS Quality Advantage software to help us validate Misys v6.1. I am hoping to have it installed sometime in the next month. So I can't answer any of your questions yet. Once we have it up and running, you are welcome to come check it out.

Bev

Our facility is going to upgrade from (Misys) Sunquest 6.1 to V 6.3. This validation software is new to me. I am very much interested if having a software validate the blood bank information system is acceptable by FDA and AABB.

If it does meet the regulation, I would like to know about STS Quality Advantage and Cyrano software. How much is the cost, contact information or websites.

Will this take the place of running test validation script performed by staff?

I appreciate any information from anyone.

[Franklyn]

We have looked at this validation software in the past and the only reason why we haven't purchased it was lack of capital. I have, however, used home-rolled scripts as part of a build and validation with great success. Yes, you need to test the software as it will be used but no you do not need to use the (validation) software for ongoing data entry. The software takes the place a live body (call it a robot, if you will) and sends pre-defined scenario to the system and captures screen shots at each point. So it provides all the documentation you could every want and the software we looked at flagged unexpected results as well.

My refrigeration alarm system used a built in validation module that validates the software of the system in an analogous manner. The resulting documentation is greatly in excess of what I would have done myself, but it is finished in about 30 minutes and is FDA approved.

If you can get your management team to invest in one of these system you have my undying envy because they can be HUGE time savers and will not run you afoul of regulations provided you generate and review sufficient documentation of each step in the validation.

[John C. Staley]

Franklyn, what alarm system are you referring to? We are considering getting such a system and a validation aid could swing the vote.

[WVK]

John, if one of your goals is to minimize time spent on validation of a alarm system, you may want to look into a out sourcing service for the complete function and there is little on site to validate...www.tutelasystems.com

Walter

[i bleed crimson]

is anyone using validation software in converting to isbt 128?

is there a real savings in time? how long does it take to set the software up and start validating?

[Franklyn]

Franklyn, what alarm system are you referring to? We are considering getting such a system and a validation aid could swing the vote.

We are using CIMScan from CIMTechniques. I know, you have never heard of them... We used Rees Scientific for almost 15 years until we switched to this new vendor and became the first Blood Bank to adopt this system. It has been in place for a year now and we have expanded it to twice the original install and still growing. I may have 300 units being monitored by the end of 2008. I have re-validated 4 times and the validation engine has worked like a charm and output as a PDF file so I could dump it on a disk, review it and then just write a summary. http://www.cimtechniques.com

[Dawn]

Back to the topic of scripting tools for validation...

We are looking at Boston Workstation. http://www.bostonworkstation.com/workflow_automation/product_scripting_features.aspx

It is very impressive. It seems very easy to set up scripts. During the demo we thought of all kinds of uses for it besides validation.

Contact them for a really great webex demo.

[matt14802]

It has been my experience that developing automated scripts to perform validation is not worth the effort. It is labor intensive to develop automated scripts for the purpose of validation and the time spent in the development of the scripts would be better served by having an experienced blood banker stress the system. There is a tendency when developing automated scripts to myopically test known pathways through a system while missing subtle idiosyncrasies that are captured through manual testing. The argument can be made that scripts are reusable; therefore, the cost of the initial development may be saved in subsequent usage of the scripts. This may be true; however, in order to develop truly robust and reusable scripts a tremendous amount of time must expended in order to develop a "testing harness". A "testing harness" is the backbone of the automated test script environment and includes defined functions, parameters, and variables that will be used when running the automated scripts. If a "testing harness" is not properly set up when developing the automated scripts, then the "reusability" of the automated scripts will be greatly diminished. In other words if you go to run your scripts a year from when you initially developed them without a good "testing harness", you will have to dive into them again and tweak them extensively to get them to run.

[Franklyn]

The STS system (mentioned previously) imports the setting from the your LIS directly and then tests those parameters. So you don't have to write or build your scripts, it does it for you automatically.

I attended a demo of STS and was really impressed, unfortunately we have been unsuccessful at acquiring the capital to purchase it (yet).

Their website is http://www.sts-healthcare.com if you would like more detail on their system. Again, I have seen the demo, but don't have any firsthand knowledge of the system. It sure looks sweet...

[Dawn]

I agree that a tool like this cannot be used to create a stand-alone validation process. Our thought is that some basic scripts will be developed that can be used as part of the initial validation and then later used to validate small changes.

It has been my experience that developing automated scripts to perform validation is not worth the effort. It is labor intensive to develop automated scripts for the purpose of validation and the time spent in the development of the scripts would be better served by having an experienced blood banker stress the system. There is a tendency when developing automated scripts to myopically test known pathways through a system while missing subtle idiosyncrasies that are captured through manual testing. The argument can be made that scripts are reusable; therefore, the cost of the initial development may be saved in subsequent usage of the scripts. This may be true; however, in order to develop truly robust and reusable scripts a tremendous amount of time must expended in order to develop a "testing harness". A "testing harness" is the backbone of the automated test script environment and includes defined functions, parameters, and variables that will be used when running the automated scripts. If a "testing harness" is not properly set up when developing the automated scripts, then the "reusability" of the automated scripts will be greatly diminished. In other words if you go to run your scripts a year from when you initially developed them without a good "testing harness", you will have to dive into them again and tweak them extensively to get them to run.

[rayjbell]

Our facility is going to upgrade from (Misys) Sunquest 6.1 to V 6.3. This validation software is new to me. I am very much interested if having a software validate the blood bank information system is acceptable by FDA and AABB.

If it does meet the regulation, I would like to know about STS Quality Advantage and Cyrano software. How much is the cost, contact information or websites.

Will this take the place of running test validation script performed by staff?

I appreciate any information from anyone.

I have worked in services for Cyrano (Vedant Health as we are now called -- www.vedanthealth.com/products) for 23 years now. I currently run the Healthcare Validation Group. I do not want this to be a sales pitch but wanted to answer some of the various questions and concerns here.

The way that our and STS's software works is that we provide the testing by covering a large degree of validation by emulating a human being. This is not a medical device so the software itself does not need to be validated by the FDA. However, it must follow very specific rules, such as including 'sequenced' screen prints (and many others) -- just as you would provide screen prints as part of your manual validation. This approach saves months of work and provides a complete validation that the FDA and AABB have signed off on for many, many years.

However, you cannot just go out a buy a standard test tool and write a bunch of scripts to validation your blood bank system. A couple of the original manual validation companies have tried that. The problem here is that you have a programmer, not a blood banker, writing the scripts on a regular scripting tool. This means that there could be issues in the validation coverage. This automated validation system must come from a specific knowledge set and must be tried and tested. You cannot have a bunch of scripts written in a test tool.

Obviously, I would highly recommend this approach. When you look at such automation just ask the right questions (how long have you provided the automation approach? Do you have a basic test tool underneath (bad answer is 'yes' here)?) and ensure that you feel comfortable in its approach. After all, it is your blood bank that is being validated.

I hope that this helps - Ray

[angonzalez]

Yes. We have used Loadrunner to load test PROGESA. That is to stress the system with large load of simulated users. Other blood banks have as well. ePROGESA uses openSTA for load testing. You cannot do load testing without these automation tool.

We also have Quicktest Pro but it has been used sparingly for unit testing/validation because that requires a lot of time to set up for the benefit it can provide. The problem is time. Essentially to use an automated validating tool, someone in your validation team needs to perform the validation manually and write it up. Then the script has to be written to do the same thing. The benefit comes when you regression rest the same functionality, you don't have to perform the manual process again.

This is very useful if your doing a lot of regression testing, for example if you are development shop. But from my experience it has limited value in your typical blood bank setting. In such a setting, major new implementations or upgrades of commercial software do a one time validation. Minor upgrades or patches after that are not fully regression tested. The next major validation would only come with a new implementaion or major upgrade which would probably make any previous automation script obsolete.

Another set of tools that could be used would be a test management or change management suite. A test management suite like Mercury Quality Center is software that lets you manage the manual or automated validation process electronically as opposed to using Excel/Word/file folders. It's useful but like with automation tools, it takes some time to set up and use and under the time crunch experienced in most blood bank deployments, this may be hard to pull off. Test management software also requires buy in from QA and management so this is another hurdle.

[smartesoft-biased]

All good points.

There has been a great deal of success with automation of validation, pre-test data seeding, data migration, performance testing, and various other areas, using SmarteScript and Smarteload. Users include the American Red Cross, the UK National Health Service, and many more.

SmarteScript does not require any scripting skills, and is quick and easy for building and maintain validation automation. There are versions of SmarteScript optimized for specific blood-bank software (eProgesa etc), making it easier yet.

The testing harness capability you mention is already built in to SmarteScript.

SmarteSoft, the vendor of SmarteScript, works proactively with validation specialists (e.g. RF Nozick) and blood bank professionals to help them develop automated validation capabilities.

Gordon

- Disclaimer: I work for SmarteSoft and am thus likely to be biased no matter how hard I try to be neutral.