Technical Validation through Automated Testing

Technical Validation through Automated Testing
Computational Phenotyping and Software Engineering
S36
Martin Chapman
King’s College London
#IS24
AMIA 2024 Informatics Summit | amia.org 1

1. Background

Automated testing
Floor 1 Æ ¥
r

Automated testing
Floor 1 Æ ¥
r
Floor 2 Æ ¥

Automated testing
Floor 1 Æ ¥
r
Floor 2 Æ ¥
Floor 3 Æ ¥
W Æ q

Automated testing
Floor 1 Æ ¥
r
Floor 2 Æ ¥
Floor 3 Æ ¥
W Æ q
W Æ q

Automated testing
Floor 1 Æ ¥
r
Floor 2 Æ ¥
Floor 3 Æ ¥
https://github.com/sindresorhus/got

Unit tests vs. integration tests
W Æ q
W Æ q
Floor 1 Æ ¥
Floor 2 Æ ¥
Floor 3 Æ ¥
r

W
Æ
q
W
Æ
q
Floor
1
Æ
¥
Floor
2
Æ
¥
Floor
3
Æ
¥
r

W
Æ
q
W
Æ
q
Floor
1
Æ
¥
Floor
2
Æ
¥
Floor
3
Æ
¥
r
Æ q

W
Æ
q
W
Æ
q
Floor
1
Æ
¥
Floor
2
Æ
¥
Floor
3
Æ
¥
r
Æ q
https://github.com/mattermost/mattermost

Automated testing and Health Informatics
https://github.com/OHDSI/CommonDataModel

2. Automated testing and phenotyping

The phenotype development lifecycle
Health informatics applications, libraries and packages that are tested are more likely to operate
correctly. We want the same to be true of phenotypes.
There are several places in the phenotype development lifecycle where we can apply testing tools:
Author
1. Modelling
languages
2. Multi-
dimensional
descriptions
4. Modular
relationships
Refine
3. Versioning and
provenance
13. Comprehensive
metadata
Implement
5. Implementation
details in model
6. Prog. language
agnostic
7. Multiple data
formats/standards
Validate
8. Defined
validation process
9. Automate
multiple validation
techniques
10. Enable
feedback
Publish
11. Expose
standard API
12. Advanced
search
capabilities

Running example: CQL phenotypes
Pascal S. Brandt, Jennifer A. Pacheco, and Luke V. Rasmussen. “Development of a repository of
computable phenotype definitions using the clinical quality language”. In: JAMIA Open (2021)

1. Testing during phenotype authoring
We’ve seen that there is a necessary difference between a phenotype definition (abstract logic;
generalisable) and a computable phenotype (programming code; specific).

Phenotype
definition:
Computable
phenotype:

This does not, however, mean that phenotype definitions cannot be computable. If they are, we
can apply testing tools to this abstract logic as another measure of correctness. In our running
example, a CQL phenotype definition can itself be tested using the CQL Testing Framework
(CTF) and YAML-based test definitions.

This does not, however, mean that phenotype definitions cannot be computable. If they are, we
can apply testing tools to this abstract logic as another measure of correctness. In our running
example, a CQL phenotype definition can itself be tested using the CQL Testing Framework
(CTF) and YAML-based test definitions.
The application of unit tests during phenotype authoring is likely to be particularly effective, as
phenotype definitions are often designed as distinct units of logic.

https://github.com/PheMA/phekb-phenotypes

2. Testing during phenotype implementation
Phenotype
definition:
Computable
phenotype:

2. Testing during phenotype implementation
Testing tools can be applied during phenotype
implementation in much the same way testing
tools are applied during general software de-
velopment.
For example, CQL connects with the popular
Mocha Javascript testing framework. At this
level, the tests we see are more akin to inte-
gration tests.
https://github.com/cqframework/cql-execution

3. Testing during phenotype validation
Recall that the reason we are motivated to apply (automated) testing tools to the development of
phenotypes is to ensure they are operating correctly.
Phenotype validation has the same goal: to determine whether a given phenotype can correctly
identify which individuals exhibit the true phenotype and which do not.
Therefore, testing can be applied and used as an additional form of validation. We term this
technical validation. Practically, technical validation is likely to manifest as writing and
executing test cases for existing phenotypes.
,
Technical validation and phenotype validation are still very distinct processes.

Summary
• Automated testing tools, which include both unit and integration testing, are commonplace in
software development, including software developed in the Health Informatics domain.
• However, they are relatively underused in the development of phenotypes.
• We identify huge potential for automated testing tools to be applied to phenotype authoring
(running testing tools against computable definitions), phenotype implementation (to test
whole definitions) and phenotype validation (as an additional way to assess the correctness
of a definition).

Thank you!

Technical Validation through Automated Testing

More Related Content

Similar to Technical Validation through Automated Testing (20)

More from Martin Chapman (20)

Recently uploaded (20)

Technical Validation through Automated Testing