Author
Erdil, Ö. F., Can, İrfan, Sözer, Hasan
Publication Date
2017
Publication Place
-
CEUR-WS
Subject
Image comparison, Image comparison algorithms, Test oracle, Test automation, Industrial case study, Image comparison, Image comparison algorithms, Test oracles, Test automation, Industrial case study
Type
Document
Language
Turkish
Digital
Yes
Manuscript
No
Library
Özyeğin University
Library Asset ID
1613-0073
Record ID
3338da78-df76-4e3f-89ba-bdb2dc8a00cb
Library Location
Computer Science
Date
2017
Sample Text
Black box tests of software-intensive embedded systems such as televisions are carried out through graphical user interfaces (GUIs). As part of the automation of these tests, a series of user actions are triggered externally.
Meanwhile, an automatic test oracle is needed that distinguishes between correct and incorrect system behavior and thus decides whether the tests pass or fail. Image comparison tools are commonly used for this purpose.
These tools compare the observed GUI with a previously saved reference GUI screenshot. In this study, 9 different image comparison tools were evaluated through an industrial case study. 1000 pairs of reference and snapshot GCA images were collected from actual test runs of a television system and these images were labeled as pass/fail test. In addition, the collected data set was classified according to various effects such as pixel shift, color tone/saturation difference and stretching (growth, shrinkage, expansion, contraction) in the image body. Then, this data
The tools compared to the set were evaluated for accuracy and performance. It has been observed that the tools give different results depending on the parameter values and the effects to which the compared images are subjected. The tool that gave the best results for the prepared data set and the parameter values of this tool were determined., Black box testing of software intensive embedded systems such as TVs is performed via their graphical user interfaces (GUI). A series of user events are triggered for automating these tests. In the meantime, there is a need for a test oracle, which decides if tests pass or fail by differentiating between correct and incorrect system behavior. Image comparison tools are commonly used for this purpose. These tools compare the observed GUI screen during tests with respect to a previously recorded snapshot of a reference GUI screen. In this work, we evaluated 9 image comparison tools with an industrial case study. We collected 1000 pairs of reference and runtime GUI images during test activities performed on a real TV system and we labeled these image pairs as passed and failed tests. In addition, we categorized the data set according to various effects observed on images such as pixel shifting, color saturation and scaling. Then, this data set is used for comparing tools in terms of accuracy and performance. We observed that results are dependent on tool parameters and various image effects that take place. We identified the best tool and its parameter set for the collected data set.
Editör
Turhan, Ç., Coşkunçay, A., Yazıcı, A., Oğuztüzün, H.
Cilt
1980
