ADVISOR: An adjustable framework for test oracle automation of visual output systems

Name: ADVISOR: An adjustable framework for test oracle automation of visual output systems
Author: Genç, A. E., Sözer, Hasan, Kıraç, Mustafa Furkan, Aktemur, Tankut Barış

İsim	ADVISOR: An adjustable framework for test oracle automation of visual output systems
Yazar	Genç, A. E., Sözer, Hasan, Kıraç, Mustafa Furkan, Aktemur, Tankut Barış
Basım Tarihi:	2020-09
Basım Yeri	- IEEE
Konu	Adjustable framework, Black-box testing, Com-puter vision, Test automation, Test oracle
Tür	Süreli Yayın
Dil	İngilizce
Dijital	Evet
Yazma	Hayır
Kütüphane:	Özyeğin Üniversitesi
Demirbaş Numarası	0018-9529
Kayıt Numarası	5cd867db-4997-412b-ad75-4c6e19b8ab24
Lokasyon	Computer Science
Tarih	2020-09
Örnek Metin	Test oracles differentiate between the correct and incorrect system behavior. Automation of test oracles for visual output systems mainly involves image comparison, where a snapshot of the output is compared with respect to a reference image. Hereby, the captured snapshot can be subject to variations such as scaling and shifting. These variations lead to incorrect evaluations. Existing approaches employ computer vision techniques to address a specific set of variations. In this article, we introduce ADVISOR, an adjustable framework for test oracle automation of visual output systems. It allows the use of a flexible combination and configuration of computer vision techniques. We evaluated a set of valid configurations with a benchmark dataset collected during the tests of commercial digital TV systems. Some of these configurations achieved up to 3% better overall accuracy with respect to state-of-the-art tools. Further, we observed that there is no configuration that reaches the best accuracy for all types of image variations. We also empirically investigated the impact of significant parameters. One of them is a threshold regarding image matching score that determines the final verdict. This parameter is automatically tuned by offline training. We evaluated runtime performance as well. Results showed that differences among the ADVISOR configurations and state-of-the-art tools are in the order of seconds per image comparison.
DOI	10.1109/TR.2019.2957507
Cilt	69

Kaynağa git Özyeğin Üniversitesi

Aramaya Dön

Özyeğin Üniversitesi

Kaynağa git

ADVISOR: An adjustable framework for test oracle automation of visual output systems

Yazar Genç, A. E., Sözer, Hasan, Kıraç, Mustafa Furkan, Aktemur, Tankut Barış

Basım Tarihi 2020-09

Basım Yeri - IEEE

Konu Adjustable framework, Black-box testing, Com-puter vision, Test automation, Test oracle

Tür Süreli Yayın

Dil İngilizce

Dijital Evet

Yazma Hayır

Kütüphane Özyeğin Üniversitesi

Demirbaş Numarası 0018-9529

Kayıt Numarası 5cd867db-4997-412b-ad75-4c6e19b8ab24

Lokasyon Computer Science

Tarih 2020-09

Örnek Metin Test oracles differentiate between the correct and incorrect system behavior. Automation of test oracles for visual output systems mainly involves image comparison, where a snapshot of the output is compared with respect to a reference image. Hereby, the captured snapshot can be subject to variations such as scaling and shifting. These variations lead to incorrect evaluations. Existing approaches employ computer vision techniques to address a specific set of variations. In this article, we introduce ADVISOR, an adjustable framework for test oracle automation of visual output systems. It allows the use of a flexible combination and configuration of computer vision techniques. We evaluated a set of valid configurations with a benchmark dataset collected during the tests of commercial digital TV systems. Some of these configurations achieved up to 3% better overall accuracy with respect to state-of-the-art tools. Further, we observed that there is no configuration that reaches the best accuracy for all types of image variations. We also empirically investigated the impact of significant parameters. One of them is a threshold regarding image matching score that determines the final verdict. This parameter is automatically tuned by offline training. We evaluated runtime performance as well. Results showed that differences among the ADVISOR configurations and state-of-the-art tools are in the order of seconds per image comparison.

DOI 10.1109/TR.2019.2957507

Cilt 69