TY - GEN

T1 - A Means to what End? Evaluating the Explainability of Software Systems using Goal-Oriented Heuristics

AU - Deters, Hannah Luca

AU - Droste, Jakob Richard Christian

AU - Schneider, Kurt

N1 - Funding Information: This work was funded by the Deutsche Forschungsgemeinschaft (DFG, German Research Foundation) under Grant No.: 470146331, project softXplain (2022-2025).

PY - 2023/6/14

Y1 - 2023/6/14

N2 - Explainability is an emerging quality aspect of software systems. Explanations offer a solution approach for achieving a variety of quality goals, such as transparency and user satisfaction. Therefore, explainability should be considered a means to an end. The evaluation of quality aspects is essential for successful software development. Evaluating explainability allows an assessment of the quality of explanations and enables the comparison of different explanation variants. As the evaluation depends on what quality goals the explanations are supposed to achieve, evaluating explainability is non-trivial. To address this problem, we combine the already well-established method of expert evaluation with goal-oriented heuristics. Goal-oriented heuristics are heuristics that are grouped with respect to the goals that the explanations are meant to achieve.By establishing appropriate goal-oriented heuristics, software engineers are enabled to evaluate explanations and identify problems with affordable resources. To show that this way of evaluating explainability is suitable, we conducted an interactive user study, using a high-fidelity software prototype. The results suggest that the alignment of heuristics with specific goals can enable an effective assessment of explainability.

AB - Explainability is an emerging quality aspect of software systems. Explanations offer a solution approach for achieving a variety of quality goals, such as transparency and user satisfaction. Therefore, explainability should be considered a means to an end. The evaluation of quality aspects is essential for successful software development. Evaluating explainability allows an assessment of the quality of explanations and enables the comparison of different explanation variants. As the evaluation depends on what quality goals the explanations are supposed to achieve, evaluating explainability is non-trivial. To address this problem, we combine the already well-established method of expert evaluation with goal-oriented heuristics. Goal-oriented heuristics are heuristics that are grouped with respect to the goals that the explanations are meant to achieve.By establishing appropriate goal-oriented heuristics, software engineers are enabled to evaluate explanations and identify problems with affordable resources. To show that this way of evaluating explainability is suitable, we conducted an interactive user study, using a high-fidelity software prototype. The results suggest that the alignment of heuristics with specific goals can enable an effective assessment of explainability.

KW - explainability

KW - metrics

KW - software evaluation

KW - heuristics

KW - explainable systems

KW - Metrics

KW - Software Evaluation

KW - Heuristics

KW - Explainability

KW - Explainable Systems

UR - http://www.scopus.com/inward/record.url?scp=85162207646&partnerID=8YFLogxK

U2 - 10.1145/3593434.3593444

DO - 10.1145/3593434.3593444

M3 - Conference contribution

T3 - ACM International Conference Proceeding Series

SP - 329

EP - 338

BT - EASE '23: Proceedings of the 27th International Conference on Evaluation and Assessment in Software Engineering

ER -