joshua/phd-thesis
Archived
1
Fork 0
Code Activity

Test methods: shuffle content around

Browse source
This commit is contained in:
Joshua Moerman 2018-08-14 16:18:14 +02:00
parent ba248a3aa4
commit 5f24d96ec3
4 changed files with 277 additions and 211 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

5
content.tex
View file

@ -7,14 +7,9 @@
\completecontent
\completelistoffigures
\completelistoftables
\completelistofalgorithms
\showbodyfont
\chapter{Intro}
\section{Learning and Testing}
\section{Nominal Techniques}

3
content/learning-nominal-automata.tex
View file

@ -1305,7 +1305,8 @@ There is plenty of potential for running time optimization, ranging from improve
\startsubject
[title={Acknowledgements}]
We thank Frits Vaandrager and Gerco van Heerdt for useful comments and discussions. We also thank the anonymous reviewers.
We thank Frits Vaandrager and Gerco van Heerdt for useful comments and discussions.
We also thank the anonymous reviewers.
\stopsubject
\referencesifcomponent

476
content/test-methods.tex
View file

@ -1,14 +1,15 @@
\project thesis
\startcomponent test-methods
%\usepackage[vlined]{algorithm2e}
%\newcommand{\tot}[1]{\xrightarrow{\,\,{#1}\,\,}}
\startchapter
[title={FSM-based Test Methods},
reference=chap:test-methods]
\startchapter[title={A Framework for FSM-based Test Methods}]
\startsection
[title={Preliminaries}]
\startsection[title={Preliminaries}]
\startsubsection[title={Mealy machines}]
\startsubsection
[title={Words and Mealy machines}]
We will focus on Mealy machines, as those capture many protocol specifications and reactive systems.
Note that we restrict ourselves to deterministic and complete machines.
@ -32,9 +33,13 @@ The \emph{behaviour} of a state $s$ is given by the output function $\lambda(s,
Two states $s$ and $t$ are \emph{equivalent} if they have equal behaviours, written $s \sim t$, and two Mealy machines are equivalent if their initial states are equivalent.
\stopdefinition
{\todo{Voorbeeld gebruiken in komende sectie: uios: $0: aa$, $1: a$, $2$ geen uio, $3: v$ en $4: ac$. Een char. set is $\{ aa, c, ac \}$. Geen ADS (want $2$ geen uio).}}
An example Mealy machine is given in \in{Figure}[fig:running-example].
\startplacefigure[title={An example specification with input $I=\{a,b,c\}$ and output $O=\{0,1\}$.},reference=fig:running-example,list={An example specification.}]
\startplacefigure
[title={An example specification with input $I=\{a,b,c\}$ and output $O=\{0,1\}$.},
list={An example specification.},
reference=fig:running-example]
\hbox{
\starttikzpicture[shorten >=2pt,node distance=0.9cm and 3cm,bend angle=20]
\tikzstyle{every state}=[draw=black,text=black,inner
@ -64,10 +69,11 @@ An example Mealy machine is given in \in{Figure}[fig:running-example].
}
\stopplacefigure
\stopsubsection
\startsubsection
[title={Testing}]
\startsubsection[title={Testing}]
In conformance testing we have a specification modelled as a Mealy machine and an implementation (the system under test, or SUT) which we assume to behave as a Mealy machine \cite[DBLP:journals/tc/LeeY94].
Tests are generated from the specification and applied to the implementation. We assume that we can reset the implementation before every test.
If the output is different than the specified output, then we know the implementation is flawed.
@ -92,9 +98,10 @@ The \emph{size} of a test suite $T$ is defined to be
$||T|| = \sum\limits_{t \in max(T)} (|t| + 1)$.
\stopdefinition
\stopsubsection
\startsubsection[title={Completeness of test suites}]
\stopsubsection
\startsubsection
[title={Completeness of test suites}]
\startexample[reference=incompleteness]
{\bf No test suite is complete.}
Consider the specification in \in{Figure}{a}[fig:incompleteness-example].
@ -103,7 +110,10 @@ For any test suite we can make a faulty implementation which passes the test sui
Such an implementation might look like \in{Figure}{b}[fig:incompleteness-example] with $n$ big enough.
This justifies the following definition.
\startplacefigure[title={A basic example showing that finite test suites are incomplete. The implementation on the right will pass any test suite if we choose $n$ big enough.},reference=fig:incompleteness-example]
\startplacefigure
[title={A basic example showing that finite test suites are incomplete. The implementation on the right will pass any test suite if we choose $n$ big enough.},
list={A basic example showing that finite test suites are incomplete.},
reference=fig:incompleteness-example]
\startcombination[2*1]
{\hbox{
\starttikzpicture[shorten >=2pt,node distance=2cm]
@ -113,7 +123,7 @@ This justifies the following definition.
\path[->]
(0) edge [loop] node [below] {${a}/0$} (0);
\stoptikzpicture
}} {a}
}} {(a)}
{\hbox{
\starttikzpicture[shorten >=2pt,node distance=2cm]
\tikzstyle{every state}=[draw=black,text=black,inner
@ -128,7 +138,7 @@ This justifies the following definition.
(2) edge [bend left=20 ] node [below] {${a}/0$} (3)
(3) edge [loop ] node [below] {${a}/1$} (3);
\stoptikzpicture
}} {b}
}} {(b)}
\stopcombination
\stopplacefigure
\stopexample
@ -143,25 +153,75 @@ We are often interested in the case of $m$-completeness, where $m = n + k$ for s
Here $k$ will stand for the number of \emph{extra states} we can test.
The issue of an unknown bound is addressed later in the paper.
\stopsubsection
\startsubsection[title={State identifiers, access sequences and sets of words}]
\startsubsection
[title={Separating Sequences}]
Before we construct test suites, we discuss several types of useful sequences.
All the following notions are standard in the literature, and the corresponding references will be given in \in{Section}[sec:methods], where we discuss the test generation methods using these notions.
We fix a Mealy machine $M$.
For convenience we assume $M$ to be minimal, this implies that distinct states are, in fact, inequivalent.
All definitions can be generalised to non-minimal $M$, by replacing distinct (or other) with inequivalent.
\startitemize[after, before]
\item Given two states $s, t$ of $M$ we say that $w$ is a \defn{separating sequence} if $\lambda(s, w) \neq \lambda(t, w)$.
\item For a single state $s$, a sequence $w$ is a \defn{unique input output sequence (UIO)} if for every other state $t$ we have $\lambda(s, w) \neq \lambda(t, w)$.
\item Finally, a \defn{(preset) distinguishing sequence (DS)} is a single sequence $w$ which separates all states, i.e., for every distinct pair $s, t$ of $M$ we have $\lambda(s, w) \neq \lambda(t, w)$.
\stopitemize
\todo{Misschien in een definition env zetten? Of in descriptions.}
The above list is ordered from weaker to stronger notions, i.e., every distinguishing sequence is an UIO sequence for every state.
Similarly, an UIO for a state $s$ is a separating sequence for $s$ and any other $t$.
Separating sequences always exist for inequivalent states and finding them efficiently is the topic of \in{Chapter}[chap:separating-sequences].
On the other hand, UIOs and DSs do not always exist for a machine.
\todo{For example, ...}.
In order to separate multiple states at once, we might need sets of states.
This brings us to the following notions.
\startitemize[after, before]
\item A set of sequences $W$ is a called a \defn{characterisation set} if it contains a separating sequence for each pair of (distinct) states.
\item A \defn{state identifier} for a state $s$ is a set $W$ which contains a separating sequence for every other state $t$.
\item A set of state identifiers $\{ W_s \}_{s}$ is \defn{harmonised} if a separating sequence $w$ for states $s$ and $t$ exists in both $W_s$ and $W_t$.
This is sometimes called a \defn{separating family}.
\todo{Equivalently: $x \sim_\Fam{X} y$ implies $x \sim y$}
\todo{Preciezer zijn over prefixes...}
\item Following the definitions in \cite[DBLP:journals/tc/LeeY94], a separating family where each set is a singleton is an \defn{adaptive distinguishing sequence} (ads).
An ads is of special interest since they can identify a state using a single word.
\stopitemize
These notions are again related.
We obtain a characterisation set by taking the union of state identifiers for each state.
For every machine we can construct a set of harmonized state identifiers as will be shown in \in{Chapter}[chap:separating-sequences] and hence every machine has a characterisation set.
However, an adaptive distinguishing sequence may not exist.
\todo{Voorbeeld}
Besides sequences which separate states, we also need sequences which brings a machine to specified states.
\startdefinition
An \emph{access sequence for state $s$} is a word $w$ such that $\delta(q_0, w) = s$.
A set $P$ consisting of an access sequence for each state is called a \emph{state cover}.
If $P$ is a state cover, $P \cdot I$ is called a \emph{transition cover}.
\stopdefinition
\stopsubsection
\startsubsection
[title={Constructions on sets of sequences}]
In order to define a test suite modularly, we introduce notation for combining sets of words.
We require all sets to be \emph{prefix-closed}, this is very convenient in later proofs.
For sets of words $X$ and $Y$, we define:
\startitemize[after]
\item their concatenation $X \cdot Y = \{ xy \mid x \in X, y \in Y \}$,
\item iterated concatenation $X^0 = \{ \epsilon \}$ and $X^{n+1} = X \cdot X^{n}$,
\item bounded concatenation $X^{\leq n} = \bigcup_{i \leq n} X^i$, and
\item prefix closure $Pref(X) = \{ y \mid y \text{ is a prefix of } x, x \in X \}$.
\item prefix closure $\pref(X) = \{ y \mid y \text{ is a prefix of } x, x \in X \}$.
\stopitemize
For a set $S$, a family of sets $\Fam{X}$ is a set of sets indexed by $S$:
$\Fam{X} = \{ X_s \}_{s \in S}$.
We will use families to define words relevant for a state $s$.
We define:
\startitemize
On families we define:
\startitemize[after]
\item flattening: $\bigcup \Fam{X} = \{ x \mid x \in X_s, s \in S \}$,
\item union: $\Fam{X} \cup \Fam{Y}$ is defined point-wise:
$(\Fam{X} \cup \Fam{Y})_s = X_s \cup Y_s$, and
@ -170,33 +230,19 @@ $x \sim_\Fam{X} y$ if $\lambda(x,w) = \lambda(y,w)$ for all $w \in X_x \cap X_y$
\todo{Hangt af van $M$, maar is nog niet gefixed}
\stopitemize
\startdefinition
An \emph{access sequence for state $s$} is a word $w$ such that $\delta(q_0, w) = s$.
A set $P$ consisting of an access sequence for each state is called a \emph{state cover}.
If $P$ is a state cover, $P \cdot I$ is called a \emph{transition cover}.
\stopdefinition
\startdefinition
A family of sets $\Fam{X}$ is a \emph{separating family} (or set of \emph{harmonized state identifiers} \cite[YevtushenkoP90]) if $x \sim_\Fam{X} y$ implies $x \sim y$.
\stopdefinition
In other words, a family of words is a separating family if for each pair of inequivalent states it contains a word separating those states.
Following the definitions in \cite[DBLP:journals/tc/LeeY94], a separating family where each set is a singleton is an \emph{adaptive distinguishing sequence} (ads).
An ads is of special interest since they can identify a state using a single word.
Separating families always exist for Mealy machines, but an ads might not exist.
Given a specification $M$ (with states $S$) we define two operations:
Given a specification $M$ (with states $S$) we define two operations.
We omit $M$ from the notation as the specification is always clear from the context.
\startitemize[after]
\item concatenation:
$X \odot \Fam{Y} = \{ xy \mid x \in X, y \in Y_{\delta(s_0, x)} \}$ and
\item refinement: $\Fam{X} ; \Fam{Y}$ defined by
$(\Fam{X} ; \Fam{Y})_s = X_s \cup \bigcup_{y \sim_\Fam{X} x} D(\Fam{Y}, x, y)$, where
$D(\Fam{Y}, x, y) = Pref\{w \mid \lambda(x,w) \neq \lambda(y,w), w \in Y_x \cap Y_y\}$.
\startformula
(\Fam{X} ; \Fam{Y})_s = X_s \cup \bigcup_{y \sim_\Fam{X} x} D(\Fam{Y}, x, y),
\stopformula
where $D(\Fam{Y}, x, y) = Pref\{w \mid \lambda(x,w) \neq \lambda(y,w), w \in Y_x \cap Y_y\}$.
\stopitemize
Note that these operations depend on the specification $M$.
We omit $M$ from the notation as the specification is always clear from the context.
We will use the following facts in defining our new test method.
The latter construction is new and will be used to define a hybrid test generation method in \in{Section}[sec:hybrid].
\startlemma[reference=lemma:refinement-props]
For all families $\Fam{X}$ and $\Fam{Y}$:
@ -207,24 +253,134 @@ For all families $\Fam{X}$ and $\Fam{Y}$:
\stopitemize
\stoplemma
\stopsubsection
\stopsection
\startsection[title={Hybrid adaptive distinguishing sequences}]
\startsection
[title={Test generation methods},
reference=sec:methods]
In this section, we briefly review the classical conformance testing methods:
the W, Wp, UIO, UIOv, HSI, ADS methods.
Our method described is very similar to some of these methods, so we will relate them by describing them uniformly.
There are many more test generation methods.
Literature shows, however, that not all of them are complete.
For example, the methods in \cite[DBLP:journals/tosem/Bernhard94] are falsified by \cite[DBLP:journals/tosem/Petrenko97] and the UIO-method \cite[DBLP:journals/cn/SabnaniD88] is shown to be incomplete in \cite[DBLP:conf/sigcomm/ChanVO89].
For that reason, completeness of the correct methods is shown (again) in the next section.
We fix a state cover $P$ throughout this section and take the transition cover $Q = P \cdot I$.
\todo{Gebruik de volgorde en uitleg zoals in versie 1 van dit paper. Inclusief tegenvoorbeeld voor de UIO methode.}
\startsubsection
[title={W-method \cite[DBLP:journals/tse/Chow78, Vasilevskii73]},
reference=sec:w]
Possibly one of the earliest $m$-complete test methods.
\startdefinition
[reference=w-method]
Let $W$ be a characterization set, the \defn{W test suite} is
defined as $(P \cup Q) \cdot I^{\leq k} \cdot W$.
\stopdefinition
If we have a separating family $\Fam{W}$, we can obtain a characterization
set by flattening: take $W = \bigcup \Fam{W}$.
\stopsubsection
\startsubsection
[title={The Wp-method \cite[DBLP:journals/tse/FujiwaraBKAG91]},
reference=sec:wp]
The W-method was refined by Fujiwara to use smaller sets when identifying states.
In order to do that he defined state-local sets of words.
\startdefinition
[reference={state-identifier,wp-method}]
Let $\Fam{W}$ be a family of state identifiers, the \defn{Wp test suite} is
defined as $P \cdot I^{\leq k} \cdot \bigcup \Fam{W} \,\cup\, Q \cdot I^{\leq k}
\odot \Fam{W}$.
\stopdefinition
Note that $\bigcup \Fam{W}$ is a characterization set as defined for the W-method.
It is needed for completeness to test states with the whole set $\bigcup \Fam{W}$.
Once states are tested as such, we can use the smaller sets $W_s$ for testing transitions.
\stopsubsection
\startsubsection
[title={The HSI-method \cite[LuoPB95, YevtushenkoP90]},
reference=sec:hsi]
The Wp-method in turn was refined by Yevtushenko and Petrenko.
They make use of so called \emph{harmonized} state identifiers (which are called separating families in \cite[DBLP:journals/tc/LeeY94] and in present paper).
By having this global property of the family, less tests need to be executing when testing a state.
\startdefinition
[reference=hsi-method]
Let $\Fam{H}$ be a separating family, the \defn{HSI test suite} is defined as
$(P \cup Q) \cdot I^{\leq k} \odot \Fam{H}$.
\stopdefinition
Our newly described test method is an instance of the HSI-method.
However, in \cite[LuoPB95, YevtushenkoP90] they describe the HSI-method together with a specific way of generating the separating families.
Namely, the set obtained by a splitting tree with shortest witnesses.
In present paper that is generalized, allowing for our extension to be an instance.
\stopsubsection
\startsubsection
[title={The ADS-method \cite[DBLP:journals/tc/LeeY94]},
reference=sec:ads]
As discussed before, when a Mealy machine admits a adaptive distinguishing sequence, only one test has to be performed for identifying a state.
This is exploited in the ADS-method.
\startdefinition
[reference=ds-method]
Let $\Fam{Z}$ be a separating family where every set is a singleton, the \defn{ADS test suite} is defined as
$(P \cup Q) \cdot I^{\leq k} \odot \Fam{Z}$.
\stopdefinition
\stopsubsection
\startsubsection
[title={The UIOv-method \cite[DBLP:conf/sigcomm/ChanVO89]},
reference=sec:uiov]
Some Mealy machines which do not admit an adaptive distinguishing sequence,
may still admit state identifiers which are singletons.
These are exactly UIO sequences and gives rise to the UIOv-method.
In a way this is a generalization of the ADS-method, since the requirement that state identifiers are harmonized is dropped.
\startdefinition
[reference={uio, uiov-method}]
Let $\Fam{U} = \{ \text{a single UIO for } s \}_{s \in S}$ be a family of UIO sequences, the \defn{UIOv test suite} is defined as
$P \cdot I^{\leq k} \cdot \bigcup \Fam{U} \,\cup\, Q \cdot I^{\leq k} \odot \Fam{U}$.
\stopdefinition
\stopsubsection
\startsubsection
[title={Hybrid ADS method},
reference=sec:hybrid]
\todo{Referentie naar volgend hoofdstuk over Oce}
In this section we describe a new test generation method for Mealy machines.
Its completeness will be proven in a later section, together with completeness for related, known, methods.
From a high level perspective, the method uses the algorithm by Lee and Yannakakis \cite[DBLP:journals/tc/LeeY94] to obtain an ads.
Its completeness will be proven in a later section, together with completeness for all methods defined in this section.
From a high level perspective, the method uses the algorithm by \cite[authoryears][DBLP:journals/tc/LeeY94] to obtain an ads.
If no ads exists, their algorithm still provides some sequences which separates some inequivalent states.
Our extension is to refine the set of sequences by using pairwise separating sequences.
The reason we do this is the fact that the ADS-method generally constructs small test suites \cite[DBLP:journals/infsof/DorofeevaEMCY10].
Hence, this method is a hybrid between the ADS-method and HSI-method.
The reason we do this is the fact that the ADS-method generally constructs small test suites as experiments by \cite[authoryears][DBLP:journals/infsof/DorofeevaEMCY10] suggest.
The test suites are small since an ads can identify a state with a single word, instead of a set of words which is generally needed.
Even if the ads does not exist, using the partial result of Lee and Yannakakis' algorithm can reduce the size of test suites.
We will describe how to obtain a separating families and adaptive distinguishing sequences for Mealy machines.
For the former, one typically uses Moore's or Hopcroft's minimization algorithm \cite[DBLP:conf/lata/SmetsersMJ16].
For the latter, an efficient algorithm is described in \cite[DBLP:journals/tc/LeeY94].
Both algorithms use a splitting tree as data structure.
Instead of manipulating separating families directly, we use a \emph{splitting tree}.
This is a data structure which is used to construct separating families or adaptive distinguishing sequences.
\startdefinition[reference=splitting-tree]
A \defn{splitting tree (for $M$)} is a rooted tree where each node $u$ has
@ -242,20 +398,22 @@ We require that if a node $u$ has children $C(u)$ then
A splitting tree is called \defn{complete} if all inequivalent states belong to different leaves.
\stopdefinition
The splitting tree records the execution of a partition refinement algorithm (such as Moore's or Hopcroft's algorithm).
Each non-leaf node encode a \defn{split} together with a witness.
Since a complete splitting tree contains witnesses for all inequivalences, we can extract a separating family from it.
Briefly, for each state we define the set of words as follows: locate the leaf containing the state and collect all the sequences you read when traversing to the root of the tree.
We refer to \cite[DBLP:conf/lata/SmetsersMJ16] for more details on these algorithms.
Efficient construction of a splitting tree is described in more detail in \in{Chapter}[chap:separating-sequences].
Briefly, the splitting tree records the execution of a partition refinement algorithm (such as Moore's or Hopcroft's algorithm).
Each non-leaf node encode a \defn{split} together with a witness, which is a separating sequence for its children.
From such a tree we can construct a state identifier for a state by locating the leaf containing that state and collecting all the sequences you read when traversing to the root.
For adaptive distinguishing sequences an additional requirement is put on the splitting tree:
for each non-leaf node $u$, the sequence $\sigma(u)$ defines an injective map $x \mapsto (\delta(x, \sigma(u)), \lambda(x, \sigma(u)))$ on the set $l(u)$.
In \cite[DBLP:journals/tc/LeeY94] such splits are called \defn{valid}.
\in{Figure}[fig:example-splitting-tree] shows both a valid and invalid split.
\cite[authoryears][DBLP:journals/tc/LeeY94] call such splits \defn{valid}.
\in{Figure}[fig:example-splitting-tree] shows both valid and invalid splits.
Validity precisely ensures that after performing a split, the states are still distinguishable.
\startplacefigure
[title={A complete splitting tree with shortest witnesses for the specification of \in{Figure}[fig:running-example]. Only the splits $a$ and $aa$ are valid.},
[title={A complete splitting tree with shortest witnesses for the specification of \in{Figure}[fig:running-example].
Only the splits $a$ and $aa$ are valid.
\todo{Commands maken voor deze plaatjes.}},
list={Complete splitting tree with shortest witnesses for \in{Figure}[fig:running-example].},
reference=fig:example-splitting-tree]
\hbox{
\starttikzpicture[node distance=1.5cm]
@ -281,11 +439,11 @@ Validity precisely ensures that after performing a split, the states are still d
}
\stopplacefigure
\todo{Commands maken voor deze plaatjes.}
The following lemma is a result of \cite[authoryears][DBLP:journals/tc/LeeY94].
\startfact
A complete splitting tree with valid splits exists if and only if there exists an adaptive distinguishing sequence \cite[DBLP:journals/tc/LeeY94].
\stopfact
\startlemma
A complete splitting tree with only valid splits exists if and only if there exists an adaptive distinguishing sequence.
\stoplemma
Our method uses the exact same algorithm as the one by Lee and Yannakakis.
However, we also apply it in the case when the splitting tree with valid splits is not complete (and hence no adaptive distinguishing sequence exists).
@ -294,21 +452,24 @@ Their algorithm still produces a family of sets, but is not necessarily a separa
In order to recover separability, we refine that family of sets.
Let $\Fam{Z'}$ be the result of Lee and Yannakakis' algorithm (to distinguish from their notation, we add a prime) and let $\Fam{H}$ be a separating family extracted from an ordinary splitting tree.
The hybrid ADS family is defined as $\Fam{Z'} ; \Fam{H}$, and can be computed as sketched in \in{Algorithm}[alg:hybrid].
\todo{Niet precies wat ik in de code doe (wel equivalent), omdat ik efficienter de boom doorloop. En $\Fam{Z,H}$ gegeven door splitting tree.}
By \in{Lemma}[lemma:refinement-props] we note the following: in the best case this family is an adaptive distinguishing sequence; in the worst case it is equal to $\Fam{H}$; and in general it is a combination of the two families.
For this reason we call the method a hybrid method.
\startplacealgorithm[title={Obtaining the hybrid separating family $\Fam{Z'} ; \Fam{H}$},reference=alg:hybrid]
\startplacealgorithm
[title={Obtaining the hybrid separating family $\Fam{Z'} ; \Fam{H}$},
reference=alg:hybrid]
\startalgorithmic
\REQUIRE{A Mealy machine $M$}
\ENSURE{A separating family $Z'$}
\startlinenumbering
\STATE{$T_1 \leftarrow$ splitting tree for Moore's minimization algorithm}
\STATE{$\Fam{H} \leftarrow$ separating family extracted from $T_1$}
\STATE{$T_2 \leftarrow$ splitting tree with valid splits (see \cite[DBLP:journals/tc/LeeY94])}
\STATE{$\Fam{Z'} \leftarrow$ (incomplete) family as constructed from $T_2$}
\STATE $T_1 \gets$ splitting tree for Moore's minimization algorithm
\STATE $\Fam{H} \gets$ separating family extracted from $T_1$
\STATE $T_2 \gets$ splitting tree with valid splits (see \cite[DBLP:journals/tc/LeeY94])
\STATE $\Fam{Z'} \gets$ (incomplete) family as constructed from $T_2$
\FORALL{inequivalent states $s, t$ in the same leaf of $T_2$}{
\STATE{$Z'_s \leftarrow Z_s \cup H_s$}
\STATE{$Z'_t \leftarrow Z_t \cup H_t$}
\STATE $u \gets lca(s, t)$
\STATE $Z'_s \gets Z_s \cup \{ \sigma(u) \}$
\STATE $Z'_t \gets Z_t \cup \{ \sigma(u) \}$
}
\ENDFOR
\RETURN{$Z'$}
@ -330,6 +491,8 @@ T = P \cdot I^{\leq k+1} \odot (\Fam{Z'} ; \Fam{H}).
\stopformula
\stopdefinition
\stopsubsection
\startsubsection[title={Example}]
\todo{Beter opschrijven, en uiteindelijke test suite geven.
@ -339,7 +502,8 @@ We note that the states $s_2, s_3, s_4$ are not split, so we need to refine the
\startplacefigure
[title={(a): Largest splitting tree with only valid splits for \in{Figure}[fig:running-example].
(b): Its adaptive distinguishing tree in notation of \cite[DBLP:journals/tc/LeeY94]},
(b): Its adaptive distinguishing tree in notation of \cite[DBLP:journals/tc/LeeY94].},
list={Splitting tree and adaptive distinguishing sequence.},
reference=fig:example-splitting-tree]
\startcombination[2*1]{
\hbox{
@ -355,7 +519,7 @@ We note that the states $s_2, s_3, s_4$ are not split, so we need to refine the
(2) edge (3)
(2) edge (4);
\stoptikzpicture
}} {a}
}} {(a)}
{\hbox{
\starttikzpicture[node distance=2.0cm]
\node (0) [text width=2cm, align=center, ] {$s_0, s_1, s_2, s_3, s_4$ $s_0, s_1, s_2, s_3, s_4$ $a$};
@ -369,7 +533,7 @@ We note that the states $s_2, s_3, s_4$ are not split, so we need to refine the
(2) edge (3)
(2) edge (4);
\stoptikzpicture
}} {b}
}} {(b)}
\stopcombination
\stopplacefigure
@ -386,127 +550,6 @@ These families and the refinement $\Fam{Z'};\Fam{H}$ are given below:
\stoptabulate
\todo{In startformula/startalign zetten}
\stopsubsection
\startsubsection
[title={When $k$ is not known}]
In many of the applications described in \in{Section}[sec:applications] no bound on the number of states of the SUT was known.
In such cases it is possible to randomly select test cases from an infinite test suite.
Unfortunately, we lose the theoretical guarantees of completeness with random generation.
Still, for the applications in \in{Section}[sec:applications] it has worked well in finding flaws.
We can randomly test cases as follows.
In the above definition for the hybrid ADS test suite we replace $I^{\leq k}$ by $I^{\ast}$ to obtain an infinite test suite.
Then we sample tests as follows:
\startitemize[n]
\item sample an element $p$ from $P$ uniformly,
\item sample a word $w$ from $I^{\ast}$ with a geometric distribution, and
\item sample uniformly from $(\Fam{Z'} ; \Fam{H})_s$ for the state $s = \delta(s_0, pw)$.
\stopitemize
\stopsection
\startsection
[title={Related test methods},
reference=sec:methods]
In this section, we briefly review the classical conformance testing methods:
the W, Wp, UIO, UIOv, HSI, ADS methods.
Our method described is very similar to some of these methods, so we will relate them by describing them uniformly.
There are many more test generation methods.
Literature shows, however, that not all of them are complete.
For example, the methods in \cite[DBLP:journals/tosem/Bernhard94] are falsified by \cite[DBLP:journals/tosem/Petrenko97] and the UIO-method \cite[DBLP:journals/cn/SabnaniD88] is shown to be incomplete in \cite[DBLP:conf/sigcomm/ChanVO89].
For that reason, completeness of the correct methods is shown (again) in the next section.
We fix a state cover $P$ throughout this section and take the transition cover $Q = P \cdot I$.
\startsubsection
[title={W-method \cite[DBLP:journals/tse/Chow78, Vasilevskii73]}]
Possibly one of the earliest $m$-complete test methods.
\startdefinition
[reference=w-method]
A set of words $W$ is a \defn{characterization set} if for each pair of inequivalent states $s$ and $t$ there exists a word $w \in W$ with $\lambda(s,w) \neq \lambda(t,w)$.
Let $W$ be a characterization set, the \defn{W test suite} is
defined as $(P \cup Q) \cdot I^{\leq k} \cdot W$.
\stopdefinition
If we have a separating family $\Fam{W}$, we can obtain a characterization
set by flattening: take $W = \bigcup \Fam{W}$.
\stopsubsection
\startsubsection
[title={The Wp-method \cite[DBLP:journals/tse/FujiwaraBKAG91]},
reference=sec:wp]
The W-method was refined by Fujiwara to use smaller sets when identifying states.
In order to do that he defined state-local sets of words.
\startdefinition
[reference={state-identifier,wp-method}]
A \defn{state identifier for $s$} is a set $W_s$ such that for every inequivalent
state $t$ there is a $w \in W_s$ such that $\lambda(s, w) \neq \lambda(t, w)$.
Let $\Fam{W}$ be a family of state identifiers, the \defn{Wp test suite} is
defined as $P \cdot I^{\leq k} \cdot \bigcup \Fam{W} \,\cup\, Q \cdot I^{\leq k}
\odot \Fam{W}$.
\stopdefinition
Note that $\bigcup \Fam{W}$ is a characterization set as defined for the W-method.
It is needed for completeness to test states with the whole set $\bigcup \Fam{W}$.
Once states are tested as such, we can use the smaller sets $W_s$ for testing transitions.
\stopsubsection
\startsubsection
[title={The HSI-method \cite[LuoPB95, YevtushenkoP90]},
reference=sec:hsi]
The Wp-method in turn was refined by Yevtushenko and Petrenko.
They make use of so called \emph{harmonized} state identifiers (which are called separating families in \cite[DBLP:journals/tc/LeeY94] and in present paper).
By having this global property of the family, less tests need to be executing when testing a state.
\startdefinition
[reference=hsi-method]
Let $\Fam{H}$ be a separating family, the \defn{HSI test suite} is defined as
$(P \cup Q) \cdot I^{\leq k} \odot \Fam{H}$.
\stopdefinition
Our newly described test method is an instance of the HSI-method.
However, in \cite[LuoPB95, YevtushenkoP90] they describe the HSI-method together with a specific way of generating the separating families.
Namely, the set obtained by a splitting tree with shortest witnesses.
In present paper that is generalized, allowing for our extension to be an instance.
\stopsubsection
\startsubsection
[title={The ADS-method \cite[DBLP:journals/tc/LeeY94]},
reference=sec:ads]
As discussed before, when a Mealy machine admits a adaptive distinguishing sequence, only one test has to be performed for identifying a state.
This is exploited in the ADS-method.
\startdefinition
[reference=ds-method]
Let $\Fam{Z}$ be a separating family where every set is a singleton, the \defn{ADS test suite} is defined as
$(P \cup Q) \cdot I^{\leq k} \odot \Fam{Z}$.
\stopdefinition
\stopsubsection
\startsubsection
[title={The UIOv-method \cite[DBLP:conf/sigcomm/ChanVO89]},
reference=sec:uiov]
Some Mealy machines which do not admit an adaptive distinguishing sequence,
may still admit state identifiers which are singletons.
These are exactly UIO sequences and gives rise to the UIOv-method.
In a way this is a generalization of the ADS-method, since the requirement that state identifiers are harmonized is dropped.
\startdefinition
[reference={uio, uiov-method}]
Given $s \in M$, we say that a word $w \in I^{\ast}$ is an \defn{UIO sequence for $s$} if for all inequivalent $t \in M$ we have $\lambda(s, w) \neq \lambda(t, w)$.
Let $\Fam{U} = \{ \text{a single UIO for } s \}_{s \in S}$ be a family of UIO sequences, the \defn{UIOv test suite} is defined as
$P \cdot I^{\leq k} \cdot \bigcup \Fam{U} \,\cup\, Q \cdot I^{\leq k} \odot \Fam{U}$.
\stopdefinition
\stopsubsection
\startsubsection
@ -548,10 +591,11 @@ The following test suites are all $n+k$-complete:
\stoptabulate
\stoptheorem
\todo{Iets zeggen over de hybrid UIO method}
\todo{Geef groottes van suites voor het voorbeeld. Merk op dat ADS and UIOv niet van toepassing zijn (state 3 heeft geen UIO)}
\todo{Iets zeggen over de hybrid UIO method.}
\todo{Geef groottes van suites voor het voorbeeld. Merk op dat ADS and UIOv niet van toepassing zijn (state 2 heeft geen UIO)}
The incomplete \defn{UIO test suite} is defined as $(P \cup Q) \cdot I^{\leq k} \odot \Fam{U}$, incompleteness is shown in \cite[DBLP:conf/sigcomm/ChanVO89].
\todo{In UIOv sectie.}
It should be noted that the ADS-method is a specific instance of the HSI-method and similarly the UIOv-method is an instance of the Wp-method.
What is generally meant by the Wp-method and HSI-method is the above formula together with a particular way to obtain the (harmonised) state identifiers.
@ -563,12 +607,14 @@ Also we expect the bottom row to perform better as there is a single test for ea
Small experimental results confirm this intuition
\cite[DBLP:journals/infsof/DorofeevaEMCY10].
\stopsubsection
\stopsection
\startsection
[title={Proof of completeness},
reference=sec:completeness]
\todo{Stukje over bisimulaties?}
We fix a specification $M$ which has a minimal representative with $n$ states and an implementation $M'$ with at most $n+k$ states.
We assume that all states are reachable from the initial state in both machines (i.e., both are \defn{connected}).
We define the following notation:
@ -593,6 +639,7 @@ If
\item the machines $M$ and $M'$ agree on $T$,
\stopitemize
then $M$ and $M'$ are equivalent.
\todo{Puntje 2 verdient meer aandacht?}
\stoplemma
\startproof
First, we prove that $P \cdot I^{\leq k}$ reaches all states in $M'$.
@ -626,6 +673,7 @@ And so the machines $M$ and $M'$ are equivalent.
Before we show that the conditions hold for the test methods described in this paper, we reflect on the above proof first.
This proof is very similar to the completeness proof in \cite[DBLP:journals/tse/Chow78].
(In fact, it is also similar to Lemma 4 in \cite[DBLP:journals/iandc/Angluin87] which proves termination in the L* learning algorithm. This correspondence was noted in \cite[DBLP:conf/fase/BergGJLRS05].)
\todo{Hoofdstuk over leren van nom. aut. heeft ook deze stelling.}
In the first part we argue that all states are visited by using some sort of counting and reachability argument.
Then in the second part we show the actual equivalence.
To the best of the authors knowledge, this is first $m$-completeness proof which explicitly uses the concept of a bisimulation.
@ -645,7 +693,7 @@ By transitivity we get $x \sim_{W'_y} y$ and so by definition of state identifie
\stopproof
\startcorollary
The W, Wp, UIOv and hybrid UIOv test suites are $n+k$-complete.
The W, Wp, and UIOv test suites are $n+k$-complete.
\stopcorollary
\startlemma
@ -665,9 +713,9 @@ The HSI, ADS and hybrid ADS test suites are $n+k$-complete.
\stopsection
\startsection
[title={Related Work}]
[title={Related Work and discussion}]
\todo{Opnieuw lezen, want verouderd}
\todo{Opnieuw lezen, want verouderd. Voeg toe: non-det, no-reset.}
Comparison of test methods already appeared in the recent papers \cite[DBLP:journals/infsof/DorofeevaEMCY10] and \cite[DBLP:journals/infsof/EndoS13].
Their work is mainly evaluated on randomly generated Mealy machines.
We continue their work by evaluating on many specifications from industry.
@ -697,19 +745,41 @@ However, they show that finding the minimal adaptive distinguishing sequence is
We would like to incorporate their greedy algorithms in our implementation.
\startsubsection
[title={When $k$ is not known}]
In many of the applications described in \in{Section}[sec:applications] no bound on the number of states of the SUT was known.
In such cases it is possible to randomly select test cases from an infinite test suite.
Unfortunately, we lose the theoretical guarantees of completeness with random generation.
Still, for the applications in \in{Section}[sec:applications] it has worked well in finding flaws.
We can randomly test cases as follows.
In the above definition for the hybrid ADS test suite we replace $I^{\leq k}$ by $I^{\ast}$ to obtain an infinite test suite.
Then we sample tests as follows:
\startitemize[n]
\item sample an element $p$ from $P$ uniformly,
\item sample a word $w$ from $I^{\ast}$ with a geometric distribution, and
\item sample uniformly from $(\Fam{Z'} ; \Fam{H})_s$ for the state $s = \delta(s_0, pw)$.
\stopitemize
\stopsubsection
\todo{Enkele resultaten bespreken, test-suite-groottes vergelijken}
\todo{Future work? Meer benchmarks? Andere automaat-modellen?}
\stopsection
\startsection
[title={Applications},
reference=sec:applications]
\todo{Kan waarschijnlijk weg. In de introductie wordt gepraat over toepassingen van leren (zie Vaandrager (2017).}
The presented test generation methods is implemented and used in a couple of applications.
The implementation can be found on {\tt https://gitlab.science.ru.nl/moerman/hybrid-ads}.
This implementations has been used in several model learning applications:
learning embedded controller software \cite[DBLP:conf/icfem/SmeenkMVJ15], learning the TCP protocol \cite[DBLP:conf/cav/Fiterau-Brostean16] and learning the MQTT protocol \cite[DBLP:conf/icst/TapplerAB17].
\todo{Enkele resultaten bespreken, test-suite-groottes vergelijken}
\todo{Future work? Meer benchmarks? Andere automaat-modellen?}
\stopsection
\referencesifcomponent

4
environment.tex
View file

@ -21,8 +21,7 @@
\defineenumeration[lemma][text=Lemma]
\defineenumeration[theorem][text=Theorem]
\defineenumeration[corollary][text=Corollary]
\defineenumeration[fact][text=Fact?] % niet nodig?
\setupenumeration[definition,example,lemma,theorem,corollary,fact][alternative=serried,width=fit,right=.]
\setupenumeration[definition,example,lemma,theorem,corollary][alternative=serried,width=fit,right=.]
\definestartstop[proof][before={{\it Proof. }}, after={\hfill$\square$}]
@ -70,6 +69,7 @@
%\showboxes
%\showframe
%\setupparagraphnumbering[state=start,style=italic,distance=0pt]
%\showbodyfont
\stopenvironment