joshua/phd-thesis
Archived
1
Fork 0
Code Activity
This repository has been archived on 2025-04-09. You can view files and clone it, but cannot push or open issues or pull requests.
phd-thesis/content/test-methods.tex
Joshua Moerman 54bbd389e0 Voorbeeld ADS in elkaar gepleurd
2018-11-06 21:53:55 +01:00

1048 lines
52 KiB
TeX
Raw Blame History

\project thesis
\startcomponent test-methods
\startchapter
[title={FSM-based Test Methods},
reference=chap:test-methods]
In this chapter we will discuss some of the theory of test generation methods for black box conformance testing.
From a theoretical discussion we derive a new algorithm: the \emph{hybrid ADS methods}, which is applied on a case study in the next chapter (\in{Chapter}[chap:applying-automata-learning]).
A key aspect of such methods is the size of the obtained test suite.
On one hand we want to cover as much as the specification as possible.
On the other hand: testing takes time, so we want to minimise the size of a test suite.
The test methods described here are well-known in the literature of FSM-based testing.
They all share similar concepts, such as \emph{access sequences} and \emph{state identifiers}.
In this chapter we will define these concepts, relate them with one another and show how to build test suites from these concepts.
\startsection
[title={Mealy machines and sequences}]
We will focus on Mealy machines, as those capture many protocol specifications and reactive systems.
We fix finite alphabets $I$ and $O$ of inputs respectively outputs.
We use the usual notation for operations on \emph{sequences} (also called \emph{words}):
$uv$ for the concatenation of two sequences $u, v \in I^{\ast}$ and $|u|$ for the length of $u$.
For a sequence $w = uv$ we say that $u$ and $v$ are a prefix and suffix respectively.
\startdefinition
A \emph{Mealy machine} $M$ consists of a finite set of \emph{states} $S$, an \emph{initial state} $s_0 \in S$ and two functions:
\startitemize
\item a \emph{transition function} $\delta \colon S \times I \to S$, and
\item an \emph{output function} $\lambda \colon S \times I \to O$.
\stopitemize
Both the transition function and output function are extended inductively to sequences as $\delta \colon S \times I^{\ast} \to S$ and $\lambda \colon S \times I^{\ast} \to O^{\ast}$:
\startformula\startmathmatrix[n=4, align={right,left,right,left},distance=1pt]
\NC \delta(s, \epsilon) \NC = s \NC\quad \lambda(s, \epsilon) \NC = \epsilon \NR
\NC \delta(s, aw) \NC = \delta(\delta(s, a), w) \NC\qquad \lambda(s, aw) \NC = \lambda(s, a)\lambda(\delta(s, a), w) \NR
\stopmathmatrix\stopformula
The \emph{behaviour} of a state $s$ is given by the output function $\lambda(s, -) \colon I^{\ast} \to O^{\ast}$.
Two states $s$ and $t$ are \emph{equivalent} if they have equal behaviours, written $s \sim t$, and two Mealy machines are equivalent if their initial states are equivalent.
\stopdefinition
\startremark
We will use the following conventions and notation.
We often write $s \in M$ instead of $s \in S$ and for a second Mealy machine $M'$ its members are denoted $S', s'_0, \delta'$ and $\lambda'$.
Moreover, if we have a state $s \in M$, we silently assume that $s$ is not a member of any other Mealy machine $M'$.
(In other words, the behaviour of $s$ is determined by the state itself.)
This eases the notation since we can write $s \sim t$ without needing to introduce a context.
\stopremark
An example Mealy machine is given in \in{Figure}[fig:running-example].
We note that a Mealy machine is deterministic and complete by definition.
This means that for each state $s$ and each word $w$, there is a unique state $t$ by running the word $w$ from $s$.
\startplacefigure
[title={An example specification with input $I=\{a,b,c\}$ and output $O=\{0,1\}$.},
list={An example specification.},
reference=fig:running-example]
\hbox{
\starttikzpicture[node distance=0.9cm and 3cm,bend angle=20]
\node[state] (0) {$s_0$};
\node[state] (1) [above left = of 0] {$s_1$};
\node[state] (2) [below left = of 0] {$s_2$};
\node[state] (3) [above right = of 0] {$s_3$};
\node[state] (4) [below right = of 0] {$s_4$};
\node (5) [above = of 0] {};
\path[->]
(5) edge (0)
(0) edge [bend left=] node [below] {${a}/1$} (1)
(0) edge [bend right] node [below] {${b}/0$} (4)
(0) edge [loop below] node [below] {${c}/0$} (0)
(1) edge node [left ] {${a}/0$} (2)
(1) edge [bend left] node [above] {${b}/0$, ${c}/0$} (0)
(2) edge [bend right] node [below] {${b}/0$, ${c}/0$} (0)
(2) edge [loop left] node [left ] {${a}/1$} (1)
(3) edge [bend left=30] node [right] {${a}/1$} (4)
(3) edge [bend right] node [above] {${b}/0$} (0)
(3) edge [loop right] node [right] {${c}/1$} (3)
(4) edge [bend left] node [right] {${a}/1$} (3)
(4) edge [loop right] node [right] {${b}/0$} (4)
(4) edge [bend right] node [above] {${c}/0$} (0);
\stoptikzpicture
}
\stopplacefigure
\startsubsection
[title={Testing}]
In conformance testing we have a specification modelled as a Mealy machine and an implementation (the system under test, or SUT) which we assume to behave as a Mealy machine.
Tests, or experiments, are generated from the specification and applied to the implementation. We assume that we can reset the implementation before every test.
If the output is different than the specified output, then we know the implementation is flawed.
The goals is to test as little as possible, while covering as much as possible.
We assume the following
\startitemize[after]
\item
The system can be modelled as Mealy machine.
In particular, this means it is \emph{deterministic} and \emph{complete}.
\item
We are able to reset the system, i.e., bring it back to the initial state.
\stopitemize
A test suite is nothing more than a set of sequences.
We do not have to encode outputs in the test suite, as those follow from the deterministic specification.
\startdefinition[reference=test-suite]
A \emph{test suite} is a finite subset $T \subseteq I^{\ast}$.
\stopdefinition
A test $t \in T$ is called \emph{maximal} if it is not a proper prefix of another test $s \in T$.
We denote the set of maximal tests of $T$ by $\max(T)$.
The maximal tests are the only tests in $T$ we actually have to apply to our SUT as we can record the intermediate outputs.
In the examples of this chapter we will show $\max(T)$ instead of $T$.
We define the size of a test suite as usual \citep[DBLP:journals/infsof/DorofeevaEMCY10, DBLP:conf/hase/Petrenko0GHO14].
The size of a test suite is measured as the sum of the lengths of all its maximal tests plus one reset per test.
\startdefinition[reference=test-suite-size]
The \emph{size} of a test suite $T$ is defined to be
$||T|| = \sum\limits_{t \in max(T)} (|t| + 1)$.
\stopdefinition
\stopsubsection
\startsubsection
[title={Completeness of test suites}]
\startexample[reference=incompleteness]
{\bf No test suite is complete.}
Consider the specification in \in{Figure}{a}[fig:incompleteness-example].
This machine will always produce zeroes.
For any test suite we can make a faulty implementation which passes the test suite.
Such an implementation might look like \in{Figure}{b}[fig:incompleteness-example] with $n$ big enough.
This shows that no test-suite can be complete and it justifies the following definition.
\startplacefigure
[title={A basic example showing that finite test suites are incomplete. The implementation on the right will pass any test suite if we choose $n$ big enough.},
list={A basic example showing that finite test suites are incomplete.},
reference=fig:incompleteness-example]
\startcombination[2*1]
{\hbox{
\starttikzpicture
\node[state] (0) {$s_0$};
\path[->]
(0) edge [loop] node [below] {${a}/0$} (0);
\stoptikzpicture
}} {(a)}
{\hbox{
\starttikzpicture
\node[state] (0) {$s'_0$};
\node[state] (1) [right of=0] {$s'_1$};
\node (2) [right of=1] {$\cdots$};
\node[state] (3) [right of=2] {$s'_n$};
\path[->]
(0) edge [bend left=20 ] node [below] {${a}/0$} (1)
(1) edge [bend left=20 ] node [below] {${a}/0$} (2)
(2) edge [bend left=20 ] node [below] {${a}/0$} (3)
(3) edge [loop ] node [below] {${a}/1$} (3);
\stoptikzpicture
}} {(b)}
\stopcombination
\stopplacefigure
\stopexample
\startdefinition[reference=completeness]
Let $M$ be a Mealy machine and $T$ be a test suite.
We say that $T$ is \emph{$m$-complete (for $M$)} if for all inequivalent machines $M'$ with at most $m$ states there exists a $t \in T$ such that $\lambda(s_0, t) \neq \lambda'(s'_0, t)$.
\stopdefinition
We are often interested in the case of $m$-completeness, where $m = n + k$ for some $k \in \naturalnumbers$ and $n$ is the number of states in the specification.
Here $k$ will stand for the number of \emph{extra states} we can test.
The issue of an unknown bound is addressed later in the paper.
Note the order of the quantifiers in the above definition.
We ask for a single test suite which works for all implementations of bounded size.
This is crucial for black box testing, as we do not know the implementation, so the test suite has to work for all of them.
\stopsubsection
\startsubsection
[title={Separating Sequences}]
Before we construct test suites, we discuss several types of useful sequences.
All the following notions are standard in the literature, and the corresponding references will be given in \in{Section}[sec:methods], where we discuss the test generation methods using these notions.
We fix a Mealy machine $M$.
For convenience we assume $M$ to be minimal, this implies that distinct states are, in fact, inequivalent.
All definitions can be generalised to non-minimal $M$, by replacing the word \quote{distinct} (or \quote{other}) with \quote{inequivalent}.
\startdefinition
Given a Mealy machine $M$ we define the following kinds of sequences.
\startitemize
\item Given two states $s, t$ in $M$ we say that $w$ is a \defn{separating sequence} if $\lambda(s, w) \neq \lambda(t, w)$.
\item For a single state $s$ in $M$, a sequence $w$ is a \defn{unique input output sequence (UIO)} if for every other state $t$ in $M$ we have $\lambda(s, w) \neq \lambda(t, w)$.
\item Finally, a \defn{(preset) distinguishing sequence (DS)} is a single sequence $w$ which separates all states of $M$, i.e., for every distinct pair $s, t$ in $M$ we have $\lambda(s, w) \neq \lambda(t, w)$.
\stopitemize
\stopdefinition
The above list is ordered from weaker to stronger notions, i.e., every distinguishing sequence is an UIO sequence for every state.
Similarly, an UIO for a state $s$ is a separating sequence for $s$ and any other $t$.
Separating sequences always exist for inequivalent states and finding them efficiently is the topic of \in{Chapter}[chap:separating-sequences].
On the other hand, UIOs and DSs do not always exist for a machine.
\startexample
For the machine in \in{Figure}[fig:running-example], we note that state $s_0$ and $s_2$ are separated by the sequence $aa$ (but not by any shorter sequence).
In fact, the sequence $aa$ is an UIO for state $s_0$ since it is the only state outputting $10$ on that input.
However, state $s_2$ has no UIO:
If the sequence were to start with $b$ or $c$, state $s_3$ and $s_4$ respectively have equal transition, which makes it impossible to separate those states after the first symbol.
If it starts with an $a$, states $s_3$ and $s_4$ are swapped and we make no progress in distinguishing these states from $s_2$.
Since $s_2$ has no UIO, the machine as a whole does not admit a DS.
In this example, all other states actually have UIOs.
For the states $s_0, s_1, s_3$ and $s_4$, we can pick the sequences $aa, a, c$ and $ac$ respectively.
In order to separate $s_2$ from the other state, we have to pick multiple sequences.
For instance, the set $\{aa, ac, c\}$ will separate $s_2$ from all other states.
\stopexample
\stopsubsection
\startsubsection
[title={Sets of separating sequences}]
As the example shows, we need sets of sequences and sometimes even sets of sets of sequences -- called \emph{families}.
\footnote{A family of often written as $\{X_s\}_{s \in M}$ or simply $\{X_s\}_{s}$, meaning that for each state $s \in M$ we have a set $X_s$.}
\startdefinition
Given a Mealy machine $M$, we define the following kinds of sets of sequences.
We require that all sets are \emph{prefix-closed}, however, we only give the maximal sequences in examples.
\footnote{Taking these sets to be prefix-closed makes many proofs easier.}
\startitemize
\item A set of sequences $W$ is a called a \defn{characterisation set} if it contains a separating sequence for each pair of distinct states in $M$.
\item A \defn{state identifier} for a state $s \in M$ is a set $W_s$ which contains a separating sequence for every other state $t \in M$.
\item A set of state identifiers $\{ W_s \}_{s}$ is \defn{harmonised} if a separating sequence $w$ for states $s$ and $t$ exists in both $W_s$ and $W_t$.
This is also called a \defn{separating family}.
\stopitemize
\stopdefinition
The property of being harmonised might seem a bit strange.
This property ensure that the same tests are used for different states.
This extra consistency within a test suite is necessary for some test methods.
We return to this notion in more detail in \in{Example}[ex:uio-counterexample].
We may obtain a characterisation set by taking the union of state identifiers for each state.
For every machine we can construct a set of harmonised state identifiers as will be shown in \in{Chapter}[chap:separating-sequences] and hence every machine has a characterisation set.
\startexample
As mentioned before, state $s_2$ from \in{Figure}[fig:running-example] has a state identifier $\{aa, ac, b\}$.
In fact, this set is a characterisation set for the whole machine.
Since the other states have UIOs, we can pick singleton sets as state identifiers.
For example, state $s_0$ has the UIO $aa$, so a state identifier for $s_0$ is $W_0 = \{ aa \}$.
Similarly, we can take $W_1 = \{ a \}$ and $W_3 = \{ c \}$.
But note that such a family will \emph{not} be harmonised since the sets $\{ a \}$ and $\{ c \}$ have no common separating sequence.
\stopexample
One more type of state identifier is of our interest: the \emph{adaptive distinguishing sequence}.
It it the strongest type of state identifier, and as a result not many machine have one.
\startdefinition
Following the results of \citet[DBLP:journals/tc/LeeY94], a separating family where each set is the prefix-closure of a single word is called an \defn{adaptive distinguishing sequence} (ADS).
\stopdefinition
An ADS is of special interest since they can identify a state using a single word.
It is called an adaptive sequence, since it has a tree structure which depends on the output of the machine.
To see this, consider the first symbols of each of the sequences in the family.
Since the family is harmonised and each set is given by a single word, there is only one first symbol.
So we test each state with one symbol, observe the output, and then continue with the remainder of the sequence (which depends on the output).
\todo{Verwijs naar plaatje}
Given an ADS, there exists an UIO for every state.
The converse -- if every state has an UIO, then the machine admits an ADS-- does not hold.
The machine in \in{Figure}[fig:running-example] admits no ADS, since $s_2$ has no UIO.
\startplacefigure
[title={(a): A Mealy machine with alphabets $I = \{a, b\}$ and $O = \{0, 1\}$ and (b): an ADS for (a). },
list={A Mealy machine and its ADS.}
reference=fig:ads-example]
\startcombination[nx=2, ny=1, align=center]
{\hbox{
\starttikzpicture
\node[initial, state] (0) {$s_0$};
\node[state] (1) [right=of 0] {$s_1$};
\node[state] (2) [below=of 1] {$s_2$};
\node[state] (3) [left =of 2] {$s_3$};
\path[->]
(0) edge node [above] {$a/0, b/0$} (1)
(1) edge node [right] {$a/1$} (2)
(2) edge node [above] {$a/0, b/0$} (3)
(3) edge node [left, align=left] {$a/1,$ \\ $b/0$} (0)
(1) edge [loop right] node {$b/0$} (1);
\stoptikzpicture}} {(a)}
{\hbox{
\starttikzpicture[node distance=.75cm]
\node (0) {$a$};
\node [below left=of 0] (1) {$b$};
\node [below=of 1] (3) {$a$};
\node [below left=of 3] (5) {$2$};
\node [below right=of 3] (6) {$0$};
\node [below right=of 0] (2) {$a$};
\node [below=of 2] (4) {$b$};
\node [below=of 4] (7) {$a$};
\node [below left=of 7] (8) {$1$};
\node [below right=of 7] (9) {$3$};
\path[->]
(0)+(0,.75) edge (0)
(0) edge node [above left] {$0$} (1)
(0) edge node [above right] {$1$} (2)
(1) edge node [left] {$0$} (3)
(3) edge node [above left] {$0$} (5)
(3) edge node [above right] {$1$} (6)
(2) edge node [right] {$0$} (4)
(4) edge node [right] {$0$} (7)
(7) edge node [above left] {$0$} (8)
(7) edge node [above right] {$1$} (9);
\stoptikzpicture}} {(b)}
\stopcombination
\stopplacefigure
\todo{Dit staat een beetje random}
Besides sequences which separate states, we also need sequences which brings a machine to specified states.
\startdefinition
An \emph{access sequence for $s$} is a word $w$ such that $\delta(q_0, w) = s$.
A set $P$ consisting of an access sequence for each state is called a \emph{state cover}.
If $P$ is a state cover, then $P \cdot I$ is called a \emph{transition cover}.
\stopdefinition
\stopsubsection
\startsubsection
[title={Constructions on sets of sequences}]
In order to define a test suite modularly, we introduce notation for combining sets of words.
We fix a specification $M$.
For sets of words $X$ and $Y$, we define
\startitemize[after]
\item their concatenation $X \cdot Y = \{ xy \mid x \in X, y \in Y \}$,
\item iterated concatenation $X^0 = \{ \epsilon \}$ and $X^{n+1} = X \cdot X^{n}$, and
\item bounded concatenation $X^{\leq n} = \bigcup_{i \leq n} X^i$.
\stopitemize
On families we define
\startitemize[after]
\item flattening: $\bigcup \Fam{X} = \{ x \mid x \in X_s, s \in S \}$,
\item union: $\Fam{X} \cup \Fam{Y}$ is defined point-wise:
$(\Fam{X} \cup \Fam{Y})_s = X_s \cup Y_s$, and
\item $\Fam{X}$-equivalence:
$x \sim_\Fam{X} y$ if $\lambda(x,w) = \lambda(y,w)$ for all $w \in X_x \cap X_y$.
\item concatenation:
$X \odot \Fam{Y} = \{ xy \mid x \in X, y \in Y_{\delta(s_0, x)} \}$ and
\item refinement: $\Fam{X} ; \Fam{Y}$ defined by
\startformula
(\Fam{X} ; \Fam{Y})_s = X_s \,\cup\, Y_s \cap \bigcup_{s \sim_{\Fam{X}} t} Y_t.
\stopformula
\stopitemize
The latter construction is new and will be used to define a hybrid test generation method in \in{Section}[sec:hybrid].
It refines a family $\Fam{X}$, which need not be separating, by including sequences from a second family $\Fam{Y}$.
It only adds those sequences to states if $\Fam{X}$ does not distinguish those states.
\startlemma[reference=lemma:refinement-props]
For all families $\Fam{X}$ and $\Fam{Y}$:
\startitemize
\item $\Fam{X} ; \Fam{X} = \Fam{X}$,
\item $\Fam{X} ; \Fam{Y} = \Fam{X}$, whenever $\Fam{X}$ is a separating family, and
\item $\Fam{X} ; \Fam{Y}$ is a separating family whenever $\Fam{Y}$ is a separating family.
\stopitemize
\stoplemma
\startproof
\todo{Kort uitschrijven}
\stopproof
\stopsubsection
\stopsection
\startsection
[title={Test generation methods},
reference=sec:methods]
In this section, we review the classical conformance testing methods:
the W, Wp, UIO, UIOv, HSI, ADS methods.
Our hybrid ADS method uses a similar construction.
There are many more test generation methods.
Literature shows, however, that not all of them are complete.
For example, the method by \citet[DBLP:journals/tosem/Bernhard94] are falsified by \citet[DBLP:journals/tosem/Petrenko97] and the UIO-method from \citet[DBLP:journals/cn/SabnaniD88] is shown to be incomplete by \citet[DBLP:conf/sigcomm/ChanVO89].
For that reason, completeness of the correct methods is shown in \in{Theorem}[thm:completeness].
The proof is general enough to capture all the methods at once.
We fix a state cover $P$ throughout this section and take the transition cover $Q = P \cdot I$.
\startsubsection
[title={W-method \cite[DBLP:journals/tse/Chow78, Vasilevskii73]},
reference=sec:w]
Possibly one of the earliest $m$-complete test methods.
\todo{Opmerking over Moore's vermoeden?}
\startdefinition
[reference=w-method]
Let $W$ be a characterisation set, the \defn{W test suite} is
defined as
\startformula
(P \cup Q) \cdot I^{\leq k} \cdot W .
\stopformula
\stopdefinition
\todo{Uitleggen}
\stopsubsection
\startsubsection
[title={The Wp-method \cite[DBLP:journals/tse/FujiwaraBKAG91]},
reference=sec:wp]
The W-method was refined by Fujiwara et al. to use smaller sets when identifying states.
In order to do that he defined state-local sets of words.
\startdefinition
[reference={state-identifier,wp-method}]
Let $\Fam{W}$ be a family of state identifiers, the \defn{Wp test suite} is
defined as
\startformula
P \cdot I^{\leq k} \cdot \bigcup \Fam{W} \,\cup\, Q \cdot I^{\leq k}
\odot \Fam{W}.
\stopformula
\stopdefinition
Note that $\bigcup \Fam{W}$ is a characterisation set as defined for the W-method.
It is needed for completeness to test states with the whole set $\bigcup \Fam{W}$.
Once states are tested as such, we can use the smaller sets $W_s$ for testing transitions.
\stopsubsection
\startsubsection
[title={The HSI-method \cite[LuoPB95, YevtushenkoP90]},
reference=sec:hsi]
The Wp-method in turn was refined by Yevtushenko and Petrenko.
They make use of so called \emph{harmonised} state identifiers.
By having this global property of the family, less tests need to be executing when testing a state.
\startdefinition
[reference=hsi-method]
Let $\Fam{H}$ be a separating family, the \defn{HSI test suite} is defined as
\startformula
(P \cup Q) \cdot I^{\leq k} \odot \Fam{H}.
\stopformula
\stopdefinition
Our newly described test method is an instance of the HSI-method.
However, \citet[LuoPB95, YevtushenkoP90] describe the HSI-method together with a specific way of generating the separating families.
Namely, the set obtained by a splitting tree with shortest witnesses.
\stopsubsection
\startsubsection
[title={The ADS-method \cite[DBLP:journals/tc/LeeY94]},
reference=sec:ads]
As discussed before, when a Mealy machine admits a adaptive distinguishing sequence, only one test has to be performed for identifying a state.
This is exploited in the ADS-method.
\startdefinition
[reference=ds-method]
Let $\Fam{Z}$ be a separating family where every set is a singleton, the \defn{ADS test suite} is defined as
\startformula
(P \cup Q) \cdot I^{\leq k} \odot \Fam{Z}.
\stopformula
\stopdefinition
\stopsubsection
\startsubsection
[title={The UIOv-method \cite[DBLP:conf/sigcomm/ChanVO89]},
reference=sec:uiov]
Some Mealy machines which do not admit an adaptive distinguishing sequence,
may still admit state identifiers which are singletons.
These are exactly UIO sequences and gives rise to the UIOv-method.
In a way this is a generalisation of the ADS-method, since the requirement that state identifiers are harmonised is dropped.
\startdefinition
[reference={uio, uiov-method}]
Let $\Fam{U} = \{ \text{a single UIO for } s \}_{s \in S}$ be a family of UIO sequences, the \defn{UIOv test suite} is defined as
\startformula
P \cdot I^{\leq k} \cdot \bigcup \Fam{U} \,\cup\, Q \cdot I^{\leq k} \odot \Fam{U}.
\stopformula
\stopdefinition
One might think that using a single UIO sequence instead of the set $\bigcup \Fam{U}$ to verify the state is enough.
In fact, this idea was used for the \emph{UIO-method} which defines the test suite $(P \cup Q) \cdot I^{\leq k} \odot \Fam{U}$.
The following is a counterexample to such conjecture.
(This counterexample is due to \citenp[DBLP:conf/sigcomm/ChanVO89].)
\startexample
[reference=ex:uio-counterexample]
The example in \in{Figure}[fig:uio-counterexample] shows that UIO-method does not define a 3-complete test suite.
Take for example the UIOs $u_0 = aa, u_1 = a, u_2 = ba$ for the states $s_0, s_1, s_2$ respectively.
The test suite then becomes $\{ aaaa, abba, baaa, bba \}$ and the faulty implementation passes this suite.
This happens because the sequence $u_2$ is not an UIO in the implementation, and the state $s'_2$ simulates both UIOs $u_1$ and $u_2$.
Hence we also want to check that a state does not behave as one of the other states, and therefore we use $\bigcup \Fam{U}$.
With the same UIOs as above, the resulting UIOv test suite for the specification in \in{Figure}[fig:uio-counterexample] is $\{aaaa, aba, abba, baaa, bba\}$ of size $23$.
\startplacefigure
[title={An example where the UIO method is not complete.},
reference=fig:uio-counterexample]
\startcombination[2*1]
{\hbox{\starttikzpicture
\node[state,initial,initial text=] (0) {$s_0$};
\node[state] (1) [above right of=0] {$s_1$};
\node[state] (2) [below right of=1] {$s_2$};
\path[->]
(0) edge [bend left=20 ] node [left ] {${a}/0$} (1)
(0) edge [bend right=15] node [below] {${b}/1$} (2)
(1) edge [bend left=20 ] node [left ] {${a}/1$} (0)
(1) edge [bend right=20] node [right] {${b}/1$} (2)
(2) edge [bend right=20] node [below] {${a}/0$} (0)
(2) edge [bend right=15] node [right] {${b}/1$} (1);
\stoptikzpicture}} {Specification}
{\hbox{\starttikzpicture
\node[state,initial,initial text=] (0) {$s'_0$};
\node[state] (1) [above right of=0] {$s'_1$};
\node[state] (2) [below right of=1] {$s'_2$};
\path[->]
(0) edge [bend left=20 ] node [left ] {${a}/0$} (1)
(0) edge [bend right=15] node [below] {${b}/1$} (2)
(1) edge [bend left=20 ] node [left ] {${a}/1$} (0)
(1) edge [loop ] node [above] {${b}/1$} (1)
(2) edge [bend right=20] node [below] {${a}/0$} (0)
(2) edge [bend right=15] node [right] {${b}/1$} (1);
\stoptikzpicture}} {Implementation}
\stopcombination
\stopplacefigure
\stopexample
\stopsubsection
\startsubsection
[title={Example},
reference=sec:all-example]
Let us compute all the previous test suites on the specification in \in{Figure}[fig:running-example].
We will be testing without extra states, i.e., we construct 5-complete test suites.
We start by defining the state and transition cover.
For this, we simply take all shortest sequences from the initial state to the other states:
\startformula\startalign
\NC P \NC = \{ \epsilon, a, aa, b, ba \} \NR
\NC Q = P \cdot I \NC = \{ a, b, c, aa, ab, ac, aaa, aab, aac, ba, bb, bc, baa, bab, bac \} \NR
\stopalign\stopformula
As shown earlier, the set $W = \{ aa, ac, c \}$ is a characterisation set.
The the W-method gives the following test suite of size 169:
\startformula\startmathmatrix[n=2,align={left,left}]
\NC T_{\text{W}} = \{ \NC aaaaa, aaaac, aaac, aabaa, aabac, aabc, aacaa, \NR
\NC \NC aacac, aacc, abaa, abac, abc, acaa, acac, acc, baaaa, \NR
\NC \NC baaac, baac, babaa, babac, babc, bacaa, bacac, bacc, \NR
\NC \NC bbaa, bbac, bbc, bcaa, bcac, bcc, caa, cac, cc \quad \} \NR
\stopmathmatrix\stopformula
With the Wp-method we get to choose a different state identifier per state.
Since many states have an UIO, we can use them as state identifiers.
This defines the following family $\Fam{W}$:
\startformulas
\startformula W_0 = \{ aa \} \stopformula
\startformula W_1 = \{ a \} \stopformula
\startformula W_2 = \{ aa, ac, c \} \stopformula
\startformula W_3 = \{ c \} \stopformula
\startformula W_4 = \{ ac \} \stopformula
\stopformulas
For the first part of the Wp test suite we need $\bigcup \Fam{W} = \{ aa, ac, c \}$.
For the second part, we only combine the sequences in the transition cover with the corresponding suffixes.
All in al we get a test suite of size 75:
\startformula\startmathmatrix[n=2,align={left,left}]
\NC T_{\text{Wp}} = \{ \NC aaaaa, aaaac, aaac, aabaa, aacaa, abaa, \NR
\NC \NC acaa, baaac, baac, babaa, bacc, bbac, bcaa, caa \quad \} \NR
\stopmathmatrix\stopformula
For the HSI method we need a separating family.
We pick the following sets:
\startformulas
\startformula H_0 = \{ aa, c \} \stopformula
\startformula H_1 = \{ a \} \stopformula
\startformula H_2 = \{ aa, ac, c \} \stopformula
\startformula H_3 = \{ a, c \} \stopformula
\startformula H_4 = \{ aa, ac, c \} \stopformula
\stopformulas
Note that these sets are harmonised, unlike the family $\Fam{W}$.
For example, the separating sequence $a$ is contained in both $H_1$ and $H_3$.
This ensures that we do not have to consider $\bigcup \Fam{H}$ in the first part of the test suite.
When combining this with the corresponding prefixes, we obtain the HSI test suite of size 125:
\startformula\startmathmatrix[n=2,align={left,left}]
\NC T_{\text{HSI}} = \{ \NC aaaaa, aaaac, aaac, aabaa, aabc, aacaa, aacc, \NR
\NC \NC abaa, abc, acaa, acc, baaaa, baaac, baac, babaa, \NR
\NC \NC babc, baca, bacc, bbaa, bbac, bbc, bcaa, bcc, caa, cc \quad \} \NR
\stopmathmatrix\stopformula
On this particular example the Wp method outperforms the HSI method.
The reason is that many states have UIOs and we picked those to be the state identifiers.
In general, however, UIOs may not exist (and finding them is hard).
The UIO method and ADS method are not applicable in this example because state $s_2$ does not have an UIO.
\todo{Draaien op faulty implementation}
\stopsubsection
\stopsection
\startsection
[title={Hybrid ADS method},
reference=sec:hybrid]
In this section we describe a new test generation method for Mealy machines.
Its completeness will be proven in a later section, together with completeness for all methods defined in this section.
From a high level perspective, the method uses the algorithm by \cite[authoryears][DBLP:journals/tc/LeeY94] to obtain an ADS.
If no ADS exists, their algorithm still provides some sequences which separates some inequivalent states.
Our extension is to refine the set of sequences by using pairwise separating sequences.
Hence, this method is a hybrid between the ADS-method and HSI-method.
The reason we do this is the fact that the ADS-method generally constructs small test suites as experiments by \cite[authoryears][DBLP:journals/infsof/DorofeevaEMCY10] suggest.
The test suites are small since an ads can identify a state with a single word, instead of a set of words which is generally needed.
Even if the ADS does not exist, using the partial result of Lee and Yannakakis' algorithm can reduce the size of test suites.
We will now see the construction of this hybrid method.
Instead of manipulating separating families directly, we use a \emph{splitting tree}.
This is a data structure which is used to construct separating families or adaptive distinguishing sequences.
\startdefinition[reference=splitting-tree]
A \defn{splitting tree (for $M$)} is a rooted tree where each node $u$ has
\startitemize
\item a set of states $l(u) \subseteq M$, and
\item if $u$ is not a leaf, a sequence $\sigma(u) \in I^{\ast}$.
\stopitemize
We require that if a node $u$ has children $C(u)$ then
\startitemize
\item the sets of states of the children of $u$ partition $l(u)$, i.e., the set
$P(u) = \{l(v) \,|\, v \in C(u)\}$ is a \emph{non-trivial} partition of $l(u)$, and
\item the sequence $\sigma(u)$ witnesses the partition $P(u)$, meaning that
$\lambda(s, \sigma(u)) = \lambda(t, \sigma(u))$ iff $p = q$ for all $s \in p, t \in q$ for all $p, q \in P(u)$.
\stopitemize
A splitting tree is called \defn{complete} if all inequivalent states belong to different leaves.
\stopdefinition
Efficient construction of a splitting tree is described in more detail in \in{Chapter}[chap:minimal-separating-sequences].
Briefly, the splitting tree records the execution of a partition refinement algorithm (such as Moore's or Hopcroft's algorithm).
Each non-leaf node encodes a \defn{split} together with a witness, which is a separating sequence for its children.
From such a tree we can construct a state identifier for a state by locating the leaf containing that state and collecting all the sequences you read when traversing to the root.
For adaptive distinguishing sequences an additional requirement is put on the splitting tree:
for each non-leaf node $u$, the sequence $\sigma(u)$ defines an injective map $x \mapsto (\delta(x, \sigma(u)), \lambda(x, \sigma(u)))$ on the set $l(u)$.
\cite[authoryears][DBLP:journals/tc/LeeY94] call such splits \defn{valid}.
\in{Figure}[fig:example-splitting-tree] shows both valid and invalid splits.
Validity precisely ensures that after performing a split, the states are still distinguishable.
Hence, sequences of such splits can be concatenated.
\startplacefigure
[title={A complete splitting tree with shortest witnesses for the specification of \in{Figure}[fig:running-example].
Only the splits $a$, $aa$, and $ac$ are valid.},
list={Complete splitting tree with shortest witnesses for \in{Figure}[fig:running-example].},
reference=fig:example-splitting-tree]
\hbox{
\starttikzpicture[node distance=1.75cm]
\splitnode{0}{s_0, s_1, s_2, s_3, s_4}{a}{}
\splitnode{3}{s_0, s_2, s_3, s_4}{c}{below right of=0}
\splitnode{5}{s_0, s_2, s_4}{aa}{below left of=3}
\splitnode{8}{s_2, s_4}{ac}{below right of=5}
\node (1) [below left of=0 ] {$s_1$};
\node (6) [below right of=3] {$s_3$};
\node (7) [below left of=5 ] {$s_0$};
\node (8l)[below left of=8 ] {$s_2$};
\node (8r)[below right of=8] {$s_4$};
\path[->]
(0) edge (1)
(0) edge (3)
(3) edge (5)
(3) edge (6)
(5) edge (7)
(5) edge (8)
(8) edge (8l)
(8) edge (8r);
\stoptikzpicture
}
\stopplacefigure
The following lemma is a result of \cite[authoryears][DBLP:journals/tc/LeeY94].
\startlemma
A complete splitting tree with only valid splits exists if and only if there exists an adaptive distinguishing sequence.
\stoplemma
Our method uses the exact same algorithm as the one by Lee and Yannakakis.
However, we also apply it in the case when the splitting tree with valid splits is not complete (and hence no adaptive distinguishing sequence exists).
Their algorithm still produces a family of sets, but is not necessarily a separating family.
In order to recover separability, we refine that family.
Let $\Fam{Z'}$ be the result of Lee and Yannakakis' algorithm (to distinguish from their notation, we add a prime) and let $\Fam{H}$ be a separating family extracted from an ordinary splitting tree.
The hybrid ADS family is defined as $\Fam{Z'} ; \Fam{H}$, and can be computed as sketched in \in{Algorithm}[alg:hybrid] (the algorithm works on splitting trees instead of separating families).
By \in{Lemma}[lemma:refinement-props] we note the following: in the best case this family is an adaptive distinguishing sequence; in the worst case it is equal to $\Fam{H}$; and in general it is a combination of the two families.
In all cases, the result is a separating family because $\Fam{H}$ is.
\startplacealgorithm
[title={Obtaining the hybrid separating family $\Fam{Z'} ; \Fam{H}$},
reference=alg:hybrid]
\startalgorithmic
\REQUIRE{A Mealy machine $M$}
\ENSURE{A separating family $Z$}
\startlinenumbering
\STATE $T_1 \gets$ splitting tree for Moore's minimisation algorithm
\STATE $T_2 \gets$ splitting tree with valid splits (see \citenp[DBLP:journals/tc/LeeY94])
\STATE $\Fam{Z'} \gets$ (incomplete) family constructed from $T_2$
\FORALL{inequivalent states $s, t$ in the same leaf of $T_2$}{
\STATE $u \gets lca(T_1, s, t)$
\STATE $Z_s \gets Z'_s \cup \{ \sigma(u) \}$
\STATE $Z_t \gets Z'_t \cup \{ \sigma(u) \}$
}
\ENDFOR
\RETURN{$Z$}
\stoplinenumbering
\stopalgorithmic
\stopplacealgorithm
With the hybrid family we can define the test suite as follows.
Its $m$-completeness is proven in \in{Section}[sec:completeness].
\startdefinition
Let $P$ be a state cover, $\Fam{Z'}$ be a family of sets constructed with the Lee and Yannakakis algorithm, and $\Fam{H}$ be a separating family.
The \defn{hybrid ADS} test suite is
\startformula
T = P \cdot I^{\leq k+1} \odot (\Fam{Z'} ; \Fam{H}).
\stopformula
\stopdefinition
\startsubsection
[title={Example},
reference=sec:hads-example]
In the figure we see the (unique) result of Lee and Yannakakis' algorithm.
We note that the states $s_2, s_3, s_4$ are not split, so we need to refine the family for those states.
\startplacefigure
[title={(a): Largest splitting tree with only valid splits for \in{Figure}[fig:running-example].
(b): Its adaptive distinguishing tree in notation of \citet[DBLP:journals/tc/LeeY94].},
list={Splitting tree and adaptive distinguishing sequence.},
reference=fig:example-splitting-tree]
\startcombination[2*1]{
\hbox{
\starttikzpicture[node distance=2cm]
\splitnode{0}{s_0, s_1, s_2, s_3, s_4}{a}{}
\splitnode{2}{s_0, s_2, s_3, s_4}{aa}{below right of=0}
\node (1) [below left of=0 ] {$s_1$};
\node (3) [below left of=2 ] {$s_0$};
\node (4) [below right of=2] {$s_2, s_3, s_4$};
\path[->]
(0) edge (1)
(0) edge (2)
(2) edge (3)
(2) edge (4);
\stoptikzpicture
}} {(a)}
{\hbox{
\starttikzpicture[node distance=2cm]
\adsnode{0}{s_0, s_1, s_2, s_3, s_4}{s_0, s_1, s_2, s_3, s_4}{a}{}
\adsnode{1}{s_1}{s_2}{}{below left of=0}
\adsnode{2}{s_0, s_2, s_3, s_4}{s_1, s_2, s_4, s_3}{aa}{below right of=0}
\adsnode{3}{s_0}{s_2}{}{below left of=2}
\adsnode{4}{s_2, s_3, s_4}{s_2, s_3, s_4}{}{below right of=2}
\path[->]
(0) edge (1)
(0) edge (2)
(2) edge (3)
(2) edge (4);
\stoptikzpicture
}} {(b)}
\stopcombination
\stopplacefigure
\todo{Dit eerder noemen. En zeggen: we take the same $\Fam{H}$ as before.}
From the splitting tree in \in{Figure}[fig:example-splitting-tree], we obtain the following separating family $\Fam{H}$.
From the figure above we obtain the family $\Fam{Z'}$.
These families and the refinement $\Fam{Z'}; \Fam{H}$ are given below:
\startformula\startmathmatrix[n=3,align={left,left,left},distance=1cm]
\NC H_0 = \{aa,c\} \NC Z'_0 = \{aa\} \NC (Z';H)_0 = \{aa\} \NR
\NC H_1 = \{a\} \NC Z'_1 = \{a\} \NC (Z';H)_1 = \{a\} \NR
\NC H_2 = \{aa,ac,c\} \NC Z'_2 = \{aa\} \NC (Z';H)_2 = \{aa,ac,c\} \NR
\NC H_3 = \{a,c\} \NC Z'_3 = \{aa\} \NC (Z';H)_3 = \{aa,c\} \NR
\NC H_4 = \{aa,ac,c\} \NC Z'_4 = \{aa\} \NC (Z';H)_4 = \{aa,ac,c\} \NR
\stopmathmatrix\stopformula
With the separating family $\Fam{Z}'; \Fam{H}$ we obtain the following test suite of size 96:
\startformula\startmathmatrix[n=2,align={left,left}]
\NC T_{\text{h-ADS}} = \{ \NC aaaaa, aaaac, aaac, aabaa, aacaa, abaa, acaa, \NR
\NC \NC baaaa, baaac, baac, babaa, bacaa, bacc, bbaa, bbac, \NR
\NC \NC bbc, bcaa, caa \quad \} \NR
\stopmathmatrix\stopformula
We note that this is indeed smaller than the HSI test suite.
In particular, we have a smaller state identifier for $s_0$: $\{ aa \}$ instead of $\{ aa, c \}$.
As a consequence, there are less combinations of prefixes and suffixes.
We also observe that one of the state identifiers grew in length: $\{ aa, c \}$ instead of $\{ a, c \}$ for state $s_3$.
\stopsubsection
\startsubsection
[title={Implementation}]
All the algorithms concerning the hybrid ADS-method have been implemented and can be found at
\href{https://gitlab.science.ru.nl/moerman/hybrid-ads}.
We note that \in{Algorithm}[alg:hybrid] is implemented a bit more efficiently, as we can walk the splitting trees in a particular order.
For constructing the splitting trees in the first place, we use Moore's minimisation algorithm and the algorithms by \citet[DBLP:journals/tc/LeeY94].
We keep all relevant sets prefix-closed by maintaining a \emph{trie data structure}.
A trie data structure also allows us to immediately obtain the set of maximal tests only.
\stopsubsection
\startsubsection
[title={Randomisation}]
In many of the applications such as learning, no bound on the number of states of the SUT was known.
In such cases it is possible to randomly select test cases from an infinite test suite.
Unfortunately, we lose the theoretical guarantees of completeness with random generation.
Still, as we will see in \in{Chapter}[chap:applying-automata-learning], this can work really well.
We can randomly test cases as follows.
In the above definition for the hybrid ADS test suite we replace $I^{\leq k}$ by $I^{\ast}$ to obtain an infinite test suite.
Then we sample tests as follows:
\startitemize[n]
\item sample an element $p$ from $P$ uniformly,
\item sample a word $w$ from $I^{\ast}$ with a geometric distribution, and
\item sample uniformly from $(\Fam{Z'} ; \Fam{H})_s$ for the state $s = \delta(s_0, pw)$.
\stopitemize
\stopsubsection
\stopsection
\startsection
[title={Overview},
reference=sec:overview]
We give an overview of the aforementioned test methods.
We classify them in two directions,
\startitemize
\item whether they use harmonised state identifiers or not and
\item whether it used singleton state identifiers or not.
\stopitemize
\starttheorem
[reference=thm:completeness]
The following test suites are all $n+k$-complete:
\starttabulate[|c|c|c|]
\NC \NC Arbitrary \NC Harmonised
\NR \HL %----------------------------------------
\NC Many / pairwise \NC Wp \NC HSI
\NR
\NC % empty cell
\NC $ P \cdot I^{\leq k} \cdot \bigcup \Fam{W} \,\cup\, Q \cdot I^{\leq k} \odot \Fam{W} $
\NC $ (P \cup Q) \cdot I^{\leq k} \odot \Fam{H} $
\NR \HL %----------------------------------------
\NC Hybrid \NC \NC Hybrid ADS
\NR
\NC % empty cell
\NC % empty cell
\NC $ (P \cup Q) \cdot I^{\leq k} \odot (\Fam{Z'} ; \Fam{H}) $
\NR \HL %----------------------------------------
\NC Single / global \NC UIOv \NC ADS
\NR
\NC % empty cell
\NC $ P \cdot I^{\leq k} \cdot \bigcup \Fam{U} \,\cup\, Q \cdot I^{\leq k} \odot \Fam{U} $
\NC $ (P \cup Q) \cdot I^{\leq k} \odot \Fam{Z} $
\NR \HL %----------------------------------------
\stoptabulate
\stoptheorem
\todo{Iets zeggen over de hybrid UIO method?}
It should be noted that the ADS-method is a specific instance of the HSI-method and similarly the UIOv-method is an instance of the Wp-method.
What is generally meant by the Wp-method and HSI-method is the above formula together with a particular way to obtain the (harmonised) state identifiers.
We are often interested in the size of the test suite.
In the worst case, all methods generate a test suite with a size in $\bigO(pn^3)$ and this bound is tight \cite[Vasilevskii73].
Nevertheless we expect intuitively that the right column performs better, as we are using a more structured set (given a separating family for the HSI-method, we can always forget about the common prefixes and apply the Wp-method, which will never be smaller if constructed in this way).
Also we expect the bottom row to perform better as there is a single test for each state.
Small experimental results confirm this intuition
\cite[DBLP:journals/infsof/DorofeevaEMCY10].
On the example in \in{Figure}[fig:running-example], we computed all applicable test suites in \in{Sections}[sec:all-example] and \in{}[sec:hads-example].
The UIO and ADS methods are not applicable.
For the W, Wp, HSI and hybrid ADS methods we obtained test suites of size 169, 75, 125 and 96 respectively.
\stopsection
\startsection
[title={Proof of completeness},
reference=sec:completeness]
\todo{Stukje over bisimulaties?}
We fix a specification $M$ which has a minimal representative with $n$ states and an implementation $M'$ with at most $n+k$ states.
We assume that all states are reachable from the initial state in both machines (i.e., both are \defn{connected}).
We define the following notation:
\startdefinition
Let $W \subseteq I^{\ast}$ be a set of words.
Two states $x, y$ (of possibly different machines) are $W$-equivalent, written $x \sim_W y$, if $\lambda(x, w) = \lambda(y, w)$ for all $w \in W$.
\stopdefinition
We note that for a fixed set $W$ the relation $\sim_W$ is a equivalence relation and that $W \subseteq V$ gives $x \sim_V y \implies x \sim_W y$.
The next lemma gives sufficient conditions for a test suite of the given anatomy to be
complete.
We then prove that these conditions hold for the test suites in this paper.
\startproposition
[reference=prop:completeness]
Let $\Fam{W}$ and $\Fam{W'}$ be two families of words and $P$ a state cover for $M$.
Let $T = P \cdot I^{\leq k} \odot \Fam{W} \,\cup\, P \cdot I^{\leq k+1} \odot \Fam{W'}$ be a test suite.
If
\startitemize[n]
\item for all $x, y \in M:$ $x \sim_{W_x \cap W_y} y$ implies $x \sim y$,
\item for all $x, y \in M$ and $z \in M'$: $x \sim_{W_x} z$ and $z \sim_{W'_y} y$ implies $x \sim y$, and
\item the machines $M$ and $M'$ agree on $T$,
\stopitemize
then $M$ and $M'$ are equivalent.
\todo{Puntje 2 verdient meer aandacht?}
\stopproposition
\startproof
First, we prove that $P \cdot I^{\leq k}$ reaches all states in $M'$.
For $p, q \in P$ and $x = \delta(s_0, p), y = \delta(s_0, q)$ such that $x \not\sim_{W_x \cap W_y} y$, we also have $\delta'(m'_0, p) \not\sim_{W_x \cap W_y} \delta'(m'_0, q)$ in the implementation $M'$.
By (1) this means that there are at least $n$ different behaviours in $M'$, hence at least $n$ states.
Now $n$ states are reached by the previous argument (using the set $P$).
By assumption $M'$ has at most $k$ extra states and so we can reach all those extra states by using $I^{\leq k}$ after $P$.
Second, we show that the reached states are bisimilar.
Define the relation $R = \{(\delta(q_0, p), \delta'(q_0', p)) \mid p \in P \cdot I^{\leq k}\}$.
Note that for each $(s, i) \in R$ we have $s \sim_{W_s} i$.
For each state $i \in M'$ there is a state $s \in M$ such that $(s, i) \in R$, since we reach all states in both machines by $P \cdot I^{\leq k}$.
We will prove that this relation is in fact a bisimulation up-to equivalence.
For output, we note that $(s, i) \in R$ implies $\lambda(s, a) = \lambda'(i, a)$ for all $a$,
since the machines agree on $P \cdot I^{\leq k+1}$.
For the successors, let $(s, i) \in R$ and $a \in I$ and consider the successors $s_2 = \delta(s, a)$ and $i_2 = \delta'(i, a)$.
We know that there is some $t \in M$ with $(t, i_2) \in R$.
We also know that we tested $i_2$ with the set $W'_t$.
So we have:
\startformula s_2 \sim_{W'_{s_2}} i_2 \sim_{W_t} t. \stopformula
By the second assumption, we conclude that $s_2 \sim t$.
So $s_2 \sim t$ and $(t, i) \in R$, which means that $R$ is a bisimulation up-to equivalence.
Using the theory of up-to techniques \cite[DBLP:journals/cacm/BonchiP15] we know that there exists a bisimulation relation containing $R$.
\todo{Up to is best nieuw. Misschien meer aandacht aan besteden. Ook nog Jurriaan citeren?}
In particular $q_0$ and $q'_0$ are bisimilar.
And so the machines $M$ and $M'$ are equivalent.
\stopproof
Before we show that the conditions hold for the test methods described in this paper, we reflect on the above proof first.
This proof is very similar to the completeness proof by \citet[DBLP:journals/tse/Chow78].
(In fact, it is also similar to Lemma 4 by \citet[DBLP:journals/iandc/Angluin87] which proves termination in the L* learning algorithm. This correspondence was noted by \citet[DBLP:conf/fase/BergGJLRS05].)
In the first part we argue that all states are visited by using some sort of counting and reachability argument.
Then in the second part we show the actual equivalence.
To the best of the authors knowledge, this is first $m$-completeness proof which explicitly uses the concept of a bisimulation.
Using a bisimulation allows us to slightly generalise and use bisimulation up-to equivalence, dropping the the often-assumed requirement that $M$ is minimal.
\startlemma
Let $\Fam{W'}$ be a family of state identifiers for $M$.
Define the family $\Fam{W}$ by $W_s = \bigcup \Fam{W'}$.
Then the conditions (1) and (2) in the previous lemma are satisfied.
\stoplemma
\startproof
The first condition we note that $W_x \cap W_y = W_x = W_y$, and so $x \sim_{W_x \cap W_y} y$ implies $x \sim_{W_x} y$, now by definition of state identifier we get $x \sim y$.
For the second condition, let $x \sim_{\bigcup \Fam{W'}} z \sim_{W'_y} y$.
Then we note that $W'_y \subseteq \bigcup{W'}$ and so we get $x \sim_{W'_y} z \sim_{W'_y} y$.
By transitivity we get $x \sim_{W'_y} y$ and so by definition of state identifier we get $x \sim y$.
\stopproof
\startcorollary
The W, Wp, and UIOv test suites are $n+k$-complete.
\stopcorollary
\startlemma
Let $\Fam{H}$ be a separating family and take $\Fam{W} = \Fam{W'} = \Fam{H}$.
Then the conditions (1) and (2) in the previous lemma are satisfied.
\stoplemma
\startproof
Let $x \sim_{H_x \cap H_y} y$, then by definition of separating family $x \sim y$.
For the second condition, let $x \sim_{H_x} z \sim_{H_y} y$.
Then we get $x \sim_{H_x \cap H_y} z \sim_{H_x \cap H_y} y$ and so by transitivity $x \sim_{H_x \cap H_y} y$, hence again $x \sim y$.
\stopproof
\startcorollary
The HSI, ADS and hybrid ADS test suites are $n+k$-complete.
\stopcorollary
\stopsection
\startsection
[title={Related Work and Discussion}]
\todo{Veel gerelateerd werk kan naar de grote intro.}
In this chapter we have mostly considered classical test methods which are all based on prefixes and state identifiers.
There are more recent methods which almost fit in the same framework.
We mention the P \citep[SimaoP10], H \citep[DorofeevaEY05], and SPY \citep[SimaoPY09] methods.
The P method construct a test suite by carefully considering sufficient conditions for a $p$-complete test suite (here $p \leq n$, where $n$ is the number of states).
It does not generalise to extra states, but it seems to construct very small test suites.
The H method is a refinement of the HSI method where state identifiers for a testing transitions are reconsidered.
(Note that \in{Proposition}[prop:completeness] allows for a different family when testing transitions.)
Last, the SPY method builds upon the HSI-method and changes the prefixes in order to minimise the size of a test suite, exploiting overlap in test sequences.
We believe that this technique is independent of the HSI-method and can in fact be applied to all methods presented in this paper.
As such, the SPY method should be considered as an optimisation technique, orthogonal to the work in this paper.
Recently, \citet[DBLP:journals/cj/HieronsT15] devise a novel test method which is based on incomplete distinguishing sequences and is similar to the hybrid ADS method.
They use sequences which can be considered to be adaptive distinguishing sequences on a subset of the state space.
With several of those one can cover the whole state space, obtaining a $m$-complete test suite.
This is a bit dual to our approach, as our \quotation{incomplete} adaptive distinguishing sequences define a course partition of the complete state space.
Our method becomes complete by refining the tests with pairwise separating sequences.
Some work is put into minimising the adaptive distinguishing sequences themselves.
\citet[DBLP:journals/fmsd/TurkerY14] describe greedy algorithms which construct small adaptive distinguishing sequences.
Moreover, they show that finding the minimal adaptive distinguishing sequence is NP-complete in general, even approximation is NP-complete.
We expect that similar heuristics also exist for the other test methods and that it will improve the performance.
Note that minimal separating sequences do not guarantee a minimal test suite.
In fact, we see that the hybrid ADS method outperforms the HSI method on the example in \in{Figure}[fig:running-example] since it prefers longer, but fewer, sequences.
Some of the assumptions made at the start of this chapter have also been challenged.
For non-deterministic Mealy machine, we mention the work of \citet[PetrenkoY14].
For input/output transition systems with the \emph{ioco} relation we mention the work of \citet[BosJM17, SimaoP14].
In both cases, the test suites are still defined in the same way as in this chapter: prefixes followed by state identifiers.
However, for non-deterministic systems, guiding an implementation into a state is harder as the implementation may choose its own path.
For that reason, sequences are often replaced by automata, so that the testing can be adaptive.
This game theoretic point of view is further investigated by \citet[BosS18].
The test suites generally are of exponential size, depending on how non-deterministic the systems are.
The assumption that the implementation is resettable is also challenged early on.
If the machine has no reliable reset (or the reset is too expensive), one tests the system with a single \emph{checking sequence}.
\citet[DBLP:journals/tc/LeeY94] give a randomised algorithm for constructing such a checking sequence using adaptive distinguishing sequences.
There is a similarity with the randomised algorithm by \citet[DBLP:journals/iandc/RivestS93] for learning non-resettable automata.
Recently, \citet[GrozSPO16] give a deterministic learning algorithm for non-resettable machines based on adaptive distinguishing sequences.
Many of the methods described here are benchmarked on small or random Mealy machines by \citet[DBLP:journals/infsof/DorofeevaEMCY10, DBLP:journals/infsof/EndoS13].
The benchmarks are of limited scope, the machine from \in{Chapter}[chap:applying-automata-learning], for instance, is neither small nor random.
For this reason, we started to collect more realistic benchmarks at \href{http://automata.cs.ru.nl/}.
\stopsection
\referencesifcomponent
\stopchapter
\stopcomponent
View git blame Copy permalink