phd-thesis/content/test-methods.tex

\project thesis
\startcomponent test-methods

\startchapter
  [title={FSM-based Test Methods},
   reference=chap:test-methods]

In this chapter, we will discuss some of the theory of test generation methods for black box conformance checking.
Since the systems we consider are black box, we cannot simply determine equivalence with a specification.
The only way to gain confidence is to perform experiments on the system.
A key aspect of test generation methods is the size and completeness of the test suites.
On one hand, we want to cover as much as the specification as possible, hopefully ensuring that we find mistakes in any faulty implementation.
On the other hand: testing takes time, so we want to minimise the size of a test suite.

The test methods described here are well-known in the literature of FSM-based testing.
They all share similar concepts, such as \emph{access sequences} and \emph{state identifiers}.
In this chapter we will define these concepts, relate them with one another and show how to build test suites from these concepts.
This theoretically discussion is new and enables us to compare the different methods uniformly.
For instance, we can prove all these methods to be $n$-complete with a single proof.

The discussion also inspired a new algorithm: the \emph{hybrid ADS methods}.
This method is applied to an industrial case study in \in{Chapter}[chap:applying-automata-learning].
It combines the strength of the ADS method (which is not always applicable) with the generality of the HSI method.

This chapter starts with the basics: Mealy machines, sequences and what it means to test a black box system.
Then, starting from \in{Section}[sec:separating-sequences] we define several concepts, such as state identifiers, in order to distinguish one state from another.
These concepts are then combined in \in{Section}[sec:methods] to derive test suites.
In a similar vein, we define a novel test method in \in{Section}[sec:hybrid] and we discuss some of the implementation details of the \kw{hybrid-ads} tool.
We summarise the various test methods in \in{Section}[sec:overview].
All methods are proven to be $n$-complete in \in{Section}[sec:completeness].
Finally, in \in{Section}[sec:discussion], we discuss related work.


\startsection
  [title={Mealy machines and sequences}]

We will focus on Mealy machines, as those capture many protocol specifications and reactive systems.

We fix finite alphabets $I$ and $O$ of inputs respectively outputs.
We use the usual notation for operations on \emph{sequences} (also called \emph{words}):
$uv$ for the concatenation of two sequences $u, v \in I^{\ast}$ and $|u|$ for the length of $u$.
For a sequence $w = uv$ we say that $u$ and $v$ are a prefix and suffix respectively.

\startdefinition
A (deterministic and complete) \emph{Mealy machine} $M$ consists of a finite set of \emph{states} $S$, an \emph{initial state} $s_0 \in S$ and two functions:
\startitemize
\item a \emph{transition function} $\delta \colon S \times I \to S$, and
\item an \emph{output function} $\lambda \colon S \times I \to O$.
\stopitemize

Both the transition function and output function are extended inductively to sequences as $\delta \colon S \times I^{\ast} \to S$ and $\lambda \colon S \times I^{\ast} \to O^{\ast}$:
\startformula\startmathmatrix[n=4, align={right,left,right,left},distance=1pt]
\NC \delta(s, \epsilon) \NC = s \NC\quad \lambda(s, \epsilon) \NC = \epsilon \NR
\NC \delta(s, aw) \NC = \delta(\delta(s, a), w) \NC\qquad \lambda(s, aw) \NC = \lambda(s, a)\lambda(\delta(s, a), w) \NR
\stopmathmatrix\stopformula

The \emph{behaviour} of a state $s$ is given by the output function $\lambda(s, -) \colon I^{\ast} \to O^{\ast}$.
Two states $s$ and $t$ are \emph{equivalent} if they have equal behaviours, written $s \sim t$, and two Mealy machines are equivalent if their initial states are equivalent.
\stopdefinition

\startremark
We will use the following conventions and notation.
We often write $s \in M$ instead of $s \in S$ and for a second Mealy machine $M'$ its constituents are denoted $S', s'_0, \delta'$ and $\lambda'$.
Moreover, if we have a state $s \in M$, we silently assume that $s$ is not a member of any other Mealy machine $M'$.
(In other words, the behaviour of $s$ is determined by the state itself.)
This eases the notation since we can write $s \sim t$ without needing to introduce a context.
\stopremark

An example Mealy machine is given in \in{Figure}[fig:running-example].

\startplacefigure
  [title={An example specification with input $I=\{a,b,c\}$ and output $O=\{0,1\}$.},
   list={An example specification.},
   reference=fig:running-example]
\hbox{
\starttikzpicture[node distance=0.9cm and 3cm,bend angle=20]
	\node[state] (0) {$s_0$};
	\node[state] (1) [above left = of 0] {$s_1$};
	\node[state] (2) [below left = of 0] {$s_2$};
	\node[state] (3) [above right = of 0] {$s_3$};
	\node[state] (4) [below right = of 0] {$s_4$};
	\node        (5) [above = of 0] {};
	\path[trans]
	(5) edge (0)
	(0) edge [bend left=]    node [below] {${a}/1$} (1)
	(0) edge [bend right]    node [below] {${b}/0$} (4)
	(0) edge [loop below]    node [below] {${c}/0$} (0)
	(1) edge                 node [left ] {${a}/0$} (2)
	(1) edge [bend left]     node [above] {${b}/0$, ${c}/0$} (0)
	(2) edge [bend right]    node [below] {${b}/0$, ${c}/0$} (0)
	(2) edge [loop left]     node [left ] {${a}/1$} (1)
	(3) edge [bend left=30]  node [right] {${a}/1$} (4)
	(3) edge [bend right]    node [above] {${b}/0$} (0)
	(3) edge [loop right]    node [right] {${c}/1$} (3)
	(4) edge [bend left]     node [right] {${a}/1$} (3)
	(4) edge [loop right]    node [right] {${b}/0$} (4)
	(4) edge [bend right]    node [above] {${c}/0$} (0);
\stoptikzpicture
}
\stopplacefigure


\startsubsection
  [title={Testing}]

In conformance testing we have a specification modelled as a Mealy machine and an implementation (the system under test, or SUT) which we assume to behave as a Mealy machine.
Tests, or experiments, are generated from the specification and applied to the implementation. We assume that we can reset the implementation before every test.
If the output is different from the specified output, then we know the implementation is flawed.
The goals is to test as little as possible, while covering as much as possible.

A test suite is nothing more than a set of sequences.
We do not have to encode outputs in the test suite, as those follow from the deterministic specification.

\startdefinition[reference=test-suite]
A \emph{test suite} is a finite subset $T \subseteq I^{\ast}$.
\stopdefinition

A test $t \in T$ is called \emph{maximal} if it is not a proper prefix of another test $s \in T$.
We denote the set of maximal tests of $T$ by $\max(T)$.
The maximal tests are the only tests in $T$ we actually have to apply to our SUT as we can record the intermediate outputs.
In the examples of this chapter we will show $\max(T)$ instead of $T$.

We define the size of a test suite as usual \citep[DorofeevaEMCY10, Petrenko0GHO14].
The size of a test suite is measured as the sum of the lengths of all its maximal tests plus one reset per test.

\startdefinition[reference=test-suite-size]
The \emph{size} of a test suite $T$ is defined to be
$||T|| = \sum\limits_{t \in max(T)} (|t| + 1)$.
\stopdefinition


\stopsubsection
\startsubsection
  [title={Completeness of test suites}]
\startexample[reference=incompleteness]
{\bf No test suite is complete.}
Consider the specification in \in{Figure}{a}[fig:incompleteness-example].
This machine will always outputs a cup of coffee -- when given money.
For any test suite we can make a faulty implementation which passes the test suite.
A faulty implementation might look like \in{Figure}{b}[fig:incompleteness-example], where the machine starts to output beers after $n$ steps (signalling that it's the end of the day), where $n$ is larger than the length of the longest sequence in the suite.
This shows that no test-suite can be complete and it justifies the following definition.

\startplacefigure
  [title={A basic example showing that finite test suites are incomplete. The implementation on the right will pass any test suite if we choose $n$ big enough.},
   list={A basic example showing that finite test suites are incomplete.},
   reference=fig:incompleteness-example]
\startcombination[2*1]
{\hbox{
	\starttikzpicture
	\node[state] (0) {$s_0$};
	\path[trans]
	(0) edge [loop] node [below] {\money $/$ \coffee} (0);
	\stoptikzpicture
}} {(a)}
{\hbox{
	\starttikzpicture[bend angle=20]
	\node[state] (0) {$s'_0$};
	\node[state] (1) [right of=0] {$s'_1$};
	\node        (2) [right of=1] {$\cdots$};
	\node[state] (3) [right of=2] {$s'_n$};
	\path[trans]
	(0) edge [bend left] node [below] {\money $/$ \coffee} (1)
	(1) edge [bend left] node [below] {\money $/$ \coffee} (2)
	(2) edge [bend left] node [below] {\money $/$ \coffee} (3)
	(3) edge [loop]      node [below] {\money $/$ \beer}   (3);
	\stoptikzpicture
}} {(b)}
\stopcombination
\stopplacefigure
\stopexample


\startdefinition[reference=completeness]
Let $M$ be a Mealy machine and $T$ be a test suite.
We say that $T$ is \emph{$m$-complete (for $M$)} if for all inequivalent machines $M'$ with at most $m$ states there exists a $t \in T$ such that $\lambda(s_0, t) \neq \lambda'(s'_0, t)$.
\stopdefinition

We are often interested in the case of $m$-completeness, where $m = n + k$ for some $k \in \naturalnumbers$ and $n$ is the number of states in the specification.
Here $k$ will stand for the number of \emph{extra states} we can test.

Note the order of the quantifiers in the above definition.
We ask for a single test suite which works for all implementations of bounded size.
This is crucial for black box testing, as we do not know the implementation, so the test suite has to work for all of them.


\stopsubsection
\startsubsection
  [title={Separating Sequences},
   reference=sec:separating-sequences]

Before we construct test suites, we discuss several types of useful sequences.
All the following notions are standard in the literature, and the corresponding references will be given in \in{Section}[sec:methods], where we discuss the test generation methods using these notions.
We fix a Mealy machine $M$ for the remainder of this chapter.

\startdefinition
We define the following kinds of sequences.
\startitemize
\item Given two states $s, t$ in $M$ we say that $w$ is a \defn{separating sequence} if $\lambda(s, w) \neq \lambda(t, w)$.
\item For a single state $s$ in $M$, a sequence $w$ is a \defn{unique input output sequence (UIO)} if for every inequivalent state $t$ in $M$ we have $\lambda(s, w) \neq \lambda(t, w)$.
\item Finally, a \defn{(preset) distinguishing sequence (DS)} is a single sequence $w$ which separates all states of $M$, i.e., for every pair of inequivalent states $s, t$ in $M$ we have $\lambda(s, w) \neq \lambda(t, w)$.
\stopitemize
\stopdefinition

The above list is ordered from weaker to stronger notions, i.e., every distinguishing sequence is an UIO sequence for every state.
Similarly, an UIO for a state $s$ is a separating sequence for $s$ and any inequivalent $t$.
Separating sequences always exist for inequivalent states and finding them efficiently is the topic of \in{Chapter}[chap:minimal-separating-sequences].
On the other hand, UIOs and DSs do not always exist for a machine.

A machine $M$ is \emph{minimal} if every distinct pair of states is inequivalent (i.e.,\break $s \sim t \implies s = t$).
We will not require $M$ te be minimal, although this is often done in literature.
Minimality is sometimes convenient, as one can write \quote{every other state $t$} instead of \quote{every inequivalent state $t$}.

\startexample
For the machine in \in{Figure}[fig:running-example], we note that state $s_0$ and $s_2$ are separated by the sequence $aa$ (but not by any shorter sequence).
In fact, the sequence $aa$ is an UIO for state $s_0$ since it is the only state outputting $10$ on that input.
However, state $s_2$ has no UIO:
If the sequence were to start with $b$ or $c$, state $s_3$ and $s_4$ respectively have equal transition, which makes it impossible to separate those states after the first symbol.
If it starts with an $a$, states $s_3$ and $s_4$ are swapped and we make no progress in distinguishing these states from $s_2$.
Since $s_2$ has no UIO, the machine as a whole does not admit a DS.

In this example, all other states actually have UIOs.
For the states $s_0, s_1, s_3$ and $s_4$, we can pick the sequences $aa, a, c$ and $ac$ respectively.
In order to separate $s_2$ from the other state, we have to pick multiple sequences.
For instance, the set $\{aa, ac, c\}$ will separate $s_2$ from all other states.
\stopexample


\stopsubsection
\startsubsection
  [title={Sets of separating sequences}]

As the example shows, we need sets of sequences and sometimes even sets of sets of sequences -- called \emph{families}.
\footnote{A family is often written as $\{X_s\}_{s \in M}$ or simply $\{X_s\}_{s}$, meaning that for each state $s \in M$ we have a set $X_s$.}

\startdefinition
We define the following kinds of sets of sequences.
We require that all sets are \emph{prefix-closed}, however, we only show the maximal sequences in examples.
\footnote{Taking these sets to be prefix-closed makes many proofs easier.}
\startitemize
\item A set of sequences $W$ is a called a \defn{characterisation set} if it contains a separating sequence for each pair of inequivalent states in $M$.
\item A \defn{state identifier} for a state $s \in M$ is a set $W_s$ such that for every inequivalent $t \in M$ a separating sequence for $s$ and $t$ exists in $W_s$.
\item A set of state identifiers $\{ W_s \}_{s}$ is \defn{harmonised} if $W_s \cap W_t$ contains a separating sequence for inequivalent states $s$ and $t$.
This is also called a \defn{separating family}.
\stopitemize
\stopdefinition

A state identifier $W_s$ will be used to test against a single state.
In contrast to a characterisation set, it only include sequences which are relevant for $s$.
The property of being harmonised might seem a bit strange.
This property ensures that the same tests are used for different states.
This extra consistency within a test suite is necessary for some test methods.
We return to this notion in more detail in \in{Example}[ex:uio-counterexample].

We may obtain a characterisation set by simply considering every pair of states and look for a difference.
However, it turns out a harmonised set of state identifiers exists for every machine and this can be constructed very efficiently (\in{Chapter}[chap:minimal-separating-sequences]).
From a set of state identifiers we may obtain a characterisation set by taking the union of all those sets.

\startexample
As mentioned before, state $s_2$ from \in{Figure}[fig:running-example] has a state identifier $\{aa, ac, b\}$.
In fact, this set is a characterisation set for the whole machine.
Since the other states have UIOs, we can pick singleton sets as state identifiers.
For example, state $s_0$ has the UIO $aa$, so a state identifier for $s_0$ is $W_0 = \{ aa \}$.
Similarly, we can take $W_1 = \{ a \}$ and $W_3 = \{ c \}$.
But note that such a family will \emph{not} be harmonised since the sets $\{ a \}$ and $\{ c \}$ have no common separating sequence.
\stopexample

One more type of state identifier is of our interest: the \emph{adaptive distinguishing sequence}.
It it the strongest type of state identifier, and as a result not many machines have one.
Like DSs, adaptive distinguishing sequences can identify a state using a single word.
We give a slightly different (but equivalent) definition than the one of \citet[LeeY94].

\startdefinition
A separating family $\Fam{H}$ is an \defn{adaptive distinguishing sequence} (ADS) if each set $\max(H_s)$ is a singleton.
\stopdefinition

It is called an adaptive sequence, since it has a tree structure which depends on the output of the machine.
To see this tree structure, consider the first symbols of each of the sequences in the family.
Since the family is harmonised and each set is essentially given by a single word, there is only one first symbol.
Depending on the output after the first symbol, the sequence continues.

\startexample
In \in{Figure}[fig:ads-example] we see a machine with an ADS.
The ADS is given as follows:
\startformulas
\startformula H_0 = \{aba\}  \stopformula
\startformula H_1 = \{aaba\} \stopformula
\startformula H_2 = \{aba\}  \stopformula
\startformula H_3 = \{aaba\} \stopformula
\stopformulas
Note that all sequences start with $a$.
This already separates $s_0, s_2$ from $s_1, s_3$.
To further separate the states, the sequences continues with either a $b$ or another $a$.
And so on.

\startplacefigure
  [title={(a): A Mealy machine with an ADS and (b): the tree structure of this ADS.},
   list={A Mealy machine and its ADS.},
   reference=fig:ads-example]
\startcombination[nx=2, location=middle, distance=.5cm]
{\hbox{
\starttikzpicture
	\node[initial, state] (0)     {$s_0$};
	\node[state] (1) [right=of 0] {$s_1$};
	\node[state] (2) [below=of 1] {$s_2$};
	\node[state] (3) [left =of 2] {$s_3$};
	\path[trans]
	(0) edge node [above] {$a/0, b/0$} (1)
	(1) edge node [right] {$a/1$}      (2)
	(2) edge node [above] {$a/0, b/0$} (3)
	(3) edge node [left, align=left] {$a/1,$ \\ $b/0$} (0)
	(1) edge [loop right] node {$b/0$} (1);
\stoptikzpicture}} {(a)}
{\hbox{
\starttikzpicture[node distance=.75cm]
	\node (0) {$a$};
	\node [below left=of 0]  (1) {$b$};
	\node [below=of 1]       (3) {$a$};
	\node [below left=of 3]  (5) {$s_2$};
	\node [below right=of 3] (6) {$s_0$};
	\node [below right=of 0] (2) {$a$};
	\node [below=of 2]       (4) {$b$};
	\node [below=of 4]       (7) {$a$};
	\node [below left=of 7]  (8) {$s_1$};
	\node [below right=of 7] (9) {$s_3$};
	
	\path[->]
	(0)+(0,.75) edge (0)
	(0) edge node [above left]  {$0$} (1)
	(0) edge node [above right] {$1$} (2)
	(1) edge node [left]        {$0$} (3)
	(3) edge node [above left]  {$0$} (5)
	(3) edge node [above right] {$1$} (6)
	(2) edge node [right]       {$0$} (4)
	(4) edge node [right]       {$0$} (7)
	(7) edge node [above left]  {$0$} (8)
	(7) edge node [above right] {$1$} (9);
\stoptikzpicture}} {(b)}
\stopcombination
\stopplacefigure
\stopexample

Given an ADS, there exists an UIO for every state.
The converse -- if every state has an UIO, then the machine admits an ADS -- does not hold.
The machine in \in{Figure}[fig:running-example] admits no ADS, since $s_2$ has no UIO.


\stopsubsection
\startsubsection
  [title={Partial equivalence}]

\startdefinition
We define the following notation.
\startitemize
\item
Let $W$ be a set of sequences.
Two states $x, y$ are \emph{$W$-equivalent}, written $x \sim_W y$, if $\lambda(x, w) = \lambda(y, w)$ for all $w \in W$.
\item
Let $\Fam{W}$ be a family.
Two states $x, y$ are \emph{$\Fam{W}$-equivalent}, written $x \sim_{\Fam{W}} y$, if $\lambda(x, w) = \lambda(y, w)$ for all $w \in W_x \cap W_y$.
\stopitemize
\stopdefinition

The relation $\sim_W$ is an equivalence relation and $W \subseteq V$ implies that $V$ separates more states than $W$, i.e., $x \sim_V y \implies x \sim_W y$.
Clearly, if two states are equivalent (i.e., $s \sim t$), then for any set $W$ we have $s \sim_W t$.

\startlemma[reference=lem:relation-characterisation]
The relations $\sim_W$ and $\sim_{\Fam{W}}$ can be used to \emph{define} characterisation sets and separating families.
Concretely:
\startitemize
\item
$W$ is a characterisation set if and only if for all $s, t$ in $M$, $s \sim_W t$ implies $s \sim t$.
\item
$\Fam{W}$ is a separating family if and only if for all $s, t$ in $M$, $s \sim_{\Fam{W}} t$ implies $s \sim t$.
\stopitemize
\stoplemma
\startproofnoqed
\startitemize
\item
$W$ is a characterisation set by definition means $s \not\sim t \implies s \not\sim_W t$ as $W$ contains a separating sequence (if it exists at all).
This is equivalent to $s \sim_W t \implies s \sim t$.
\item
Let $\Fam{W}$ be a separating family and $s \not\sim t$.
Then there is a sequence $w \in W_s \cap W_t$ such that $\lambda(s, w) \neq \lambda(t, w)$, i.e., $s \not\sim_{\Fam{W}} t$.
We have shown $s \not\sim t \implies s \not\sim_{\Fam{W}} t$, which is equivalent to $s \sim_{\Fam{W}} t \implies s \sim t$.
The converse is proven similarly.
\QED
\stopitemize
\stopproofnoqed


\stopsubsection
\startsubsection
  [title={Access sequences}]

Besides sequences which separate states, we also need sequences which brings a machine to specified states.

\startdefinition
An \emph{access sequence for $s$} is a word $w$ such that $\delta(s_0, w) = s$.
A set $P$ consisting of an access sequence for each state is called a \emph{state cover}.
If $P$ is a state cover, then the set $\{ p a \mid p \in P, a \in I \}$ is called a \emph{transition cover}.
\stopdefinition


\stopsubsection
\startsubsection
  [title={Constructions on sets of sequences}]

In order to define a test suite modularly, we introduce notation for combining sets of words.
For sets of words $X$ and $Y$, we define
\startitemize[after]
\item their concatenation $X \cdot Y = \{ xy \mid x \in X, y \in Y \}$,
\item iterated concatenation $X^0 = \{ \epsilon \}$ and $X^{n+1} = X \cdot X^{n}$, and
\item bounded concatenation $X^{\leq n} = \bigcup_{i \leq n} X^i$.
\stopitemize

On families we define
\startitemize[after]
\item flattening: $\bigcup \Fam{X} = \{ x \mid x \in X_s, s \in S \}$,
\item union: $\Fam{X} \cup \Fam{Y}$ is defined point-wise:
$(\Fam{X} \cup \Fam{Y})_s = X_s \cup Y_s$,
\item concatenation\footnote{We will often see the combination $P \cdot I \odot \Fam{X}$, this should be read as $(P \cdot I) \odot \Fam{X}$.}:
$X \odot \Fam{Y} = \{ xy \mid x \in X, y \in Y_{\delta(s_0, x)} \}$, and
\item refinement: $\Fam{X} ; \Fam{Y}$ defined by
\footnote{We use the convention that $\cap$ binds stronger than $\cup$.
In fact, all the operators here bind stronger than $\cup$.}
\startformula
(\Fam{X} ; \Fam{Y})_s
  = X_s \,\cup\, Y_s \!\cap\! \bigcup_{\startsubstack s \sim_{\Fam{X}} t \NR s \not\sim_{\Fam{Y}} t \stopsubstack} Y_t.
\stopformula
\stopitemize

The latter construction is new and will be used to define a hybrid test generation method in \in{Section}[sec:hybrid].
It refines a family $\Fam{X}$, which need not be separating, by including sequences from a second family $\Fam{Y}$.
It only adds those sequences to states if $\Fam{X}$ does not distinguish those states.
This is also the reason behind the $;$-notation: first the tests from $\Fam{X}$ are used to distinguish states, \emph{and then} for the remaining states $\Fam{Y}$ is used.

\startlemma[reference=lemma:refinement-props]
For all families $\Fam{X}$ and $\Fam{Y}$:
\startitemize
\item $\Fam{X} ; \Fam{X} = \Fam{X}$,
\item $\Fam{X} ; \Fam{Y} = \Fam{X}$, whenever $\Fam{X}$ is a separating family, and
\item $\Fam{X} ; \Fam{Y}$ is a separating family whenever $\Fam{Y}$ is a separating family.
\stopitemize
\stoplemma
\startproof
For the first item, note that there are no states $t$ such that $s \sim_{\Fam{X}} t$ and $s \not\sim_{\Fam{X}} t$.
Consequently, the union is empty, and the expression simplifies to
\startformula (\Fam{X} ; \Fam{X})_s = X_s \cup (X_s \cap \emptyset) = X_s . \stopformula

If $\Fam{X}$ is a separating family, then the only $t$ for which $s \sim_{\Fam{X}} t$ hold are $t$ such that $s \sim t$ (\in{Lemma}[lem:relation-characterisation]).
But $s \sim t$ is ruled out by $s \not\sim_{\Fam{Y}} t$, and again so
\startformula (\Fam{X} ; \Fam{Y})_s = X_s \cup (Y_s \cap \emptyset) = X_s . \stopformula

For the last item, suppose that $s \sim_{\Fam{X} ; \Fam{Y}} t$.
Then $s$ and $t$ agree on every sequence in $(\Fam{X} ; \Fam{Y})_s \cap (\Fam{X} ; \Fam{Y})_t$.
We distinguish two cases:
\startitemize
\item
Suppose $s \sim_{\Fam{X}} t$, then $Y_s \cap Y_t \subseteq (\Fam{X} ; \Fam{Y})_s \cap (\Fam{X} ; \Fam{Y})_t$.
And so $s$ and $t$ agree on $Y_s \cap Y_t$, meaning $s \sim_{\Fam{Y}} t$.
Since $\Fam{Y}$ is a separating family, we have $s \sim t$.
\item
Suppose $s \not\sim_{\Fam{X}} t$.
This contradicts $s \sim_{\Fam{X} ; \Fam{Y}} t$, since $X_s \cap X_t \subseteq (\Fam{X} ; \Fam{Y})_s \cap (\Fam{X} ; \Fam{Y})_t$.
\stopitemize
We conclude that $s \sim t$.
This proves that $\Fam{X}; \Fam{Y}$ is a separating family.
\stopproof


\stopsubsection
\stopsection
\startsection
  [title={Test generation methods},
   reference=sec:methods]

In this section, we review the classical conformance testing methods:
the W, Wp, UIO, UIOv, HSI, ADS methods.
At the end of this section, we construct the test suites for the running example.
Our hybrid ADS method uses a similar construction.

There are many more test generation methods.
Literature shows, however, that not all of them are complete.
For example, the method by \citet[Bernhard94] is falsified by \citet[Petrenko97], and the UIO-method from \citet[SabnaniD88] is shown to be incomplete by \citet[ChanVO89].
For that reason, completeness of the correct methods is shown in \in{Theorem}[thm:completeness].
The proof is general enough to capture all the methods at once.
We fix a state cover $P$ throughout this section and take the transition cover $Q = P \cdot I$.


\startsubsection
  [title={W-method \cite[Chow78, Vasilevskii73]},
   reference=sec:w]

After the work of \citet[Moore56], it was unclear whether a test suite of polynomial size could exist.
He presented a finite test suite which was complete, however it was exponential in size.
Both \citet[Chow78, Vasilevskii73] independently prove that test suites of polynomial size exist.
\footnote{More precisely: the size of $T_{\text{W}}$ is polynomial in the size of the specification for each fixed $k$.}
The W-method is a very structured test suite construction.
It is called the W-method as the characterisation set is often called $W$.

\startdefinition
  [reference=w-method]
Given a characterisation set $W$, we define the \defn{W test suite} as
\startformula
T_{\text{W}} = (P \cup Q) \cdot I^{\leq k} \cdot W .
\stopformula
\stopdefinition

This -- and all following methods -- tests the machine in two phases.
For simplicity, we explain these phases when $k = 0$.
The first phase consists of the tests $P \cdot W$ and tests whether all states of the specification are (roughly) present in the implementation.
The second phase is $Q \cdot W$ and tests whether the successor states are correct.
Together, these two phases put enough constraints on the implementation to know that the implementation and specification coincide (provided that the implementation has no more states than the specification).


\stopsubsection
\startsubsection
  [title={The Wp-method \cite[FujiwaraBKAG91]},
   reference=sec:wp]

\citet[FujiwaraBKAG91] realised that one needs fewer tests in the second phase of the W-method.
Since we already know the right states are present after phase one, we only need to check if the state after a transition is consistent with the expected state.
This justifies the use of state identifiers for each state.

\startdefinition
  [reference={state-identifier,wp-method}]
Let $\Fam{W}$ be a family of state identifiers.
The \defn{Wp test suite} is defined as
\startformula
T_{\text{Wp}} = P \cdot I^{\leq k} \cdot \bigcup \Fam{W} \,\cup\, Q \cdot I^{\leq k}
\odot \Fam{W}.
\stopformula
\stopdefinition

Note that $\bigcup \Fam{W}$ is a characterisation set as defined for the W-method.
It is needed for completeness to test states with the whole set $\bigcup \Fam{W}$.
Once states are tested as such, we can use the smaller sets $W_s$ for testing transitions.


\stopsubsection
\startsubsection
  [title={The HSI-method \cite[LuoPB95, PetrenkoYLD93]},
   reference=sec:hsi]

The Wp-method in turn was refined by \citet[LuoPB95, PetrenkoYLD93].
They make use of harmonised state identifiers, allowing to take state identifiers in the initial phase of the test suite.

\startdefinition
  [reference=hsi-method]
Let $\Fam{H}$ be a separating family.
We define the \defn{HSI test suite} by
\startformula
T_{\text{HSI}} = (P \cup Q) \cdot I^{\leq k} \odot \Fam{H}.
\stopformula
\stopdefinition

Our hybrid ADS method is an instance of the HSI-method as we define it here.
However, \citet[LuoPB95, PetrenkoYLD93] describe the HSI-method together with a specific way of generating the separating families.
Namely, the set obtained by a splitting tree with shortest witnesses.
The hybrid ADS method does not refine the HSI-method defined in the more restricted sense.


\stopsubsection
\startsubsection
  [title={The ADS-method \cite[LeeY94]},
   reference=sec:ads]

As discussed before, when a Mealy machine admits a adaptive distinguishing sequence, only a single test has to be performed for identifying a state.
This is exploited in the ADS-method.

\startdefinition
  [reference=ds-method]
Let $\Fam{Z}$ be an adaptive distinguishing sequence.
The \defn{ADS test suite} is defined as
\startformula
T_{\text{ADS}} = (P \cup Q) \cdot I^{\leq k} \odot \Fam{Z}.
\stopformula
\stopdefinition


\stopsubsection
\startsubsection
  [title={The UIOv-method \cite[ChanVO89]},
   reference=sec:uiov]

Some Mealy machines which do not admit an adaptive distinguishing sequence,
may still admit state identifiers which are singletons.
These are exactly UIO sequences and gives rise to the UIOv-method.
In a way this is a generalisation of the ADS-method, since the requirement that state identifiers are harmonised is dropped.

\startdefinition
  [reference={uio, uiov-method}]
Let $\Fam{U} = \{ \text{a single UIO for } s \}_{s \in S}$ be a family of UIO sequences, the \defn{UIOv test suite} is defined as
\startformula
T_{\text{UIOv}} = P \cdot I^{\leq k} \cdot \bigcup \Fam{U} \,\cup\, Q \cdot I^{\leq k} \odot \Fam{U}.
\stopformula
\stopdefinition

One might think that using a single UIO sequence instead of the set $\bigcup \Fam{U}$ to verify the state is enough.
In fact, this idea was used for the \emph{UIO-method} which defines the test suite $(P \cup Q) \cdot I^{\leq k} \odot \Fam{U}$.
The following is a counterexample, due to \citet[ChanVO89], to such conjecture.

\startexample
  [reference=ex:uio-counterexample]
The Mealy machines in \in{Figure}[fig:uio-counterexample] shows that UIO-method does not define a 3-complete test suite.
Take for example the UIOs $u_0 = aa, u_1 = a, u_2 = ba$ for the states $s_0, s_1, s_2$ respectively.
The test suite then becomes $\{ aaaa, abba, baaa, bba \}$ and the faulty implementation passes this suite.
This happens because the sequence $u_2$ is not an UIO in the implementation, and the state $s'_2$ simulates both UIOs $u_1$ and $u_2$.
Hence we also want to check that a state does not behave as one of the other states, and therefore we use $\bigcup \Fam{U}$.
With the same UIOs as above, the resulting UIOv test suite for the specification in \in{Figure}[fig:uio-counterexample] is $\{aaaa, aba, abba, baaa, bba\}$ of size $23$.
(Recall that we also count resets when measuring the size.)

\startplacefigure
  [title={An example where the UIO-method is not complete.},
   reference=fig:uio-counterexample]
\startcombination[nx=2, distance=1cm]
{\hbox{\starttikzpicture[node distance=2.5cm, bend angle=18]
  \node[state,initial,initial text=] (0) {$s_0$};
  \node[state] (1) [above right of=0] {$s_1$};
  \node[state] (2) [below right of=1] {$s_2$};
  \path[trans]
  (0) edge [bend left ] node [left ] {${a}/0$} (1)
  (0) edge [bend right] node [below] {${b}/1$} (2)
  (1) edge [bend left ] node [left ] {${a}/1$} (0)
  (1) edge [bend right] node [right] {${b}/1$} (2)
  (2) edge [bend right] node [below] {${a}/0$} (0)
  (2) edge [bend right] node [right] {${b}/1$} (1);
\stoptikzpicture}} {Specification}
{\hbox{\starttikzpicture[node distance=2.5cm, bend angle=18]
  \node[state,initial,initial text=] (0) {$s'_0$};
  \node[state] (1) [above right of=0] {$s'_1$};
  \node[state] (2) [below right of=1] {$s'_2$};
  \path[trans]
  (0) edge [bend left ] node [left ] {${a}/0$} (1)
  (0) edge [bend right] node [below] {${b}/1$} (2)
  (1) edge [bend left ] node [left ] {${a}/1$} (0)
  (1) edge [loop      ] node [below] {${b}/1$} (1)
  (2) edge [bend right] node [below] {${a}/0$} (0)
  (2) edge [bend right] node [right] {${b}/1$} (1);
\stoptikzpicture}} {Implementation}
\stopcombination
\stopplacefigure
\stopexample


\stopsubsection
\startsubsection
  [title={All test suites for \in{Figure}[fig:running-example]},
   reference=sec:all-example]

Let us compute all the previous test suites on the specification in \in{Figure}[fig:running-example].
We will be testing without extra states, i.e., we construct 5-complete test suites.
We start by defining the state and transition cover.
For this, we take all shortest sequences from the initial state to the other states.
This state cover is depicted in \in{Figure}[fig:running-example-prefixes].
The transition cover is simply constructed by extending each access sequence with another symbol.

\startformula\startalign
\NC P \NC = \{ \epsilon, a, aa, b, ba \} \NR
\NC Q = P \cdot I \NC = \{ a, b, c, aa, ab, ac, aaa, aab, aac, ba, bb, bc, baa, bab, bac \} \NR
\stopalign\stopformula

\startplacefigure
  [title={A state cover for the specification from \in{Figure}[fig:running-example].},
   reference=fig:running-example-prefixes]
\hbox{
\starttikzpicture[node distance=0.9cm and 3cm,bend angle=20]
	\node[state] (0) {$s_0$};
	\node[state] (1) [above left = of 0] {$s_1$};
	\node[state] (2) [below left = of 0] {$s_2$};
	\node[state] (3) [above right = of 0] {$s_3$};
	\node[state] (4) [below right = of 0] {$s_4$};
	\node        (5) [above = of 0] {};
	\path[trans]
	(5) edge node [right] {$\epsilon$} (0)
	(0) edge [bend left=] node [below] {${a}/1$} (1)
	(0) edge [bend right] node [below] {${b}/0$} (4)
	(1) edge              node [left ] {${a}/0$} (2)
	(4) edge [bend left]  node [right] {${a}/1$} (3);
	\path[trans, color=grijs]
	(0) edge [loop below] node [below] {${c}/0$} (0)
	(1) edge [bend left]  node [above] {${b}/0$, ${c}/0$} (0)
	(2) edge [bend right] node [below] {${b}/0$, ${c}/0$} (0)
	(2) edge [loop left]  node [left ] {${a}/1$} (1)
	(3) edge [bend left=30] node [right] {${a}/1$} (4)
	(3) edge [bend right] node [above] {${b}/0$} (0)
	(3) edge [loop right] node [right] {${c}/1$} (3)
	(4) edge [loop right] node [right] {${b}/0$} (4)
	(4) edge [bend right] node [above] {${c}/0$} (0);
\stoptikzpicture
}
\stopplacefigure

As shown earlier, the set $W = \{ aa, ac, c \}$ is a characterisation set.
The W-method, which simply combines $P \cup Q$ with $W$, gives the following test suite of size 169:

\startformula\startmathmatrix[n=2,align={left,left}]
\NC T_{\text{W}} = \{ \NC aaaaa, aaaac, aaac, aabaa, aabac, aabc, aacaa, \NR
\NC \NC aacac, aacc, abaa, abac, abc, acaa, acac, acc, baaaa, \NR
\NC \NC baaac, baac, babaa, babac, babc, bacaa, bacac, bacc, \NR
\NC \NC bbaa, bbac, bbc, bcaa, bcac, bcc, caa, cac, cc \quad \} \NR
\stopmathmatrix\stopformula

With the Wp-method we get to choose a different state identifier per state.
Since many states have an UIO, we can use them as state identifiers.
This defines the following family $\Fam{W}$:
\startformulas
\startformula W_0 = \{ aa \}        \stopformula
\startformula W_1 = \{ a \}         \stopformula
\startformula W_2 = \{ aa, ac, c \} \stopformula
\startformula W_3 = \{ c \}         \stopformula
\startformula W_4 = \{ ac \}        \stopformula
\stopformulas
For the first part of the Wp test suite we need $\bigcup \Fam{W} = \{ aa, ac, c \}$.
For the second part, we only combine the sequences in the transition cover with the corresponding suffixes.
All in all we get a test suite of size 75:
\startformula\startmathmatrix[n=2,align={left,left}]
\NC T_{\text{Wp}} = \{ \NC aaaaa, aaaac, aaac, aabaa, aacaa, abaa, \NR
\NC \NC acaa, baaac, baac, babaa, bacc, bbac, bcaa, caa \quad \} \NR
\stopmathmatrix\stopformula

For the HSI-method we need a separating family $\Fam{H}$.
We pick the following sets:
\startformulas
\startformula H_0 = \{ aa, c \}     \stopformula
\startformula H_1 = \{ a \}         \stopformula
\startformula H_2 = \{ aa, ac, c \} \stopformula
\startformula H_3 = \{ a, c \}      \stopformula
\startformula H_4 = \{ aa, ac, c \} \stopformula
\stopformulas
(We repeat that these sets are prefix-closed, but we only show the maximal sequences.)
Note that these sets are harmonised, unlike the family $\Fam{W}$.
For example, the separating sequence $a$ is contained in both $H_1$ and $H_3$.
This ensures that we do not have to consider $\bigcup \Fam{H}$ in the first part of the test suite.
When combining this with the corresponding prefixes, we obtain the HSI test suite of size 125:
\startformula\startmathmatrix[n=2,align={left,left}]
\NC T_{\text{HSI}} = \{ \NC aaaaa, aaaac, aaac, aabaa, aabc, aacaa, aacc, \NR
\NC \NC abaa, abc, acaa, acc, baaaa, baaac, baac, babaa, \NR
\NC \NC babc, baca, bacc, bbaa, bbac, bbc, bcaa, bcc, caa, cc \quad \} \NR
\stopmathmatrix\stopformula
On this particular example the Wp-method outperforms the HSI-method.
The reason is that many states have UIOs and we picked those to be the state identifiers.
In general, however, UIOs may not exist (and finding them is hard).

The UIO-method and ADS-method are not applicable in this example because state $s_2$ does not have an UIO.

\startplacefigure
  [title={A faulty implementation for the specification in \in{Figure}[fig:running-example].},
   list={A faulty implementation.},
   reference=fig:running-example-faulty]
\hbox{
\starttikzpicture[node distance=0.9cm and 3cm,bend angle=20]
	\node[state] (0) {$s'_0$};
	\node[state] (1) [above left = of 0] {$s'_1$};
	\node[state] (2) [below left = of 0] {$s'_2$};
	\node[state] (3) [above right = of 0] {$s'_3$};
	\node[state] (4) [below right = of 0] {$s'_4$};
	\node        (5) [above = of 0] {};
	\path[trans]
	(5) edge (0)
	(0) edge [bend left]  node [below] {$a/1$} (1)
	(0) edge [bend right] node [below] {$b/0$} (4)
	(0) edge [loop below] node [below] {$c/0$} (0)
	(1) edge              node [left ] {$a/0$} (2)
	(1) edge [bend left]  node [above] {$b/0$, $c/0$} (0)
	(2) edge [bend right] node [below] {\color[darkred]{$a/1$}, $b/0$, $c/0$} (0)
	(3) edge [bend left]  node [right] {$a/1$} (4)
	(3) edge [bend right] node [above] {$b/0$} (0)
	(3) edge [loop right] node [right] {$c/1$} (3)
	(4) edge [bend left]  node [left ] {$a/1$} (3)
	(4) edge [loop right] node [right] {$b/0$} (4)
	(4) edge [bend right] node [above] {$c/0$} (0);
\stoptikzpicture
}
\stopplacefigure

We can run these test suites on the faulty implementation shown in \in{Figure}[fig:running-example-faulty].
Here, the $a$-transition from state $s'_2$ transitions to the wrong target state.
It is not an obvious mistake, since the faulty target $s'_0$ has very similar transitions as $s_2$.
Yet, all the test suites detect this error.
When choosing the prefix $aaa$ (included in the transition cover), and suffix $aa$ (included in the characterisation set and state identifiers for $s_2$), we see that the specification outputs $10111$ and the implementation outputs $10110$.
The sequence $aaaaa$ is the only sequence (in any of the test suites here) which detects this fault.

Alternatively, the $a$-transition from $s'_2$ would transition to $s'_4$, we need the suffix $ac$ as $aa$ will not detect the fault.
Since the sequences $ac$ is included in the state identifier for $s_2$, this fault would also be detected.
This shows that it is sometimes necessary to include multiple sequences in the state identifier.

Another approach to testing would be to enumerate all sequences up to a certain length.
In this example, we need sequences of at least length 5.
Consequently, the test suite contains 243 sequences and this boils down to a size of 1458.
Such a brute-force approach is not scalable.


\stopsubsection
\stopsection
\startsection
  [title={Hybrid ADS method},
   reference=sec:hybrid]

In this section, we describe a new test generation method for Mealy machines.
Its completeness will be proven in \in{Theorem}[thm:completeness], together with completeness for all methods defined in the previous section.
From a high level perspective, the method uses the algorithm by \cite[authoryears][LeeY94] to obtain an ADS.
If no ADS exists, their algorithm still provides some sequences which separates some inequivalent states.
Our extension is to refine the set of sequences by using pairwise separating sequences.
Hence, this method is a hybrid between the ADS-method and HSI-method.

The reason we do this is the fact that the ADS-method generally constructs small test suites as experiments by \cite[authoryears][DorofeevaEMCY10] suggest.
The test suites are small since an ADS can identify a state with a single word, instead of a set of words which is generally needed.
Even if the ADS does not exist, using the partial result of Lee and Yannakakis' algorithm can reduce the size of test suites.

We will now see the construction of this hybrid method.
Instead of manipulating separating families directly, we use a \emph{splitting tree}.
This is a data structure which is used to construct separating families or adaptive distinguishing sequences.

\startdefinition[reference=splitting-tree]
A \defn{splitting tree (for $M$)} is a rooted tree where each node $u$ has
\startitemize
	\item a non-empty set of states $l(u) \subseteq M$, and
	\item if $u$ is not a leaf, a sequence $\sigma(u) \in I^{\ast}$.
\stopitemize
We require that if a node $u$ has children $C(u)$ then
\startitemize
	\item the sets of states of the children of $u$ partition $l(u)$, i.e., the set
	$P(u) = \{l(v) \,|\, v \in C(u)\}$ is a \emph{non-trivial} partition of $l(u)$, and
	\item the sequence $\sigma(u)$ witnesses the partition $P(u)$, meaning that
	for all $p, q \in P(u)$ we have $p = q$ iff
	$\lambda(s, \sigma(u)) = \lambda(t, \sigma(u))$ for all $s \in p, t \in q$.
\stopitemize
A splitting tree is called \defn{complete} if all inequivalent states belong to different leaves.
\stopdefinition

Efficient construction of a splitting tree is described in more detail in \in{Chapter}[chap:minimal-separating-sequences].
Briefly, the splitting tree records the execution of a partition refinement algorithm (such as Moore's or Hopcroft's algorithm).
Each non-leaf node encodes a \defn{split} together with a witness, which is a separating sequence for its children.
From such a tree we can construct a state identifier for a state by locating the leaf containing that state and collecting all the sequences you read when traversing to the root.

For adaptive distinguishing sequences an additional requirement is put on the splitting tree:
for each non-leaf node $u$, the sequence $\sigma(u)$ defines an injective map $x \mapsto (\delta(x, \sigma(u)), \lambda(x, \sigma(u)))$ on the set $l(u)$.
\cite[authoryears][LeeY94] call such splits \defn{valid}.
\in{Figure}[fig:example-splitting-tree] shows both valid and invalid splits.
Validity precisely ensures that after performing a split, the states are still distinguishable.
Hence, sequences of such splits can be concatenated.

\startplacefigure
  [title={A complete splitting tree with shortest witnesses for the specification of \in{Figure}[fig:running-example].
          Only the splits $a$, $aa$, and $ac$ are valid.},
   list={Complete splitting tree with shortest witnesses for \in{Figure}[fig:running-example].},
   reference=fig:example-splitting-tree]
\hbox{
\starttikzpicture[node distance=1.75cm]
\splitnode{0}{s_0, s_1, s_2, s_3, s_4}{a}{}
\splitnode{3}{s_0, s_2, s_3, s_4}{c}{below right of=0}
\splitnode{5}{s_0, s_2, s_4}{aa}{below left of=3}
\splitnode{8}{s_2, s_4}{ac}{below right of=5}
\node (1) [below left of=0 ] {$s_1$};
\node (6) [below right of=3] {$s_3$};
\node (7) [below left of=5 ] {$s_0$};
\node (8l)[below left of=8 ] {$s_2$};
\node (8r)[below right of=8] {$s_4$};
\path[->]
(0) edge (1)
(0) edge (3)
(3) edge (5)
(3) edge (6)
(5) edge (7)
(5) edge (8)
(8) edge (8l)
(8) edge (8r);
\stoptikzpicture
}
\stopplacefigure

The following lemma is a result of \cite[authoryears][LeeY94].

\startlemma
A complete splitting tree with only valid splits exists if and only if there exists an adaptive distinguishing sequence.
\stoplemma

Our method uses the exact same algorithm as the one by Lee and Yannakakis.
However, we also apply it in the case when the splitting tree with valid splits is not complete (and hence no adaptive distinguishing sequence exists).
Their algorithm still produces a family of sets, but is not necessarily a separating family.

In order to recover separability, we refine that family.
Let $\Fam{Z'}$ be the result of Lee and Yannakakis' algorithm (to distinguish from their notation, we add a prime) and let $\Fam{H}$ be a separating family extracted from an ordinary splitting tree.
The hybrid ADS family is defined as $\Fam{Z'} ; \Fam{H}$, and can be computed as sketched in \in{Algorithm}[alg:hybrid] (the algorithm works on splitting trees instead of separating families).
By \in{Lemma}[lemma:refinement-props] we note the following: in the best case this family is an adaptive distinguishing sequence; in the worst case it is equal to $\Fam{H}$; and in general it is a combination of the two families.
In all cases, the result is a separating family because $\Fam{H}$ is.

\startplacealgorithm
  [title={Obtaining the hybrid separating family $\Fam{Z'} ; \Fam{H}$},
   reference=alg:hybrid]
\startalgorithmic
	\REQUIRE{A Mealy machine $M$}
	\ENSURE{A separating family $Z$}
	\startlinenumbering
	\STATE $T_1 \gets$ splitting tree for Moore's minimisation algorithm
	\STATE $T_2 \gets$ splitting tree with valid splits (see \citenp[LeeY94])
	\STATE $\Fam{Z'} \gets$ (incomplete) family constructed from $T_2$
	\FORALL{inequivalent states $s, t$ in the same leaf of $T_2$}{
		\STATE $u \gets lca(T_1, s, t)$
		\STATE $Z_s \gets Z'_s \cup \{ \sigma(u) \}$
		\STATE $Z_t \gets Z'_t \cup \{ \sigma(u) \}$
	}
	\ENDFOR
	\RETURN{$Z$}
	\stoplinenumbering
\stopalgorithmic
\stopplacealgorithm

With the hybrid family we can define the test suite as follows.
Its $m$-completeness is proven in \in{Section}[sec:completeness].

\startdefinition
Let $P$ be a state cover, $\Fam{Z'}$ be a family of sets constructed with the Lee and Yannakakis algorithm, and $\Fam{H}$ be a separating family.
The \defn{hybrid ADS} test suite is
\startformula
T_{\text{h-ADS}} = (P \cup Q) \cdot I^{\leq k} \odot (\Fam{Z'} ; \Fam{H}).
\stopformula
\stopdefinition


\startsubsection
  [title={Example},
   reference=sec:hads-example]
In the figure we see the (unique) result of Lee and Yannakakis' algorithm.
We note that the states $s_2, s_3, s_4$ are not split, so we need to refine the family for those states.

\startplacefigure
  [title={(a): Largest splitting tree with only valid splits for \in{Figure}[fig:running-example].
          (b): Its incomplete adaptive distinguishing tree.},
   list={Splitting tree and incomplete adaptive distinguishing sequence.},
   reference=fig:example-ads]
\startcombination[nx=2, ny=1, location=middle]{
\hbox{
\starttikzpicture[node distance=2cm]
\splitnode{0}{s_0, s_1, s_2, s_3, s_4}{a}{}
\splitnode{2}{s_0, s_2, s_3, s_4}{aa}{below right of=0}
\node (1) [below left of=0 ] {$s_1$};
\node (3) [below left of=2 ] {$s_0$};
\node (4) [below right of=2] {$s_2, s_3, s_4$};
\path[->]
(0) edge (1)
(0) edge (2)
(2) edge (3)
(2) edge (4);
\stoptikzpicture
}} {(a)}
{\hbox{
\starttikzpicture[node distance=.75cm]
	\node (0) {$a$};
	\node [below left =of 0] (1) {$s_1$};
	\node [below right=of 0] (2) {$a$};
	\node [below left =of 2] (3) {$s_0$};
	\node [below right=of 2] (4) {$s_2, s_3, s_4$};
	
	\path[->]
	(0)+(0,.75) edge (0)
	(0) edge node [above left ] {$0$} (1)
	(0) edge node [above right] {$1$} (2)
	(2) edge node [above left ] {$0$} (3)
	(2) edge node [above right] {$1$} (4);
\stoptikzpicture
}} {(b)}
\stopcombination
\stopplacefigure

We take the separating family $\Fam{H}$ from before.
From the incomplete ADS in \in{Figure}{b}[fig:example-ads] above we obtain the family $\Fam{Z'}$.
These families and the refinement $\Fam{Z'}; \Fam{H}$ are given below.

\startformula\startmathmatrix[n=3,align={left,left,left},distance=1cm]
\NC H_0 = \{aa,c\}    \NC Z'_0 = \{aa\} \NC (Z';H)_0 = \{aa\}      \NR
\NC H_1 = \{a\}       \NC Z'_1 = \{a\}  \NC (Z';H)_1 = \{a\}       \NR
\NC H_2 = \{aa,ac,c\} \NC Z'_2 = \{aa\} \NC (Z';H)_2 = \{aa,ac,c\} \NR
\NC H_3 = \{a,c\}     \NC Z'_3 = \{aa\} \NC (Z';H)_3 = \{aa,c\}    \NR
\NC H_4 = \{aa,ac,c\} \NC Z'_4 = \{aa\} \NC (Z';H)_4 = \{aa,ac,c\} \NR
\stopmathmatrix\stopformula

With the separating family $\Fam{Z}'; \Fam{H}$ we obtain the following test suite of size 96:
\startformula\startmathmatrix[n=2,align={left,left}]
\NC T_{\text{h-ADS}} = \{ \NC aaaaa, aaaac, aaac, aabaa, aacaa, abaa, acaa, \NR
\NC \NC baaaa, baaac, baac, babaa, bacaa, bacc, bbaa, bbac, \NR
\NC \NC bbc, bcaa, caa \quad \} \NR
\stopmathmatrix\stopformula

We note that this is indeed smaller than the HSI test suite.
In particular, we have a smaller state identifier for $s_0$: $\{ aa \}$ instead of $\{ aa, c \}$.
As a consequence, there are less combinations of prefixes and suffixes.
We also observe that one of the state identifiers grew in length: $\{ aa, c \}$ instead of $\{ a, c \}$ for state $s_3$.

\stopsubsection
\startsubsection
  [title={Implementation},
   reference=sec:hybrid-ads-impementation]

All the algorithms concerning the hybrid ADS-method have been implemented and can be found at
\citeurl[hybrid-ads-code].
We note that \in{Algorithm}[alg:hybrid] is implemented a bit more efficiently, as we can walk the splitting trees in a particular order.
For constructing the splitting trees in the first place, we use Moore's minimisation algorithm and the algorithms by \citet[LeeY94].
We keep all relevant sets prefix-closed by maintaining a \emph{trie data structure}.
A trie data structure also allows us to immediately obtain the set of maximal tests only.


\stopsubsection
\startsubsection
  [title={Randomisation}]

Many constructions of the test suite generation can be randomised.
There may exist many shortest access sequences to a state and we can randomly pick any.
Also in the construction of state identifiers many steps in the algorithm are non-deterministic:
the algorithm may ask to find any input symbol which separates a set of states.
The tool randomises many such choices.
We have noticed that this can have a huge influence on the size of the test suite.
However, a decent statistical investigation still lacks at the moment.

In many of the applications such as learning, no bound on the number of states of the SUT is known.
In such cases it is possible to randomly select test cases from an infinite test suite.
Unfortunately, we lose the theoretical guarantees of completeness with random generation.
Still, as we will see in \in{Chapter}[chap:applying-automata-learning], this can work really well.
We can randomly test cases as follows.
In the above definition for the hybrid ADS test suite we replace $I^{\leq k}$ by $I^{\ast}$ to obtain an infinite test suite.
Then we sample tests as follows:
\startitemize[n]
\item sample an element $p$ from $P$ uniformly,
\item sample a word $w$ from $I^{\ast}$ with a geometric distribution, and
\item sample uniformly from $(\Fam{Z'} ; \Fam{H})_s$ for the state $s = \delta(s_0, pw)$.
\stopitemize


\stopsubsection
\stopsection
\startsection
  [title={Overview},
   reference=sec:overview]

We give an overview of the aforementioned test methods.
We classify them in two directions,
\startitemize
\item whether they use harmonised state identifiers or not and
\item whether it used singleton state identifiers or not.
\stopitemize

\starttheorem
  [reference=thm:completeness]
Assume $M$ to be minimal, reachable, and of size $n$.
The following test suites are all $n+k$-complete:

\placetable[force,none]{}{
\starttabulate[|c|c|c|]
\NC                 \NC Arbitrary \NC Harmonised
\NR \HL %----------------------------------------
\NC Many / pairwise \NC Wp        \NC HSI
\NR
\NC % empty cell
\NC $ P \cdot I^{\leq k} \cdot \bigcup \Fam{W} \,\cup\, Q \cdot I^{\leq k} \odot \Fam{W} $
\NC $ (P \cup Q) \cdot I^{\leq k} \odot \Fam{H} $
\NR \HL %----------------------------------------
\NC Hybrid          \NC           \NC Hybrid ADS
\NR
\NC % empty cell
\NC % empty cell
\NC $ (P \cup Q) \cdot I^{\leq k} \odot (\Fam{Z'} ; \Fam{H}) $
\NR \HL %----------------------------------------
\NC Single / global \NC UIOv      \NC ADS
\NR
\NC % empty cell
\NC $ P \cdot I^{\leq k} \cdot \bigcup \Fam{U} \,\cup\, Q \cdot I^{\leq k} \odot \Fam{U} $
\NC $ (P \cup Q) \cdot I^{\leq k} \odot \Fam{Z} $
\NR \HL %----------------------------------------
\stoptabulate}
\stoptheorem
\startproof
See \in{Corollary}[cor:w-wp-uiov-coplete] and \in{}[cor:hsi-ads-coplete].
\stopproof

Each of the methods in the right column can be written simpler as $P \cdot I^{\leq k+1} \odot \Fam{H}$, since $Q = P \cdot I$.
This makes them very easy to implement.

It should be noted that the ADS-method is a specific instance of the HSI-method and similarly the UIOv-method is an instance of the Wp-method.
What is generally meant by the Wp-method and HSI-method is the above formula together with a particular way to obtain the (harmonised) state identifiers.

We are often interested in the size of the test suite.
In the worst case, all methods generate a test suite with a size in $\bigO(pn^3)$ and this bound is tight \cite[Vasilevskii73].
Nevertheless we expect intuitively that the right column performs better, as we are using a more structured set (given a separating family for the HSI-method, we can always forget about the common prefixes and apply the Wp-method, which will never be smaller if constructed in this way).
Also we expect the bottom row to perform better as there is a single test for each state.
Small experimental results confirm this intuition
\cite[DorofeevaEMCY10].

On the example in \in{Figure}[fig:running-example], we computed all applicable test suites in \in{Sections}[sec:all-example] and \in{}[sec:hads-example].
The UIO and ADS methods are not applicable.
For the W, Wp, HSI and hybrid ADS methods we obtained test suites of size 169, 75, 125 and 96 respectively.


\stopsection
\startsection
  [title={Proof of completeness},
   reference=sec:completeness]

In this section, we will prove $n$-completeness of the discussed test methods.
Before we dive into the proof, we give some background on the proof-principle of bisimulation.
The original proofs of completeness often involve an inductive argument (on the length of words) inlined with arguments about characterisation sets.
This can be hard to follow and so we prefer a proof based on bisimulations, which defers the inductive argument to a general statement about bisimulation.
Many notions of bisimulation exist in the theory of labelled transition systems, but for Mealy machines there is just one simple definition.
We give the definition and the main proof principle, all of which can be found in a paper by \citet[Rutten98].

\startdefinition
Let $M$ be a Mealy machine.
A relation $R \subseteq S \times S$ is called a \defn{bisimulation} if for every $(s, t) \in R$ we have
\startitemize
\item equal outputs: $\lambda(s, a) = \lambda(t, a)$ for all $a \in I$, and
\item related successor states: $(\delta(s, a), \delta(t, a)) \in R$ for all $a \in I$.
\stopitemize
\stopdefinition

\startlemma[reference=lem:bisim]
If two states $s, t$ are related by a bisimulation, then $s \sim t$.
\footnote{The converse -- which we do not need here -- also holds, as $\sim$ is a bisimulation.}
\stoplemma

We use a slight generalisation of the bisimulation proof technique, called \emph{bisimulation up-to}.
This allows one to give a smaller set $R$ which extends to a bisimulation.
A good introduction of these up-to techniques is given by \citet[BonchiP15] or the thesis of \citet[Rot17].
In our case we use bisimulation \emph{up-to ${\sim}$-union}.
The following lemma can be found in the given references.

\startdefinition
Let $M$ be a Mealy machine.
A relation $R \subseteq S \times S$ is called a \defn{bisimulation up-to ${\sim}$-union} if for every $(s, t) \in R$ we have
\startitemize
\item equal outputs: $\lambda(s, a) = \lambda(t, a)$ for all $a \in I$, and
\item related successor states: $(\delta(s, a), \delta(t, a)) \in R$ \emph{or} $\delta(s, a) \sim \delta(t, a)$ for all $a \in I$.
\stopitemize
\stopdefinition

\startlemma[reference=lem:bisim-upto]
Any bisimulation up-to ${\sim}$-union is contained in a bisimulation.
\stoplemma

We fix a specification $M$ which has a minimal representative with $n$ states and an implementation $M'$ with at most $n+k$ states.
We assume that all states are reachable from the initial state in both machines (i.e., both are \defn{connected}).

The next proposition gives sufficient conditions for a test suite of a certain shape to be
complete.
We then prove that these conditions hold for the test suites in this chapter.

\startproposition
  [reference=prop:completeness]
Let $\Fam{W}$ and $\Fam{W'}$ be two families of words and $P$ a state cover for $M$.
Let $T = P \cdot I^{\leq k} \odot \Fam{W} \,\cup\, P \cdot I^{\leq k+1} \odot \Fam{W'}$ be a test suite.
If
\startitemize[n]
\item for all $x, y \in M:$ $x \sim_{W_x \cap W_y} y$ implies $x \sim y$,
\item for all $x, y \in M$ and $z \in M'$: $x \sim_{W_x} z$ and $z \sim_{W'_y} y$ implies $x \sim y$, and
\item the machines $M$ and $M'$ agree on $T$,
\stopitemize
then $M$ and $M'$ are equivalent.
\stopproposition
\startproof
First, we prove that $P \cdot I^{\leq k}$ reaches all states in $M'$.
For $p, q \in P$ and $x = \delta(s_0, p), y = \delta(s_0, q)$ such that $x \not\sim_{W_x \cap W_y} y$, we also have $\delta'(s'_0, p) \not\sim_{W_x \cap W_y} \delta'(s'_0, q)$ in the implementation $M'$.
By (1) this means that there are at least $n$ different behaviours in $M'$, hence at least $n$ states.

Now $n$ states are reached by the previous argument (using the set $P$).
By assumption $M'$ has at most $k$ extra states.
If those extra states are reachable, they are reachable from an already visited state in at most $k$ steps.
So we can reach all states of $M'$ by using $I^{\leq k}$ after $P$.

Second, we show that the reached states are bisimilar.
Define the relation $R = \{(\delta(s_0, p), \delta'(s_0', p)) \mid p \in P \cdot I^{\leq k}\}$.
Note that for each $(s, i) \in R$ we have $s \sim_{W_s} i$.
For each state $i \in M'$ there is a state $s \in M$ such that $(s, i) \in R$, since we reach all states in both machines by $P \cdot I^{\leq k}$.
We will prove that this relation is in fact a bisimulation up-to ${\sim}$-union.

For output, we note that $(s, i) \in R$ implies $\lambda(s, a) = \lambda'(i, a)$ for all $a$,
since the machines agree on $P \cdot I^{\leq k+1}$.
For the successors, let $(s, i) \in R$ and $a \in I$ and consider the successors $s_2 = \delta(s, a)$ and $i_2 = \delta'(i, a)$.
We know that there is some $t \in M$ with $(t, i_2) \in R$.
We also know that we tested $i_2$ with the set $W_t$.
So we have:
\startformula s_2 \sim_{W'_{s_2}} i_2 \sim_{W_t} t. \stopformula
By the second assumption, we conclude that $s_2 \sim t$.
So $s_2 \sim t$ and $(t, i) \in R$, which means that $R$ is a bisimulation up-to ${\sim}$-union.
Moreover, $R$ contains the pair $(s_0, s'_0)$.
By using \in{Lemmas}[lem:bisim-upto] and \in{}[lem:bisim], we conclude that the initial $s_0$ and $s'_0$ are equivalent.
\stopproof

Before we show that the conditions hold for the test methods, we reflect on the above proof first.
This proof is very similar to the completeness proof by \citet[Chow78].
\footnote{In fact, it is also similar to Lemma 4 by \citet[Angluin87] which proves termination in the L* learning algorithm. This correspondence was noted by \citet[BergGJLRS05].}
In the first part we argue that all states are visited by using some sort of counting and reachability argument.
Then in the second part we show the actual equivalence.
To the best of the authors knowledge, this is first $m$-completeness proof which explicitly uses the concept of a bisimulation.
Using a bisimulation allows us to slightly generalise and use bisimulation up-to ${\sim}$-union, dropping the the often-assumed requirement that $M$ is minimal.

\startlemma
Let $\Fam{W'}$ be a family of state identifiers for $M$.
Define the family $\Fam{W}$ by $W_s = \bigcup \Fam{W'}$.
Then the conditions (1) and (2) in \in{Proposition}[prop:completeness] are satisfied.
\stoplemma
\startproof
The first condition we note that $W_x \cap W_y = W_x = W_y$, and so $x \sim_{W_x \cap W_y} y$ implies $x \sim_{W_x} y$, now by definition of state identifier we get $x \sim y$.

For the second condition, let $x \sim_{\bigcup \Fam{W'}} z \sim_{W'_y} y$.
Then we note that $W'_y \subseteq \bigcup{W'}$ and so we get $x \sim_{W'_y} z \sim_{W'_y} y$.
By transitivity we get $x \sim_{W'_y} y$ and so by definition of state identifier we get $x \sim y$.
\stopproof

\startcorollary[reference=cor:w-wp-uiov-coplete]
The W, Wp, and UIOv test suites are $n+k$-complete.
\stopcorollary

\startlemma
Let $\Fam{H}$ be a separating family and take $\Fam{W} = \Fam{W'} = \Fam{H}$.
Then the conditions (1) and (2) in \in{Proposition}[prop:completeness] are satisfied.
\stoplemma
\startproof
Let $x \sim_{H_x \cap H_y} y$, then by definition of separating family $x \sim y$.
For the second condition, let $x \sim_{H_x} z \sim_{H_y} y$.
Then we get $x \sim_{H_x \cap H_y} z \sim_{H_x \cap H_y} y$ and so by transitivity $x \sim_{H_x \cap H_y} y$, hence again $x \sim y$.
\stopproof

\startcorollary[reference=cor:hsi-ads-coplete]
The HSI, ADS and hybrid ADS test suites are $n+k$-complete.
\stopcorollary


\stopsection
\startsection
  [title={Related Work and Discussion},
   reference=sec:discussion]

In this chapter, we have mostly considered classical test methods which are all based on prefixes and state identifiers.
There are more recent methods which almost fit in the same framework.
We mention the P \citep[SimaoP10], H \citep[DorofeevaEY05], and SPY \citep[SimaoPY09] methods.
The P method construct a test suite by carefully considering sufficient conditions for a $p$-complete test suite (here $p \leq n$, where $n$ is the number of states).
It does not generalise to extra states, but it seems to construct very small test suites.
The H method is a refinement of the HSI-method where state identifiers for a testing transitions are reconsidered.
(Note that \in{Proposition}[prop:completeness] allows for a different family when testing transitions.)
Last, the SPY method builds upon the HSI-method and changes the prefixes in order to minimise the size of a test suite, exploiting overlap in test sequences.
We believe that this technique is independent of the HSI-method and can in fact be applied to all methods presented in this chapter.
As such, the SPY method should be considered as an optimisation technique, orthogonal to the work in this chapter.

Recently, \citet[HieronsT15] devise a novel test method which is based on incomplete distinguishing sequences and is similar to the hybrid ADS method.
They use sequences which can be considered to be adaptive distinguishing sequences on a subset of the state space.
With several of those one can cover the whole state space, obtaining a $m$-complete test suite.
This is a bit dual to our approach, as our \quotation{incomplete} adaptive distinguishing sequences define a course partition of the complete state space.
Our method becomes complete by refining the tests with pairwise separating sequences.

Some work is put into minimising the adaptive distinguishing sequences themselves.
\citet[TurkerY14] describe greedy algorithms which construct small adaptive distinguishing sequences.
Moreover, they show that finding the minimal adaptive distinguishing sequence is NP-complete in general, even approximation is NP-complete.
We expect that similar heuristics also exist for the other test methods and that they will improve the performance.
Note that minimal separating sequences do not guarantee a minimal test suite.
In fact, we see that the hybrid ADS method outperforms the HSI-method on the example in \in{Figure}[fig:running-example] since it prefers longer, but fewer, sequences.

Some of the assumptions made at the start of this chapter have also been challenged.
For non-deterministic Mealy machine, we mention the work of \citet[PetrenkoY14].
We also mention the work of \citet[BosJM17, SimaoP14] for input/output transition systems with the \emph{ioco} relation.
In both cases, the test suites are still defined in the same way as in this chapter: prefixes followed by state identifiers.
However, for non-deterministic systems, guiding an implementation into a state is harder as the implementation may choose its own path.
For that reason, sequences are often replaced by automata, so that the testing can be adaptive.
This adaptive testing is game-theoretic and the automaton provides a strategy.
This game theoretic point of view is further investigated by \citet[BosS18].
The test suites generally are of exponential size, depending on how non-deterministic the systems are.

The assumption that the implementation is resettable is also challenged early on.
If the machine has no reliable reset (or the reset is too expensive), one tests the system with a single \emph{checking sequence}.
\citet[LeeY94] give a randomised algorithm for constructing such a checking sequence using adaptive distinguishing sequences.
There is a similarity with the randomised algorithm by \citet[RivestS93] for learning non-resettable automata.
Recently, \citet[GrozSPO16] give a deterministic learning algorithm for non-resettable machines based on adaptive distinguishing sequences.

Many of the methods described here are benchmarked on small or random Mealy machines by \citet[DorofeevaEMCY10, EndoS13].
The benchmarks are of limited scope, the machine from \in{Chapter}[chap:applying-automata-learning], for instance, is neither small nor random.
For this reason, we started to collect more realistic benchmarks at \citeurl[automata-wiki].


\stopsection
\referencesifcomponent
\stopchapter
\stopcomponent