joshua/phd-thesis
Archived
1
Fork 0
Code Activity

Nog meer kleine dingetjes

Browse source
This commit is contained in:
Joshua Moerman 2019-01-11 12:41:33 +01:00
parent 4a92b104bb
commit a2022293cc
6 changed files with 76 additions and 38 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

32
biblio.bib
View file

@ -796,6 +796,22 @@
bibsource = {dblp computer science bibliography, https://dblp.org}
}
@article{EmersonS96,
author = {E. Allen Emerson and
A. Prasad Sistla},
title = {Symmetry and Model Checking},
journal = {Formal Methods in System Design},
volume = {9},
number = {1/2},
pages = {105--131},
year = {1996},
url = {https://doi.org/10.1007/BF00625970},
doi = {10.1007/BF00625970},
timestamp = {Sat, 27 May 2017 14:24:05 +0200},
biburl = {https://dblp.org/rec/bib/journals/fmsd/EmersonS96},
bibsource = {dblp computer science bibliography, https://dblp.org}
}
@article{EndoS13,
author = {Andr{\'{e}} Takeshi Endo and
Adenilso da Silva Sim{\~{a}}o},
@ -1239,6 +1255,22 @@
bibsource = {dblp computer science bibliography, https://dblp.org}
}
@article{IpD96,
author = {C. Norris Ip and
David L. Dill},
title = {Better Verification Through Symmetry},
journal = {Formal Methods in System Design},
volume = {9},
number = {1/2},
pages = {41--75},
year = {1996},
url = {https://doi.org/10.1007/BF00625968},
doi = {10.1007/BF00625968},
timestamp = {Sat, 27 May 2017 14:24:05 +0200},
biburl = {https://dblp.org/rec/bib/journals/fmsd/IpD96},
bibsource = {dblp computer science bibliography, https://dblp.org}
}
@inproceedings{IsbernerHS13,
author = {Malte Isberner and
Falk Howar and

5
content.tex
View file

@ -4,14 +4,13 @@
\starttext
\startfrontmatter
\start
\switchtobodyfont[20pt]
\midaligned{\framed[width=12cm,align=center,offset=2cm,frame=off]{Nominal Techniques \& Black Box Testing for Automata Learning}}
\midaligned{\framed[width=12cm,align=center,offset=2cm,frame=off]{Nominal Techniques and Black Box Testing for Automata Learning}}
\switchtobodyfont[12pt]
\midaligned{Joshua Moerman}
\midaligned{8 January 2019}
\midaligned{11 January 2019}
\stop
\page[yes]

45
content/introduction.tex
View file

@ -6,7 +6,7 @@
reference=chap:introduction]
When I was younger, I often learned how to play with new toys by messing about with them, i.e., by pressing buttons at random, observing their behaviour, pressing more buttons, and so on.
Only resorting to the manual -- or asking \quotation{experts} -- to confirm my beliefs on how the toy works.
Only resorting to the manual -- or asking \quotation{experts} -- to confirm my beliefs on how the toys work.
Now that I am older, I do mostly the same with new devices, new tools, and new software.
However, now I know that this is an established computer science technique, called \emph{model learning}.
@ -42,7 +42,7 @@ For an alphabet $\Sigma$, we denote the set of words by $\Sigma^{*}$.
The learning problem is defined as follows.
There is some fixed, but unknown, language $\lang \subseteq \Sigma^{*}$.
This language may define the behaviour of a software component, a property in model checking, a set of traces from a protocol, etc.
We wish to infer a description of $\lang$ by only having observed a small part of this language.
We wish to infer a description of $\lang$ after only having observed a small part of this language.
For example, we may have seen hundred words belonging to the language and a few which do not belong to the language.
Then concluding with a good description of $\lang$ is difficult, as we are missing information about the infinitely many words we have not observed.
@ -62,7 +62,7 @@ This type of query is often generalised to more general output, in these cases w
In some papers, such a query is then called an \emph{output query}.
\description{Equivalence queries (EQs)}
The learner can provide a hypothesised description of $\lang$ to the teacher.
The learner can provide a hypothesised description $H$ of $\lang$ to the teacher.
If the hypothesis is correct, the teacher replies with \kw{yes}.
If, however, the hypothesis is incorrect, the teacher replies with \kw{no} together with a counterexample, i.e., a word which is in $\lang$ but not in the hypothesis or vice versa.
@ -130,7 +130,7 @@ Yet, as applications show, this is a useful way of learning software behaviour.
\startsection
[title={Applications of Model Learning}]
Since this thesis only contains one real-world application on learning in \in{Chapter}[chap:applying-automata-learning], it is good to mention a few others.
Since this thesis only contains one real-world application of learning in \in{Chapter}[chap:applying-automata-learning], it is good to mention a few others.
Although we remain in the context of learning software behaviour, the applications are quite different from each other.
This is by no means a complete list.
@ -138,7 +138,7 @@ This is by no means a complete list.
A prominent example is by \citet[Fiterau-Brostean16].
They learn models of TCP implementations -- both clients and server sides.
Interestingly, they found bugs in the (closed source) Windows implementation.
Later, \citet[Fiterau-BrosteanH17] also found a bug in the sliding window of the Linux implementation.
Later, \citet[Fiterau-BrosteanH17] also found a bug in the sliding window of the Linux implementation of TCP.
Other protocols have been learned as well, such as the MQTT protocol by \citet[TapplerAB17], TLS by \citet[RuiterP15], and SSH by \citet[Fiterau-BrosteanLPRVV17].
Many of these applications reveal bugs by learning a model and consequently apply model checking.
The combination of learning and model checking was first described by \citet[PeledVY02].
@ -147,9 +147,9 @@ The combination of learning and model checking was first described by \citet[Pel
\citet[AartsRP13] learn the software on smart cards of several Dutch and German banks.
These cards use the EMV protocol, which is run on the card itself.
So this is an example of a real black box system, where no other monitoring is possible and no code is available.
No vulnerabilities were found, although each card had a slightly different protocol.
Card readers have been learned by \citet[ChaluparPPR14].
They built a Lego machine which could automatically press buttons and the researchers found a security flaw.
No vulnerabilities were found, although each card had a slightly different state machine.
The e.dentifier, a card reader implementing a challenge-response protocol, has been learned by \citet[ChaluparPPR14].
They built a Lego machine which could automatically press buttons and the researchers found a security flaw in this card reader.
\description{Regression testing.}
\citet[HungarNS03] describe the potential of automata learning in regression testing.
@ -201,7 +201,7 @@ These abstractions are automatically refined when a counterexample is presented
\citet[Fiterau-Brostean18] extends this approach to cope with \quotation{fresh values}, crucial for protocols such as TCP.
In the thesis by \citet[Cassel15], another approach is taken.
The queries are changed to tree queries.
The approach in my thesis will be based on symmetries, which gives yet another insight into the problem of learning such automata.
The approach in my thesis will be based on symmetries, which gives yet another perspective into the problem of learning such automata.
\stopdescription
@ -211,7 +211,7 @@ The approach in my thesis will be based on symmetries, which gives yet another i
An important step in automata learning is equivalence checking.
Normally, this is abstracted away and done by an oracle, but we intend to implement such an oracle ourselves for our applications.
Concretely, the problem we need to solve is that of \emph{conformance checking}\footnote{Also known as \emph{machine verification} or \emph{fault detection}.} and this was first described by \citet[Moore56].
Concretely, the problem we need to solve is that of \emph{conformance checking}\footnote{Also known as \emph{machine verification} or \emph{fault detection}.} as it was first described by \citet[Moore56].
The problem is as follows:
Given the description of a finite state machine and a black box system, does the system behave exactly as the description?
@ -223,7 +223,7 @@ Under these conditions, \citet[Moore56] already solved the problem.
Unfortunately, his experiment is exponential in size, or in his own words: \quotation{fantastically large.}
Years later, \citet[Chow78, Vasilevskii73] independently designed efficient experiments.
In particular, the set of experiments is polynomial in size.
In particular, the set of experiments is polynomial in the number of states.
These techniques will be discussed in detail in \in{Chapter}[chap:test-methods].
More background and other related problems, as well as their complexity results, are well exposed in a survey of \citet[LeeY94].
@ -261,13 +261,13 @@ In the first (\in{Figure}{a}[fig:faulty-record-players]), the sequence \playstop
However, this is not immediately observable, the turntable is in a non-spinning state as it should be.
The fault is only visible when we press \playstop\, once more: now the turntable is spinning fast instead of slow.
The sequence \playstop\,\spin\,\spin\,\playstop\, is a \emph{counterexample}.
In the second example (\in{Figure}{b}[fig:faulty-record-players]), the fault is again not immediately obvious: after pressing \spin\,\playstop\, we are in the wrong state as observably by pressing \playstop.
In the second example (\in{Figure}{b}[fig:faulty-record-players]), the fault is again not immediately obvious: after pressing \spin\,\playstop\, we are in the wrong state as observed by pressing \playstop.
Here, the counterexample is \spin\,\playstop\,\playstop.
When a model of the implementation is given, it is not hard to find counterexamples.
However, in a black box setting we do not have such a model.
In order test whether a black box system is equivalent to a model, we somehow need to test all possible counterexamples.
In this example, a test suite should at least include the sequences \playstop\,\spin\,\spin\,\playstop\, and \spin\,\playstop\,\playstop.
In this example, a test suite should include sequences such as \playstop\,\spin\,\spin\,\playstop\, and \spin\,\playstop\,\playstop.
\startplacefigure
[title={Two faulty record players.},
@ -318,7 +318,7 @@ Usually, nominal techniques are introduced in order to solve problems involving
However, we use them in automata theory, specifically to model \emph{register automata}.
These are automata which have an infinite alphabet, often thought of as input actions with \emph{data}.
The control flow of the automaton may actually depend on the data.
However, the data cannot be used in an arbitrary way as this would lead to many problems, such as emptiness, being undecidable.
However, the data cannot be used in an arbitrary way as this would lead to many problems, such as emptiness and equivalence, being undecidable.
\footnote{The class of automata with arbitrary data operations is sometimes called \emph{extended finite state machines}.}
A principal concept in nominal techniques is that of \emph{symmetries}.
@ -349,7 +349,7 @@ We will only informally discuss its semantics for now.
(0) edge node [above, align=center] {$\kw{sign-up}(p)$ / \checkok \\ set $r := p$} (1)
(1) edge [loop below] node [below] {$\ast$ / \checknok}
(1) edge [bend left, align=center] node [above] {$\kw{login}(p)$ / \checkok \\ if $r = p$} (2)
(2) edge [loop below] node [right, align=left] {$\kw{view}()$ / \checkok \\ $\ast$ / \checknok}
(2) edge [loop below] node [right, xshift=2mm, align=left] {$\kw{view}()$ / \checkok \\ $\ast$ / \checknok}
(2) edge [bend left] node [below] {$\kw{logout}()$ / \checkok} (1);
\stoptikzpicture}
\stopplacefigure
@ -372,7 +372,9 @@ So we see that, despite the symmetry, we cannot simply identify the value \quota
For this reason, we keep the alphabet infinite and explicitly mention its symmetries.
Using symmetries in automata theory is not a new idea.
One of the first to use symmetries was \citet[Sakamoto97].
In the context of model checking, the first to use symmetries were \citet[EmersonS96, IpD96].
But only \citet[IpD96] used it to deal with infinite data domains.
For automata learning with infinite domains, symmetries were used by \citet[Sakamoto97].
He devised an \LStar{} learning algorithm for register automata, much like the one presented in \in{Chapter}[chap:learning-nominal-automata].
The symmetries are crucial to reduce the problem to a finite alphabet and use the regular \LStar{} algorithm.
(\in{Chapter}[chap:learning-nominal-automata] shows how to do it with more general symmetries.)
@ -383,7 +385,7 @@ However, nominal sets are defined in a more elementary way.
The nominal sets we will soon see are introduced by \citet[GabbayP02] to solve certain problems in name binding in abstract syntaxes.
Although this is not really related to automata theory, it was picked up by \citet[BojanczykKL14],
who provide an equivalence between register automata and nominal automata.
(This equivalence is exposed in more detail in the book of \citet[Bojanczyk18].)
(This equivalence is exposed in more detail in the book of \citenp[Bojanczyk18].)
Additionally, they generalise the work on nominal sets to other symmetries.
The symmetries we encounter in this thesis are listed below, but other symmetries can be found in the literature.
@ -457,7 +459,7 @@ However, the chapters do get more technical and mathematical -- especially in \i
Detailed discussion on related work and future directions of research are presented in each chapter.
\startcontribution[title={Chapter \ref[default][chap:test-methods]: FSM-based test methods.}]
This chapter introduces test generation methods which can be used for learning.
This chapter introduces test generation methods which can be used for learning or conformance testing.
The methods are presented in a uniform way, which allows to give a single proof of completeness for all these methods.
Moreover, the uniform presentation gives room to develop new test generation methods.
One particular contribution (besides the theoretical framework) is the new \emph{hybrid ADS} test generation method.
@ -584,7 +586,7 @@ An extended version will appear soon:
\stopcontribution
\startcontribution[title={Learning Product Automata.}]
In this article we consider Moore machines with multiple outputs.
In this article, we consider Moore machines with multiple outputs.
These machines can be decomposed by projecting on each output,
resulting in smaller components that can be learned with fewer queries.
We give experimental evidence that this is a useful technique which can reduce the number of queries substantially.
@ -601,7 +603,7 @@ This work was presented at ICGI 2018:
With the current tools for model learning, it is possible to learn big state machines of black box systems.
It involves using the clever algorithms for learning (such as the TTT algorithm by \citenp[Isberner15]) and efficient testing methods (see \in{Chapter}[chap:test-methods]).
However, as the industrial case study from \in{Chapter}[chap:applying-automata-theory] shows, the bottleneck is often in conformance testing.
However, as the industrial case study from \in{Chapter}[chap:applying-automata-learning] shows, the bottleneck is often in conformance testing.
In order to improve on this bottleneck, one possible direction is to consider \quote{grey box testing.}
The methods discussed in this thesis are all black box methods, this could be considered as \quote{too pessimistic.}
@ -621,9 +623,10 @@ Nevertheless, we show that the nominal algorithms can be implemented and that th
The advantage of using the foundations of nominal sets is that the algorithms are closely related to the original \LStar{} algorithm.
Consequently, variations of \LStar{} can easily be implemented.
For instance, we show that \NLStar{} algorithm for non-deterministic automata works in the nominal case too.
(We have not attempted to implement more recent algorithms such as TTT.)
The nominal learning algorithms can be implemented in just a few hundreds lines of code, much less than the approach taken by, e.g., \citet[Fiterau-Brostean18].
In this thesis we tackle some efficiency issues when computing with nominal sets.
In this thesis, we tackle some efficiency issues when computing with nominal sets.
In \in{Chapter}[chap:ordered-nominal-sets] we characterise orbits in order to give a efficient representation (for the total-order symmetry).
Another result is the fact that some nominal automata can be \quote{compressed} to separated automata, which can be exponentially smaller (\in{Chapter}[chap:separated-nominal-automata]).
However, the nominal tools still leave much to desired in terms of efficiency.

24
content/test-methods.tex
View file

@ -230,7 +230,7 @@ For instance, the set $\{aa, ac, c\}$ will separate $s_2$ from all other states.
[title={Sets of separating sequences}]
As the example shows, we need sets of sequences and sometimes even sets of sets of sequences -- called \emph{families}.
\footnote{A family of often written as $\{X_s\}_{s \in M}$ or simply $\{X_s\}_{s}$, meaning that for each state $s \in M$ we have a set $X_s$.}
\footnote{A family is often written as $\{X_s\}_{s \in M}$ or simply $\{X_s\}_{s}$, meaning that for each state $s \in M$ we have a set $X_s$.}
\startdefinition
We define the following kinds of sets of sequences.
@ -252,7 +252,7 @@ This extra consistency within a test suite is necessary for some test methods.
We return to this notion in more detail in \in{Example}[ex:uio-counterexample].
We may obtain a characterisation set by simply considering every pair of states and look for a difference.
However, it turns out a harmonised set of state identifiers exists for every machine and this can be constructed very efficiently (\in{Chapter}[chap:separating-sequences]).
However, it turns out a harmonised set of state identifiers exists for every machine and this can be constructed very efficiently (\in{Chapter}[chap:minimal-separating-sequences]).
From a set of state identifiers we may obtain a characterisation set by taking the union of all those sets.
\startexample
@ -356,7 +356,7 @@ Let $W$ be a set of sequences.
Two states $x, y$ are \emph{$W$-equivalent}, written $x \sim_W y$, if $\lambda(x, w) = \lambda(y, w)$ for all $w \in W$.
\item
Let $\Fam{W}$ be a family.
Two states $x, y$ are \emph{$\Fam{W}$-equivalent}, written $x \sim_{\Fam{W}} y$, if $\lambda(x, w) = \lambda(y, w)$ for all $w \in X_x \cap X_y$.
Two states $x, y$ are \emph{$\Fam{W}$-equivalent}, written $x \sim_{\Fam{W}} y$, if $\lambda(x, w) = \lambda(y, w)$ for all $w \in W_x \cap W_y$.
\stopitemize
\stopdefinition
@ -823,7 +823,8 @@ We require that if a node $u$ has children $C(u)$ then
\item the sets of states of the children of $u$ partition $l(u)$, i.e., the set
$P(u) = \{l(v) \,|\, v \in C(u)\}$ is a \emph{non-trivial} partition of $l(u)$, and
\item the sequence $\sigma(u)$ witnesses the partition $P(u)$, meaning that
$\lambda(s, \sigma(u)) = \lambda(t, \sigma(u))$ iff $p = q$ for all $s \in p, t \in q$ for all $p, q \in P(u)$.
for all $p, q \in P(u)$ we have $p = q$ iff
$\lambda(s, \sigma(u)) = \lambda(t, \sigma(u))$ for all $s \in p, t \in q$.
\stopitemize
A splitting tree is called \defn{complete} if all inequivalent states belong to different leaves.
\stopdefinition
@ -1041,7 +1042,7 @@ We classify them in two directions,
\starttheorem
[reference=thm:completeness]
Assume $M$ to be minimal and of size $n$.
Assume $M$ to be minimal, reachable, and of size $n$.
The following test suites are all $n+k$-complete:
\placetable[force,none]{}{
@ -1113,9 +1114,10 @@ A relation $R \subseteq S \times S$ is called a \defn{bisimulation} if for every
\startlemma[reference=lem:bisim]
If two states $s, t$ are related by a bisimulation, then $s \sim t$.
\footnote{The converse -- which we do not need here -- also holds, as $\sim$ is a bisimulation.}
\stoplemma
We use a slight generalisation of the bisimulation proof technique, callend \emph{bisimulation up-to}.
We use a slight generalisation of the bisimulation proof technique, called \emph{bisimulation up-to}.
This allows one to give a smaller set $R$ which extends to a bisimulation.
A good introduction of these up-to techniques is given by \citet[BonchiP15] or the thesis of \citet[Rot17].
In our case we use bisimulation \emph{up-to ${\sim}$-union}.
@ -1155,11 +1157,13 @@ then $M$ and $M'$ are equivalent.
\stopproposition
\startproof
First, we prove that $P \cdot I^{\leq k}$ reaches all states in $M'$.
For $p, q \in P$ and $x = \delta(s_0, p), y = \delta(s_0, q)$ such that $x \not\sim_{W_x \cap W_y} y$, we also have $\delta'(m'_0, p) \not\sim_{W_x \cap W_y} \delta'(m'_0, q)$ in the implementation $M'$.
For $p, q \in P$ and $x = \delta(s_0, p), y = \delta(s_0, q)$ such that $x \not\sim_{W_x \cap W_y} y$, we also have $\delta'(s'_0, p) \not\sim_{W_x \cap W_y} \delta'(s'_0, q)$ in the implementation $M'$.
By (1) this means that there are at least $n$ different behaviours in $M'$, hence at least $n$ states.
Now $n$ states are reached by the previous argument (using the set $P$).
By assumption $M'$ has at most $k$ extra states and so we can reach all those extra states by using $I^{\leq k}$ after $P$.
By assumption $M'$ has at most $k$ extra states.
If those extra states are reachable, they are reachable from an already visited state in at most $k$ steps.
So we can reach all states of $M'$ by using $I^{\leq k}$ after $P$.
Second, we show that the reached states are bisimilar.
Define the relation $R = \{(\delta(s_0, p), \delta'(s_0', p)) \mid p \in P \cdot I^{\leq k}\}$.
@ -1171,13 +1175,13 @@ For output, we note that $(s, i) \in R$ implies $\lambda(s, a) = \lambda'(i, a)$
since the machines agree on $P \cdot I^{\leq k+1}$.
For the successors, let $(s, i) \in R$ and $a \in I$ and consider the successors $s_2 = \delta(s, a)$ and $i_2 = \delta'(i, a)$.
We know that there is some $t \in M$ with $(t, i_2) \in R$.
We also know that we tested $i_2$ with the set $W'_t$.
We also know that we tested $i_2$ with the set $W_t$.
So we have:
\startformula s_2 \sim_{W'_{s_2}} i_2 \sim_{W_t} t. \stopformula
By the second assumption, we conclude that $s_2 \sim t$.
So $s_2 \sim t$ and $(t, i) \in R$, which means that $R$ is a bisimulation up-to ${\sim}$-union.
Moreover, $R$ contains the pair $(s_0, s'_0)$.
By using \in{Lemma}[lem:bisim-upto] and \in{Lemma}[lem:bisim], we conclude that the initial $s_0$ and $s'_0$ are equivalent.
By using \in{Lemmas}[lem:bisim-upto] and \in{}[lem:bisim], we conclude that the initial $s_0$ and $s'_0$ are equivalent.
\stopproof
Before we show that the conditions hold for the test methods, we reflect on the above proof first.

6
environment.tex
View file

@ -5,7 +5,7 @@
\setupexternalfigures[directory={images,../images}]
\definemode[afloop][yes]
% \definemode[afloop][yes]
% \definemode[draft][yes]
\environment bib
@ -42,10 +42,10 @@ This chapter is based on the following publication:
[headstyle=bold, alternative=serried, width=fit, indenting=yes, text={Step }]
% The black dot is missing from Euler?
\setupitemize[2,joinedup,nowhite,autointro]
\setupitemize[each][2,joinedup,nowhite,autointro][indentnext=no]
% For more whitespace between items
\defineitemgroup[bigitemize]
\setupitemgroup[bigitemize][2,autointro,indentnext]
\setupitemgroup[bigitemize][each][2,autointro][indentnext=no]
% Standaard \colon had niet veel ruimte
\define\colon{{\,{:}\,\,}}

2
environment/paper.tex
View file

@ -28,7 +28,7 @@
\setuplayout[marking=on]
% Assymetrisch
\setuppagenumbering[state=start, style=normal, location={header, right}, alternative=doublesided]
\setuppagenumbering[state=stop, style=normal, location={header, right}, alternative=doublesided]
\startmode[draft]
\setuppapersize[A5klein]