tag:blogger.com,1999:blog-39381663353946836062024-02-20T12:47:51.273+01:00Yet another developer's blogVolker Stampahttp://www.blogger.com/profile/03277007007448976514noreply@blogger.comBlogger5125tag:blogger.com,1999:blog-3938166335394683606.post-40453296491077792012014-06-19T15:28:00.000+02:002014-06-19T15:28:49.764+02:00Journal Migration<?xml version="1.0" encoding="utf-8" ?>
<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">
<html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en" lang="en">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8" />
<meta name="generator" content="Docutils 0.11: http://docutils.sourceforge.net/" />
<title></title>
<style type="text/css">
/*
:Author: David Goodger (goodger@python.org)
:Id: $Id: html4css1.css 7614 2013-02-21 15:55:51Z milde $
:Copyright: This stylesheet has been placed in the public domain.
Default cascading style sheet for the HTML output of Docutils.
See http://docutils.sf.net/docs/howto/html-stylesheets.html for how to
customize this style sheet.
*/
/* used to remove borders from tables and images */
.borderless, table.borderless td, table.borderless th {
border: 0 }
table.borderless td, table.borderless th {
/* Override padding for "table.docutils td" with "! important".
The right padding separates the table cells. */
padding: 0 0.5em 0 0 ! important }
.first {
/* Override more specific margin styles with "! important". */
margin-top: 0 ! important }
.last, .with-subtitle {
margin-bottom: 0 ! important }
.hidden {
display: none }
a.toc-backref {
text-decoration: none ;
color: black }
blockquote.epigraph {
margin: 2em 5em ; }
dl.docutils dd {
margin-bottom: 0.5em }
object[type="image/svg+xml"], object[type="application/x-shockwave-flash"] {
overflow: hidden;
}
/* Uncomment (and remove this text!) to get bold-faced definition list terms
dl.docutils dt {
font-weight: bold }
*/
div.abstract {
margin: 2em 5em }
div.abstract p.topic-title {
font-weight: bold ;
text-align: center }
div.admonition, div.attention, div.caution, div.danger, div.error,
div.hint, div.important, div.note, div.tip, div.warning {
margin: 2em ;
border: medium outset ;
padding: 1em }
div.admonition p.admonition-title, div.hint p.admonition-title,
div.important p.admonition-title, div.note p.admonition-title,
div.tip p.admonition-title {
font-weight: bold ;
font-family: sans-serif }
div.attention p.admonition-title, div.caution p.admonition-title,
div.danger p.admonition-title, div.error p.admonition-title,
div.warning p.admonition-title, .code .error {
color: red ;
font-weight: bold ;
font-family: sans-serif }
/* Uncomment (and remove this text!) to get reduced vertical space in
compound paragraphs.
div.compound .compound-first, div.compound .compound-middle {
margin-bottom: 0.5em }
div.compound .compound-last, div.compound .compound-middle {
margin-top: 0.5em }
*/
div.dedication {
margin: 2em 5em ;
text-align: center ;
font-style: italic }
div.dedication p.topic-title {
font-weight: bold ;
font-style: normal }
div.figure {
margin-left: 2em ;
margin-right: 2em }
div.footer, div.header {
clear: both;
font-size: smaller }
div.line-block {
display: block ;
margin-top: 1em ;
margin-bottom: 1em }
div.line-block div.line-block {
margin-top: 0 ;
margin-bottom: 0 ;
margin-left: 1.5em }
div.sidebar {
margin: 0 0 0.5em 1em ;
border: medium outset ;
padding: 1em ;
background-color: #ffffee ;
width: 40% ;
float: right ;
clear: right }
div.sidebar p.rubric {
font-family: sans-serif ;
font-size: medium }
div.system-messages {
margin: 5em }
div.system-messages h1 {
color: red }
div.system-message {
border: medium outset ;
padding: 1em }
div.system-message p.system-message-title {
color: red ;
font-weight: bold }
div.topic {
margin: 2em }
h1.section-subtitle, h2.section-subtitle, h3.section-subtitle,
h4.section-subtitle, h5.section-subtitle, h6.section-subtitle {
margin-top: 0.4em }
h1.title {
text-align: center }
h2.subtitle {
text-align: center }
hr.docutils {
width: 75% }
img.align-left, .figure.align-left, object.align-left {
clear: left ;
float: left ;
margin-right: 1em }
img.align-right, .figure.align-right, object.align-right {
clear: right ;
float: right ;
margin-left: 1em }
img.align-center, .figure.align-center, object.align-center {
display: block;
margin-left: auto;
margin-right: auto;
}
.align-left {
text-align: left }
.align-center {
clear: both ;
text-align: center }
.align-right {
text-align: right }
/* reset inner alignment in figures */
div.align-right {
text-align: inherit }
/* div.align-center * { */
/* text-align: left } */
ol.simple, ul.simple {
margin-bottom: 1em }
ol.arabic {
list-style: decimal }
ol.loweralpha {
list-style: lower-alpha }
ol.upperalpha {
list-style: upper-alpha }
ol.lowerroman {
list-style: lower-roman }
ol.upperroman {
list-style: upper-roman }
p.attribution {
text-align: right ;
margin-left: 50% }
p.caption {
font-style: italic }
p.credits {
font-style: italic ;
font-size: smaller }
p.label {
white-space: nowrap }
p.rubric {
font-weight: bold ;
font-size: larger ;
color: maroon ;
text-align: center }
p.sidebar-title {
font-family: sans-serif ;
font-weight: bold ;
font-size: larger }
p.sidebar-subtitle {
font-family: sans-serif ;
font-weight: bold }
p.topic-title {
font-weight: bold }
pre.address {
margin-bottom: 0 ;
margin-top: 0 ;
font: inherit }
pre.literal-block, pre.doctest-block, pre.math, pre.code {
margin-left: 2em ;
margin-right: 2em }
pre.code .ln { color: grey; } /* line numbers */
pre.code, code { background-color: #eeeeee }
pre.code .comment, code .comment { color: #5C6576 }
pre.code .keyword, code .keyword { color: #3B0D06; font-weight: bold }
pre.code .literal.string, code .literal.string { color: #0C5404 }
pre.code .name.builtin, code .name.builtin { color: #352B84 }
pre.code .deleted, code .deleted { background-color: #DEB0A1}
pre.code .inserted, code .inserted { background-color: #A3D289}
span.classifier {
font-family: sans-serif ;
font-style: oblique }
span.classifier-delimiter {
font-family: sans-serif ;
font-weight: bold }
span.interpreted {
font-family: sans-serif }
span.option {
white-space: nowrap }
span.pre {
white-space: pre }
span.problematic {
color: red }
span.section-subtitle {
/* font-size relative to parent (h1..h6 element) */
font-size: 80% }
table.citation {
border-left: solid 1px gray;
margin-left: 1px }
table.docinfo {
margin: 2em 4em }
table.docutils {
margin-top: 0.5em ;
margin-bottom: 0.5em }
table.footnote {
border-left: solid 1px black;
margin-left: 1px }
table.docutils td, table.docutils th,
table.docinfo td, table.docinfo th {
padding-left: 0.5em ;
padding-right: 0.5em ;
vertical-align: top }
table.docutils th.field-name, table.docinfo th.docinfo-name {
font-weight: bold ;
text-align: left ;
white-space: nowrap ;
padding-left: 0 }
/* "booktabs" style (no vertical lines) */
table.docutils.booktabs {
border: 0px;
border-top: 2px solid;
border-bottom: 2px solid;
border-collapse: collapse;
}
table.docutils.booktabs * {
border: 0px;
}
table.docutils.booktabs th {
border-bottom: thin solid;
text-align: left;
}
h1 tt.docutils, h2 tt.docutils, h3 tt.docutils,
h4 tt.docutils, h5 tt.docutils, h6 tt.docutils {
font-size: 100% }
ul.auto-toc {
list-style-type: none }
</style>
<style type="text/css">
/* example stylesheet for Docutils */
/* :Author: Günter Milde */
/* :Copyright: © 2012 G. Milde */
/* :License: This stylesheet is placed in the public domain. */
/* Syntax highlight rules for HTML documents generated with Docutils */
/* using the ``--syntax-highlight=long`` option (new in v. 0.9). */
/* This stylesheet implements Pygment's "default" style with less rules than */
/* pygments-default using class hierarchies. */
/* Use it as example for "handcrafted" styles with only few rules. */
.code { background: #f8f8f8; }
.code .comment { color: #008800; font-style: italic }
.code .error { border: 1px solid #FF0000 }
.code .generic.deleted { color: #A00000 }
.code .generic.emph { font-style: italic }
.code .generic.error { color: #FF0000 }
.code .generic.heading { color: #000080; font-weight: bold }
.code .generic.inserted { color: #00A000 }
.code .generic.output { color: #808080 }
.code .generic.prompt { color: #000080; font-weight: bold }
.code .generic.strong { font-weight: bold }
.code .generic.subheading { color: #800080; font-weight: bold }
.code .generic.traceback { color: #0040D0 }
.code .keyword { color: #AA22FF; font-weight: bold }
.code .keyword.pseudo { font-weight: normal }
.code .literal.number { color: #666666 }
.code .literal.string { color: #BB4444 }
.code .literal.string.doc { color: #BB4444; font-style: italic }
.code .literal.string.escape { color: #BB6622; font-weight: bold }
.code .literal.string.interpol { color: #BB6688; font-weight: bold }
.code .literal.string.other { color: #008000 }
.code .literal.string.regex { color: #BB6688 }
.code .literal.string.symbol { color: #B8860B }
.code .name.attribute { color: #BB4444 }
.code .name.builtin { color: #AA22FF }
.code .name.class { color: #0000FF }
.code .name.constant { color: #880000 }
.code .name.decorator { color: #AA22FF }
.code .name.entity { color: #999999; font-weight: bold }
.code .name.exception { color: #D2413A; font-weight: bold }
.code .name.function { color: #00A000 }
.code .name.label { color: #A0A000 }
.code .name.namespace { color: #0000FF; font-weight: bold }
.code .name.tag { color: #008000; font-weight: bold }
.code .name.variable { color: #B8860B }
.code .operator { color: #666666 }
.code .operator.word { color: #AA22FF; font-weight: bold }
.code .ln { margin-right: -0px }
</style>
</head>
<body>
<div class="document">
<p>The last two posts were about testing recovery of the application state of an
<a class="reference external" href="http://doc.akka.io/docs/akka/2.3.3/scala/persistence.html">akka-persistence</a> based application from a <a class="reference external" href="/2014/05/testing-recovery.html">journal</a> or a
<a class="reference external" href="/2014_05_01_archive.html">snapshot</a>. This post concludes the
series of posts on testing akka-persistence based application with some thoughts on testing
migration of journals.</p>
<div class="section" id="journal-migration">
<h1>Journal Migration</h1>
<p>The migration problem has already been addressed in the first <a class="reference external" href="/2014/04/akka-persistence-and-testing.html">post</a>
of this series. When the application evolves, the objects that are written to the
journal evolve and thus their serialized representation has to change, too. To ensure that
even if the evolved objects have a new serialized representation the deserialization
is still able to read and process the old representation, custom akka-serializers are applied.
So the idea is to maintain compatibility to existing journals and to restore the changed command-objects
from the old serialized representation. Alternatively one could write a migration-tool that
reads an existing journal and writes the content in the new serialization format. While this
post focuses on the first approach the testing principles can be applied to both.</p>
<div class="section" id="snapshot-migration">
<h2>Snapshot Migration</h2>
<p>Before we get into details, one word about migration of snapshots. In principle
we have the same problems here as in case of journals. As the application evolves, the
state of processors evolve and so does their serialized representation, and one
has to make sure that the evolved application is still able to read existing snapshots.
Like for the journal akka-persistence uses <a class="reference external" href="http://doc.akka.io/docs/akka/2.3.2/scala/serialization.html">akka-serialization</a> when it comes to writing
the snapshot to the storage and that is why it makes sense to employ custom
serializers for maintaining compatibility between versions. However, there is an
important difference between the journal and snapshots. Snapshots can be considered a pure
performance optimization and are not important from a functional point of view for a
successful recovery of application state. That is why it might be a valid alternative
to do without custom serializers and backwards compatibility in case of snapshots.
Incompatible snapshots could simply be deleted in case of an upgrade. Of course this results
in a longer
recovery time for the first restart after an upgrade.</p>
<p>Because of this and because of the fact that maintaining backwards compatibility
for snapshots is a very similar challenge (for implementation and test) as in case of
journals, this post only considers migration of journals.</p>
</div>
<div class="section" id="general-idea">
<h2>General Idea</h2>
<p>Once again the idea is to reuse as much of the existing tests as possible. Basically
we want to know if the test for recovery from a journal still works even when an
old journal is used. So instead of writing the journal and recover from it in a single test-run,
we rather save the journals produced by the tests when development has reached a state
compatibility has to be maintained to (e.g. right after a release) and make the
test read from this journal when testing recovery instead of reading from a journal produced in the same test.</p>
<p>Let's have a quick look at one of the recover-tests:</p>
<pre class="code scala literal-block">
<span class="ln"> 1 </span> <span class="keyword">val</span> <span class="name">created</span> <span class="keyword">=</span> <span class="name">startApplication</span><span class="operator">(</span><span class="name">persistDir</span><span class="operator">)</span> <span class="operator">{</span> <span class="name">application</span> <span class="keyword">=></span>
<span class="ln"> 2 </span> <span class="name">application</span><span class="operator">.</span><span class="name">itemServiceTestExtension</span><span class="operator">.</span><span class="name">createNewItem</span><span class="operator">()</span>
<span class="ln"> 3 </span> <span class="operator">}</span>
<span class="ln"> 4 </span> <span class="name">restartApplication</span><span class="operator">(</span><span class="name">persistDir</span><span class="operator">)</span> <span class="operator">{</span> <span class="name">application</span> <span class="keyword">=></span>
<span class="ln"> 5 </span> <span class="name">application</span><span class="operator">.</span><span class="name">itemServiceTestExtension</span><span class="operator">.</span><span class="name">findItem</span><span class="operator">(</span><span class="name">created</span><span class="operator">.</span><span class="name">id</span><span class="operator">)</span> <span class="name">should</span> <span class="name">be</span> <span class="operator">(</span>
<span class="ln"> 6 </span> <span class="name class">Some</span><span class="operator">(</span><span class="name">created</span><span class="operator">))</span>
<span class="ln"> 7 </span> <span class="operator">}</span>
</pre>
<p>In lines 1-3 the journal is produced and the relevant state is kept locally in <tt class="docutils literal">created</tt>.
For our migration-test we do not need to produce a journal as this has been saved before.
We rather just need
to initialize <tt class="docutils literal">created</tt> with a value that corresponds to the saved journal.</p>
<p>In lines 4-7 the application is recovered. For our migration-test we need to pass in
the folder that contains the saved journal.</p>
</div>
<div class="section" id="saving-old-journals">
<h2>Saving old Journals</h2>
<p>So the first challenge is to write a tool that saves the journals produced by the
tests to a dedicated place. You may have noticed that the test-code above differs slightly
from the initial-version shown in the post about testing recovery. The <tt class="docutils literal">withApplication</tt>
used before for both creating the journal and testing recovery are replaced by <tt class="docutils literal">startApplication</tt> (line 1)
and <tt class="docutils literal">restartApplication</tt> (line 4) respectively. This allows us to inject different actions
in either case.</p>
<p>For our use-case to save the journal to a dedicated place we simply copy the journal
from the temporary folder to the dedicated place after the first application shutdown.
For this we create a trait <tt class="docutils literal">WithItemApplicationWithSaveJournal</tt> extending the well known
<tt class="docutils literal">WithItemApplication</tt> and overwriting <tt class="docutils literal">startApplication</tt> like this:</p>
<pre class="code scala literal-block">
<span class="ln"> 1 </span> <span class="keyword">override</span> <span class="keyword">def</span> <span class="name">startApplication</span><span class="operator">[</span><span class="keyword type">A</span><span class="operator">](</span><span class="name">persistDir</span><span class="keyword">:</span> <span class="keyword type">File</span><span class="operator">)(</span><span class="name">block</span><span class="keyword">:</span> <span class="keyword type">TestApplication</span> <span class="operator">=></span> <span class="name">A</span><span class="operator">)</span><span class="keyword">:</span> <span class="keyword type">A</span> <span class="operator">=</span> <span class="operator">{</span>
<span class="ln"> 2 </span> <span class="keyword">val</span> <span class="name">result</span> <span class="keyword">=</span> <span class="keyword">super</span><span class="operator">.</span><span class="name">startApplication</span><span class="operator">(</span><span class="name">persistDir</span><span class="operator">)(</span><span class="name">block</span><span class="operator">)</span>
<span class="ln"> 3 </span> <span class="name">saveData</span><span class="operator">(</span><span class="name">persistDir</span><span class="operator">,</span> <span class="name">result</span><span class="operator">)</span>
<span class="ln"> 4 </span> <span class="name">result</span>
<span class="ln"> 5 </span> <span class="operator">}</span>
<span class="ln"> 6 </span>
<span class="ln"> 7 </span> <span class="keyword">def</span> <span class="name">saveData</span><span class="operator">[</span><span class="keyword type">A</span><span class="operator">](</span><span class="name">persistDir</span><span class="keyword">:</span> <span class="keyword type">File</span><span class="operator">,</span> <span class="name">result</span><span class="keyword">:</span> <span class="keyword type">A</span><span class="operator">)</span><span class="keyword">:</span> <span class="keyword type">Unit</span> <span class="operator">=</span> <span class="operator">{</span>
<span class="ln"> 8 </span> <span class="keyword">val</span> <span class="name">destinationDir</span> <span class="keyword">=</span> <span class="name">migrationDataDir</span>
<span class="ln"> 9 </span> <span class="keyword">if</span><span class="operator">(</span><span class="name">destinationDir</span><span class="operator">.</span><span class="name">isDirectory</span><span class="operator">)</span>
<span class="ln">10 </span> <span class="name class">FileUtils</span><span class="operator">.</span><span class="name">deleteDirectoryContents</span><span class="operator">(</span><span class="name">destinationDir</span><span class="operator">)</span>
<span class="ln">11 </span> <span class="name class">FileUtils</span><span class="operator">.</span><span class="name">copyDirectoryContents</span><span class="operator">(</span><span class="name">persistDir</span><span class="operator">,</span> <span class="name">destinationDir</span><span class="operator">)</span>
<span class="ln">12 </span> <span class="name class">Files</span><span class="operator">.</span><span class="name">write</span><span class="operator">(</span><span class="name">result</span><span class="operator">.</span><span class="name">toString</span><span class="operator">,</span> <span class="keyword">new</span> <span class="name class">File</span><span class="operator">(</span><span class="name">destinationDir</span><span class="operator">,</span> <span class="literal string">"expected.txt"</span><span class="operator">),</span>
<span class="ln">13 </span> <span class="name class">Charsets</span><span class="operator">.</span><span class="name class">UTF_8</span><span class="operator">)</span>
<span class="ln">14 </span> <span class="operator">}</span>
</pre>
<p>When <tt class="docutils literal">startApplication</tt> (line 2) returns, the application has not only been started, but the test-code (<tt class="docutils literal">block</tt>)
has been executed and the application is shutdown. Before it simply returns the result (line 4),
it copies the temporary folder containing the journal (<tt class="docutils literal">persistDir</tt>) to a dedicated place (line 3).
In addition to this it writes a text file (<em>expected.txt</em>) to this folder with a
string-representation of the result (line 12-13). We will see below how this helps
us to prepare the <em>relevant state</em> for verification.</p>
<p>The dedicated place is
<tt class="docutils literal">migrationDataDir</tt> (line 8) which computes to the folder <em>src/test/saved-journals/<id>/<test-class-name>/<test-name>/</em>.
The variables in this path are basically provided by a dummy-test that derives from
the original <tt class="docutils literal">ItemApplicationRecoverSpec</tt> and mixes in the trait <tt class="docutils literal">WithItemApplicationWithSaveJournal</tt> like follows:</p>
<pre class="code scala literal-block">
<span class="ln">1 </span><span class="keyword">class</span> <span class="name class">SaveItemApplicationJournal</span>
<span class="ln">2 </span> <span class="operator">(</span><span class="keyword">protected</span> <span class="keyword">val</span> <span class="name">runId</span><span class="keyword">:</span> <span class="keyword type">String</span><span class="operator">)</span>
<span class="ln">3 </span> <span class="operator">(</span><span class="keyword">implicit</span> <span class="keyword">val</span> <span class="name">tag</span><span class="keyword">:</span> <span class="keyword type">ClassTag</span><span class="operator">[</span><span class="keyword type">ItemApplicationRecoverSpec</span><span class="operator">])</span>
<span class="ln">4 </span> <span class="keyword">extends</span> <span class="name class">ItemApplicationRecoverSpec</span> <span class="keyword">with</span> <span class="name class">JournalMigrationRecoverSource</span>
<span class="ln">5 </span> <span class="keyword">with</span> <span class="name class">WithItemApplicationWithSaveJournal</span><span class="operator">[</span><span class="keyword type">ItemApplicationRecoverSpec</span><span class="operator">]</span>
</pre>
<p><em><id></em> becomes <tt class="docutils literal">runId</tt> (line 2), <em><test-class-name></em> is determined by <tt class="docutils literal">tag</tt> (line 3) and
<em><test-name></em> stands for the names of the individual tests in <tt class="docutils literal">ItemApplicationRecoverSpec</tt>. This dummy
test is run by a dedicated scala-application:</p>
<pre class="code scala literal-block">
<span class="ln">1 </span><span class="keyword">object</span> <span class="name class">SaveJournalApp</span> <span class="keyword">extends</span> <span class="name class">App</span> <span class="operator">{</span>
<span class="ln">2 </span> <span class="keyword">val</span> <span class="name">runId</span> <span class="keyword">=</span> <span class="name">args</span><span class="operator">.</span><span class="name">headOption</span><span class="operator">.</span><span class="name">getOrElse</span><span class="operator">(</span><span class="literal string">"generic"</span><span class="operator">)</span>
<span class="ln">3 </span>
<span class="ln">4 </span> <span class="name">run</span><span class="operator">(</span><span class="keyword">new</span> <span class="name class">SaveItemApplicationJournal</span><span class="operator">(</span><span class="name">runId</span><span class="operator">))</span>
<span class="ln">5 </span><span class="operator">}</span>
</pre>
<p>This application simply instantiates the dummy-test with a certain <tt class="docutils literal">runId</tt> and
runs it using a scala-test test-runner (line 4). It could for example be used after each release
of the application and actually also be included in a release-process. In that case
the <tt class="docutils literal">runId</tt> could be the version of the application. In our case it produces three folders,
one for each test in <tt class="docutils literal">ItemApplicationRecoverSpec</tt>, containing the journal
created by the respective test and the file <em>expected.txt</em> containing the
string-representation of the value returned by <tt class="docutils literal">startApplication</tt>. So for our three
tests these <em>expected.txt</em> files contain the created, updated or deleted item.</p>
<p>The files produced by <tt class="docutils literal">SaveJournalApp</tt> should be added to the source-control
as resources required to run tests as we cannot easily reproduce those
files once the application evolves.</p>
</div>
<div class="section" id="running-tests-against-old-journals">
<h2>Running tests against old journals</h2>
<p>The next step is to execute the test of <tt class="docutils literal">ItemApplicationRecoverSpec</tt> while making
sure that it reads the saved journals when restarting the application. Once again
we basically just need to provide another variation of the <tt class="docutils literal">WithItemApplication</tt>
trait for this:</p>
<pre class="code scala literal-block">
<span class="ln"> 1 </span> <span class="keyword">protected</span> <span class="keyword">def</span> <span class="name">expectedValueFor</span><span class="operator">(</span><span class="name">test</span><span class="keyword">:</span> <span class="keyword type">TestData</span><span class="operator">)</span><span class="keyword">:</span> <span class="keyword type">Any</span>
<span class="ln"> 2 </span>
<span class="ln"> 3 </span> <span class="keyword">override</span> <span class="keyword">def</span> <span class="name">startApplication</span><span class="operator">[</span><span class="keyword type">A</span><span class="operator">](</span><span class="name">persistDir</span><span class="keyword">:</span> <span class="keyword type">File</span><span class="operator">)(</span><span class="name">block</span><span class="keyword">:</span> <span class="operator">(</span><span class="keyword type">TestApplication</span><span class="operator">)</span> <span class="operator">=></span> <span class="name">A</span><span class="operator">)</span><span class="keyword">:</span> <span class="keyword type">A</span> <span class="operator">=</span>
<span class="ln"> 4 </span> <span class="name">expectedValueFor</span><span class="operator">(</span><span class="name">currentTest</span><span class="operator">).</span><span class="name">asInstanceOf</span><span class="operator">[</span><span class="keyword type">A</span><span class="operator">]</span>
<span class="ln"> 5 </span>
<span class="ln"> 6 </span> <span class="keyword">override</span> <span class="keyword">def</span> <span class="name">restartApplication</span><span class="operator">[</span><span class="keyword type">A</span><span class="operator">](</span><span class="name">persistDir</span><span class="keyword">:</span> <span class="keyword type">File</span><span class="operator">)(</span><span class="name">block</span><span class="keyword">:</span> <span class="operator">(</span><span class="keyword type">TestApplication</span><span class="operator">)</span> <span class="operator">=></span> <span class="name">A</span><span class="operator">)</span><span class="keyword">:</span> <span class="keyword type">A</span> <span class="operator">=</span>
<span class="ln"> 7 </span> <span class="keyword">super</span><span class="operator">.</span><span class="name">restartApplication</span><span class="operator">(</span><span class="name">migrationDataDir</span><span class="operator">)(</span><span class="name">block</span><span class="operator">)</span>
</pre>
<p>This overwrites <tt class="docutils literal">startApplication</tt> and <tt class="docutils literal">restartApplication</tt>. In case of
<tt class="docutils literal">startApplication</tt> it simply returns prepared values without executing the
test-code (<tt class="docutils literal">block</tt>) (line 4). These values have to be provided by the test-implementation
in form of the method
<tt class="docutils literal">expectedValueFor</tt> (line 1).</p>
<p><tt class="docutils literal">restartApplication</tt> executes the <tt class="docutils literal">block</tt> as usual, however instead
of providing the temporary folder <tt class="docutils literal">persistDir</tt> it passes the folder with the
previously saved journal <tt class="docutils literal">migrationDataDir</tt> (line 7).</p>
<p>Having this, a migration test basically
just needs to provide the expected values and the metadata required to find the
correct folder containing the previously saved journal:</p>
<pre class="code scala literal-block">
<span class="ln"> 1 </span> <span class="keyword">protected</span> <span class="keyword">def</span> <span class="name">runId</span> <span class="keyword">=</span> <span class="literal string">"0.1"</span>
<span class="ln"> 2 </span>
<span class="ln"> 3 </span> <span class="keyword">protected</span> <span class="keyword">def</span> <span class="name">tag</span> <span class="keyword">=</span> <span class="name">classTag</span><span class="operator">[</span><span class="keyword type">ItemApplicationRecoverSpec</span><span class="operator">]</span>
<span class="ln"> 4 </span>
<span class="ln"> 5 </span> <span class="keyword">protected</span> <span class="keyword">val</span> <span class="name class">Id</span> <span class="keyword">=</span> <span class="name class">ItemId</span><span class="operator">(</span><span class="literal number integer">1</span><span class="operator">)</span>
<span class="ln"> 6 </span> <span class="keyword">protected</span> <span class="keyword">def</span> <span class="name">description</span><span class="operator">(</span><span class="name">d</span><span class="keyword">:</span> <span class="keyword type">Long</span><span class="operator">)</span> <span class="keyword">=</span> <span class="name">s</span><span class="literal string">"$d - description"</span>
<span class="ln"> 7 </span>
<span class="ln"> 8 </span> <span class="keyword">protected</span> <span class="keyword">val</span> <span class="name">expectedValues</span><span class="keyword">:</span> <span class="keyword type">Iterator</span><span class="operator">[</span><span class="keyword type">Any</span><span class="operator">]</span> <span class="keyword">=</span> <span class="name class">Iterator</span><span class="operator">(</span>
<span class="ln"> 9 </span> <span class="name class">Item</span><span class="operator">(</span><span class="name class">Id</span><span class="operator">,</span> <span class="name">description</span><span class="operator">(</span><span class="literal number integer">1</span><span class="operator">)),</span>
<span class="ln">10 </span> <span class="name class">Item</span><span class="operator">(</span><span class="name class">Id</span><span class="operator">,</span> <span class="name">description</span><span class="operator">(</span><span class="literal number integer">3</span><span class="operator">)),</span>
<span class="ln">11 </span> <span class="name class">Item</span><span class="operator">(</span><span class="name class">Id</span><span class="operator">,</span> <span class="name">description</span><span class="operator">(</span><span class="literal number integer">4</span><span class="operator">)))</span>
<span class="ln">12 </span>
<span class="ln">13 </span> <span class="keyword">override</span> <span class="keyword">protected</span> <span class="keyword">def</span> <span class="name">expectedValueFor</span><span class="operator">(</span><span class="name">test</span><span class="keyword">:</span> <span class="keyword type">TestData</span><span class="operator">)</span><span class="keyword">:</span> <span class="keyword type">Any</span> <span class="operator">=</span>
<span class="ln">14 </span> <span class="name">expectedValues</span><span class="operator">.</span><span class="name">next</span><span class="operator">()</span>
</pre>
<p><tt class="docutils literal">runId</tt> (line 1) and <tt class="docutils literal">tag</tt> (line 3) represent this metadata for the folder-name. As you
can see, the method <tt class="docutils literal">expectedValueFor</tt> gets a scala-test
<a class="reference external" href="http://doc.scalatest.org/2.1.5/index.html#org.scalatest.TestData">TestData</a> instance (line 13) which
allows it to pick the right expected value. For simplicity reasons the implementation here ignores it
and relies on the order in which the tests are executed (line 14). The actual values (line 9-11)
can be derived from the <em>expected.txt</em> files that were generated along with the saved journals.</p>
</div>
<div class="section" id="test-in-action">
<h2>Test in Action</h2>
<p>So just like in case of the snapshot-tests we are able to provide migration tests
while fully reusing the existing test-logic. Now lets see this test in action. After the
journals have been saved by running <tt class="docutils literal">SaveJournalApp</tt>, we modify the domain
model by adding the optional field <tt class="docutils literal">rank: Option[Int] = None</tt> defaulting to <tt class="docutils literal">None</tt>
to <tt class="docutils literal">Item</tt> and <tt class="docutils literal">ItemDescription</tt>. Thanks to the <a class="reference external" href="http://www.playframework.com/documentation/2.2.x/ScalaJson">play-json</a> <a class="reference external" href="http://www.playframework.com/documentation/2.2.x/ScalaJsonInception">macro</a>
-based serialization that we used
for the custom serializers they basically adapt to these changes automagically. Of course
we expect that the <tt class="docutils literal">Item</tt>s recovered from an old journal have the field initialized to
<tt class="docutils literal">None</tt> and as this is also the default we do not even have to change the migration test
and can simply rerun it without any further changes and indeed it runs through just fine.</p>
<p>However if we break compatibility of
the macro-based serializer for example by adding a field <tt class="docutils literal">rank: Int = 0</tt>, the test fails accordingly:</p>
<pre class="literal-block">
akka.persistence.RecoveryException:
Recovery failure by journal (processor id = [/user/ItemActor])
...
Caused by: play.api.libs.json.JsResultException:
JsResultException(errors:List((/template/rank,
List(ValidationError(error.path.missing,WrappedArray())))))
</pre>
<p>So for this kind of change we would have to provide a json-serializer that
initializes missing <tt class="docutils literal">rank</tt>-fields with 0.</p>
</div>
<div class="section" id="conclusion">
<h2>Conclusion</h2>
<p>There is one thing to keep in mind when it comes to reusing a recovery test for
this kind of migration-test. Once the <em>original</em> recovery test evolves, it might
no longer be a suitable basis for the saved journals. Even just adding test-cases is
not a good idea as there is no <em>old</em> journal for these new test-cases. So the
migration-test and the original recovery-test might diverge when the development
continues and you have to find a suitable strategy to avoid as much redundancy as possible.</p>
<p>This post concludes the series of posts about akka-persistence and testing. It showed
that it is possible to re-use the same test for testing three different kind of
recovery scenarios: recovery from a journal, a snapshot and an <em>old</em> journal
of the previous release.</p>
</div>
</div>
</div>
</body>
</html>
Volker Stampahttp://www.blogger.com/profile/03277007007448976514noreply@blogger.com0tag:blogger.com,1999:blog-3938166335394683606.post-73208472406260842852014-05-29T15:03:00.001+02:002014-05-29T15:09:34.763+02:00Testing Recovery through Snapshots<?xml version="1.0" encoding="utf-8" ?>
<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">
<html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en" lang="en">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8" />
<meta name="generator" content="Docutils 0.11: http://docutils.sourceforge.net/" />
<title></title>
<style type="text/css">
/*
:Author: David Goodger (goodger@python.org)
:Id: $Id: html4css1.css 7614 2013-02-21 15:55:51Z milde $
:Copyright: This stylesheet has been placed in the public domain.
Default cascading style sheet for the HTML output of Docutils.
See http://docutils.sf.net/docs/howto/html-stylesheets.html for how to
customize this style sheet.
*/
/* used to remove borders from tables and images */
.borderless, table.borderless td, table.borderless th {
border: 0 }
table.borderless td, table.borderless th {
/* Override padding for "table.docutils td" with "! important".
The right padding separates the table cells. */
padding: 0 0.5em 0 0 ! important }
.first {
/* Override more specific margin styles with "! important". */
margin-top: 0 ! important }
.last, .with-subtitle {
margin-bottom: 0 ! important }
.hidden {
display: none }
a.toc-backref {
text-decoration: none ;
color: black }
blockquote.epigraph {
margin: 2em 5em ; }
dl.docutils dd {
margin-bottom: 0.5em }
object[type="image/svg+xml"], object[type="application/x-shockwave-flash"] {
overflow: hidden;
}
/* Uncomment (and remove this text!) to get bold-faced definition list terms
dl.docutils dt {
font-weight: bold }
*/
div.abstract {
margin: 2em 5em }
div.abstract p.topic-title {
font-weight: bold ;
text-align: center }
div.admonition, div.attention, div.caution, div.danger, div.error,
div.hint, div.important, div.note, div.tip, div.warning {
margin: 2em ;
border: medium outset ;
padding: 1em }
div.admonition p.admonition-title, div.hint p.admonition-title,
div.important p.admonition-title, div.note p.admonition-title,
div.tip p.admonition-title {
font-weight: bold ;
font-family: sans-serif }
div.attention p.admonition-title, div.caution p.admonition-title,
div.danger p.admonition-title, div.error p.admonition-title,
div.warning p.admonition-title, .code .error {
color: red ;
font-weight: bold ;
font-family: sans-serif }
/* Uncomment (and remove this text!) to get reduced vertical space in
compound paragraphs.
div.compound .compound-first, div.compound .compound-middle {
margin-bottom: 0.5em }
div.compound .compound-last, div.compound .compound-middle {
margin-top: 0.5em }
*/
div.dedication {
margin: 2em 5em ;
text-align: center ;
font-style: italic }
div.dedication p.topic-title {
font-weight: bold ;
font-style: normal }
div.figure {
margin-left: 2em ;
margin-right: 2em }
div.footer, div.header {
clear: both;
font-size: smaller }
div.line-block {
display: block ;
margin-top: 1em ;
margin-bottom: 1em }
div.line-block div.line-block {
margin-top: 0 ;
margin-bottom: 0 ;
margin-left: 1.5em }
div.sidebar {
margin: 0 0 0.5em 1em ;
border: medium outset ;
padding: 1em ;
background-color: #ffffee ;
width: 40% ;
float: right ;
clear: right }
div.sidebar p.rubric {
font-family: sans-serif ;
font-size: medium }
div.system-messages {
margin: 5em }
div.system-messages h1 {
color: red }
div.system-message {
border: medium outset ;
padding: 1em }
div.system-message p.system-message-title {
color: red ;
font-weight: bold }
div.topic {
margin: 2em }
h1.section-subtitle, h2.section-subtitle, h3.section-subtitle,
h4.section-subtitle, h5.section-subtitle, h6.section-subtitle {
margin-top: 0.4em }
h1.title {
text-align: center }
h2.subtitle {
text-align: center }
hr.docutils {
width: 75% }
img.align-left, .figure.align-left, object.align-left {
clear: left ;
float: left ;
margin-right: 1em }
img.align-right, .figure.align-right, object.align-right {
clear: right ;
float: right ;
margin-left: 1em }
img.align-center, .figure.align-center, object.align-center {
display: block;
margin-left: auto;
margin-right: auto;
}
.align-left {
text-align: left }
.align-center {
clear: both ;
text-align: center }
.align-right {
text-align: right }
/* reset inner alignment in figures */
div.align-right {
text-align: inherit }
/* div.align-center * { */
/* text-align: left } */
ol.simple, ul.simple {
margin-bottom: 1em }
ol.arabic {
list-style: decimal }
ol.loweralpha {
list-style: lower-alpha }
ol.upperalpha {
list-style: upper-alpha }
ol.lowerroman {
list-style: lower-roman }
ol.upperroman {
list-style: upper-roman }
p.attribution {
text-align: right ;
margin-left: 50% }
p.caption {
font-style: italic }
p.credits {
font-style: italic ;
font-size: smaller }
p.label {
white-space: nowrap }
p.rubric {
font-weight: bold ;
font-size: larger ;
color: maroon ;
text-align: center }
p.sidebar-title {
font-family: sans-serif ;
font-weight: bold ;
font-size: larger }
p.sidebar-subtitle {
font-family: sans-serif ;
font-weight: bold }
p.topic-title {
font-weight: bold }
pre.address {
margin-bottom: 0 ;
margin-top: 0 ;
font: inherit }
pre.literal-block, pre.doctest-block, pre.math, pre.code {
margin-left: 2em ;
margin-right: 2em }
pre.code .ln { color: grey; } /* line numbers */
pre.code, code { background-color: #eeeeee }
pre.code .comment, code .comment { color: #5C6576 }
pre.code .keyword, code .keyword { color: #3B0D06; font-weight: bold }
pre.code .literal.string, code .literal.string { color: #0C5404 }
pre.code .name.builtin, code .name.builtin { color: #352B84 }
pre.code .deleted, code .deleted { background-color: #DEB0A1}
pre.code .inserted, code .inserted { background-color: #A3D289}
span.classifier {
font-family: sans-serif ;
font-style: oblique }
span.classifier-delimiter {
font-family: sans-serif ;
font-weight: bold }
span.interpreted {
font-family: sans-serif }
span.option {
white-space: nowrap }
span.pre {
white-space: pre }
span.problematic {
color: red }
span.section-subtitle {
/* font-size relative to parent (h1..h6 element) */
font-size: 80% }
table.citation {
border-left: solid 1px gray;
margin-left: 1px }
table.docinfo {
margin: 2em 4em }
table.docutils {
margin-top: 0.5em ;
margin-bottom: 0.5em }
table.footnote {
border-left: solid 1px black;
margin-left: 1px }
table.docutils td, table.docutils th,
table.docinfo td, table.docinfo th {
padding-left: 0.5em ;
padding-right: 0.5em ;
vertical-align: top }
table.docutils th.field-name, table.docinfo th.docinfo-name {
font-weight: bold ;
text-align: left ;
white-space: nowrap ;
padding-left: 0 }
/* "booktabs" style (no vertical lines) */
table.docutils.booktabs {
border: 0px;
border-top: 2px solid;
border-bottom: 2px solid;
border-collapse: collapse;
}
table.docutils.booktabs * {
border: 0px;
}
table.docutils.booktabs th {
border-bottom: thin solid;
text-align: left;
}
h1 tt.docutils, h2 tt.docutils, h3 tt.docutils,
h4 tt.docutils, h5 tt.docutils, h6 tt.docutils {
font-size: 100% }
ul.auto-toc {
list-style-type: none }
</style>
<style type="text/css">
/* example stylesheet for Docutils */
/* :Author: Günter Milde */
/* :Copyright: © 2012 G. Milde */
/* :License: This stylesheet is placed in the public domain. */
/* Syntax highlight rules for HTML documents generated with Docutils */
/* using the ``--syntax-highlight=long`` option (new in v. 0.9). */
/* This stylesheet implements Pygment's "default" style with less rules than */
/* pygments-default using class hierarchies. */
/* Use it as example for "handcrafted" styles with only few rules. */
.code { background: #f8f8f8; }
.code .comment { color: #008800; font-style: italic }
.code .error { border: 1px solid #FF0000 }
.code .generic.deleted { color: #A00000 }
.code .generic.emph { font-style: italic }
.code .generic.error { color: #FF0000 }
.code .generic.heading { color: #000080; font-weight: bold }
.code .generic.inserted { color: #00A000 }
.code .generic.output { color: #808080 }
.code .generic.prompt { color: #000080; font-weight: bold }
.code .generic.strong { font-weight: bold }
.code .generic.subheading { color: #800080; font-weight: bold }
.code .generic.traceback { color: #0040D0 }
.code .keyword { color: #AA22FF; font-weight: bold }
.code .keyword.pseudo { font-weight: normal }
.code .literal.number { color: #666666 }
.code .literal.string { color: #BB4444 }
.code .literal.string.doc { color: #BB4444; font-style: italic }
.code .literal.string.escape { color: #BB6622; font-weight: bold }
.code .literal.string.interpol { color: #BB6688; font-weight: bold }
.code .literal.string.other { color: #008000 }
.code .literal.string.regex { color: #BB6688 }
.code .literal.string.symbol { color: #B8860B }
.code .name.attribute { color: #BB4444 }
.code .name.builtin { color: #AA22FF }
.code .name.class { color: #0000FF }
.code .name.constant { color: #880000 }
.code .name.decorator { color: #AA22FF }
.code .name.entity { color: #999999; font-weight: bold }
.code .name.exception { color: #D2413A; font-weight: bold }
.code .name.function { color: #00A000 }
.code .name.label { color: #A0A000 }
.code .name.namespace { color: #0000FF; font-weight: bold }
.code .name.tag { color: #008000; font-weight: bold }
.code .name.variable { color: #B8860B }
.code .operator { color: #666666 }
.code .operator.word { color: #AA22FF; font-weight: bold }
.code .ln { margin-right: -0px }
</style>
</head>
<body>
<div class="document">
<p>In the last <a class="reference external" href="/2014/05/testing-recovery.html">post</a> I explained how to test
if the state of an <a class="reference external" href="http://doc.akka.io/docs/akka/2.3.2/scala/persistence.html">akka-persistence</a> based application is properly recovered by
replaying the journals after a system restart. This post is about testing the second
way of recovering application state supported by akka-persistence: by using snapshots.</p>
<div class="section" id="testing-recovery-through-snapshots">
<span id="testingsnapshotrecovery"></span><h1>Testing Recovery through Snapshots</h1>
<p>In addition to recover application state by replaying the entire journal akka-persistence
supports taking <a class="reference external" href="http://doc.akka.io/docs/akka/2.3.2/scala/persistence.html#Snapshots">snapshots</a> of
application state and recover from them. This typically decreases recovery time
significantly as the state is not reconstructed command-message by command-message
but rather all at once and only those commands that arrived after the snapshot was taken need
to be replayed message by message.</p>
<div class="section" id="enabling-snapshots">
<h2>Enabling Snapshots</h2>
<p>To enable the <cite>ItemActor</cite> to take snapshots we only need a few lines of code:</p>
<pre class="code scala literal-block">
<span class="ln"> 1 </span> <span class="keyword">case</span> <span class="name class">SaveSnapshot</span> <span class="keyword">=></span> <span class="name">saveSnapshot</span><span class="operator">(</span><span class="name class">ItemActorSnapshot</span><span class="operator">(</span><span class="name">itemById</span><span class="operator">,</span> <span class="name">idCounter</span><span class="operator">))</span>
<span class="ln"> 2 </span> <span class="keyword">case</span> <span class="name class">SaveSnapshotSuccess</span><span class="operator">(</span><span class="name">metadata</span><span class="operator">)</span> <span class="keyword">=></span>
<span class="ln"> 3 </span> <span class="keyword">case</span> <span class="name class">SaveSnapshotFailure</span><span class="operator">(</span><span class="name">metadata</span><span class="operator">,</span> <span class="name">cause</span><span class="operator">)</span> <span class="keyword">=></span>
<span class="ln"> 4 </span>
<span class="ln"> 5 </span> <span class="keyword">case</span> <span class="name class">SnapshotOffer</span><span class="operator">(</span><span class="keyword">_</span><span class="operator">,</span> <span class="name class">ItemActorSnapshot</span><span class="operator">(</span><span class="name">itemMap</span><span class="operator">,</span> <span class="name">lastId</span><span class="operator">))</span> <span class="keyword">=></span>
<span class="ln"> 6 </span> <span class="keyword">this</span><span class="operator">.</span><span class="name">itemById</span> <span class="keyword">=</span> <span class="name">itemMap</span>
<span class="ln"> 7 </span> <span class="keyword">this</span><span class="operator">.</span><span class="name">idCounter</span> <span class="keyword">=</span> <span class="name">lastId</span>
</pre>
<p>The actor reacts on the custom message <cite>SaveSnapshot</cite> by providing the actor's
current state to <cite>saveSnapshot</cite>. The entire state of the actor is simply modelled
as the case class <cite>ItemActorSnapshot</cite> containing the item-map and the id-counter.
<cite>saveSnapshot</cite> basically <em>responds</em> with
<cite>SaveSnapshotSuccess</cite> or <cite>SaveSnapshotFailure</cite> depending on the success of the
operation and the actor can react accordingly. We do not want to go into the
details of handling these properly here, that is why they are simply ignored.</p>
<p>When it comes to recovery of an actor (e.g. after a restart of the application)
and akka-persistence finds a snapshot for a <cite>Processor</cite> it offers that (<cite>SnapshotOffer</cite>)
before replaying messages from the journal that arrived after the snapshot was taken.</p>
</div>
<div class="section" id="testing-recovery">
<h2>Testing Recovery</h2>
<p>Testing recovery from snapshots is in principle pretty similar to testing
recovery from a journal. We have almost the same steps:</p>
<ul class="simple">
<li>start the application</li>
<li>modify the application's state by sending corresponding commands</li>
<li>take a snapshot</li>
<li>stop the application</li>
<li>restart the application (and recover from the snapshot)</li>
<li>verify the application's state by queries</li>
</ul>
<p>As these steps are almost identical we should try to reuse as much as possible from
the previous test. In fact we can reuse the entire test and just have to take care
that a snapshot is taken before the application is stopped. Let's one more time have a look at
the central method that starts/stops an application in the tests:</p>
<pre class="code scala literal-block">
<span class="ln"> 1 </span> <span class="keyword">def</span> <span class="name">withApplication</span><span class="operator">[</span><span class="keyword type">A</span><span class="operator">](</span><span class="name">persistDir</span><span class="keyword">:</span> <span class="keyword type">File</span><span class="operator">)(</span><span class="name">block</span><span class="keyword">:</span> <span class="keyword type">TestApplication</span> <span class="operator">=></span> <span class="name">A</span><span class="operator">)</span> <span class="keyword">=</span> <span class="operator">{</span>
<span class="ln"> 2 </span> <span class="keyword">val</span> <span class="name">tmpDirPersistenceConfig</span> <span class="keyword">=</span> <span class="name class">ConfigFactory</span><span class="operator">.</span><span class="name">parseMap</span><span class="operator">(</span>
<span class="ln"> 3 </span> <span class="name class">Map</span><span class="operator">(</span><span class="name class">JournalDirConfig</span> <span class="operator">-></span> <span class="keyword">new</span> <span class="name class">File</span><span class="operator">(</span><span class="name">persistDir</span><span class="operator">,</span> <span class="literal string">"journal"</span><span class="operator">).</span><span class="name">getPath</span><span class="operator">,</span>
<span class="ln"> 4 </span> <span class="name class">NativeLevelDbConfig</span> <span class="operator">-></span> <span class="keyword constant">false</span><span class="operator">.</span><span class="name">toString</span><span class="operator">,</span>
<span class="ln"> 5 </span> <span class="name class">SnapshotDirConfig</span> <span class="operator">-></span> <span class="keyword">new</span> <span class="name class">File</span><span class="operator">(</span><span class="name">persistDir</span><span class="operator">,</span> <span class="literal string">"snapshots"</span><span class="operator">).</span><span class="name">getPath</span><span class="operator">)</span>
<span class="ln"> 6 </span> <span class="operator">.</span><span class="name">asJava</span><span class="operator">)</span>
<span class="ln"> 7 </span> <span class="keyword">val</span> <span class="name">application</span> <span class="keyword">=</span> <span class="name">newItemApplication</span><span class="operator">(</span><span class="name">tmpDirPersistenceConfig</span><span class="operator">)</span>
<span class="ln"> 8 </span> <span class="name">ultimately</span><span class="operator">(</span><span class="name">application</span><span class="operator">.</span><span class="name">shutdown</span><span class="operator">())(</span><span class="name">block</span><span class="operator">(</span><span class="name">application</span><span class="operator">))</span>
<span class="ln"> 9 </span> <span class="operator">}</span>
<span class="ln">10 </span>
<span class="ln">11 </span> <span class="keyword">def</span> <span class="name">newItemApplication</span><span class="operator">(</span><span class="name">config</span><span class="keyword">:</span> <span class="keyword type">Config</span><span class="operator">)</span> <span class="keyword">=</span>
<span class="ln">12 </span> <span class="keyword">new</span> <span class="name class">ItemApplication</span><span class="operator">(</span><span class="name">config</span><span class="operator">)</span> <span class="keyword">with</span> <span class="name class">ItemApplicationTestExtensions</span>
</pre>
<p>The application is shutdown after the test by invoking the corresponding method
of the application (line 8). If we are able to inject taking a snapshot
here we are basically all set as in this case akka-persistence will find the
snapshot after the next restart and recover from that instead of the
journal. We can actually easily do this as we already amend the application
with some test-extensions and we just have to modify this a bit for taking
a snapshot at shutdown.</p>
<p>First we overwrite the <cite>newItemApplication</cite> method in the trait <cite>WithItemApplicationWithSnapshot</cite>
that extends the original one <cite>WithItemApplication</cite> to amend
the application with a snapshot-specific extension:</p>
<pre class="code scala literal-block">
<span class="ln">1 </span><span class="keyword">trait</span> <span class="name class">WithItemApplicationWithSnapshot</span> <span class="keyword">extends</span> <span class="name class">WithItemApplication</span> <span class="operator">{</span>
<span class="ln">2 </span> <span class="keyword">override</span> <span class="keyword">def</span> <span class="name">newItemApplication</span><span class="operator">(</span><span class="name">config</span><span class="keyword">:</span> <span class="keyword type">Config</span><span class="operator">)</span> <span class="keyword">=</span>
<span class="ln">3 </span> <span class="keyword">new</span> <span class="name class">ItemApplication</span><span class="operator">(</span><span class="name">config</span><span class="operator">)</span> <span class="keyword">with</span> <span class="name class">ItemApplicationWithSnapshot</span>
<span class="ln">4 </span><span class="operator">}</span>
</pre>
<p>This extension overwrites the <cite>shutdown</cite> and also amends the <cite>ItemActor</cite> by overwriting
<cite>itemActorProps</cite>:</p>
<pre class="code scala literal-block">
<span class="ln"> 1 </span><span class="keyword">trait</span> <span class="name class">ItemApplicationWithSnapshot</span> <span class="keyword">extends</span> <span class="name class">ItemApplicationTestExtensions</span>
<span class="ln"> 2 </span> <span class="keyword">with</span> <span class="name class">TestUtil</span> <span class="operator">{</span> <span class="keyword">this:</span> <span class="keyword type">ItemApplication</span> <span class="operator">=></span>
<span class="ln"> 3 </span>
<span class="ln"> 4 </span> <span class="keyword">abstract</span> <span class="keyword">override</span> <span class="keyword">def</span> <span class="name">itemActorProps</span><span class="keyword">:</span> <span class="keyword type">Props</span> <span class="operator">=</span>
<span class="ln"> 5 </span> <span class="name class">Props</span><span class="operator">(</span><span class="keyword">new</span> <span class="name class">ItemActor</span> <span class="keyword">with</span> <span class="name class">RespondToSnapshotRequest</span><span class="operator">)</span>
<span class="ln"> 6 </span>
<span class="ln"> 7 </span> <span class="keyword">abstract</span> <span class="keyword">override</span> <span class="keyword">def</span> <span class="name">shutdown</span><span class="operator">()</span> <span class="keyword">=</span> <span class="operator">{</span>
<span class="ln"> 8 </span> <span class="name">resultOf</span><span class="operator">(</span><span class="name">itemActor</span> <span class="operator">?</span> <span class="name class">SaveSnapshot</span><span class="operator">)</span> <span class="keyword">match</span> <span class="operator">{</span>
<span class="ln"> 9 </span> <span class="keyword">case</span> <span class="keyword">_:</span> <span class="keyword type">SaveSnapshotSuccess</span> <span class="operator">=></span>
<span class="ln">10 </span> <span class="keyword">case</span> <span class="name class">SaveSnapshotFailure</span><span class="operator">(</span><span class="keyword">_</span><span class="operator">,</span> <span class="name">cause</span><span class="operator">)</span> <span class="keyword">=></span>
<span class="ln">11 </span> <span class="name">sys</span><span class="operator">.</span><span class="name">error</span><span class="operator">(</span><span class="name">s</span><span class="literal string">"Saving snapshot failed with: $cause"</span><span class="operator">)</span>
<span class="ln">12 </span> <span class="operator">}</span>
<span class="ln">13 </span> <span class="keyword">super</span><span class="operator">.</span><span class="name">shutdown</span><span class="operator">()</span>
<span class="ln">14 </span> <span class="operator">}</span>
<span class="ln">15 </span><span class="operator">}</span>
</pre>
<p><cite>shutdown</cite> simply sends a <cite>SaveSnapshot</cite> message to <cite>ItemActor</cite> and waits for
a response (line 8) before it actually shuts down the application (line 13). However
to make <cite>ItemActor</cite> actually respond to this message it has to be modified slightly
and that is why <cite>itemActorProps</cite> is overwritten as well (line 4) and the returned
actor is extended with the trait <cite>RespondToSnapshotRequest</cite>:</p>
<pre class="code scala literal-block">
<span class="ln"> 1 </span><span class="keyword">trait</span> <span class="name class">RespondToSnapshotRequest</span> <span class="keyword">extends</span> <span class="name class">Actor</span> <span class="operator">{</span>
<span class="ln"> 2 </span>
<span class="ln"> 3 </span> <span class="keyword">private</span> <span class="keyword">var</span> <span class="name">lastSnapshotSender</span><span class="keyword">:</span> <span class="keyword type">Option</span><span class="operator">[</span><span class="keyword type">ActorRef</span><span class="operator">]</span> <span class="keyword">=</span> <span class="name class">None</span>
<span class="ln"> 4 </span>
<span class="ln"> 5 </span> <span class="keyword">abstract</span> <span class="keyword">override</span> <span class="keyword">def</span> <span class="name">receive</span><span class="keyword">:</span> <span class="keyword type">Receive</span> <span class="operator">=</span> <span class="name">respondToSnapshotReceive</span><span class="operator">.</span><span class="name">orElse</span><span class="operator">(</span><span class="keyword">super</span><span class="operator">.</span><span class="name">receive</span><span class="operator">)</span>
<span class="ln"> 6 </span>
<span class="ln"> 7 </span> <span class="keyword">def</span> <span class="name">respondToSnapshotReceive</span><span class="keyword">:</span> <span class="keyword type">Receive</span> <span class="operator">=</span> <span class="operator">{</span>
<span class="ln"> 8 </span> <span class="keyword">case</span> <span class="name class">SaveSnapshot</span> <span class="keyword">=></span>
<span class="ln"> 9 </span> <span class="name">lastSnapshotSender</span> <span class="keyword">=</span> <span class="name class">Some</span><span class="operator">(</span><span class="name">sender</span><span class="operator">())</span>
<span class="ln">10 </span> <span class="keyword">super</span><span class="operator">.</span><span class="name">receive</span><span class="operator">(</span><span class="name class">SaveSnapshot</span><span class="operator">)</span>
<span class="ln">11 </span>
<span class="ln">12 </span> <span class="keyword">case</span> <span class="name">message</span><span class="keyword">:</span> <span class="keyword type">SaveSnapshotSuccess</span> <span class="operator">=></span>
<span class="ln">13 </span> <span class="keyword">super</span><span class="operator">.</span><span class="name">receive</span><span class="operator">(</span><span class="name">message</span><span class="operator">)</span>
<span class="ln">14 </span> <span class="name">respondToSnapshotRequester</span><span class="operator">(</span><span class="name">message</span><span class="operator">)</span>
<span class="ln">15 </span>
<span class="ln">16 </span> <span class="keyword">case</span> <span class="name">message</span><span class="keyword">:</span> <span class="keyword type">SaveSnapshotFailure</span> <span class="operator">=></span>
<span class="ln">17 </span> <span class="keyword">super</span><span class="operator">.</span><span class="name">receive</span><span class="operator">(</span><span class="name">message</span><span class="operator">)</span>
<span class="ln">18 </span> <span class="name">respondToSnapshotRequester</span><span class="operator">(</span><span class="name">message</span><span class="operator">)</span>
<span class="ln">19 </span> <span class="operator">}</span>
<span class="ln">20 </span>
<span class="ln">21 </span> <span class="keyword">private</span> <span class="keyword">def</span> <span class="name">respondToSnapshotRequester</span><span class="operator">(</span><span class="name">response</span><span class="keyword">:</span> <span class="keyword type">AnyRef</span><span class="operator">)</span> <span class="keyword">=</span> <span class="operator">{</span>
<span class="ln">22 </span> <span class="name">lastSnapshotSender</span><span class="operator">.</span><span class="name">foreach</span><span class="operator">(</span><span class="keyword">_</span> <span class="operator">!</span> <span class="name">response</span><span class="operator">)</span>
<span class="ln">23 </span> <span class="name">lastSnapshotSender</span> <span class="keyword">=</span> <span class="name class">None</span>
<span class="ln">24 </span> <span class="operator">}</span>
<span class="ln">25 </span><span class="operator">}</span>
</pre>
<p>The <cite>receive</cite>-method of this trait intercepts <cite>SaveSnapshot</cite> messages and keeps
the sender of the message (line 9) before it continues with normal processing
(line 10). The saved sender reference is used to forward the <cite>SaveSnapshotSuccess</cite> (line 12) or
<cite>SaveSnapshotFailure</cite> (line 16) messages to it.</p>
<p>Armed with this the test for testing successful recovery from snapshots can simply extend
the one for testing recovery from the journal but mix-in the <cite>WithItemApplicationWithSnapshot</cite>-trait:</p>
<pre class="code scala literal-block">
<span class="ln">1 </span><span class="keyword">class</span> <span class="name class">ItemApplicationRecoverFromSnapshotSpec</span> <span class="keyword">extends</span> <span class="name class">ItemApplicationRecoverSpec</span>
<span class="ln">2 </span> <span class="keyword">with</span> <span class="name class">WithItemApplicationWithSnapshot</span> <span class="operator">{</span>
</pre>
<p>So there is no need to redundantly formulate the individual tests for recovery
after create, update and delete. To verify how those tests are working we
can break the implementation of taking snapshots. We could for example forget
to include the id-counter in a snapshot, so stripping down the <cite>ItemActorSnapshot</cite> to
<cite>case class ItemActorSnapshot(itemById: Map[ItemId, Item])</cite>. Running the tests
immediately shows ...oops... that all are running fine. So the bug created by this
change is not discovered by the tests. It seems we need an additional one. The difference
between recovery from the journal and recovery from a snapshot is that when the state
is recovered from the journal the same application logic is triggered as in case
of normal operation. So if tests have proven that during normal operation the
application's state is not being messed up, its hard to mess it up during recovery
(as long as all command-messages really end up in the journal).
However in case of snapshots this is different as the state is handled independently
from the processed messages and that is why we have to add more tests. The bug
we just introduced resets the id-counter to 0 after each restart. To verify
that this does not occur we need to create a new item before and after the
restart and check if both can be retrieved afterwards. The test looks like this:</p>
<pre class="code scala literal-block">
<span class="ln"> 1 </span> <span class="keyword">val</span> <span class="name">existingItem</span> <span class="keyword">=</span> <span class="name">withApplication</span><span class="operator">(</span><span class="name">persistDir</span><span class="operator">)</span> <span class="operator">{</span> <span class="name">application</span> <span class="keyword">=></span>
<span class="ln"> 2 </span> <span class="name">application</span><span class="operator">.</span><span class="name">itemServiceTestExtension</span><span class="operator">.</span><span class="name">createNewItem</span><span class="operator">()</span>
<span class="ln"> 3 </span> <span class="operator">}</span>
<span class="ln"> 4 </span> <span class="name">withApplication</span><span class="operator">(</span><span class="name">persistDir</span><span class="operator">)</span> <span class="operator">{</span> <span class="name">application</span> <span class="keyword">=></span>
<span class="ln"> 5 </span> <span class="keyword">val</span> <span class="name">service</span> <span class="keyword">=</span> <span class="name">application</span><span class="operator">.</span><span class="name">itemServiceTestExtension</span>
<span class="ln"> 6 </span>
<span class="ln"> 7 </span> <span class="keyword">val</span> <span class="name">created</span> <span class="keyword">=</span> <span class="name">service</span><span class="operator">.</span><span class="name">createNewItem</span><span class="operator">()</span>
<span class="ln"> 8 </span>
<span class="ln"> 9 </span> <span class="name">service</span><span class="operator">.</span><span class="name">findItem</span><span class="operator">(</span><span class="name">created</span><span class="operator">.</span><span class="name">id</span><span class="operator">)</span> <span class="name">should</span> <span class="name">be</span> <span class="operator">(</span><span class="name class">Some</span><span class="operator">(</span><span class="name">created</span><span class="operator">))</span>
<span class="ln">10 </span> <span class="name">service</span><span class="operator">.</span><span class="name">findItem</span><span class="operator">(</span><span class="name">existingItem</span><span class="operator">.</span><span class="name">id</span><span class="operator">)</span> <span class="name">should</span> <span class="name">be</span> <span class="operator">(</span><span class="name class">Some</span><span class="operator">(</span><span class="name">existingItem</span><span class="operator">))</span>
<span class="ln">11 </span> <span class="operator">}</span>
</pre>
<p>First an item is created before the restart (line 2) and another one after the
restart (line 7). Then the test asserts that both can be retrieved (line 9-10).
With the bug introduced above this test fails (in line 9) as the newly created
item gets the same id as the old one and thus overwrites it. Fixing the bug
makes the test run fine again.</p>
<p>This blog post showed how one can reuse the tests for recovery from the journal
for testing recovery from snapshots. The next one shows if the same ideas can
even be applied to testing successful recovery from an <em>old</em> journal.</p>
</div>
</div>
</div>
</body>
</html>
Volker Stampahttp://www.blogger.com/profile/03277007007448976514noreply@blogger.com0tag:blogger.com,1999:blog-3938166335394683606.post-2033481781895832342014-05-23T09:16:00.001+02:002014-06-19T15:33:39.022+02:00Testing Recovery<?xml version="1.0" encoding="utf-8" ?>
<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">
<html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en" lang="en">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8" />
<meta name="generator" content="Docutils 0.11: http://docutils.sourceforge.net/" />
<title></title>
<style type="text/css">
/*
:Author: David Goodger (goodger@python.org)
:Id: $Id: html4css1.css 7614 2013-02-21 15:55:51Z milde $
:Copyright: This stylesheet has been placed in the public domain.
Default cascading style sheet for the HTML output of Docutils.
See http://docutils.sf.net/docs/howto/html-stylesheets.html for how to
customize this style sheet.
*/
/* used to remove borders from tables and images */
.borderless, table.borderless td, table.borderless th {
border: 0 }
table.borderless td, table.borderless th {
/* Override padding for "table.docutils td" with "! important".
The right padding separates the table cells. */
padding: 0 0.5em 0 0 ! important }
.first {
/* Override more specific margin styles with "! important". */
margin-top: 0 ! important }
.last, .with-subtitle {
margin-bottom: 0 ! important }
.hidden {
display: none }
a.toc-backref {
text-decoration: none ;
color: black }
blockquote.epigraph {
margin: 2em 5em ; }
dl.docutils dd {
margin-bottom: 0.5em }
object[type="image/svg+xml"], object[type="application/x-shockwave-flash"] {
overflow: hidden;
}
/* Uncomment (and remove this text!) to get bold-faced definition list terms
dl.docutils dt {
font-weight: bold }
*/
div.abstract {
margin: 2em 5em }
div.abstract p.topic-title {
font-weight: bold ;
text-align: center }
div.admonition, div.attention, div.caution, div.danger, div.error,
div.hint, div.important, div.note, div.tip, div.warning {
margin: 2em ;
border: medium outset ;
padding: 1em }
div.admonition p.admonition-title, div.hint p.admonition-title,
div.important p.admonition-title, div.note p.admonition-title,
div.tip p.admonition-title {
font-weight: bold ;
font-family: sans-serif }
div.attention p.admonition-title, div.caution p.admonition-title,
div.danger p.admonition-title, div.error p.admonition-title,
div.warning p.admonition-title, .code .error {
color: red ;
font-weight: bold ;
font-family: sans-serif }
/* Uncomment (and remove this text!) to get reduced vertical space in
compound paragraphs.
div.compound .compound-first, div.compound .compound-middle {
margin-bottom: 0.5em }
div.compound .compound-last, div.compound .compound-middle {
margin-top: 0.5em }
*/
div.dedication {
margin: 2em 5em ;
text-align: center ;
font-style: italic }
div.dedication p.topic-title {
font-weight: bold ;
font-style: normal }
div.figure {
margin-left: 2em ;
margin-right: 2em }
div.footer, div.header {
clear: both;
font-size: smaller }
div.line-block {
display: block ;
margin-top: 1em ;
margin-bottom: 1em }
div.line-block div.line-block {
margin-top: 0 ;
margin-bottom: 0 ;
margin-left: 1.5em }
div.sidebar {
margin: 0 0 0.5em 1em ;
border: medium outset ;
padding: 1em ;
background-color: #ffffee ;
width: 40% ;
float: right ;
clear: right }
div.sidebar p.rubric {
font-family: sans-serif ;
font-size: medium }
div.system-messages {
margin: 5em }
div.system-messages h1 {
color: red }
div.system-message {
border: medium outset ;
padding: 1em }
div.system-message p.system-message-title {
color: red ;
font-weight: bold }
div.topic {
margin: 2em }
h1.section-subtitle, h2.section-subtitle, h3.section-subtitle,
h4.section-subtitle, h5.section-subtitle, h6.section-subtitle {
margin-top: 0.4em }
h1.title {
text-align: center }
h2.subtitle {
text-align: center }
hr.docutils {
width: 75% }
img.align-left, .figure.align-left, object.align-left {
clear: left ;
float: left ;
margin-right: 1em }
img.align-right, .figure.align-right, object.align-right {
clear: right ;
float: right ;
margin-left: 1em }
img.align-center, .figure.align-center, object.align-center {
display: block;
margin-left: auto;
margin-right: auto;
}
.align-left {
text-align: left }
.align-center {
clear: both ;
text-align: center }
.align-right {
text-align: right }
/* reset inner alignment in figures */
div.align-right {
text-align: inherit }
/* div.align-center * { */
/* text-align: left } */
ol.simple, ul.simple {
margin-bottom: 1em }
ol.arabic {
list-style: decimal }
ol.loweralpha {
list-style: lower-alpha }
ol.upperalpha {
list-style: upper-alpha }
ol.lowerroman {
list-style: lower-roman }
ol.upperroman {
list-style: upper-roman }
p.attribution {
text-align: right ;
margin-left: 50% }
p.caption {
font-style: italic }
p.credits {
font-style: italic ;
font-size: smaller }
p.label {
white-space: nowrap }
p.rubric {
font-weight: bold ;
font-size: larger ;
color: maroon ;
text-align: center }
p.sidebar-title {
font-family: sans-serif ;
font-weight: bold ;
font-size: larger }
p.sidebar-subtitle {
font-family: sans-serif ;
font-weight: bold }
p.topic-title {
font-weight: bold }
pre.address {
margin-bottom: 0 ;
margin-top: 0 ;
font: inherit }
pre.literal-block, pre.doctest-block, pre.math, pre.code {
margin-left: 2em ;
margin-right: 2em }
pre.code .ln { color: grey; } /* line numbers */
pre.code, code { background-color: #eeeeee }
pre.code .comment, code .comment { color: #5C6576 }
pre.code .keyword, code .keyword { color: #3B0D06; font-weight: bold }
pre.code .literal.string, code .literal.string { color: #0C5404 }
pre.code .name.builtin, code .name.builtin { color: #352B84 }
pre.code .deleted, code .deleted { background-color: #DEB0A1}
pre.code .inserted, code .inserted { background-color: #A3D289}
span.classifier {
font-family: sans-serif ;
font-style: oblique }
span.classifier-delimiter {
font-family: sans-serif ;
font-weight: bold }
span.interpreted {
font-family: sans-serif }
span.option {
white-space: nowrap }
span.pre {
white-space: pre }
span.problematic {
color: red }
span.section-subtitle {
/* font-size relative to parent (h1..h6 element) */
font-size: 80% }
table.citation {
border-left: solid 1px gray;
margin-left: 1px }
table.docinfo {
margin: 2em 4em }
table.docutils {
margin-top: 0.5em ;
margin-bottom: 0.5em }
table.footnote {
border-left: solid 1px black;
margin-left: 1px }
table.docutils td, table.docutils th,
table.docinfo td, table.docinfo th {
padding-left: 0.5em ;
padding-right: 0.5em ;
vertical-align: top }
table.docutils th.field-name, table.docinfo th.docinfo-name {
font-weight: bold ;
text-align: left ;
white-space: nowrap ;
padding-left: 0 }
/* "booktabs" style (no vertical lines) */
table.docutils.booktabs {
border: 0px;
border-top: 2px solid;
border-bottom: 2px solid;
border-collapse: collapse;
}
table.docutils.booktabs * {
border: 0px;
}
table.docutils.booktabs th {
border-bottom: thin solid;
text-align: left;
}
h1 tt.docutils, h2 tt.docutils, h3 tt.docutils,
h4 tt.docutils, h5 tt.docutils, h6 tt.docutils {
font-size: 100% }
ul.auto-toc {
list-style-type: none }
</style>
<style type="text/css">
/* example stylesheet for Docutils */
/* :Author: Günter Milde */
/* :Copyright: © 2012 G. Milde */
/* :License: This stylesheet is placed in the public domain. */
/* Syntax highlight rules for HTML documents generated with Docutils */
/* using the ``--syntax-highlight=long`` option (new in v. 0.9). */
/* This stylesheet implements Pygment's "default" style with less rules than */
/* pygments-default using class hierarchies. */
/* Use it as example for "handcrafted" styles with only few rules. */
.code { background: #f8f8f8; }
.code .comment { color: #008800; font-style: italic }
.code .error { border: 1px solid #FF0000 }
.code .generic.deleted { color: #A00000 }
.code .generic.emph { font-style: italic }
.code .generic.error { color: #FF0000 }
.code .generic.heading { color: #000080; font-weight: bold }
.code .generic.inserted { color: #00A000 }
.code .generic.output { color: #808080 }
.code .generic.prompt { color: #000080; font-weight: bold }
.code .generic.strong { font-weight: bold }
.code .generic.subheading { color: #800080; font-weight: bold }
.code .generic.traceback { color: #0040D0 }
.code .keyword { color: #AA22FF; font-weight: bold }
.code .keyword.pseudo { font-weight: normal }
.code .literal.number { color: #666666 }
.code .literal.string { color: #BB4444 }
.code .literal.string.doc { color: #BB4444; font-style: italic }
.code .literal.string.escape { color: #BB6622; font-weight: bold }
.code .literal.string.interpol { color: #BB6688; font-weight: bold }
.code .literal.string.other { color: #008000 }
.code .literal.string.regex { color: #BB6688 }
.code .literal.string.symbol { color: #B8860B }
.code .name.attribute { color: #BB4444 }
.code .name.builtin { color: #AA22FF }
.code .name.class { color: #0000FF }
.code .name.constant { color: #880000 }
.code .name.decorator { color: #AA22FF }
.code .name.entity { color: #999999; font-weight: bold }
.code .name.exception { color: #D2413A; font-weight: bold }
.code .name.function { color: #00A000 }
.code .name.label { color: #A0A000 }
.code .name.namespace { color: #0000FF; font-weight: bold }
.code .name.tag { color: #008000; font-weight: bold }
.code .name.variable { color: #B8860B }
.code .operator { color: #666666 }
.code .operator.word { color: #AA22FF; font-weight: bold }
.code .ln { margin-right: -0px }
</style>
</head>
<body>
<div class="document">
<p>In the last <a class="reference external" href="/2014/04/akka-persistence-and-testing.html">post</a>
I introduced the problem area of what and how to test applications
that use <a class="reference external" href="http://doc.akka.io/docs/akka/2.3.0/scala/persistence.html">akka-persistence</a> for persisting their state. I demonstrated the first
important test verifying that the custom serializers for the messages to be
persisted are actually used. This second part is about testing the successful
recovery of the application state after a restart.</p>
<div class="section" id="testing-recovery">
<span id="testingrecovery"></span><h1>Testing Recovery</h1>
<p>To test recovery we basically have to execute the following steps:</p>
<ul class="simple">
<li>start the application (with an empty journal)</li>
<li>modify the application's state by sending corresponding commands</li>
<li>stop the application</li>
<li>restart the application</li>
<li>verify the application's state by queries</li>
</ul>
<p>While it sounds a little bit odd to start, stop and restart an application in a
unit test, with the <tt class="docutils literal">ItemApplicationFixture</tt> that we have seen in the last post
it is actually not a big deal. Let's have another quick look at the central method:
<tt class="docutils literal">withApplication</tt></p>
<pre class="code scala literal-block">
<span class="ln"> 1 </span> <span class="keyword">def</span> <span class="name">withApplication</span><span class="operator">[</span><span class="keyword type">A</span><span class="operator">](</span><span class="name">persistDir</span><span class="keyword">:</span> <span class="keyword type">File</span><span class="operator">)(</span><span class="name">block</span><span class="keyword">:</span> <span class="keyword type">TestApplication</span> <span class="operator">=></span> <span class="name">A</span><span class="operator">)</span> <span class="keyword">=</span> <span class="operator">{</span>
<span class="ln"> 2 </span> <span class="keyword">val</span> <span class="name">tmpDirPersistenceConfig</span> <span class="keyword">=</span> <span class="name class">ConfigFactory</span><span class="operator">.</span><span class="name">parseMap</span><span class="operator">(</span>
<span class="ln"> 3 </span> <span class="name class">Map</span><span class="operator">(</span><span class="name class">JournalDirConfig</span> <span class="operator">-></span> <span class="keyword">new</span> <span class="name class">File</span><span class="operator">(</span><span class="name">persistDir</span><span class="operator">,</span> <span class="literal string">"journal"</span><span class="operator">).</span><span class="name">getPath</span><span class="operator">,</span>
<span class="ln"> 4 </span> <span class="name class">NativeLevelDbConfig</span> <span class="operator">-></span> <span class="keyword constant">false</span><span class="operator">.</span><span class="name">toString</span><span class="operator">,</span>
<span class="ln"> 5 </span> <span class="name class">SnapshotDirConfig</span> <span class="operator">-></span> <span class="keyword">new</span> <span class="name class">File</span><span class="operator">(</span><span class="name">persistDir</span><span class="operator">,</span> <span class="literal string">"snapshots"</span><span class="operator">).</span><span class="name">getPath</span><span class="operator">)</span>
<span class="ln"> 6 </span> <span class="operator">.</span><span class="name">asJava</span><span class="operator">)</span>
<span class="ln"> 7 </span> <span class="keyword">val</span> <span class="name">application</span> <span class="keyword">=</span> <span class="name">newItemApplication</span><span class="operator">(</span><span class="name">tmpDirPersistenceConfig</span><span class="operator">)</span>
<span class="ln"> 8 </span> <span class="name">ultimately</span><span class="operator">(</span><span class="name">application</span><span class="operator">.</span><span class="name">shutdown</span><span class="operator">())(</span><span class="name">block</span><span class="operator">(</span><span class="name">application</span><span class="operator">))</span>
<span class="ln"> 9 </span> <span class="operator">}</span>
<span class="ln">10 </span>
<span class="ln">11 </span> <span class="keyword">def</span> <span class="name">newItemApplication</span><span class="operator">(</span><span class="name">config</span><span class="keyword">:</span> <span class="keyword type">Config</span><span class="operator">)</span> <span class="keyword">=</span>
<span class="ln">12 </span> <span class="keyword">new</span> <span class="name class">ItemApplication</span><span class="operator">(</span><span class="name">config</span><span class="operator">)</span> <span class="keyword">with</span> <span class="name class">ItemApplicationTestExtensions</span>
</pre>
<p>As you can see it takes a <em>temporary</em> folder (where journals and snapshots are stored)
as one argument (<tt class="docutils literal">persistDir</tt>) and test-code (<tt class="docutils literal">block</tt>) as second argument.
The application is started (line 7),
the test is executed and in any case (failure or success) the application is shut down (line 8).</p>
<p>Armed with this we can pretty easily start, stop and restart the application in a test,
like follows:</p>
<pre class="code scala literal-block">
<span class="ln"> 1 </span> <span class="keyword">val</span> <span class="name">created</span> <span class="keyword">=</span> <span class="name">withApplication</span><span class="operator">(</span><span class="name">persistDir</span><span class="operator">)</span> <span class="operator">{</span> <span class="name">application</span> <span class="keyword">=></span>
<span class="ln"> 2 </span> <span class="name">application</span><span class="operator">.</span><span class="name">itemServiceTestExtension</span><span class="operator">.</span><span class="name">createNewItem</span><span class="operator">()</span>
<span class="ln"> 3 </span> <span class="operator">}</span>
<span class="ln"> 4 </span> <span class="name">withApplication</span><span class="operator">(</span><span class="name">persistDir</span><span class="operator">)</span> <span class="operator">{</span> <span class="name">application</span> <span class="keyword">=></span>
<span class="ln"> 5 </span> <span class="name">application</span><span class="operator">.</span><span class="name">itemServiceTestExtension</span><span class="operator">.</span><span class="name">findItem</span><span class="operator">(</span><span class="name">created</span><span class="operator">.</span><span class="name">id</span><span class="operator">)</span> <span class="name">should</span> <span class="name">be</span> <span class="operator">(</span>
<span class="ln"> 6 </span> <span class="name class">Some</span><span class="operator">(</span><span class="name">created</span><span class="operator">))</span>
<span class="ln"> 7 </span> <span class="operator">}</span>
</pre>
<p>This test starts the application (line 1), creates a new item (line 2) and
shuts the application down by ending the block.
Immediately after that it is restarted (line 4). As the directory
for storing the journal is the same as before (<tt class="docutils literal">persistDir</tt>) this should
recover the state of the application from the previously written journal
such that the following <tt class="docutils literal">findItem</tt> successfully
returns the item created before (line 5-6).</p>
<p>The application that is passed to the block
is extended with some convenience functions (like <tt class="docutils literal">itemServiceTestExtension</tt>) and also with a wrapper for the
<tt class="docutils literal">ItemService</tt> that eases invoking item-commands by waiting for the returned <tt class="docutils literal">Future</tt>s.
Let's have a quick look at <tt class="docutils literal">createNewItem</tt>:</p>
<pre class="code scala literal-block">
<span class="ln">1 </span> <span class="keyword">def</span> <span class="name">createNewItem</span><span class="operator">(</span><span class="name">template</span><span class="keyword">:</span> <span class="keyword type">ItemTemplate</span> <span class="operator">=</span> <span class="name">newItemTemplate</span><span class="operator">())</span><span class="keyword">:</span> <span class="keyword type">Item</span> <span class="operator">=</span>
<span class="ln">2 </span> <span class="name">successOf</span><span class="operator">(</span><span class="name">service</span><span class="operator">.</span><span class="name">create</span><span class="operator">(</span><span class="name">template</span><span class="operator">))</span>
</pre>
<p>And the implementation of <tt class="docutils literal">successOf</tt> looks like follows:</p>
<pre class="code scala literal-block">
<span class="ln">1 </span> <span class="keyword">def</span> <span class="name">resultOf</span><span class="operator">[</span><span class="keyword type">A</span><span class="operator">](</span><span class="name">future</span><span class="keyword">:</span> <span class="keyword type">Future</span><span class="operator">[</span><span class="keyword type">A</span><span class="operator">])</span><span class="keyword">:</span> <span class="keyword type">A</span> <span class="operator">=</span> <span class="name class">Await</span><span class="operator">.</span><span class="name">result</span><span class="operator">(</span><span class="name">future</span><span class="operator">,</span> <span class="name">timeoutDuration</span><span class="operator">)</span>
<span class="ln">2 </span>
<span class="ln">3 </span> <span class="keyword">def</span> <span class="name">successOf</span><span class="operator">[</span><span class="keyword type">A</span><span class="operator">](</span><span class="name">future</span><span class="keyword">:</span> <span class="keyword type">Future</span><span class="operator">[</span><span class="keyword type">Try</span><span class="operator">[</span><span class="keyword type">A</span><span class="operator">]])</span><span class="keyword">:</span> <span class="keyword type">A</span> <span class="operator">=</span> <span class="name">successOf</span><span class="operator">(</span><span class="name">resultOf</span><span class="operator">(</span><span class="name">future</span><span class="operator">))</span>
<span class="ln">4 </span>
<span class="ln">5 </span> <span class="keyword">def</span> <span class="name">successOf</span><span class="operator">[</span><span class="keyword type">A</span><span class="operator">](</span><span class="name">result</span><span class="keyword">:</span> <span class="keyword type">Try</span><span class="operator">[</span><span class="keyword type">A</span><span class="operator">])</span><span class="keyword">:</span> <span class="keyword type">A</span> <span class="operator">=</span> <span class="name">result</span><span class="operator">.</span><span class="name">get</span>
</pre>
<p>Factoring out this kind of code required to handle the <tt class="docutils literal">Future</tt> or <tt class="docutils literal">Try</tt> avoids polluting the
test details that do not contribute to the documentation aspect of the test, as the test does not
test the creation of new items, but rather just the proper recovery.</p>
<p>Similar tests exist for verifying that an update is recovered successfully:</p>
<pre class="code scala literal-block">
<span class="ln"> 1 </span> <span class="keyword">val</span> <span class="name">updated</span> <span class="keyword">=</span> <span class="name">withApplication</span><span class="operator">(</span><span class="name">persistDir</span><span class="operator">)</span> <span class="operator">{</span> <span class="name">application</span> <span class="keyword">=></span>
<span class="ln"> 2 </span> <span class="keyword">val</span> <span class="name">service</span> <span class="keyword">=</span> <span class="name">application</span><span class="operator">.</span><span class="name">itemServiceTestExtension</span>
<span class="ln"> 3 </span>
<span class="ln"> 4 </span> <span class="name">service</span><span class="operator">.</span><span class="name">updateItem</span><span class="operator">(</span><span class="name">service</span><span class="operator">.</span><span class="name">createNewItem</span><span class="operator">())</span>
<span class="ln"> 5 </span> <span class="operator">}</span>
<span class="ln"> 6 </span> <span class="name">withApplication</span><span class="operator">(</span><span class="name">persistDir</span><span class="operator">)</span> <span class="operator">{</span> <span class="name">application</span> <span class="keyword">=></span>
<span class="ln"> 7 </span> <span class="name">application</span><span class="operator">.</span><span class="name">itemServiceTestExtension</span><span class="operator">.</span><span class="name">findItem</span><span class="operator">(</span><span class="name">updated</span><span class="operator">.</span><span class="name">id</span><span class="operator">)</span> <span class="name">should</span> <span class="name">be</span> <span class="operator">(</span>
<span class="ln"> 8 </span> <span class="name class">Some</span><span class="operator">(</span><span class="name">updated</span><span class="operator">))</span>
<span class="ln"> 9 </span> <span class="operator">}</span>
</pre>
<p>or a delete:</p>
<pre class="code scala literal-block">
<span class="ln"> 1 </span> <span class="keyword">val</span> <span class="name">deleted</span> <span class="keyword">=</span> <span class="name">withApplication</span><span class="operator">(</span><span class="name">persistDir</span><span class="operator">)</span> <span class="operator">{</span> <span class="name">application</span> <span class="keyword">=></span>
<span class="ln"> 2 </span> <span class="keyword">val</span> <span class="name">service</span> <span class="keyword">=</span> <span class="name">application</span><span class="operator">.</span><span class="name">itemServiceTestExtension</span>
<span class="ln"> 3 </span>
<span class="ln"> 4 </span> <span class="name">service</span><span class="operator">.</span><span class="name">deleteItem</span><span class="operator">(</span><span class="name">service</span><span class="operator">.</span><span class="name">createNewItem</span><span class="operator">().</span><span class="name">id</span><span class="operator">)</span>
<span class="ln"> 5 </span> <span class="operator">}</span>
<span class="ln"> 6 </span> <span class="name">withApplication</span><span class="operator">(</span><span class="name">persistDir</span><span class="operator">)</span> <span class="operator">{</span> <span class="name">application</span> <span class="keyword">=></span>
<span class="ln"> 7 </span> <span class="name">application</span><span class="operator">.</span><span class="name">itemServiceTestExtension</span><span class="operator">.</span><span class="name">findItem</span><span class="operator">(</span><span class="name">deleted</span><span class="operator">.</span><span class="name">id</span><span class="operator">)</span> <span class="name">should</span> <span class="name">be</span> <span class="operator">(</span>
<span class="ln"> 8 </span> <span class="name class">None</span><span class="operator">)</span>
<span class="ln"> 9 </span> <span class="operator">}</span>
</pre>
<p>We can easily verify that these tests are actually significant by introducing a bug
in our code. For example, if we <em>forget</em> to wrap the <tt class="docutils literal">UpdateItem</tt> command in a
<tt class="docutils literal">Persistent</tt> when we send it in <tt class="docutils literal">ItemService</tt>:</p>
<pre class="code scala literal-block">
<span class="ln">1 </span> <span class="keyword">def</span> <span class="name">update</span><span class="operator">(</span><span class="name">item</span><span class="keyword">:</span> <span class="keyword type">Item</span><span class="operator">)</span><span class="keyword">:</span> <span class="keyword type">Future</span><span class="operator">[</span><span class="keyword type">Try</span><span class="operator">[</span><span class="keyword type">Item</span><span class="operator">]]</span> <span class="keyword">=</span>
<span class="ln">2 </span> <span class="operator">(</span><span class="name">itemActor</span> <span class="operator">?</span> <span class="name class">UpdateItem</span><span class="operator">(</span><span class="name">item</span><span class="operator">)).</span><span class="name">mapTo</span><span class="operator">[</span><span class="keyword type">Try</span><span class="operator">[</span><span class="keyword type">Item</span><span class="operator">]]</span>
</pre>
<p>as well as when we receive it in <tt class="docutils literal">ItemActor</tt>:</p>
<pre class="code scala literal-block">
<span class="ln">1 </span> <span class="keyword">case</span> <span class="name class">UpdateItem</span><span class="operator">(</span><span class="name">item</span><span class="keyword">:</span> <span class="keyword type">Item</span><span class="operator">)</span> <span class="keyword">=></span>
</pre>
<p>the corresponding test fails with:</p>
<pre class="literal-block">
Some(Item(1,2 - description)) was not equal to Some(Item(1,3 - description))
</pre>
<p>This completes the tests for application-state recovery from the journal after
a restart. What we still do not know is, if the application state can also be
recovered from a snapshot. We will have a closer look into this in the next
blog-post.</p>
</div>
</div>
</body>
</html>
Volker Stampahttp://www.blogger.com/profile/03277007007448976514noreply@blogger.com0tag:blogger.com,1999:blog-3938166335394683606.post-30434062031478407962014-04-06T09:40:00.000+02:002014-05-23T09:31:29.455+02:00Akka-Persistence and Testing<?xml version="1.0" encoding="utf-8" ?>
<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">
<html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en" lang="en">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8" />
<meta name="generator" content="Docutils 0.11: http://docutils.sourceforge.net/" />
<title></title>
<style type="text/css">
/*
:Author: David Goodger (goodger@python.org)
:Id: $Id: html4css1.css 7614 2013-02-21 15:55:51Z milde $
:Copyright: This stylesheet has been placed in the public domain.
Default cascading style sheet for the HTML output of Docutils.
See http://docutils.sf.net/docs/howto/html-stylesheets.html for how to
customize this style sheet.
*/
/* used to remove borders from tables and images */
.borderless, table.borderless td, table.borderless th {
border: 0 }
table.borderless td, table.borderless th {
/* Override padding for "table.docutils td" with "! important".
The right padding separates the table cells. */
padding: 0 0.5em 0 0 ! important }
.first {
/* Override more specific margin styles with "! important". */
margin-top: 0 ! important }
.last, .with-subtitle {
margin-bottom: 0 ! important }
.hidden {
display: none }
a.toc-backref {
text-decoration: none ;
color: black }
blockquote.epigraph {
margin: 2em 5em ; }
dl.docutils dd {
margin-bottom: 0.5em }
object[type="image/svg+xml"], object[type="application/x-shockwave-flash"] {
overflow: hidden;
}
/* Uncomment (and remove this text!) to get bold-faced definition list terms
dl.docutils dt {
font-weight: bold }
*/
div.abstract {
margin: 2em 5em }
div.abstract p.topic-title {
font-weight: bold ;
text-align: center }
div.admonition, div.attention, div.caution, div.danger, div.error,
div.hint, div.important, div.note, div.tip, div.warning {
margin: 2em ;
border: medium outset ;
padding: 1em }
div.admonition p.admonition-title, div.hint p.admonition-title,
div.important p.admonition-title, div.note p.admonition-title,
div.tip p.admonition-title {
font-weight: bold ;
font-family: sans-serif }
div.attention p.admonition-title, div.caution p.admonition-title,
div.danger p.admonition-title, div.error p.admonition-title,
div.warning p.admonition-title, .code .error {
color: red ;
font-weight: bold ;
font-family: sans-serif }
/* Uncomment (and remove this text!) to get reduced vertical space in
compound paragraphs.
div.compound .compound-first, div.compound .compound-middle {
margin-bottom: 0.5em }
div.compound .compound-last, div.compound .compound-middle {
margin-top: 0.5em }
*/
div.dedication {
margin: 2em 5em ;
text-align: center ;
font-style: italic }
div.dedication p.topic-title {
font-weight: bold ;
font-style: normal }
div.figure {
margin-left: 2em ;
margin-right: 2em }
div.footer, div.header {
clear: both;
font-size: smaller }
div.line-block {
display: block ;
margin-top: 1em ;
margin-bottom: 1em }
div.line-block div.line-block {
margin-top: 0 ;
margin-bottom: 0 ;
margin-left: 1.5em }
div.sidebar {
margin: 0 0 0.5em 1em ;
border: medium outset ;
padding: 1em ;
background-color: #ffffee ;
width: 40% ;
float: right ;
clear: right }
div.sidebar p.rubric {
font-family: sans-serif ;
font-size: medium }
div.system-messages {
margin: 5em }
div.system-messages h1 {
color: red }
div.system-message {
border: medium outset ;
padding: 1em }
div.system-message p.system-message-title {
color: red ;
font-weight: bold }
div.topic {
margin: 2em }
h1.section-subtitle, h2.section-subtitle, h3.section-subtitle,
h4.section-subtitle, h5.section-subtitle, h6.section-subtitle {
margin-top: 0.4em }
h1.title {
text-align: center }
h2.subtitle {
text-align: center }
hr.docutils {
width: 75% }
img.align-left, .figure.align-left, object.align-left {
clear: left ;
float: left ;
margin-right: 1em }
img.align-right, .figure.align-right, object.align-right {
clear: right ;
float: right ;
margin-left: 1em }
img.align-center, .figure.align-center, object.align-center {
display: block;
margin-left: auto;
margin-right: auto;
}
.align-left {
text-align: left }
.align-center {
clear: both ;
text-align: center }
.align-right {
text-align: right }
/* reset inner alignment in figures */
div.align-right {
text-align: inherit }
/* div.align-center * { */
/* text-align: left } */
ol.simple, ul.simple {
margin-bottom: 1em }
ol.arabic {
list-style: decimal }
ol.loweralpha {
list-style: lower-alpha }
ol.upperalpha {
list-style: upper-alpha }
ol.lowerroman {
list-style: lower-roman }
ol.upperroman {
list-style: upper-roman }
p.attribution {
text-align: right ;
margin-left: 50% }
p.caption {
font-style: italic }
p.credits {
font-style: italic ;
font-size: smaller }
p.label {
white-space: nowrap }
p.rubric {
font-weight: bold ;
font-size: larger ;
color: maroon ;
text-align: center }
p.sidebar-title {
font-family: sans-serif ;
font-weight: bold ;
font-size: larger }
p.sidebar-subtitle {
font-family: sans-serif ;
font-weight: bold }
p.topic-title {
font-weight: bold }
pre.address {
margin-bottom: 0 ;
margin-top: 0 ;
font: inherit }
pre.literal-block, pre.doctest-block, pre.math, pre.code {
margin-left: 2em ;
margin-right: 2em }
pre.code .ln { color: grey; } /* line numbers */
pre.code, code { background-color: #eeeeee }
pre.code .comment, code .comment { color: #5C6576 }
pre.code .keyword, code .keyword { color: #3B0D06; font-weight: bold }
pre.code .literal.string, code .literal.string { color: #0C5404 }
pre.code .name.builtin, code .name.builtin { color: #352B84 }
pre.code .deleted, code .deleted { background-color: #DEB0A1}
pre.code .inserted, code .inserted { background-color: #A3D289}
span.classifier {
font-family: sans-serif ;
font-style: oblique }
span.classifier-delimiter {
font-family: sans-serif ;
font-weight: bold }
span.interpreted {
font-family: sans-serif }
span.option {
white-space: nowrap }
span.pre {
white-space: pre }
span.problematic {
color: red }
span.section-subtitle {
/* font-size relative to parent (h1..h6 element) */
font-size: 80% }
table.citation {
border-left: solid 1px gray;
margin-left: 1px }
table.docinfo {
margin: 2em 4em }
table.docutils {
margin-top: 0.5em ;
margin-bottom: 0.5em }
table.footnote {
border-left: solid 1px black;
margin-left: 1px }
table.docutils td, table.docutils th,
table.docinfo td, table.docinfo th {
padding-left: 0.5em ;
padding-right: 0.5em ;
vertical-align: top }
table.docutils th.field-name, table.docinfo th.docinfo-name {
font-weight: bold ;
text-align: left ;
white-space: nowrap ;
padding-left: 0 }
/* "booktabs" style (no vertical lines) */
table.docutils.booktabs {
border: 0px;
border-top: 2px solid;
border-bottom: 2px solid;
border-collapse: collapse;
}
table.docutils.booktabs * {
border: 0px;
}
table.docutils.booktabs th {
border-bottom: thin solid;
text-align: left;
}
h1 tt.docutils, h2 tt.docutils, h3 tt.docutils,
h4 tt.docutils, h5 tt.docutils, h6 tt.docutils {
font-size: 100% }
ul.auto-toc {
list-style-type: none }
</style>
<style type="text/css">
/* example stylesheet for Docutils */
/* :Author: Günter Milde */
/* :Copyright: © 2012 G. Milde */
/* :License: This stylesheet is placed in the public domain. */
/* Syntax highlight rules for HTML documents generated with Docutils */
/* using the ``--syntax-highlight=long`` option (new in v. 0.9). */
/* This stylesheet implements Pygment's "default" style with less rules than */
/* pygments-default using class hierarchies. */
/* Use it as example for "handcrafted" styles with only few rules. */
.code { background: #f8f8f8; }
.code .comment { color: #008800; font-style: italic }
.code .error { border: 1px solid #FF0000 }
.code .generic.deleted { color: #A00000 }
.code .generic.emph { font-style: italic }
.code .generic.error { color: #FF0000 }
.code .generic.heading { color: #000080; font-weight: bold }
.code .generic.inserted { color: #00A000 }
.code .generic.output { color: #808080 }
.code .generic.prompt { color: #000080; font-weight: bold }
.code .generic.strong { font-weight: bold }
.code .generic.subheading { color: #800080; font-weight: bold }
.code .generic.traceback { color: #0040D0 }
.code .keyword { color: #AA22FF; font-weight: bold }
.code .keyword.pseudo { font-weight: normal }
.code .literal.number { color: #666666 }
.code .literal.string { color: #BB4444 }
.code .literal.string.doc { color: #BB4444; font-style: italic }
.code .literal.string.escape { color: #BB6622; font-weight: bold }
.code .literal.string.interpol { color: #BB6688; font-weight: bold }
.code .literal.string.other { color: #008000 }
.code .literal.string.regex { color: #BB6688 }
.code .literal.string.symbol { color: #B8860B }
.code .name.attribute { color: #BB4444 }
.code .name.builtin { color: #AA22FF }
.code .name.class { color: #0000FF }
.code .name.constant { color: #880000 }
.code .name.decorator { color: #AA22FF }
.code .name.entity { color: #999999; font-weight: bold }
.code .name.exception { color: #D2413A; font-weight: bold }
.code .name.function { color: #00A000 }
.code .name.label { color: #A0A000 }
.code .name.namespace { color: #0000FF; font-weight: bold }
.code .name.tag { color: #008000; font-weight: bold }
.code .name.variable { color: #B8860B }
.code .operator { color: #666666 }
.code .operator.word { color: #AA22FF; font-weight: bold }
.code .ln { margin-right: -0px }
</style>
</head>
<body>
<div class="document">
<p>In my first scala project I had the pleasure not only to learn scala, <a class="reference external" href="http://akka.io/">akka</a> and <a class="reference external" href="http://playframework.org/">play</a>
but also to work with <a class="reference external" href="https://github.com/eligosource/eventsourced">eventsourced</a>. Eventsourced is an akka-based open-source library
for implementing the concepts of event sourcing. The basic idea is that instead of persisting
the current state of the application - like you usually do when working with a relational database -,
you rather persist the individual messages that cause the state-changes. More details on
this concept can be found for example in the
<a class="reference external" href="https://github.com/eligosource/eventsourced/blob/master/README.md">documentation for eventsourced</a>,
its successor <a class="reference external" href="http://doc.akka.io/docs/akka/2.3.0/scala/persistence.html">akka-persistence</a> or
an <a class="reference external" href="http://martinfowler.com/eaaDev/EventSourcing.html">article</a> by Martin Fowler. This blog-post
focuses on a specific (typical) usage scenario and in particular on testing the various
aspects of it that are important for real life projects.</p>
<div class="section" id="akka-persistence-and-testing">
<h1>Akka-Persistence and Testing</h1>
<p>The example used in this blog-post is supposed to represent a complex Akka-based application (or parts of it) keeping its state in
memory and using akka-persistence to persist this state. The tests are integration style tests accessing the application
from <em>outside</em>. To keep things simple the application is just modelled as a simple CRUD service for generic items.
The service basically delegates all operations
to a stateful actor that persists its state through akka-persistence by using it as a write-ahead log (for command-sourcing).
So the service stands for the entire interface of the application (respectively the part under test) and the actor for
the arbitrarily complex logic behind this interface. In this case the interface is based on regular method calls, but the
same ideas apply for a message based interface where actors are addressed directly.</p>
<p>When it comes to testing the integration with akka-persistence, I see 4 important aspects that should be covered:</p>
<ol class="arabic simple">
<li>Are custom serializers used for serializing <tt class="docutils literal">PersistentMessages</tt> into the journal?</li>
<li>Does the application recover its state correctly after a restart when replaying the journal?</li>
<li>Does the application recover its state correctly after a restart when reading state from a snapshot?</li>
<li>Does the application recover its state correctly from an <em>old</em> journal of a previous release?</li>
</ol>
<p>In this blog-post I just cover the first point and will address the other issues in following posts.</p>
<div class="section" id="the-demo-application">
<h2>The Demo Application</h2>
<p>Let's first have a closer look at the application. As already stated the actor is
pretty simplistic:</p>
<pre class="code scala literal-block"><span class="ln"> 1 </span><span class="keyword">class</span> <span class="name class">ItemActor</span> <span class="keyword">extends</span> <span class="name class">Processor</span> <span class="operator">{</span>
<span class="ln"> 2 </span>
<span class="ln"> 3 </span> <span class="keyword">import</span> <span class="name namespace">ItemActor._</span>
<span class="ln"> 4 </span>
<span class="ln"> 5 </span> <span class="keyword">private</span> <span class="keyword">var</span> <span class="name">itemById</span><span class="keyword">:</span> <span class="keyword type">Map</span><span class="operator">[</span><span class="keyword type">ItemId</span>, <span class="keyword type">Item</span><span class="operator">]</span> <span class="keyword">=</span> <span class="name class">Map</span><span class="operator">.</span><span class="name">empty</span>
<span class="ln"> 6 </span> <span class="keyword">private</span> <span class="keyword">var</span> <span class="name">idCounter</span><span class="keyword">:</span> <span class="keyword type">Long</span> <span class="operator">=</span> <span class="literal number integer">0</span>
<span class="ln"> 7 </span>
<span class="ln"> 8 </span> <span class="keyword">def</span> <span class="name">receive</span> <span class="keyword">=</span> <span class="operator">{</span>
<span class="ln"> 9 </span> <span class="keyword">case</span> <span class="name class">Persistent</span><span class="operator">(</span><span class="name class">CreateItem</span><span class="operator">(</span><span class="name">template</span><span class="keyword">:</span> <span class="keyword type">ItemTemplate</span><span class="operator">),</span> <span class="keyword">_</span><span class="operator">)</span> <span class="keyword">=></span>
<span class="ln">10 </span> <span class="name">idCounter</span> <span class="operator">+=</span> <span class="literal number integer">1</span>
<span class="ln">11 </span> <span class="name">sender</span> <span class="operator">!</span> <span class="name">addItem</span><span class="operator">(</span><span class="name class">Item</span><span class="operator">(</span><span class="name class">ItemId</span><span class="operator">(</span><span class="name">idCounter</span><span class="operator">),</span> <span class="name">template</span><span class="operator">.</span><span class="name">description</span><span class="operator">))</span>
<span class="ln">12 </span>
<span class="ln">13 </span> <span class="keyword">case</span> <span class="name class">Persistent</span><span class="operator">(</span><span class="name class">UpdateItem</span><span class="operator">(</span><span class="name">item</span><span class="keyword">:</span> <span class="keyword type">Item</span><span class="operator">),</span> <span class="keyword">_</span><span class="operator">)</span> <span class="keyword">=></span>
<span class="ln">14 </span> <span class="name">sender</span> <span class="operator">!</span> <span class="operator">(</span><span class="keyword">if</span><span class="operator">(</span><span class="name">itemById</span><span class="operator">.</span><span class="name">isDefinedAt</span><span class="operator">(</span><span class="name">item</span><span class="operator">.</span><span class="name">id</span><span class="operator">))</span>
<span class="ln">15 </span> <span class="name">addItem</span><span class="operator">(</span><span class="name">item</span><span class="operator">)</span>
<span class="ln">16 </span> <span class="keyword">else</span>
<span class="ln">17 </span> <span class="name class">Failure</span><span class="operator">(</span><span class="keyword">new</span> <span class="name class">NonExistingItemCannotBeUpdatedException</span><span class="operator">(</span><span class="name">item</span><span class="operator">)))</span>
<span class="ln">18 </span>
<span class="ln">19 </span> <span class="keyword">case</span> <span class="name class">Persistent</span><span class="operator">(</span><span class="name class">DeleteItem</span><span class="operator">(</span><span class="name">itemId</span><span class="keyword">:</span> <span class="keyword type">ItemId</span><span class="operator">),</span> <span class="keyword">_</span><span class="operator">)</span> <span class="keyword">=></span>
<span class="ln">20 </span> <span class="keyword">val</span> <span class="name">deletedItem</span> <span class="keyword">=</span> <span class="name">itemById</span><span class="operator">.</span><span class="name">get</span><span class="operator">(</span><span class="name">itemId</span><span class="operator">)</span>
<span class="ln">21 </span> <span class="name">deletedItem</span><span class="operator">.</span><span class="name">foreach</span><span class="operator">(</span><span class="name">itemById</span> <span class="operator">-=</span> <span class="keyword">_</span><span class="operator">.</span><span class="name">id</span><span class="operator">)</span>
<span class="ln">22 </span> <span class="name">sender</span> <span class="operator">!</span> <span class="name">deletedItem</span><span class="operator">.</span><span class="name">fold</span><span class="operator">[</span><span class="keyword type">Try</span><span class="operator">[</span><span class="keyword type">Item</span><span class="operator">]](</span>
<span class="ln">23 </span> <span class="name class">Failure</span><span class="operator">(</span><span class="keyword">new</span> <span class="name class">NonExistingItemCannotBeDeletedException</span><span class="operator">(</span><span class="name">itemId</span><span class="operator">)))(</span>
<span class="ln">24 </span> <span class="name class">Success</span><span class="operator">.</span><span class="name">apply</span><span class="operator">)</span>
<span class="ln">25 </span>
<span class="ln">26 </span> <span class="keyword">case</span> <span class="name class">GetItem</span><span class="operator">(</span><span class="name">itemId</span><span class="keyword">:</span> <span class="keyword type">ItemId</span><span class="operator">)</span> <span class="keyword">=></span>
<span class="ln">27 </span> <span class="name">sender</span> <span class="operator">!</span> <span class="name">itemById</span><span class="operator">.</span><span class="name">get</span><span class="operator">(</span><span class="name">itemId</span><span class="operator">)</span>
<span class="ln">28 </span>
<span class="ln">29 </span> <span class="comment single">// SNAPSHOT BEGIN
</span><span class="ln">30 </span><span class="comment single"></span> <span class="keyword">case</span> <span class="name class">SaveSnapshot</span> <span class="keyword">=></span> <span class="name">saveSnapshot</span><span class="operator">(</span><span class="name class">ItemActorSnapshot</span><span class="operator">(</span><span class="name">itemById</span><span class="operator">,</span> <span class="name">idCounter</span><span class="operator">))</span>
<span class="ln">31 </span> <span class="keyword">case</span> <span class="name class">SaveSnapshotSuccess</span><span class="operator">(</span><span class="name">metadata</span><span class="operator">)</span> <span class="keyword">=></span>
<span class="ln">32 </span> <span class="keyword">case</span> <span class="name class">SaveSnapshotFailure</span><span class="operator">(</span><span class="name">metadata</span><span class="operator">,</span> <span class="name">cause</span><span class="operator">)</span> <span class="keyword">=></span>
<span class="ln">33 </span>
<span class="ln">34 </span> <span class="keyword">case</span> <span class="name class">SnapshotOffer</span><span class="operator">(</span><span class="keyword">_</span><span class="operator">,</span> <span class="name class">ItemActorSnapshot</span><span class="operator">(</span><span class="name">itemMap</span><span class="operator">,</span> <span class="name">lastId</span><span class="operator">))</span> <span class="keyword">=></span>
<span class="ln">35 </span> <span class="keyword">this</span><span class="operator">.</span><span class="name">itemById</span> <span class="keyword">=</span> <span class="name">itemMap</span>
<span class="ln">36 </span> <span class="keyword">this</span><span class="operator">.</span><span class="name">idCounter</span> <span class="keyword">=</span> <span class="name">lastId</span>
<span class="ln">37 </span> <span class="comment single">// SNAPSHOT END
</span><span class="ln">38 </span><span class="comment single"></span> <span class="operator">}</span>
<span class="ln">39 </span>
<span class="ln">40 </span> <span class="keyword">private</span> <span class="keyword">def</span> <span class="name">addItem</span><span class="operator">(</span><span class="name">item</span><span class="keyword">:</span> <span class="keyword type">Item</span><span class="operator">)</span><span class="keyword">:</span> <span class="keyword type">Try</span><span class="operator">[</span><span class="keyword type">Item</span><span class="operator">]</span> <span class="keyword">=</span> <span class="operator">{</span>
<span class="ln">41 </span> <span class="name">itemById</span> <span class="operator">+=</span> <span class="operator">(</span><span class="name">item</span><span class="operator">.</span><span class="name">id</span> <span class="operator">-></span> <span class="name">item</span><span class="operator">)</span>
<span class="ln">42 </span> <span class="name class">Success</span><span class="operator">(</span><span class="name">item</span><span class="operator">)</span>
<span class="ln">43 </span> <span class="operator">}</span>
<span class="ln">44 </span><span class="operator">}</span>
</pre>
<p>It extends <tt class="docutils literal">Processor</tt> so akka-persistence can take care of persisting
messages sent to it. The accepted messages are</p>
<ul class="simple">
<li><tt class="docutils literal">CreateItem</tt></li>
<li><tt class="docutils literal">UpdateItem</tt></li>
<li><tt class="docutils literal">DeleteItem</tt></li>
<li><tt class="docutils literal">GetItem</tt></li>
</ul>
<p>(The snapshot related messages can be ignored for the moment.)
They return the created, updated, deleted or requested <tt class="docutils literal">Item</tt> if the operation
was successful and a failure if it was not.
The first three are those that modify the actor's state and thus have to be logged
into the journal by akka-persistence. That is why they are wrapped
in a <tt class="docutils literal">Persistent</tt>, while the <tt class="docutils literal">GetItem</tt> can be received plainly. The state
is basically a <tt class="docutils literal">Map</tt> containing all <tt class="docutils literal">Item</tt>s by their ids (<tt class="docutils literal">itemById</tt> in line 5).</p>
<p>The corresponding domain classes <tt class="docutils literal">Item</tt> and <tt class="docutils literal">ItemTemplate</tt> are likewise simplistic:</p>
<pre class="code scala literal-block">
<span class="ln"> 1 </span><span class="keyword">class</span> <span class="name class">ItemId</span><span class="operator">(</span><span class="keyword">val</span> <span class="name">id</span><span class="keyword">:</span> <span class="keyword type">Long</span><span class="operator">)</span> <span class="keyword">extends</span> <span class="name class">AnyVal</span> <span class="keyword">with</span> <span class="name class">Serializable</span>
<span class="ln"> 2 </span><span class="keyword">object</span> <span class="name class">ItemId</span> <span class="operator">{</span>
<span class="ln"> 3 </span> <span class="keyword">def</span> <span class="name">apply</span><span class="operator">(</span><span class="name">id</span><span class="keyword">:</span> <span class="keyword type">Long</span><span class="operator">)</span><span class="keyword">:</span> <span class="keyword type">ItemId</span> <span class="operator">=</span> <span class="keyword">new</span> <span class="name class">ItemId</span><span class="operator">(</span><span class="name">id</span><span class="operator">)</span>
<span class="ln"> 4 </span><span class="operator">}</span>
<span class="ln"> 5 </span><span class="keyword">case</span> <span class="keyword">class</span> <span class="name class">ItemTemplate</span><span class="operator">(</span><span class="name">description</span><span class="keyword">:</span> <span class="keyword type">String</span><span class="operator">)</span> <span class="keyword">extends</span> <span class="name class">CommonItem</span>
<span class="ln"> 6 </span><span class="keyword">case</span> <span class="keyword">class</span> <span class="name class">Item</span><span class="operator">(</span><span class="name">id</span><span class="keyword">:</span> <span class="keyword type">ItemId</span><span class="operator">,</span> <span class="name">description</span><span class="keyword">:</span> <span class="keyword type">String</span><span class="operator">)</span> <span class="keyword">extends</span> <span class="name class">CommonItem</span> <span class="operator">{</span>
<span class="ln"> 7 </span> <span class="keyword">def</span> <span class="name">asTemplate</span><span class="keyword">:</span> <span class="keyword type">ItemTemplate</span> <span class="operator">=</span> <span class="name class">ItemTemplate</span><span class="operator">(</span><span class="name">description</span><span class="operator">)</span>
<span class="ln"> 8 </span> <span class="keyword">def</span> <span class="name">withTemplate</span><span class="operator">(</span><span class="name">template</span><span class="keyword">:</span> <span class="keyword type">ItemTemplate</span><span class="operator">)</span><span class="keyword">:</span> <span class="keyword type">Item</span> <span class="operator">=</span>
<span class="ln"> 9 </span> <span class="name">copy</span><span class="operator">(</span><span class="name">description</span> <span class="keyword">=</span> <span class="name">template</span><span class="operator">.</span><span class="name">description</span><span class="operator">)</span>
<span class="ln">10 </span><span class="operator">}</span>
</pre>
<p>An <tt class="docutils literal">Item</tt> contains an id and as representative for a more complex structure a
single field called <tt class="docutils literal">description</tt>. The <tt class="docutils literal">ItemTemplate</tt> is used for creating
new <tt class="docutils literal">Item</tt>s when the id is not yet known.</p>
<p>As described above the interface to our application is a service that wraps the
<tt class="docutils literal">ItemActor</tt>:</p>
<pre class="code scala literal-block">
<span class="ln"> 1 </span><span class="keyword">class</span> <span class="name class">ActorService</span><span class="operator">(</span><span class="name">itemActor</span><span class="keyword">:</span> <span class="keyword type">ActorRef</span><span class="operator">)(</span><span class="keyword">implicit</span> <span class="name">timeout</span><span class="keyword">:</span> <span class="keyword type">Timeout</span><span class="operator">)</span> <span class="keyword">extends</span> <span class="name class">ItemService</span> <span class="operator">{</span>
<span class="ln"> 2 </span> <span class="keyword">def</span> <span class="name">find</span><span class="operator">(</span><span class="name">itemId</span><span class="keyword">:</span> <span class="keyword type">ItemId</span><span class="operator">)</span><span class="keyword">:</span> <span class="keyword type">Future</span><span class="operator">[</span><span class="keyword type">Option</span><span class="operator">[</span><span class="keyword type">Item</span><span class="operator">]]</span> <span class="keyword">=</span>
<span class="ln"> 3 </span> <span class="operator">(</span><span class="name">itemActor</span> <span class="operator">?</span> <span class="name class">GetItem</span><span class="operator">(</span><span class="name">itemId</span><span class="operator">)).</span><span class="name">mapTo</span><span class="operator">[</span><span class="keyword type">Option</span><span class="operator">[</span><span class="keyword type">Item</span><span class="operator">]]</span>
<span class="ln"> 4 </span>
<span class="ln"> 5 </span> <span class="keyword">def</span> <span class="name">create</span><span class="operator">(</span><span class="name">template</span><span class="keyword">:</span> <span class="keyword type">ItemTemplate</span><span class="operator">)</span><span class="keyword">:</span> <span class="keyword type">Future</span><span class="operator">[</span><span class="keyword type">Try</span><span class="operator">[</span><span class="keyword type">Item</span><span class="operator">]]</span> <span class="keyword">=</span>
<span class="ln"> 6 </span> <span class="operator">(</span><span class="name">itemActor</span> <span class="operator">?</span> <span class="name class">Persistent</span><span class="operator">(</span><span class="name class">CreateItem</span><span class="operator">(</span><span class="name">template</span><span class="operator">))).</span><span class="name">mapTo</span><span class="operator">[</span><span class="keyword type">Try</span><span class="operator">[</span><span class="keyword type">Item</span><span class="operator">]]</span>
<span class="ln"> 7 </span>
<span class="ln"> 8 </span> <span class="keyword">def</span> <span class="name">delete</span><span class="operator">(</span><span class="name">itemId</span><span class="keyword">:</span> <span class="keyword type">ItemId</span><span class="operator">)</span><span class="keyword">:</span> <span class="keyword type">Future</span><span class="operator">[</span><span class="keyword type">Try</span><span class="operator">[</span><span class="keyword type">Item</span><span class="operator">]]</span> <span class="keyword">=</span>
<span class="ln"> 9 </span> <span class="operator">(</span><span class="name">itemActor</span> <span class="operator">?</span> <span class="name class">Persistent</span><span class="operator">(</span><span class="name class">DeleteItem</span><span class="operator">(</span><span class="name">itemId</span><span class="operator">))).</span><span class="name">mapTo</span><span class="operator">[</span><span class="keyword type">Try</span><span class="operator">[</span><span class="keyword type">Item</span><span class="operator">]]</span>
<span class="ln">10 </span>
<span class="ln">11 </span> <span class="keyword">def</span> <span class="name">update</span><span class="operator">(</span><span class="name">item</span><span class="keyword">:</span> <span class="keyword type">Item</span><span class="operator">)</span><span class="keyword">:</span> <span class="keyword type">Future</span><span class="operator">[</span><span class="keyword type">Try</span><span class="operator">[</span><span class="keyword type">Item</span><span class="operator">]]</span> <span class="keyword">=</span>
<span class="ln">12 </span> <span class="operator">(</span><span class="name">itemActor</span> <span class="operator">?</span> <span class="name class">Persistent</span><span class="operator">(</span><span class="name class">UpdateItem</span><span class="operator">(</span><span class="name">item</span><span class="operator">))).</span><span class="name">mapTo</span><span class="operator">[</span><span class="keyword type">Try</span><span class="operator">[</span><span class="keyword type">Item</span><span class="operator">]]</span>
<span class="ln">13 </span><span class="operator">}</span>
</pre>
<p>For each of the four commands we have seen above it offers corresponding
methods. In each case the implementation creates the corresponding command, sends
it to the actor (with ask) and returns the result wrapped in a Future to the caller.
As we have already seen on the receiving side, those commands that modify the
application's state are wrapped in a <tt class="docutils literal">Persistent</tt>.</p>
<p>An <tt class="docutils literal">ItemApplication</tt> rounds off the implementation and takes care of the initialization
and dependency injection as well as the proper configuration of the akka-system.</p>
<pre class="code scala literal-block">
<span class="ln"> 1 </span><span class="keyword">class</span> <span class="name class">ItemApplication</span><span class="operator">(</span><span class="name">overwriteConfig</span><span class="keyword">:</span> <span class="keyword type">Config</span> <span class="operator">=</span> <span class="name class">ConfigFactory</span><span class="operator">.</span><span class="name">empty</span><span class="operator">())</span> <span class="operator">{</span>
<span class="ln"> 2 </span>
<span class="ln"> 3 </span> <span class="keyword">val</span> <span class="name">akkaSystemConfig</span><span class="keyword">:</span> <span class="keyword type">Config</span> <span class="operator">=</span> <span class="name">overwriteConfig</span><span class="operator">.</span><span class="name">withFallback</span><span class="operator">(</span>
<span class="ln"> 4 </span> <span class="name">akkaSerializerConfig</span><span class="operator">[</span><span class="keyword type">Command</span>, <span class="keyword type">CommandSerializer</span><span class="operator">])</span>
<span class="ln"> 5 </span>
<span class="ln"> 6 </span> <span class="keyword">private</span> <span class="keyword">val</span> <span class="name">config</span> <span class="keyword">=</span> <span class="name">akkaSystemConfig</span><span class="operator">.</span><span class="name">withFallback</span><span class="operator">(</span><span class="name class">ConfigFactory</span><span class="operator">.</span><span class="name">load</span><span class="operator">())</span>
<span class="ln"> 7 </span> <span class="keyword">val</span> <span class="name">system</span> <span class="keyword">=</span> <span class="name class">ActorSystem</span><span class="operator">(</span><span class="name">classOf</span><span class="operator">[</span><span class="keyword type">ItemApplication</span><span class="operator">].</span><span class="name">getSimpleName</span><span class="operator">,</span> <span class="name">config</span><span class="operator">)</span>
<span class="ln"> 8 </span> <span class="keyword">protected</span> <span class="keyword">val</span> <span class="name">itemActor</span> <span class="keyword">=</span>
<span class="ln"> 9 </span> <span class="name">system</span><span class="operator">.</span><span class="name">actorOf</span><span class="operator">(</span><span class="name class">ItemActor</span><span class="operator">.</span><span class="name">props</span><span class="operator">(),</span> <span class="name">classOf</span><span class="operator">[</span><span class="keyword type">ItemActor</span><span class="operator">].</span><span class="name">getSimpleName</span><span class="operator">)</span>
<span class="ln">10 </span> <span class="keyword">val</span> <span class="name">itemService</span> <span class="keyword">=</span> <span class="keyword">new</span> <span class="name class">ActorService</span><span class="operator">(</span><span class="name">itemActor</span><span class="operator">)(</span><span class="name class">Timeout</span><span class="operator">(</span><span class="literal number float">5.</span><span class="name">seconds</span><span class="operator">))</span>
<span class="ln">11 </span>
<span class="ln">12 </span> <span class="keyword">private</span> <span class="keyword">def</span> <span class="name">akkaSerializerConfig</span><span class="operator">[</span><span class="keyword type">M</span> <span class="keyword type">:</span> <span class="keyword type">ClassTag</span>, <span class="keyword type">S</span> <span class="keyword"><:</span> <span class="keyword type">Serializer</span> <span class="keyword type">:</span> <span class="keyword type">ClassTag</span><span class="operator">]</span>
<span class="ln">13 </span> <span class="keyword">:</span> <span class="keyword type">Config</span> <span class="operator">=</span> <span class="operator">{</span>
<span class="ln">14 </span> <span class="keyword">val</span> <span class="name">messageClassName</span> <span class="keyword">=</span> <span class="name">classTag</span><span class="operator">[</span><span class="keyword type">M</span><span class="operator">].</span><span class="name">runtimeClass</span><span class="operator">.</span><span class="name">getName</span>
<span class="ln">15 </span> <span class="keyword">val</span> <span class="name">serializerClassName</span> <span class="keyword">=</span> <span class="name">classTag</span><span class="operator">[</span><span class="keyword type">S</span><span class="operator">].</span><span class="name">runtimeClass</span><span class="operator">.</span><span class="name">getName</span>
<span class="ln">16 </span> <span class="name class">ConfigFactory</span><span class="operator">.</span><span class="name">parseString</span><span class="operator">(</span><span class="name">s</span><span class="literal string">"""
</span><span class="ln">17 </span><span class="literal string"> |akka.actor {
</span><span class="ln">18 </span><span class="literal string"> | serializers {
</span><span class="ln">19 </span><span class="literal string"> | "$messageClassName" = "$serializerClassName"
</span><span class="ln">20 </span><span class="literal string"> | }
</span><span class="ln">21 </span><span class="literal string"> | serialization-bindings {
</span><span class="ln">22 </span><span class="literal string"> | "$messageClassName" = "$messageClassName"
</span><span class="ln">23 </span><span class="literal string"> | }
</span><span class="ln">24 </span><span class="literal string"> |}
</span><span class="ln">25 </span><span class="literal string"> |"""</span><span class="operator">.</span><span class="name">stripMargin</span><span class="operator">)</span>
<span class="ln">26 </span> <span class="operator">}</span>
<span class="ln">27 </span>
<span class="ln">28 </span> <span class="keyword">def</span> <span class="name">shutdown</span><span class="operator">()</span><span class="keyword">:</span> <span class="keyword type">Unit</span> <span class="operator">=</span> <span class="operator">{</span>
<span class="ln">29 </span> <span class="name">system</span><span class="operator">.</span><span class="name">shutdown</span><span class="operator">()</span>
<span class="ln">30 </span> <span class="name">system</span><span class="operator">.</span><span class="name">awaitTermination</span><span class="operator">()</span>
<span class="ln">31 </span> <span class="operator">}</span>
<span class="ln">32 </span><span class="operator">}</span>
</pre>
<p>The interesting part here is the configuration of the serializers for the akka-system.
It is created in a hard-coded manner through the method <tt class="docutils literal">akkaSerializerConfig</tt>,
but what is it good for?</p>
</div>
<div class="section" id="customer-serializers">
<h2>Customer Serializers</h2>
<p>When akka-persistence writes the persistent messages received by a <tt class="docutils literal">Processor</tt> to a
journal it makes use of standard
<a class="reference external" href="http://doc.akka.io/docs/akka/2.3.0/scala/serialization.html">akka-serialization</a>.
By default this uses <a class="reference external" href="http://docs.oracle.com/javase/7/docs/platform/serialization/spec/serialTOC.html">java-serialization</a>. That means all messages are first
converted to an <tt class="docutils literal">Array[Byte]</tt> through java-serialization and afterwards this array is
written to the journal. When the journal is replayed, the data is read from disk
as <tt class="docutils literal">Array[Byte]</tt> and this is converted back to messages again through java-(de)serialization.
If you just go with the default you can quickly run into compatibility problems when
trying to read an existing journal with a new version of the application as plain
java-serialization by default does not ensure compatibility of the serialized data
to evolved classes. To avoid
this kind of problems you can either</p>
<ul class="simple">
<li>migrate the journal files to the new format before starting the new
application. However when using java-serialization this can be a challenging
task by itself, since one application must be able to read an old serialized
form and write the new serialized form of instances of the <em>same</em> class (in
different versions) or</li>
<li>cautiously ensure that your messages stay
downwards compatible to their serialized representation (e.g. by declaring
<tt class="docutils literal">serialVersionUID</tt>
explicitly and things like implementing <tt class="docutils literal">readObject, writeObject</tt> methods)
or</li>
<li>you can make
akka use custom serializers for your messages.</li>
</ul>
<p>The latter gives you very good
control
over maintaining compatibility of the serialized representation of your message-instances
with the implementation of the corresponding classes. Based on my experience I can
recommend using custom serializers. The serializers should of course use a serial
representation that can easily be kept downwards compatible. JSON is a valid
alternative (even though certainly not one that performs best) as</p>
<ul class="simple">
<li>it is very flexible when it comes to migrating structures</li>
<li>it might be used already in a program that offers a REST interface where
the resources are represented in form of JSON documents</li>
</ul>
<p>In addition to this it was very convenient to use it for this blog-post as
the <a class="reference external" href="http://www.playframework.com/documentation/2.2.x/ScalaJson">play-json</a>-lib with
<a class="reference external" href="http://www.playframework.com/documentation/2.2.x/ScalaJsonInception">macro-inception</a>
allows to write JSON (de-)serializers of case-classes in basically one line
of code. The corresponding akka-serializer is implemented in the class
<tt class="docutils literal">CommandSerializers</tt> for all messages that derive from <tt class="docutils literal">Command</tt> (see line 3-4)
which are
all messages to be written into the journal in our case. akka allows to
configure custom serializers through the configuration object passed to the
akka-system. Typically the corresponding configuration is read from a
configuration file like <tt class="docutils literal">application.conf</tt> or <tt class="docutils literal">reference.conf</tt>. As the
application needs to ensure backwards compatibility of the serialized
representation of the message in the journal I do not consider the custom
serializers for these classes as part of the configuration that could be
modified by an administrator of the application. That is why this
<em>configuration</em> is provided in a hard-coded manner in <tt class="docutils literal">ItemApplication</tt>
(line 12-26).</p>
</div>
<div class="section" id="the-test">
<h2>The test</h2>
<p>The tricky part about this configuration is, that akka will ignore it silently if it contains errors. Thus we we will not notice any
problems during runtime, if the configuration is wrong. In that case akka will silently use the default
java-serialization, a fact that might remain unnoticed until the first migration
problem appears. Even if the code-based configuration ensures that we
cannot introduce typos in class-names there are still other things that
might go wrong as we will see soon. That is why it is important to test that
the configured
serializers are actually used by akka.</p>
<p>For this we have the following test:</p>
<pre class="code scala literal-block">
<span class="ln"> 1 </span> <span class="name">in</span> <span class="operator">{</span> <span class="name">application</span> <span class="keyword">=></span>
<span class="ln"> 2 </span> <span class="keyword">val</span> <span class="name">serializer</span> <span class="keyword">=</span> <span class="name class">SerializationExtension</span><span class="operator">(</span><span class="name">application</span><span class="operator">.</span><span class="name">system</span><span class="operator">)</span>
<span class="ln"> 3 </span>
<span class="ln"> 4 </span> <span class="name class">Seq</span><span class="operator">(</span><span class="name class">CreateItem</span><span class="operator">(</span><span class="name">newItemTemplate</span><span class="operator">()),</span>
<span class="ln"> 5 </span> <span class="name class">UpdateItem</span><span class="operator">(</span><span class="name">newItem</span><span class="operator">()),</span>
<span class="ln"> 6 </span> <span class="name class">DeleteItem</span><span class="operator">(</span><span class="name">newId</span><span class="operator">)).</span><span class="name">foreach</span> <span class="operator">{</span> <span class="name">expected</span> <span class="keyword">=></span>
<span class="ln"> 7 </span>
<span class="ln"> 8 </span> <span class="keyword">val</span> <span class="name">bytes</span> <span class="keyword">=</span> <span class="name">successOf</span><span class="operator">(</span><span class="name">serializer</span><span class="operator">.</span><span class="name">serialize</span><span class="operator">(</span><span class="name">expected</span><span class="operator">))</span>
<span class="ln"> 9 </span> <span class="keyword">val</span> <span class="name">actual</span> <span class="keyword">=</span> <span class="keyword">new</span> <span class="name class">CommandSerializer</span><span class="operator">()</span>
<span class="ln">10 </span> <span class="operator">.</span><span class="name">fromBinary</span><span class="operator">(</span><span class="name">bytes</span><span class="operator">,</span> <span class="name class">Some</span><span class="operator">(</span><span class="name">expected</span><span class="operator">.</span><span class="name">getClass</span><span class="operator">))</span>
<span class="ln">11 </span>
<span class="ln">12 </span> <span class="name">actual</span> <span class="name">should</span> <span class="name">be</span> <span class="operator">(</span><span class="name">expected</span><span class="operator">)</span>
<span class="ln">13 </span> <span class="operator">}</span>
<span class="ln">14 </span> <span class="operator">}</span>
</pre>
<p>At first this test retrieves the <tt class="docutils literal">SerializationExtension</tt> from the
akka-system of the application. Then it uses it to serialize each of the
possible command-objects and afterwards de-serializes them with the
<tt class="docutils literal">CommandSerializer</tt> that is also expected to be used by the
<tt class="docutils literal">SerializationExtension</tt>. If the de-serialized command equals the original
one
we can be sure that the serialization config was correct.</p>
<p>The first test-run discloses that there is indeed an error in the
configuration:</p>
<pre class="literal-block">
[WARN] [...] [...] [akka.serialization.Serialization (akka://ItemApplication)]
Multiple serializers found for class
de.blogspot.volkersyadb.akkapersistence.ItemActor$CreateItem,
choosing first: Vector(
(interface java.io.Serializable,akka.serialization.JavaSerializer@3545fe3b),
(interface de...ItemActor$Command,de...CommandSerializer@635eed0))
Unexpected character ('�' (code 65533 / 0xfffd)): expected a valid value
(number, String, array, object, 'true', 'false' or 'null')
</pre>
<p>As you can see <tt class="docutils literal">bytes</tt> does not contain valid JSON and the parser complains
about an unexpected character. However the more interesting output is the
line above that. akka warns about the fact that there are multiple serializers
for the command-object and it chooses an arbitrary one which happens to be
the java-serializer in this case. If there are multiple alternative
serializers for a class the akka documentation states:</p>
<blockquote>
You only need to specify the name of an interface or abstract base class of
the messages. In case of ambiguity, i.e. the message implements several of
the configured classes, the most specific configured class will be used,
i.e. the one of which all other candidates are superclasses. If this
condition cannot be met, because e.g. java.io.Serializable and
MyOwnSerializable both apply and neither is a subtype of the other,
a warning will be issued.</blockquote>
<p>That means our interface (<tt class="docutils literal">Command</tt>) must be more specific than
<tt class="docutils literal">Serializable</tt>. We can easily achieve this if we make <tt class="docutils literal">Command</tt> extend
<tt class="docutils literal">Serializable</tt>. As soon as we have this the test runs through just fine.</p>
<p>Note that for this test to be relevant, it is important that it uses the same
configuration as is used in production and not a configuration that is
specifically made up for the test. In our case this is ensured by the
<tt class="docutils literal">ItemApplicationFixture</tt> that prepares and cleans the <tt class="docutils literal">ItemApplication</tt> up
for each test. Let's have a look at the central method:</p>
<pre class="code scala literal-block">
<span class="ln"> 1 </span> <span class="keyword">def</span> <span class="name">withApplication</span><span class="operator">[</span><span class="keyword type">A</span><span class="operator">](</span><span class="name">persistDir</span><span class="keyword">:</span> <span class="keyword type">File</span><span class="operator">)(</span><span class="name">block</span><span class="keyword">:</span> <span class="keyword type">TestApplication</span> <span class="operator">=></span> <span class="name">A</span><span class="operator">)</span> <span class="keyword">=</span> <span class="operator">{</span>
<span class="ln"> 2 </span> <span class="keyword">val</span> <span class="name">tmpDirPersistenceConfig</span> <span class="keyword">=</span> <span class="name class">ConfigFactory</span><span class="operator">.</span><span class="name">parseMap</span><span class="operator">(</span>
<span class="ln"> 3 </span> <span class="name class">Map</span><span class="operator">(</span><span class="name class">JournalDirConfig</span> <span class="operator">-></span> <span class="keyword">new</span> <span class="name class">File</span><span class="operator">(</span><span class="name">persistDir</span><span class="operator">,</span> <span class="literal string">"journal"</span><span class="operator">).</span><span class="name">getPath</span><span class="operator">,</span>
<span class="ln"> 4 </span> <span class="name class">NativeLevelDbConfig</span> <span class="operator">-></span> <span class="keyword constant">false</span><span class="operator">.</span><span class="name">toString</span><span class="operator">,</span>
<span class="ln"> 5 </span> <span class="name class">SnapshotDirConfig</span> <span class="operator">-></span> <span class="keyword">new</span> <span class="name class">File</span><span class="operator">(</span><span class="name">persistDir</span><span class="operator">,</span> <span class="literal string">"snapshots"</span><span class="operator">).</span><span class="name">getPath</span><span class="operator">)</span>
<span class="ln"> 6 </span> <span class="operator">.</span><span class="name">asJava</span><span class="operator">)</span>
<span class="ln"> 7 </span> <span class="keyword">val</span> <span class="name">application</span> <span class="keyword">=</span> <span class="name">newItemApplication</span><span class="operator">(</span><span class="name">tmpDirPersistenceConfig</span><span class="operator">)</span>
<span class="ln"> 8 </span> <span class="name">ultimately</span><span class="operator">(</span><span class="name">application</span><span class="operator">.</span><span class="name">shutdown</span><span class="operator">())(</span><span class="name">block</span><span class="operator">(</span><span class="name">application</span><span class="operator">))</span>
<span class="ln"> 9 </span> <span class="operator">}</span>
<span class="ln">10 </span>
<span class="ln">11 </span> <span class="keyword">def</span> <span class="name">newItemApplication</span><span class="operator">(</span><span class="name">config</span><span class="keyword">:</span> <span class="keyword type">Config</span><span class="operator">)</span> <span class="keyword">=</span>
<span class="ln">12 </span> <span class="keyword">new</span> <span class="name class">ItemApplication</span><span class="operator">(</span><span class="name">config</span><span class="operator">)</span> <span class="keyword">with</span> <span class="name class">ItemApplicationTestExtensions</span>
</pre>
<p>As you can see it uses the <em>real</em> <tt class="docutils literal">ItemApplication</tt> (line 12) that is also used for production.
It does provide test-specific extensions (<tt class="docutils literal">ItemApplicationTestExtensions</tt>) and
config (line 2-6) to <tt class="docutils literal">ItemApplication</tt> but just to ease writing the tests and
to ensure that
the journals and snapshots for the tests are written to a temporary folder
that is cleaned up after the test and that the java-leveldb implementation is
used instead of the native one which is perfectly fine for testing and
typically comes with less problems when running the tests in different
environments.</p>
<p>This concludes the first blog-post about akka-persistence and testing. I will
cover the other important topics about recovery, snapshots and migration in
following posts so stay tuned.</p>
</div>
</div>
</div>
</body>
</html>
Volker Stampahttp://www.blogger.com/profile/03277007007448976514noreply@blogger.com0tag:blogger.com,1999:blog-3938166335394683606.post-89433897339036534362014-01-19T11:58:00.000+01:002014-04-09T21:28:19.596+02:00Using for-comprehensions to build Query-APIs<?xml version="1.0" encoding="utf-8" ?>
<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">
<html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en" lang="en">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8" />
<meta name="generator" content="Docutils 0.11: http://docutils.sourceforge.net/" />
<title></title>
<style type="text/css">
/*
:Author: David Goodger (goodger@python.org)
:Id: $Id: html4css1.css 7614 2013-02-21 15:55:51Z milde $
:Copyright: This stylesheet has been placed in the public domain.
Default cascading style sheet for the HTML output of Docutils.
See http://docutils.sf.net/docs/howto/html-stylesheets.html for how to
customize this style sheet.
*/
/* used to remove borders from tables and images */
.borderless, table.borderless td, table.borderless th {
border: 0 }
table.borderless td, table.borderless th {
/* Override padding for "table.docutils td" with "! important".
The right padding separates the table cells. */
padding: 0 0.5em 0 0 ! important }
.first {
/* Override more specific margin styles with "! important". */
margin-top: 0 ! important }
.last, .with-subtitle {
margin-bottom: 0 ! important }
.hidden {
display: none }
a.toc-backref {
text-decoration: none ;
color: black }
blockquote.epigraph {
margin: 2em 5em ; }
dl.docutils dd {
margin-bottom: 0.5em }
object[type="image/svg+xml"], object[type="application/x-shockwave-flash"] {
overflow: hidden;
}
/* Uncomment (and remove this text!) to get bold-faced definition list terms
dl.docutils dt {
font-weight: bold }
*/
div.abstract {
margin: 2em 5em }
div.abstract p.topic-title {
font-weight: bold ;
text-align: center }
div.admonition, div.attention, div.caution, div.danger, div.error,
div.hint, div.important, div.note, div.tip, div.warning {
margin: 2em ;
border: medium outset ;
padding: 1em }
div.admonition p.admonition-title, div.hint p.admonition-title,
div.important p.admonition-title, div.note p.admonition-title,
div.tip p.admonition-title {
font-weight: bold ;
font-family: sans-serif }
div.attention p.admonition-title, div.caution p.admonition-title,
div.danger p.admonition-title, div.error p.admonition-title,
div.warning p.admonition-title, .code .error {
color: red ;
font-weight: bold ;
font-family: sans-serif }
/* Uncomment (and remove this text!) to get reduced vertical space in
compound paragraphs.
div.compound .compound-first, div.compound .compound-middle {
margin-bottom: 0.5em }
div.compound .compound-last, div.compound .compound-middle {
margin-top: 0.5em }
*/
div.dedication {
margin: 2em 5em ;
text-align: center ;
font-style: italic }
div.dedication p.topic-title {
font-weight: bold ;
font-style: normal }
div.figure {
margin-left: 2em ;
margin-right: 2em }
div.footer, div.header {
clear: both;
font-size: smaller }
div.line-block {
display: block ;
margin-top: 1em ;
margin-bottom: 1em }
div.line-block div.line-block {
margin-top: 0 ;
margin-bottom: 0 ;
margin-left: 1.5em }
div.sidebar {
margin: 0 0 0.5em 1em ;
border: medium outset ;
padding: 1em ;
background-color: #ffffee ;
width: 40% ;
float: right ;
clear: right }
div.sidebar p.rubric {
font-family: sans-serif ;
font-size: medium }
div.system-messages {
margin: 5em }
div.system-messages h1 {
color: red }
div.system-message {
border: medium outset ;
padding: 1em }
div.system-message p.system-message-title {
color: red ;
font-weight: bold }
div.topic {
margin: 2em }
h1.section-subtitle, h2.section-subtitle, h3.section-subtitle,
h4.section-subtitle, h5.section-subtitle, h6.section-subtitle {
margin-top: 0.4em }
h1.title {
text-align: center }
h2.subtitle {
text-align: center }
hr.docutils {
width: 75% }
img.align-left, .figure.align-left, object.align-left {
clear: left ;
float: left ;
margin-right: 1em }
img.align-right, .figure.align-right, object.align-right {
clear: right ;
float: right ;
margin-left: 1em }
img.align-center, .figure.align-center, object.align-center {
display: block;
margin-left: auto;
margin-right: auto;
}
.align-left {
text-align: left }
.align-center {
clear: both ;
text-align: center }
.align-right {
text-align: right }
/* reset inner alignment in figures */
div.align-right {
text-align: inherit }
/* div.align-center * { */
/* text-align: left } */
ol.simple, ul.simple {
margin-bottom: 1em }
ol.arabic {
list-style: decimal }
ol.loweralpha {
list-style: lower-alpha }
ol.upperalpha {
list-style: upper-alpha }
ol.lowerroman {
list-style: lower-roman }
ol.upperroman {
list-style: upper-roman }
p.attribution {
text-align: right ;
margin-left: 50% }
p.caption {
font-style: italic }
p.credits {
font-style: italic ;
font-size: smaller }
p.label {
white-space: nowrap }
p.rubric {
font-weight: bold ;
font-size: larger ;
color: maroon ;
text-align: center }
p.sidebar-title {
font-family: sans-serif ;
font-weight: bold ;
font-size: larger }
p.sidebar-subtitle {
font-family: sans-serif ;
font-weight: bold }
p.topic-title {
font-weight: bold }
pre.address {
margin-bottom: 0 ;
margin-top: 0 ;
font: inherit }
pre.literal-block, pre.doctest-block, pre.math, pre.code {
margin-left: 2em ;
margin-right: 2em }
pre.code .ln { color: grey; } /* line numbers */
pre.code, code { background-color: #eeeeee }
pre.code .comment, code .comment { color: #5C6576 }
pre.code .keyword, code .keyword { color: #3B0D06; font-weight: bold }
pre.code .literal.string, code .literal.string { color: #0C5404 }
pre.code .name.builtin, code .name.builtin { color: #352B84 }
pre.code .deleted, code .deleted { background-color: #DEB0A1}
pre.code .inserted, code .inserted { background-color: #A3D289}
span.classifier {
font-family: sans-serif ;
font-style: oblique }
span.classifier-delimiter {
font-family: sans-serif ;
font-weight: bold }
span.interpreted {
font-family: sans-serif }
span.option {
white-space: nowrap }
span.pre {
white-space: pre }
span.problematic {
color: red }
span.section-subtitle {
/* font-size relative to parent (h1..h6 element) */
font-size: 80% }
table.citation {
border-left: solid 1px gray;
margin-left: 1px }
table.docinfo {
margin: 2em 4em }
table.docutils {
margin-top: 0.5em ;
margin-bottom: 0.5em }
table.footnote {
border-left: solid 1px black;
margin-left: 1px }
table.docutils td, table.docutils th,
table.docinfo td, table.docinfo th {
padding-left: 0.5em ;
padding-right: 0.5em ;
vertical-align: top }
table.docutils th.field-name, table.docinfo th.docinfo-name {
font-weight: bold ;
text-align: left ;
white-space: nowrap ;
padding-left: 0 }
/* "booktabs" style (no vertical lines) */
table.docutils.booktabs {
border: 0px;
border-top: 2px solid;
border-bottom: 2px solid;
border-collapse: collapse;
}
table.docutils.booktabs * {
border: 0px;
}
table.docutils.booktabs th {
border-bottom: thin solid;
text-align: left;
}
h1 tt.docutils, h2 tt.docutils, h3 tt.docutils,
h4 tt.docutils, h5 tt.docutils, h6 tt.docutils {
font-size: 100% }
ul.auto-toc {
list-style-type: none }
</style>
<style type="text/css">
/* example stylesheet for Docutils */
/* :Author: Günter Milde */
/* :Copyright: © 2012 G. Milde */
/* :License: This stylesheet is placed in the public domain. */
/* Syntax highlight rules for HTML documents generated with Docutils */
/* using the ``--syntax-highlight=long`` option (new in v. 0.9). */
/* This stylesheet implements Pygment's "default" style with less rules than */
/* pygments-default using class hierarchies. */
/* Use it as example for "handcrafted" styles with only few rules. */
.code { background: #f8f8f8; }
.code .comment { color: #008800; font-style: italic }
.code .error { border: 1px solid #FF0000 }
.code .generic.deleted { color: #A00000 }
.code .generic.emph { font-style: italic }
.code .generic.error { color: #FF0000 }
.code .generic.heading { color: #000080; font-weight: bold }
.code .generic.inserted { color: #00A000 }
.code .generic.output { color: #808080 }
.code .generic.prompt { color: #000080; font-weight: bold }
.code .generic.strong { font-weight: bold }
.code .generic.subheading { color: #800080; font-weight: bold }
.code .generic.traceback { color: #0040D0 }
.code .keyword { color: #AA22FF; font-weight: bold }
.code .keyword.pseudo { font-weight: normal }
.code .literal.number { color: #666666 }
.code .literal.string { color: #BB4444 }
.code .literal.string.doc { color: #BB4444; font-style: italic }
.code .literal.string.escape { color: #BB6622; font-weight: bold }
.code .literal.string.interpol { color: #BB6688; font-weight: bold }
.code .literal.string.other { color: #008000 }
.code .literal.string.regex { color: #BB6688 }
.code .literal.string.symbol { color: #B8860B }
.code .name.attribute { color: #BB4444 }
.code .name.builtin { color: #AA22FF }
.code .name.class { color: #0000FF }
.code .name.constant { color: #880000 }
.code .name.decorator { color: #AA22FF }
.code .name.entity { color: #999999; font-weight: bold }
.code .name.exception { color: #D2413A; font-weight: bold }
.code .name.function { color: #00A000 }
.code .name.label { color: #A0A000 }
.code .name.namespace { color: #0000FF; font-weight: bold }
.code .name.tag { color: #008000; font-weight: bold }
.code .name.variable { color: #B8860B }
.code .operator { color: #666666 }
.code .operator.word { color: #AA22FF; font-weight: bold }
.code .ln { margin-right: -0px }
</style>
</head>
<body>
<div class="document">
<p>I have been developing Java with various frameworks like J2EE, spring, hibernate or others for a couple of years now,
but a few months ago I got the chance to switch to scala, akka and play. I really had to learn a lot in the
beginning and of course I am still gaining new insights into these tools and libraries every day.
To share some of the aha effects I experienced during the last months I am starting a series of blog posts
that might be helpful to others who are thinking about a similar switch or are even already in the middle of
such a change.</p>
<div class="section" id="using-for-comprehensions-to-build-query-apis">
<h1>Using for-comprehensions to build Query-APIs</h1>
<p>When you start working with scala, you will quickly be confronted with its for-comprehensions or <a class="reference external" href="http://docs.scala-lang.org/tutorials/tour/sequence-comprehensions.html">for-expressions</a>.
If you loop for example over two collections, you can do this with a for-expression like this:</p>
<pre class="code scala literal-block">
<span class="ln">1 </span> <span class="keyword">val</span> <span class="name">as</span> <span class="keyword">=</span> <span class="name class">List</span><span class="operator">(</span><span class="literal number integer">1</span><span class="operator">,</span><span class="literal number integer">2</span><span class="operator">,</span><span class="literal number integer">3</span><span class="operator">)</span>
<span class="ln">2 </span> <span class="keyword">val</span> <span class="name">bs</span> <span class="keyword">=</span> <span class="name class">List</span><span class="operator">(</span><span class="literal number integer">4</span><span class="operator">,</span><span class="literal number integer">5</span><span class="operator">,</span><span class="literal number integer">6</span><span class="operator">)</span>
<span class="ln">3 </span> <span class="keyword">val</span> <span class="name">cs</span><span class="keyword">:</span> <span class="keyword type">List</span><span class="operator">[</span><span class="keyword type">Int</span><span class="operator">]</span> <span class="keyword">=</span> <span class="keyword">for</span> <span class="operator">{</span>
<span class="ln">4 </span> <span class="name">a</span> <span class="keyword"><-</span> <span class="name">as</span>
<span class="ln">5 </span> <span class="name">b</span> <span class="keyword"><-</span> <span class="name">bs</span>
<span class="ln">6 </span> <span class="operator">}</span> <span class="keyword">yield</span> <span class="name">a</span><span class="operator">*</span><span class="name">b</span>
</pre>
<p>This computes the products of all numbers in <tt class="docutils literal">a</tt> with all numbers in <tt class="docutils literal">b</tt> resulting in a list
of 9 <tt class="docutils literal">Int</tt>s. The scala compiler actually translates a for-expression into a cascaded set of calls to
<tt class="docutils literal">flatMap, map, withFilter</tt>. So the example above actually becomes:</p>
<pre class="code scala literal-block">
<span class="ln">1 </span> <span class="keyword">val</span> <span class="name">ds</span><span class="keyword">:</span> <span class="keyword type">List</span><span class="operator">[</span><span class="keyword type">Int</span><span class="operator">]</span> <span class="keyword">=</span> <span class="name">as</span><span class="operator">.</span><span class="name">flatMap</span><span class="operator">(</span><span class="name">a</span> <span class="keyword">=></span> <span class="name">bs</span><span class="operator">.</span><span class="name">map</span><span class="operator">(</span><span class="name">b</span> <span class="keyword">=></span> <span class="name">a</span><span class="operator">*</span><span class="name">b</span><span class="operator">))</span>
<span class="ln">2 </span>
<span class="ln">3 </span> <span class="name">ds</span> <span class="name">must</span> <span class="name">be</span> <span class="operator">(</span><span class="name">cs</span><span class="operator">)</span>
</pre>
<p><tt class="docutils literal">withFilter</tt> comes into play as soon as you add conditions to the for-expression, so:</p>
<pre class="code scala literal-block">
<span class="ln"> 1 </span> <span class="keyword">val</span> <span class="name">as</span> <span class="keyword">=</span> <span class="name class">List</span><span class="operator">(</span><span class="literal number integer">1</span><span class="operator">,</span><span class="literal number integer">2</span><span class="operator">,</span><span class="literal number integer">3</span><span class="operator">)</span>
<span class="ln"> 2 </span> <span class="keyword">val</span> <span class="name">bs</span> <span class="keyword">=</span> <span class="name class">List</span><span class="operator">(</span><span class="literal number integer">4</span><span class="operator">,</span><span class="literal number integer">5</span><span class="operator">,</span><span class="literal number integer">6</span><span class="operator">)</span>
<span class="ln"> 3 </span> <span class="keyword">val</span> <span class="name">cs</span><span class="keyword">:</span> <span class="keyword type">List</span><span class="operator">[</span><span class="keyword type">Int</span><span class="operator">]</span> <span class="keyword">=</span> <span class="keyword">for</span> <span class="operator">{</span>
<span class="ln"> 4 </span> <span class="name">a</span> <span class="keyword"><-</span> <span class="name">as</span>
<span class="ln"> 5 </span> <span class="keyword">if</span> <span class="name">a</span> <span class="operator">%</span> <span class="literal number integer">2</span> <span class="operator">==</span> <span class="literal number integer">1</span>
<span class="ln"> 6 </span> <span class="name">b</span> <span class="keyword"><-</span> <span class="name">bs</span>
<span class="ln"> 7 </span> <span class="keyword">if</span> <span class="name">b</span> <span class="operator">%</span> <span class="literal number integer">2</span> <span class="operator">==</span> <span class="literal number integer">0</span>
<span class="ln"> 8 </span> <span class="operator">}</span> <span class="keyword">yield</span> <span class="name">a</span><span class="operator">*</span><span class="name">b</span>
</pre>
<p>becomes:</p>
<pre class="code scala literal-block">
<span class="ln"> 1 </span> <span class="keyword">val</span> <span class="name">ds</span><span class="keyword">:</span> <span class="keyword type">List</span><span class="operator">[</span><span class="keyword type">Int</span><span class="operator">]</span> <span class="keyword">=</span> <span class="name">as</span><span class="operator">.</span>
<span class="ln"> 2 </span> <span class="name">withFilter</span><span class="operator">(</span><span class="name">a</span> <span class="keyword">=></span> <span class="name">a</span> <span class="operator">%</span> <span class="literal number integer">2</span> <span class="operator">==</span> <span class="literal number integer">1</span><span class="operator">).</span>
<span class="ln"> 3 </span> <span class="name">flatMap</span><span class="operator">(</span><span class="name">a</span> <span class="keyword">=></span> <span class="name">bs</span><span class="operator">.</span>
<span class="ln"> 4 </span> <span class="name">withFilter</span><span class="operator">(</span><span class="name">b</span> <span class="keyword">=></span> <span class="name">b</span> <span class="operator">%</span> <span class="literal number integer">2</span> <span class="operator">==</span> <span class="literal number integer">0</span><span class="operator">).</span>
<span class="ln"> 5 </span> <span class="name">map</span><span class="operator">(</span><span class="name">b</span> <span class="keyword">=></span> <span class="name">a</span><span class="operator">*</span><span class="name">b</span><span class="operator">))</span>
<span class="ln"> 6 </span>
<span class="ln"> 7 </span> <span class="name">ds</span> <span class="name">must</span> <span class="name">be</span> <span class="operator">(</span><span class="name">cs</span><span class="operator">)</span>
</pre>
<p>The interesting thing here is that the compiler first translates the for-expression into the cascaded form and
checks afterwards, if the provided expressions actually support the required method calls, e.g. in our case
<tt class="docutils literal">as</tt>' and <tt class="docutils literal">bs</tt>' common type (<tt class="docutils literal">List[Int]</tt>) has to have all of the three methods:
<tt class="docutils literal">flatMap</tt>, <tt class="docutils literal">map</tt> and <tt class="docutils literal">withFilter</tt>. If you use your type only in a simple for-expression like this:</p>
<pre class="code scala literal-block">
<span class="ln">1 </span> <span class="keyword">case</span> <span class="keyword">class</span> <span class="name class">Container</span><span class="operator">[</span><span class="keyword type">A</span><span class="operator">](</span><span class="name">a</span><span class="keyword">:</span> <span class="keyword type">A</span><span class="operator">)</span> <span class="operator">{</span>
<span class="ln">2 </span> <span class="keyword">def</span> <span class="name">map</span><span class="operator">[</span><span class="keyword type">B</span><span class="operator">](</span><span class="name">f</span><span class="keyword">:</span> <span class="keyword type">A</span> <span class="operator">=></span> <span class="name">B</span><span class="operator">)</span><span class="keyword">:</span> <span class="keyword type">Container</span><span class="operator">[</span><span class="keyword type">B</span><span class="operator">]</span> <span class="keyword">=</span> <span class="name class">Container</span><span class="operator">(</span><span class="name">f</span><span class="operator">(</span><span class="name">a</span><span class="operator">))</span>
<span class="ln">3 </span> <span class="operator">}</span>
<span class="ln">4 </span>
<span class="ln">5 </span> <span class="keyword">val</span> <span class="name">as</span> <span class="keyword">=</span> <span class="name class">Container</span><span class="operator">(</span><span class="literal number integer">1</span><span class="operator">)</span>
<span class="ln">6 </span> <span class="keyword">val</span> <span class="name">b</span> <span class="keyword">=</span> <span class="keyword">for</span><span class="operator">(</span><span class="name">a</span> <span class="keyword"><-</span> <span class="name">as</span><span class="operator">)</span> <span class="keyword">yield</span> <span class="name">a</span><span class="operator">+</span><span class="literal number integer">1</span>
</pre>
<p>you do not need to implement <tt class="docutils literal">flatMap</tt> or <tt class="docutils literal">withFilter</tt> at all. You do not even have to be
generic at all. If your class <tt class="docutils literal">Container</tt> only supports <tt class="docutils literal">Int</tt>s
and the expression after the yield works with <tt class="docutils literal">Int</tt>s that will compile as
well. So for the example above <tt class="docutils literal">Container</tt> could be implemented as:</p>
<pre class="code scala literal-block">
<span class="ln">1 </span> <span class="keyword">case</span> <span class="keyword">class</span> <span class="name class">Container</span><span class="operator">(</span><span class="name">a</span><span class="keyword">:</span> <span class="keyword type">Int</span><span class="operator">)</span> <span class="operator">{</span>
<span class="ln">2 </span> <span class="keyword">def</span> <span class="name">map</span><span class="operator">(</span><span class="name">f</span><span class="keyword">:</span> <span class="keyword type">Int</span> <span class="operator">=></span> <span class="name class">Int</span><span class="operator">)</span><span class="keyword">:</span> <span class="keyword type">Container</span> <span class="operator">=</span> <span class="name class">Container</span><span class="operator">(</span><span class="name">f</span><span class="operator">(</span><span class="name">a</span><span class="operator">))</span>
<span class="ln">3 </span> <span class="operator">}</span>
</pre>
<p>So far we basically simplified our custom class and made it just implement what is actually required.
In addition to that one can also change the signature of these methods in a way that may look surprising.
For example <tt class="docutils literal">withFilter</tt> takes a function as argument that maps whatever is
filtered to <tt class="docutils literal">Boolean</tt>s.
While the <tt class="docutils literal">Boolean</tt> seems
to be a very natural fit for the <tt class="docutils literal">if</tt>, there is no requirement at all, that the expression after the
<tt class="docutils literal">if</tt> evaluates to a <tt class="docutils literal">Boolean</tt> (i.e. that <tt class="docutils literal">withFilter</tt> takes an argument of
type <tt class="docutils literal">A => Boolean</tt>). Let's check out a little example that basically exchanges the
meanings of <tt class="docutils literal">map</tt> and <tt class="docutils literal">withFilter</tt>:</p>
<pre class="code scala literal-block">
<span class="ln"> 1 </span> <span class="keyword">case</span> <span class="keyword">class</span> <span class="name class">Container</span><span class="operator">(</span><span class="name">list</span><span class="keyword">:</span> <span class="keyword type">List</span><span class="operator">[</span><span class="keyword type">Int</span><span class="operator">])</span> <span class="operator">{</span>
<span class="ln"> 2 </span> <span class="keyword">def</span> <span class="name">map</span><span class="operator">(</span><span class="name">filter</span><span class="keyword">:</span> <span class="keyword type">Int</span> <span class="operator">=></span> <span class="name class">Boolean</span><span class="operator">)</span><span class="keyword">:</span> <span class="keyword type">Container</span> <span class="operator">=</span>
<span class="ln"> 3 </span> <span class="name class">Container</span><span class="operator">(</span><span class="name">list</span><span class="operator">.</span><span class="name">withFilter</span><span class="operator">(</span><span class="name">filter</span><span class="operator">).</span><span class="name">map</span><span class="operator">(</span><span class="name">identity</span><span class="operator">))</span>
<span class="ln"> 4 </span> <span class="keyword">def</span> <span class="name">withFilter</span><span class="operator">(</span><span class="name">f</span><span class="keyword">:</span> <span class="keyword type">Int</span> <span class="operator">=></span> <span class="name class">Int</span><span class="operator">)</span><span class="keyword">:</span> <span class="keyword type">Container</span> <span class="operator">=</span> <span class="name class">Container</span><span class="operator">(</span><span class="name">list</span><span class="operator">.</span><span class="name">map</span><span class="operator">(</span><span class="name">f</span><span class="operator">))</span>
<span class="ln"> 5 </span> <span class="operator">}</span>
<span class="ln"> 6 </span>
<span class="ln"> 7 </span> <span class="keyword">val</span> <span class="name">as</span> <span class="keyword">=</span> <span class="name class">Container</span><span class="operator">(</span><span class="name class">List</span><span class="operator">(</span><span class="literal number integer">1</span><span class="operator">,</span><span class="literal number integer">2</span><span class="operator">,</span><span class="literal number integer">3</span><span class="operator">))</span>
<span class="ln"> 8 </span> <span class="keyword">val</span> <span class="name">bs</span> <span class="keyword">=</span> <span class="keyword">for</span> <span class="operator">{</span>
<span class="ln"> 9 </span> <span class="name">a</span> <span class="keyword"><-</span> <span class="name">as</span>
<span class="ln">10 </span> <span class="keyword">if</span> <span class="name">a</span><span class="operator">+</span><span class="literal number integer">2</span>
<span class="ln">11 </span> <span class="keyword">if</span> <span class="name">a</span><span class="operator">*</span><span class="literal number integer">3</span>
<span class="ln">12 </span> <span class="operator">}</span> <span class="keyword">yield</span> <span class="name">a</span> <span class="operator">%</span> <span class="literal number integer">2</span> <span class="operator">==</span> <span class="literal number integer">0</span>
</pre>
<p>The for-expression translates to:</p>
<pre class="code scala literal-block">
<span class="ln">1 </span> <span class="keyword">val</span> <span class="name">cs</span> <span class="keyword">=</span> <span class="name">as</span><span class="operator">.</span>
<span class="ln">2 </span> <span class="name">withFilter</span><span class="operator">(</span><span class="name">a</span> <span class="keyword">=></span> <span class="name">a</span> <span class="operator">+</span> <span class="literal number integer">2</span><span class="operator">).</span>
<span class="ln">3 </span> <span class="name">withFilter</span><span class="operator">(</span><span class="name">a</span> <span class="keyword">=></span> <span class="name">a</span> <span class="operator">*</span> <span class="literal number integer">3</span><span class="operator">).</span>
<span class="ln">4 </span> <span class="name">map</span><span class="operator">(</span><span class="name">a</span> <span class="keyword">=></span> <span class="name">a</span> <span class="operator">%</span> <span class="literal number integer">2</span> <span class="operator">==</span> <span class="literal number integer">0</span><span class="operator">)</span>
</pre>
<p>Even if it might look like, this does not return a <tt class="docutils literal">Container</tt> with a list of <tt class="docutils literal">Boolean</tt>s
(even though the expression after yield evaluates to a <tt class="docutils literal">Boolean</tt>). It rather first adds 2 to each element of the list
(yielding <tt class="docutils literal">List(3,4,5)</tt>), then multiplies each element with 3
(yielding <tt class="docutils literal">List(9,12,15)</tt>) and after
filtering only even elements remain (yielding <tt class="docutils literal">List(12)</tt>).</p>
<p>While this seems to be a pretty useless example, you can definitely utilize this in a more meaningful way. The
container could for example store the elements in a sorted set and answer filter-requests for ranges directly without
actually looping through the whole set:</p>
<pre class="code scala literal-block">
<span class="ln"> 1 </span> <span class="keyword">class</span> <span class="name class">Container</span><span class="operator">(</span><span class="name">sortedElements</span><span class="keyword">:</span> <span class="keyword type">SortedSet</span><span class="operator">[</span><span class="keyword type">Int</span><span class="operator">])</span> <span class="operator">{</span>
<span class="ln"> 2 </span> <span class="keyword">def</span> <span class="keyword">this</span><span class="operator">(</span><span class="name">elements</span><span class="keyword">:</span> <span class="keyword type">Int*</span><span class="operator">)</span> <span class="keyword">=</span> <span class="keyword">this</span><span class="operator">(</span><span class="name class">TreeSet</span><span class="operator">(</span><span class="name">elements</span><span class="keyword">:</span> <span class="keyword">_</span><span class="keyword type">*</span><span class="operator">))</span>
<span class="ln"> 3 </span> <span class="keyword">def</span> <span class="name">withFilter</span><span class="operator">(</span><span class="name">filter</span><span class="keyword">:</span> <span class="keyword type">Int</span> <span class="operator">=></span> <span class="name class">RangeFilter</span><span class="operator">)</span><span class="keyword">:</span> <span class="keyword type">Container</span> <span class="operator">=</span> <span class="operator">{</span>
<span class="ln"> 4 </span> <span class="keyword">val</span> <span class="name">rangeFilter</span> <span class="keyword">=</span> <span class="name">filter</span><span class="operator">(</span><span class="literal number integer">0</span><span class="operator">)</span>
<span class="ln"> 5 </span> <span class="keyword">new</span> <span class="name class">Container</span><span class="operator">(</span><span class="name">sortedElements</span><span class="operator">.</span><span class="name">rangeImpl</span><span class="operator">(</span><span class="name">rangeFilter</span><span class="operator">.</span><span class="name">from</span><span class="operator">,</span> <span class="name">rangeFilter</span><span class="operator">.</span><span class="name">to</span><span class="operator">))</span>
<span class="ln"> 6 </span> <span class="operator">}</span>
<span class="ln"> 7 </span> <span class="keyword">def</span> <span class="name">map</span><span class="operator">(</span><span class="name">id</span><span class="keyword">:</span> <span class="keyword type">Int</span> <span class="operator">=></span> <span class="name class">Int</span><span class="operator">)</span><span class="keyword">:</span> <span class="keyword type">Container</span> <span class="operator">=</span> <span class="keyword">this</span>
<span class="ln"> 8 </span> <span class="keyword">def</span> <span class="name">toList</span><span class="keyword">:</span> <span class="keyword type">List</span><span class="operator">[</span><span class="keyword type">Int</span><span class="operator">]</span> <span class="keyword">=</span> <span class="name">sortedElements</span><span class="operator">.</span><span class="name">toList</span>
<span class="ln"> 9 </span> <span class="operator">}</span>
<span class="ln">10 </span>
<span class="ln">11 </span> <span class="keyword">class</span> <span class="name class">RangeFilter</span><span class="operator">(</span><span class="keyword">val</span> <span class="name">from</span><span class="keyword">:</span> <span class="keyword type">Option</span><span class="operator">[</span><span class="keyword type">Int</span><span class="operator">],</span> <span class="keyword">val</span> <span class="name">to</span><span class="keyword">:</span> <span class="keyword type">Option</span><span class="operator">[</span><span class="keyword type">Int</span><span class="operator">])</span>
<span class="ln">12 </span> <span class="keyword">case</span> <span class="keyword">class</span> <span class="name class">Within</span><span class="operator">(</span><span class="name">min</span><span class="keyword">:</span> <span class="keyword type">Int</span><span class="operator">,</span> <span class="name">max</span><span class="keyword">:</span> <span class="keyword type">Int</span><span class="operator">)</span> <span class="keyword">extends</span> <span class="name class">RangeFilter</span><span class="operator">(</span><span class="name class">Some</span><span class="operator">(</span><span class="name">min</span><span class="operator">),</span> <span class="name class">Some</span><span class="operator">(</span><span class="name">max</span><span class="operator">))</span>
<span class="ln">13 </span> <span class="keyword">case</span> <span class="keyword">class</span> <span class="name class">LessOrEqual</span><span class="operator">(</span><span class="name">max</span><span class="keyword">:</span> <span class="keyword type">Int</span><span class="operator">)</span> <span class="keyword">extends</span> <span class="name class">RangeFilter</span><span class="operator">(</span><span class="name class">None</span><span class="operator">,</span> <span class="name class">Some</span><span class="operator">(</span><span class="name">max</span><span class="operator">))</span>
<span class="ln">14 </span> <span class="keyword">case</span> <span class="keyword">class</span> <span class="name class">GreaterOrEqual</span><span class="operator">(</span><span class="name">min</span><span class="keyword">:</span> <span class="keyword type">Int</span><span class="operator">)</span> <span class="keyword">extends</span> <span class="name class">RangeFilter</span><span class="operator">(</span><span class="name class">Some</span><span class="operator">(</span><span class="name">min</span><span class="operator">),</span> <span class="name class">None</span><span class="operator">)</span>
<span class="ln">15 </span>
<span class="ln">16 </span> <span class="keyword">val</span> <span class="name">as</span> <span class="keyword">=</span> <span class="keyword">new</span> <span class="name class">Container</span><span class="operator">(</span><span class="literal number integer">6</span><span class="operator">,</span> <span class="literal number integer">10</span><span class="operator">,</span> <span class="literal number integer">3</span><span class="operator">,</span> <span class="literal number integer">5</span><span class="operator">,</span> <span class="literal number integer">15</span><span class="operator">)</span>
<span class="ln">17 </span> <span class="keyword">val</span> <span class="name">bs</span> <span class="keyword">=</span> <span class="keyword">for</span> <span class="operator">{</span>
<span class="ln">18 </span> <span class="name">a</span> <span class="keyword"><-</span> <span class="name">as</span>
<span class="ln">19 </span> <span class="keyword">if</span> <span class="name class">Within</span><span class="operator">(</span><span class="literal number integer">5</span><span class="operator">,</span> <span class="literal number integer">10</span><span class="operator">)</span>
<span class="ln">20 </span> <span class="keyword">if</span> <span class="name class">LessOrEqual</span><span class="operator">(</span><span class="literal number integer">9</span><span class="operator">)</span>
<span class="ln">21 </span> <span class="operator">}</span> <span class="keyword">yield</span> <span class="name">a</span>
</pre>
<p>The for-expression translates to:</p>
<pre class="code scala literal-block">
<span class="ln">1 </span> <span class="keyword">val</span> <span class="name">cs</span> <span class="keyword">=</span> <span class="name">as</span><span class="operator">.</span>
<span class="ln">2 </span> <span class="name">withFilter</span><span class="operator">(</span><span class="keyword">_</span> <span class="keyword">=></span> <span class="name class">Within</span><span class="operator">(</span><span class="literal number integer">5</span><span class="operator">,</span><span class="literal number integer">10</span><span class="operator">)).</span>
<span class="ln">3 </span> <span class="name">withFilter</span><span class="operator">(</span><span class="keyword">_</span> <span class="keyword">=></span> <span class="name class">LessOrEqual</span><span class="operator">(</span><span class="literal number integer">9</span><span class="operator">)).</span>
<span class="ln">4 </span> <span class="name">map</span><span class="operator">(</span><span class="name">identity</span><span class="operator">)</span>
</pre>
<p>As the function that is passed to <tt class="docutils literal">map</tt> is basically ignored, any function
can be passed.</p>
<p>In this example the <tt class="docutils literal">Container</tt> keeps its elements in a <tt class="docutils literal">SortedSet</tt>. The
method <tt class="docutils literal">withFilter</tt>
takes a function that maps the container-elements (<tt class="docutils literal">Int</tt>) to a <tt class="docutils literal">RangeFilter</tt>. Instead of looping
through all elements and passing them to the provided function <tt class="docutils literal">filter</tt> we simply want to retrieve
the <tt class="docutils literal">RangeFilter</tt> instance that is provided after the <tt class="docutils literal">if</tt> in the for-comprehension. As the expressions
after the <tt class="docutils literal">if</tt>s do not depend on <tt class="docutils literal">a</tt>, <tt class="docutils literal">filter</tt> returns for all
integers the same <tt class="docutils literal">RangeFilter</tt>, so we can pick any integer, e.g. <tt class="docutils literal">0</tt> (see
line 4). Having the RangeFilter-instance
we can get the from/to values to pass them to <tt class="docutils literal">SortedSet.rangeImpl</tt> and compute the filtered result
without looping through the set.</p>
<p>With an implementation like this you can provide the user with an elegant and efficient way to query your container with
for-comprehensions.</p>
<p>Another use case for this kind of <em>surprising</em> implementation for <tt class="docutils literal">withFilter</tt> and <tt class="docutils literal">map</tt>
is to use the for-comprehension to build a query-object that is afterwards applied to a dataset.
<a class="reference external" href="http://slick.typesafe.com/">Slick</a> uses
this approach to build SQL-queries in a typesafe manner. I will give a simple example mirroring the code from the
previous example (with the same definitions of the <tt class="docutils literal">RangeFilter</tt> class and
its decsendents):</p>
<pre class="code scala literal-block">
<span class="ln"> 1 </span> <span class="keyword">class</span> <span class="name class">QueryBuilder</span><span class="operator">(</span><span class="name">filters</span><span class="keyword">:</span> <span class="keyword type">List</span><span class="operator">[</span><span class="keyword type">RangeFilter</span><span class="operator">]</span><span class="keyword">=</span> <span class="name class">Nil</span><span class="operator">)</span> <span class="operator">{</span>
<span class="ln"> 2 </span> <span class="keyword">def</span> <span class="name">withFilter</span><span class="operator">(</span><span class="name">filter</span><span class="keyword">:</span> <span class="keyword type">Int</span> <span class="operator">=></span> <span class="name class">RangeFilter</span><span class="operator">)</span><span class="keyword">:</span> <span class="keyword type">QueryBuilder</span> <span class="operator">=</span>
<span class="ln"> 3 </span> <span class="keyword">new</span> <span class="name class">QueryBuilder</span><span class="operator">(</span><span class="name">filters</span> <span class="operator">:+</span> <span class="name">filter</span><span class="operator">(</span><span class="literal number integer">0</span><span class="operator">))</span>
<span class="ln"> 4 </span> <span class="keyword">def</span> <span class="name">map</span><span class="operator">(</span><span class="name">id</span><span class="keyword">:</span> <span class="keyword type">Int</span> <span class="operator">=></span> <span class="name class">Int</span><span class="operator">)</span><span class="keyword">:</span> <span class="keyword type">QueryBuilder</span> <span class="operator">=</span> <span class="keyword">this</span>
<span class="ln"> 5 </span> <span class="keyword">def</span> <span class="name">apply</span><span class="operator">(</span><span class="name">set</span><span class="keyword">:</span> <span class="keyword type">SortedSet</span><span class="operator">[</span><span class="keyword type">Int</span><span class="operator">])</span><span class="keyword">:</span> <span class="keyword type">List</span><span class="operator">[</span><span class="keyword type">Int</span><span class="operator">]</span> <span class="keyword">=</span> <span class="operator">{</span>
<span class="ln"> 6 </span> <span class="name">filters</span><span class="operator">.</span><span class="name">foldLeft</span><span class="operator">(</span><span class="name">set</span><span class="operator">)</span> <span class="operator">{</span> <span class="operator">(</span><span class="name">filteredSet</span><span class="operator">,</span> <span class="name">filter</span><span class="operator">)</span> <span class="keyword">=></span>
<span class="ln"> 7 </span> <span class="name">filteredSet</span><span class="operator">.</span><span class="name">rangeImpl</span><span class="operator">(</span><span class="name">filter</span><span class="operator">.</span><span class="name">from</span><span class="operator">,</span> <span class="name">filter</span><span class="operator">.</span><span class="name">to</span><span class="operator">)</span>
<span class="ln"> 8 </span> <span class="operator">}.</span><span class="name">toList</span>
<span class="ln"> 9 </span> <span class="operator">}</span>
<span class="ln">10 </span> <span class="operator">}</span>
<span class="ln">11 </span>
<span class="ln">12 </span> <span class="keyword">val</span> <span class="name">query</span> <span class="keyword">=</span> <span class="keyword">for</span> <span class="operator">{</span>
<span class="ln">13 </span> <span class="name">t</span> <span class="keyword"><-</span> <span class="keyword">new</span> <span class="name class">QueryBuilder</span><span class="operator">()</span>
<span class="ln">14 </span> <span class="keyword">if</span> <span class="name class">Within</span><span class="operator">(</span><span class="literal number integer">5</span><span class="operator">,</span> <span class="literal number integer">10</span><span class="operator">)</span>
<span class="ln">15 </span> <span class="keyword">if</span> <span class="name class">LessOrEqual</span><span class="operator">(</span><span class="literal number integer">9</span><span class="operator">)</span>
<span class="ln">16 </span> <span class="operator">}</span> <span class="keyword">yield</span> <span class="name">t</span>
<span class="ln">17 </span>
<span class="ln">18 </span> <span class="keyword">val</span> <span class="name">result</span> <span class="keyword">=</span> <span class="name">query</span><span class="operator">(</span><span class="name class">TreeSet</span><span class="operator">(</span><span class="literal number integer">6</span><span class="operator">,</span> <span class="literal number integer">10</span><span class="operator">,</span> <span class="literal number integer">3</span><span class="operator">,</span> <span class="literal number integer">5</span><span class="operator">,</span> <span class="literal number integer">15</span><span class="operator">))</span>
</pre>
<p>In this case the class that is used in the for-comprehension as <em>source</em> (<tt class="docutils literal">QueryBuilder</tt>)
is not a container any more.
Having no elements to filter <tt class="docutils literal">withFilter</tt> just collects the passed <tt class="docutils literal">RangeFilter</tt>s. Output
of the for-comprehension is a <tt class="docutils literal">QueryBuilder</tt>-instance that can be applied to arbitrary SortedSets to
perform the previously recorded filtering.</p>
<p>As you can see for-comprehensions are even more powerful as you may think in the beginning. They are not only useful
as generic query-language for all kinds of containers, but you have all options to
implement filtering (or mapping) in a more efficient way than simply looping over all container elements. In addition
to this you can even split building the query from executing it and use them as generic query-builder whose output
is applied to arbitrary datasets.</p>
</div>
</div>
</body>
</html>Volker Stampahttp://www.blogger.com/profile/03277007007448976514noreply@blogger.com0