graph LR Request --> Plugin1[Plugin] Plugin1 --> Handler Handler --> Plugin2[Plugin] Plugin2 --> Response</pre>

Ktor request/response pipeline</figcaption> </figure>

Let's write a plugin that transforms a specific type of files - going with the previous example of injecting a script to every served HTML file. Our plugin needs to:</p>

Take a script URL as an input. If not provided, it won't do anything.</li>

Add that script as a <script></code> element in the <head></code> of any HTML file served by this Ktor server.</li>

Obviously, not interfere with any other file format.</li> </ul> Let's start by defining an empty plugin and its configuration:</p>

class PluginConfiguration {
  var scriptUrl: String? = null
}

val ScriptInjectionPlugin = createApplicationPlugin(
  name = "ScriptInjectionPlugin",
  createConfiguration = ::PluginConfiguration,
) {
  val scriptUrl = pluginConfig.scriptUrl

  // The rest of our plugin goes here.
}
</code></pre>
With this simple definition, we can add our plugin to a Ktor server:</p>
embeddedServer(Netty, port = 8080) {
  install(ScriptInjectionPlugin) {
    scriptUrl = "http://foo.bar/my/injected/script.js"
  }
}
</code></pre>
Now, let's see how we can transform the body. Ktor offers a few handlers to hook into the pipeline
shown above. The main ones are:</p>

onCall</code> is fairly high level, and is mostly useful to get information about a request (e.g. to
log a request).</li>
onCallReceive</code> allows to transform data received from the client before it's processed.</li>
onCallRespond</code> allows to transform data before sending it to the client</em>. That's the one we're
after.</li>
</ul>
There are a few other handlers for specific use-cases, detailed
here</a>.</p>
Let's hook into onCallRespond</code>, and its helper transformBody</code>. We also need to check if the
content type we're sending is HTML, otherwise, we just forward the response body as-is:</p>
onCallRespond { _ ->
  transformBody { data ->
    if (data is OutgoingContent.ReadChannelContent &&
      data.contentType?.withoutParameters() == ContentType.Text.Html &&
      scriptUrl != null
    ) {
      // We are serving an HTML file.
      transform(data, scriptUrl)
    } else {
      data
    }
  }
}
</code></pre>
We can then define our transform()</code> helper, which needs to:</p>

Read the body,</li>
Transform it,</li>
Store it into a OutgoingContent</code> for Ktor to continue its pipeline.</li>
</ul>
First, let's get the injection itself out of the way. Let's assume we have our HTML file into a
string, and want a new string with the injected HTML. Jsoup</a> comes in handy for
this kind of operations:</p>
private fun injectLiveReloadFragment(target: String, scriptUrl: String): String {
  return Jsoup.parse(target).apply {
    head().appendElement("script").attributes().add("src", scriptUrl)
  }.toString()
}
</code></pre>
Now, let's actually read and return the body:</p>

  Note</div>
  
    This transformation assumes that the HTML files processed there are fairly small. As such, it
fully reads the channel Ktor provides in memory before transforming it. Any high-traffic server or
longer files should probably not</em> read the content in that way.</p>

  </div>
</aside>
private suspend fun transform(
  data: OutgoingContent.ReadChannelContent,
  scriptUrl: String,
): OutgoingContent {
  // This channel provided by Ktor gives a view of the file about to be returned.
  val channel = data.readFrom()

  // Let's ready everything in the channel into a String.
  val content = StringBuilder().run {
    while (!channel.isClosedForRead) {
      channel.readUTF8LineTo(this)
      append('\n')
    }
    toString()
  }

  // Inject our script URL.
  val htmlContent = injectLiveReloadFragment(content, scriptUrl)

  // Prepare our new content (htmlContent contains it as a string) for Ktor to process.
  return WriterContent(
    body = {
      withContext(Dispatchers.IO) {
        write(htmlContent)
      }
    },
    contentType = ContentType.Text.Html,
  )
}
</code></pre>

Writing a Bazel rule set

2020-05-16T15:55:00+11:00

This post will cover two things:</p>

How to run an arbitrary tool with Bazel (in this case, PlantUML</a>, a tool to generate diagrams), by writing a rule set</li>
How to test this rule set.</li> </ul>
It should be mentioned that while I was working on this rule set, it became more and more apparent PlantUML is not a great candidate for this kind of integration, as its output is platform-dependent (the font rendering). Despite that, it's still a simple tool and as such its integration is simple, albeit not perfect (the rendering tests I wrote need to run on the same platform every time).</p>
PlantUML usage</h2>
PlantUML is a tool that takes a text input looking like this:</p>
@startuml Alice -> Bob: SYN @enduml </code></pre> And outputs an image looking like this:</p>

sequenceDiagram Alice->>Bob: SYN</pre>
PlantUML sample output</figcaption> </figure>
PlantUML has multiple way of being invoked (CLI, GUI, as well as a lot</em> of integrations with different tools), but we'll go with the easiest: a one-shot CLI invocation. It takes as inputs:</p>

A text file, representing a diagram</li>
An optional configuration file, giving control over the output</li> </ul>
It then outputs a single image file, which can be of different formats (we'll just cover SVG and PNG in this article, but adding support for other formats is trivial).</p>
PlantUML ships as a JAR file, which needs to be run with Java. An invocation generating the sample image above would look like that:</p>
java -jar plantuml.jar -tpng -p < 'mysource.puml' > 'dir/myoutput.png' </code></pre> Pretty straightforward: run the JAR, with a single option for the image type, pipe the content of the input file and get the output file back. The -p</code> flag is the short form of -pipe</code>, which we're using as using pipes is the only way of properly controlling the output path (without that, PlantUML tries to be smart and places the output next to the input).</p> With a configuration file:</p> java -jar plantuml.jar -tpng -config config.puml -p < 'mysource.puml' > 'dir/myoutput.png' </code></pre> Simple enough, right? Well, not really. PlantUML actually integrates some metadata in the files it generates. For example, when generating an SVG:</p>  <svg><g> </g></svg> </code></pre> This makes PlantUML non hermetic by default (in addition to the fonts issue mentioned earlier). While PlantUML has a simple way of working around that (in the form of a -nometadata</code> flag), this is something to keep in mind when integrating a tool with Bazel: is this tool usable in a hermetic way? If not, how to minimise the impact of this non-hermeticity?</p> From there, here is the invocation we'll work with:</p> java -jar plantuml.jar -tpng -nometadata -config config.puml \ -p < 'mysource.puml' > 'dir/myoutput.png' </code></pre> Getting PlantUML</h2> PlantUML is a Java application, available as a JAR on Maven. As such, it can be fetched with the help of rules_jvm_external</a>, as was explained in a previous article</a>. The Maven rules will expose the JAR as a library, but we need a binary to be able to run it. In e.g. //third_party/plantuml/BUILD</code>:</p> load("@rules_java//java:defs.bzl", "java_binary") java_binary( name = "plantuml", main_class = "net.sourceforge.plantuml.Run", visibility = ["//visibility:public"], runtime_deps = [ "@maven//:net_sourceforge_plantuml_plantuml", ], ) </code></pre> From there, we can use //third_party/plantuml</code> as any Bazel binary target - we can run it with bazel run</code>, and we can pass it as a tool for rule actions.</p> This is a pattern that works well for any JVM-based tool. Other kinds of tools will need a different preparation step to make them available through Bazel - but as long as you can get a binary, you should be good.</p> Rule set structure</h2> This rule set will follow the same structure we previously used for Ktlint</a>:</p> Based in //tools/plantuml</code></li> A public interface exposed in //tools/plantuml/defs.bzl</code></li> Internal actions definition in //tools/plantuml/internal/actions.bzl</code></li> Internal rule definition in //tools/plantuml/internal/rules.bzl</code></li> </ul> But in addition:</p> Tests for the actions in //tools/plantuml/internal/actions_test.bzl</code></li> Integration tests in //tools/plantuml/tests</code></li> </ul> Let's start by defining our actions.</p> Actions</h2> Implementation</h3> We need only one action for our rule: one that takes a source file, an optional configuration file, the PlantUML binary, and emits the output file by calling PlantUML. Let's assume for a moment we have a helper function which, given the proper input, returns the PlantUML command line to call, called plantuml_command_line</code>, and write the action from there:</p> def plantuml_generate(ctx, src, format, config, out): """Generates a single PlantUML graph from a puml file. Args: ctx: analysis context. src: source file to be read. format: the output image format. config: the configuration file. Optional. out: output image file. """ command = plantuml_command_line( executable = ctx.executable._plantuml_tool.path, config = config.path if config else None, src = src.path, output = out.path, output_format = format, ) inputs = [src] if config: inputs.append(config) ctx.actions.run_shell( outputs = [out], inputs = inputs, tools = [ctx.executable._plantuml_tool], command = command, mnemonic = "PlantUML", progress_message = "Generating %s" % out.basename, ) </code></pre> This is pretty straightforward: we generate the command line, passing either the attributes' respective paths (or None</code> for the configuration file if it's not provided, since it's optional), as well as the requested image format. We define that both our source file and configuration files are inputs, and PlantUML is a requested tool.</p> Now let's implement our helper function. It's there again really straightforward: it gets a bunch of paths as input, and needs to generate a command line call (in the form of a simple string) from them:</p> def plantuml_command_line(executable, config, src, output, output_format): """Formats the command line to call PlantUML with the given arguments. Args: executable: path to the PlantUML binary. config: path to the configuration file. Optional. src: path to the source file. output: path to the output file. output_format: image format of the output file. Returns: A command to invoke PlantUML """ command = "%s -nometadata -p -t%s " % ( shell.quote(executable), output_format, ) if config: command += " -config %s " % shell.quote(config) command += " < %s > %s" % ( shell.quote(src), shell.quote(output), ) return command </code></pre> An interesting note is that because PlantUML is already integrated as an executable Bazel target, we don't care that it's a JAR, a C++ binary or a shell script: Bazel knows exactly what this executable is made of, how to prepare (e.g. compile) it if necessary, its runtime dependencies (in this case, a JRE) and, more importantly in this context, how to run it. We can treat our tool target as a single executable file, and run it as such just from its path. Bazel will automatically make sure to provide us with everything we need. (For more details: the target actually points to a shell script generated by Bazel, through the Java rules, which in the case of a java_binary</code> target is responsible for defining the classpath, among other things. The JAR file is merely a dependency of this shell script, and as such is provided as a runtime dependency.)</p> Writing this as a helper function rather than directly in the action definition serves two purposes: not only does it make the whole thing slightly easier to read, but this function, which contains the logic (even though in this case it's really simple), is easily testable: it takes only strings as arguments, and returns a string. It's also a pure function: it doesn't have any side effect, and as such it will always return the same output given the same set of inputs.</p> Tests</h3> To test Starlark functions like this one, Bazel's Skylib</a> provides a test framework which, while requiring a bit of boilerplate, is pretty simple to use. In this specific case, we only have two different cases to test: with and without configuration file provided. Error cases should be unreachable due to the way the rule will be defined: Bazel will be responsible for enforcing the presence of an executable target for PlantUML's binary, a valid image format... Let's see how that works. In //tools/plantuml/internal/actions_test.bzl</code>:</p> """Unit tests for PlantUML action""" load("@bazel_skylib//lib:unittest.bzl", "asserts", "unittest") load(":actions.bzl", "plantuml_command_line") def _no_config_impl(ctx): env = unittest.begin(ctx) asserts.equals( env, "'/bin/plantuml' -nometadata -p -tpng < 'mysource.puml' > 'dir/myoutput.png'", plantuml_command_line( executable = "/bin/plantuml", config = None, src = "mysource.puml", output = "dir/myoutput.png", output_format = "png", ), ) return unittest.end(env) no_config_test = unittest.make(_no_config_impl) def _with_config_impl(ctx): env = unittest.begin(ctx) asserts.equals( env, "'/bin/plantuml' -nometadata -p -tpng -config 'myskin.skin' < 'mysource.puml' > 'dir/myoutput.png'", plantuml_command_line( executable = "/bin/plantuml", config = "myskin.skin", src = "mysource.puml", output = "dir/myoutput.png", output_format = "png", ), ) return unittest.end(env) with_config_test = unittest.make(_with_config_impl) def actions_test_suite(): unittest.suite( "actions_tests", no_config_test, with_config_test, ) </code></pre> First, we define two functions, which are the actual test logic: _no_config_impl</code> and _with_config_impl</code>. Their content is pretty simple: we start a unit test environment, we invoke our test function and assert that the result is indeed what we expected, and we close the unit test environment. The return value is needed by the test framework, as it's what carries what assertions passed or failed.</p> Next, we declare those two functions as actual unit tests, wrapping them with a call to unittest.make</code>. We can then add those two test targets to a test suite, which is what actually generates a test target when invoked. Which means that this macro needs to be invoked, in the BUILD</code> file:</p> load(":actions_test.bzl", "actions_test_suite") actions_test_suite() </code></pre> We can run our tests, and hopefully everything should pass:</p> $ bazel test //tools/plantuml/internal:actions_tests INFO: Invocation ID: 112bd049-7398-4b23-b62b-1398e9731eb7 INFO: Analyzed 2 targets (5 packages loaded, 927 targets configured). INFO: Found 2 test targets... INFO: Elapsed time: 0.238s, Critical Path: 0.00s INFO: 0 processes. //tools/plantuml/internal:actions_tests_test_0 PASSED in 0.4s //tools/plantuml/internal:actions_tests_test_1 PASSED in 0.3s Executed 0 out of 2 tests: 2 tests pass. INFO: Build completed successfully, 1 total action </code></pre> Rules definition</h2> Similarly as the actions definition, we only have one rule to define here. Let's call it plantuml_graph()</code>. It needs our usual set of inputs, and outputs a single file, which name will be ${target_name}.{image_format}</code>. It's also where we define the set of acceptable image formats, the fact that the input file is mandatory but the configuration file optional, and the actual executable target to use for PlantUML. The only thing we actually do is, as expected, calling our plantuml_generate</code> action defined above.</p> load( ":actions.bzl", "plantuml_generate", ) def _plantuml_graph_impl(ctx): output = ctx.actions.declare_file("{name}.{format}".format( name = ctx.label.name, format = ctx.attr.format, )) plantuml_generate( ctx, src = ctx.file.src, format = ctx.attr.format, config = ctx.file.config, out = output, ) return [DefaultInfo( files = depset([output]), )] plantuml_graph = rule( _plantuml_graph_impl, attrs = { "config": attr.label( doc = "Configuration file to pass to PlantUML. Useful to tweak the skin", allow_single_file = True, ), "format": attr.string( doc = "Output image format", default = "png", values = ["png", "svg"], ), "src": attr.label( allow_single_file = [".puml"], doc = "Source file to generate the graph from", mandatory = True, ), "_plantuml_tool": attr.label( default = "//third_party/plantuml", executable = True, cfg = "host", ), }, outputs = { "graph": "%{name}.%{format}", }, doc = "Generates a PlantUML graph from a puml file", ) </code></pre> Public interface</h2> As we only have a single rule, and nothing else specific to do, the public interface is dead simple:</p> load("//tools/plantuml/internal:rules.bzl", _plantuml_graph = "plantuml_graph") plantuml_graph = _plantuml_graph </code></pre> You might then be wondering: why is this useful, and why shouldn't I just import the rule definition from //tools/plantuml/internal:rules.bzl</code> directly? Having this kind of public interface allows you to tweak the actual rule definition without breaking any consumer site, as long as you respect the public interface. You can also add features to every consumer site in a really simple way. Let's imagine for example that you have a view_image</code> rule which, given an image file, generates a script to view it, you could then transform your public interface like this:</p> load("//tools/plantuml/internal:rules.bzl", _plantuml_graph = "plantuml_graph") load("//tools/utils:defs.bzl", _view_image = "view_image") def plantuml_graph(name, src, config, format): _plantuml_graph( name = name, src = src, config = config, format = format, ) _view_image( name = "%s.view" % name, src = ":%s.%s" % (name, format), ) </code></pre> And suddenly, all your PlantUML graphs have an implicit .view</code> target defined automatically, allowing you to see the output directly without having to dig in Bazel's output directories.</p> A set of Bazel rules for LaTeX actually provides such a feature to view the PDF output: they have a view_pdf.sh</code> script</a>, used by their main latex_document</code> macro</a>.</p> Further testing</h2> For a rule this simple, I took just a simple further step: having a few reference PlantUML graphs, as well as their expected rendered output, which I compare through Phosphorus, a really simple tool I wrote to help compare two images, covered in the previous article (I told you it would be useful!). But for more complex cases, Skylib offer more utilities like an analysis test</a>, and a build test</a>.</p> Closing thoughts</h2> While writing this kind of tools might look like a lot of works, it's actually pretty mechanical for a lot of cases. I worked on a few others like markdownlint</a>, which now runs on all my Markdown files as regular Bazel test targets, or pngcrush</a>, which is ran on the PNG files hosted on this blog. In a monorepo, writing such a rule is the kind of task that you do once, and it just keeps on giving - you can easily compose different rules with a main use-case, with a bunch of test targets generated for virtually free.</p> On another note, I'm aware that having all this in a public repository would make things much simpler to follow. Sadly, it's part of a larger mono-repository which makes open-sourcing only the relevant parts tricky. Dumping a snapshot somewhere would be an option, but I'd rather have an actual living repository.</p> Now that we have all the tools we need (that was kind of convoluted, I'll give you that), there are only two steps left to cover:</p> Generating the actual blog (ironically enough, this will be a really quick step, despite being the only really important one)</li> Managing the deployment.</li> </ul> We're getting there!</p>

Compiling a Kotlin application with Bazel

2019-12-08T11:30:00+11:00

This post will describe how to compile a small application written in Kotlin using Bazel</a>, tests, as well as how to use static analyzers.</p>

Phosphorus</h2>

Phosphorus is the application that this post will cover. It's a small utility that I wrote to check if an image matches a reference. If it doesn't, Phosphorus generates an image highlighting the differences. The goal is to be able to check that something generates an image in a given way, and doesn't change - at least if it's not expected. The actual usage will be covered later in this series. While it's not open-source yet, it's something I intend to do at some point.</p>

It's written in Kotlin, as a couple external dependencies ( Clikt</a> and Dagger</a>), as well as a few tests. This is the structure:</p>

classDiagram
    namespace loader {
        class ImageLoader {
            <<interface>>
        }
        class ImageIoLoader {
        }
    }

    namespace differ {
        class ImageDiffer {
            <<interface>>
        }
        class ImageDifferImpl {
        }
    }

    namespace data {
        class Image
        class DiffResult
    }

    class Phosphorus

    ImageIoLoader ..|> ImageLoader
    ImageDifferImpl ..|> ImageDiffer
    Phosphorus --> ImageLoader
    Phosphorus --> ImageDiffer</pre>
  Phosphorus's class diagram</figcaption>
</figure>
The differ</code> module contains the core logic - comparing two images, and
generating a DiffResult</code>. This DiffResult</code> contains both the straightforward
result of the comparison (are the two images identical?) and an image
highlighting the differences, if any. The loader</code> package is responsible for
loading and writing images. Finally, the Phosphorus</code> class orchestrates all
that, in addition to processing command line arguments with Clikt.</p>
Dependencies</h2>
Phosphorus has two dependencies: Clikt, and Dagger. Both of them are available
as Maven artifacts. In order to pull Maven artifacts, the Bazel team provides a
set of rules called
rules_jvm_external</a>. The
idea is the following: you list a bunch of Maven coordinates and repositories,
the rule will fetch all of them (and their transitive dependencies) during the
loading phase, and generate Bazel targets corresponding to those Maven
artifacts, on which you can depend. Let's see how we can use them. The first
step is to load the rules, in the WORKSPACE</code>:</p>
load("@bazel_tools//tools/build_defs/repo:http.bzl", "http_archive")

http_archive(
    name = "rules_jvm_external",
    sha256 = "62133c125bf4109dfd9d2af64830208356ce4ef8b165a6ef15bbff7460b35c3a",
    strip_prefix = "rules_jvm_external-3.0",
    url = "https://github.com/bazelbuild/rules_jvm_external/archive/3.0.zip",
)
</code></pre>
Then, we can load and call maven_install</code> with the list of Maven coordinates we
want, in the WORKSPACE</code> too:</p>
load("@rules_jvm_external//:defs.bzl", "maven_install")

maven_install(
    artifacts = [
        "com.github.ajalt:clikt:2.2.0",
        "com.google.dagger:dagger:2.25.2",
        "com.google.dagger:dagger-compiler:2.25.2",
        "com.google.truth:truth:1.0",
        "javax.inject:javax.inject:1",
        "junit:junit:4.12",
    ],
    fetch_sources = True,
    repositories = [
        "https://maven.google.com",
        "https://repo1.maven.org/maven2",
        "https://jcenter.bintray.com/",
    ],
    strict_visibility = True,
)
</code></pre>
A couple of things to note:</p>

We're also downloading JUnit</a> and
Truth</a>, that we're going to use in tests</li>
maven_install</code> can try to download the sources, if they're available on
Maven, to be able to see them directly from the IDE</li>
</ul>
At this point, Clikt, JUnit and Truth are ready to be used. They are exposed
respectively as @maven//:com_github_ajalt_clikt</code>, @maven//:junit_junit</code> and
@maven//:com_google_truth_truth</code>.</p>
Dagger, on the other hand, comes with an annotation processor and, as such,
needs some more work: it needs to be exposed as a Java Plugin. Because it's a
third party dependency, this will be defined in //third_party/dagger/BUILD</code>:</p>
java_plugin(
    name = "dagger_plugin",
    processor_class = "dagger.internal.codegen.ComponentProcessor",
    deps = [
        "@maven//:com_google_dagger_dagger_compiler",
    ],
)

java_library(
    name = "dagger",
    exported_plugins = [":dagger_plugin"],
    visibility = ["//visibility:public"],
    exports = [
        "@maven//:com_google_dagger_dagger",
        "@maven//:com_google_dagger_dagger_compiler",
        "@maven//:javax_inject_javax_inject",
    ],
)
</code></pre>
It can now be used as //third_party/dagger</code>.</p>
Compilation</h2>
Bazel doesn't support Kotlin out of the box (the few languages natively
supported, Java and C++, are currently getting extracted from Bazel's core, so
all languages will soon share a similar integration). In order to compile some
Kotlin code, we'll have to use some Starlark rules describing how to use
kotlinc</code>. A set of rules is available
here</a>. While they don't support
Kotlin/Native, they do support targeting both the JVM (including Android) and
JavaScript.</p>
In order to use those rules, we need to declare them in the WORKSPACE</code>:</p>
load("@bazel_tools//tools/build_defs/repo:http.bzl", "http_archive")

http_archive(
    name = "io_bazel_rules_kotlin",
    sha256 = "54678552125753d9fc0a37736d140f1d2e69778d3e52cf454df41a913b964ede",
    strip_prefix = "rules_kotlin-legacy-1.3.0-rc3",
    url = "https://github.com/bazelbuild/rules_kotlin/archive/legacy-1.3.0-rc3.zip",
)

load("@io_bazel_rules_kotlin//kotlin:kotlin.bzl", "kotlin_repositories", "kt_register_toolchains")

kotlin_repositories()

kt_register_toolchains()
</code></pre>
Once that's done, we have access to a few rules:</p>

kt_js_library</code></li>
kt_js_import</code></li>
kt_jvm_binary</code></li>
kt_jvm_import</code></li>
kt_jvm_library</code></li>
kt_jvm_test</code></li>
kt_android_library</code></li>
</ul>
We're going to use kt_jvm_binary</code>, kt_jvm_library</code> as well as kt_jvm_test</code>.</p>
As JVM-based languages have a strong correlation between packages and folder
structure, we need to be careful about where we store our source code. Bazel
handles a few names as potential Java "roots": java</code>, javatests</code> and src</code>.
Anything inside a directory named like this needs to follow the package/folder
correlation. For example, a class
fr.enoent.phosphorus.client.matcher.Phosphorus</code> can be stored at those
locations:</p>

//java/fr/enoent/phosphorus/Phosphorus.kt</code></li>
//tools/images/java/fr/enoent/phosphorus/Phosphorus.kt</code></li>
//java/tools/images/src/fr/enoent/phosphorus/Phosphorus.kt</code></li>
</ul>
In my repo, everything Java-related is stored under //java</code>, and the
corresponding tests are in //javatests</code> (following the same structure).
Phosphorus will hence be in //java/fr/enoent/phosphorus</code>.</p>
Let's see how we can define a simple Kotlin library, with the data</code> module. In
//java/fr/enoent/phosphorus/data/BUILD</code>:</p>
load("@io_bazel_rules_kotlin//kotlin:kotlin.bzl", "kt_jvm_library")

kt_jvm_library(
    name = "data",
    srcs = [
        "DiffResult.kt",
        "Image.kt",
    ],
    visibility = [
        "//java/fr/enoent/phosphorus:__subpackages__",
        "//javatests/fr/enoent/phosphorus:__subpackages__",
    ],
)
</code></pre>
And that's it, we have our first library ready to be compiled! I won't describe
all the modules as it's pretty repetitive and there's not a lot of value into
doing that, but let's see what the main binary looks like. Defined in
//java/fr/enoent/phosphorus/BUILD</code>, we have:</p>
load("@io_bazel_rules_kotlin//kotlin:kotlin.bzl", "kt_jvm_binary")

kt_jvm_binary(
    name = "phosphorus",
    srcs = [
        "Phosphorus.kt",
    ],
    main_class = "fr.enoent.phosphorus.PhosphorusKt",
    visibility = ["//visibility:public"],
    deps = [
        "//java/fr/enoent/phosphorus/differ",
        "//java/fr/enoent/phosphorus/differ/impl:module",
        "//java/fr/enoent/phosphorus/loader",
        "//java/fr/enoent/phosphorus/loader/io_impl:module",
        "//third_party/dagger",
        "@maven//:com_github_ajalt_clikt",
    ],
)
</code></pre>
Note the name of the main_class</code>: because it's a Kotlin class, the compiler
will append Kt</code> at the end of its name. Once this is defined, we can run
Phosphorus with this command:</p>
bazel run //java/fr/enoent/phosphorus -- arguments passed to Phosphorus directly
</code></pre>
Tests</h2>
As mentioned previously, the test root will be //javatests</code>. Because we need to
follow the packages structure, the tests themselves will be under
//javatests/fr/enoent/phosphorus</code>. They are regular JUnit 4 tests, using Truth
for the assertions.</p>
Defining unit tests is really straightforward, and follows really closely the
pattern we saw with libraries and binaries. For example, the ImageTest</code> test is
defined like this, in //javatests/fr/enoent/phosphorus/data/BUILD</code>:</p>
load("@io_bazel_rules_kotlin//kotlin:kotlin.bzl", "kt_jvm_test")

kt_jvm_test(
    name = "ImageTest",
    srcs = ["ImageTest.kt"],
    deps = [
        "//java/fr/enoent/phosphorus/data",
        "@maven//:com_google_truth_truth",
        "@maven//:junit_junit",
    ],
)

</code></pre>
This will define a Bazel target that we can invoke like this:</p>
bazel test //javatests/fr/enoent/phosphorus/data:ImageTest
</code></pre>
Hopefully, the output should look like this:</p>
//javatests/fr/enoent/phosphorus/data:ImageTest                          PASSED in 0.3s
</code></pre>
Once this is done, it's possible to run
ibazel test //javatests/fr/enoent/phosphorus/...</code> - Bazel will then monitor all
the test targets defined under that path, as well as their dependencies, and
re-run all the affected tests as soon as something is edited. Because Bazel
encourages small build targets, has some great caching, and the Kotlin compiler
uses a persistent worker, the feedback loop is really quick.</p>
Static analysis</h2>
For Kotlin, two tools are quite useful:
Detekt</a>, and
Ktlint</a>. The idea to run them will be really
similar: having two supporting test targets for each actual Kotlin target,
running Detekt and Ktlint on its sources. In order to do that easily, we'll
define some wrappers around the kt_jvm_*</code> set of rules. Those wrappers will be
responsible for generating the two supporting test targets, as well as calling
the original kt_jvm_*</code> rule. The resulting macro will be entirely transparent
to use, the only difference being the load</code> call.</p>
Let's see what those macros could look like. In //java/rules/defs.bzl</code>:</p>
load(
    "@io_bazel_rules_kotlin//kotlin:kotlin.bzl",
    upstream_kt_jvm_binary = "kt_jvm_binary",
    upstream_kt_jvm_library = "kt_jvm_library",
    upstream_kt_jvm_test = "kt_jvm_test",
)
def kt_jvm_binary(name, srcs, **kwargs):
    upstream_kt_jvm_binary(
        name = name,
        srcs = srcs,
        **kwargs
    )

    _common_tests(name = name, srcs = srcs)

def kt_jvm_library(name, srcs, **kwargs):
    upstream_kt_jvm_library(
        name = name,
        srcs = srcs,
        **kwargs
    )

    _common_tests(name = name, srcs = srcs)

def kt_jvm_test(name, srcs, size = "small", **kwargs):
    upstream_kt_jvm_test(
        name = name,
        srcs = srcs,
        size = size,
        **kwargs
    )

    _common_tests(name = name, srcs = srcs)

def _common_tests(name, srcs):
    # This will come soon, no-op for now
</code></pre>
With those wrappers defined, we need to actually call them. Because we're
following the same signature and name as the upstream rules, we just need to
update our load</code> calls in the different BUILD</code> files.
load("@io_bazel_rules_kotlin//kotlin:kotlin.bzl", "kt_jvm_test")</code> will become
load("//java/rules:defs.bzl", "kt_jvm_test")</code>, and so on. _common_tests</code> will
be responsible for calling Detekt and Ktlint, let's see how.</p>
Detekt</h3>
Artem Zinnatullin</a> published a
set of rules</a> to run
Detekt a week before I started writing this, making things way easier. As usual,
let's start by loading this in the WORKSPACE</code>:</p>
http_file(
    name = "detekt_cli_jar",
    sha256 = "e9710fb9260c0824b3a9ae7d8326294ab7a01af68cfa510cab66de964da80862",
    urls = ["https://jcenter.bintray.com/io/gitlab/arturbosch/detekt/detekt-cli/1.2.0/detekt-cli-1.2.0-all.jar"],
)

http_archive(
    name = "rules_detekt",
    sha256 = "f1632c2492291f5144a5e0f5e360a094005e20987518d228709516cc935ad1a1",
    strip_prefix = "bazel_rules_detekt-0.2.0",
    url = "https://github.com/buildfoundation/bazel_rules_detekt/archive/v0.2.0.zip",
)
</code></pre>
This exposes a rule named detekt</code>, which defines a build target, generating the
Detekt report. While there are a few options, we'll keep things simple. This is
what a basic invocation looks like, in any BUILD</code> file:</p>
detekt(
    name = "detekt_report",
    srcs = glob(["**/*.kt"]),
)
</code></pre>
We can integrate that in our _common_tests</code> macro, to generate a Detekt target
automatically for every Kotlin target:</p>
def _common_tests(name, srcs):
    detekt(
        name = "%s_detekt_report" % name,
        srcs = srcs,
        config = "//java/rules/internal:detekt-config.yml",
    )
</code></pre>
All our Kotlin targets now have a $name_detekt_report</code> target generated
automatically, using a common Detekt configuration.</p>
The way this detekt</code> rule work is by creating a build target, that generates
the report. Which means that it's not actually a test - which is what we were
trying to achieve. In order to do this, we can use
Bazel Skylib</a>'s build_test</code>. This
test rule generates a test target that just has a dependency on other targets -
if any of those dependencies fails to build, then the test fails. Otherwise, it
passes. Our macro becomes:</p>
def _common_tests(name, srcs):
    detekt(
        name = "%s_detekt_report" % name,
        srcs = srcs,
        config = "//java/rules/internal:detekt-config.yml",
    )

    build_test(
        name = "%s_detekt_test" % name,
        targets = [":%s_detekt_report" % name],
    )
</code></pre>
And there we have it - a $name_detekt_test</code> that is actually a test, and will
fail if Detekt raises errors.</p>
Ktlint</h3>
Ktlint doesn't have any existing open-source rules. Let's see how we can write
our own minimal one. It will take as inputs the list of files to check, as well
as an optional editorconfig</a> configuration, that
Ktlint supports natively.</p>
The definition of the rules will be split in three files: two internal files
defining respectively the action</em> (how to invoke Ktlint) and the rule
interface</em> (what's its name, its arguments...), as well as a third, public file,
meant to be consumed by users.</p>
Let's start by downloading Ktlint itself. In the WORKSPACE</code>, as usual:</p>
http_file(
    name = "com_github_pinterest_ktlint",
    executable = True,
    sha256 = "a656342cfce5c1fa14f13353b84b1505581af246638eb970c919fb053e695d5e",
    urls = ["https://github.com/pinterest/ktlint/releases/download/0.36.0/ktlint"],
)
</code></pre>
Let's move onto the action definition. It's a simple macro returning a string,
which defines how to invoke Ktlint, given some arguments. In
//tools/ktlint/internal/actions.bzl</code>:</p>
def ktlint(ctx, srcs, editorconfig):
    """Generates a test action linting the input files.

    Args:
      ctx: analysis context.
      srcs: list of source files to be checked.
      editorconfig: editorconfig file to use (optional)

    Returns:
      A script running ktlint on the input files.
    """

    args = []

    if editorconfig:
        args.append("--editorconfig={file}".format(file = editorconfig.short_path))


    for f in srcs:
        args.append(f.path)

    return "{linter} {args}".format(
        linter = ctx.executable._ktlint_tool.short_path,
        args = " ".join(args),
    )
</code></pre>
Pretty straightforward - we combine both Ktlint's executable path, the
editorconfig file if it's provided, and the list of source files.</p>
Now for the rule interface, we will define a rule named ktlint_test</code>. Building
a ktlint_test</code> target will mean generating a shell script to invoke Ktlint with
the given set of argument, and running it will invoke that script - hence
running Ktlint as well. In //tools/ktlint/internal/rules.bzl</code>:</p>
load(":actions.bzl", "ktlint")

def _ktlint_test_impl(ctx):
    script = ktlint(
        ctx,
        srcs = ctx.files.srcs,
        editorconfig = ctx.file.editorconfig,
    )

    ctx.actions.write(
        output = ctx.outputs.executable,
        content = script,
    )

    files = [ctx.executable._ktlint_tool] + ctx.files.srcs

    if ctx.file.editorconfig:
        files.append(ctx.file.editorconfig)

    return [
        DefaultInfo(
            runfiles = ctx.runfiles(
                files = files,
            ).merge(ctx.attr._ktlint_tool[DefaultInfo].default_runfiles),
            executable = ctx.outputs.executable,
        ),
    ]

ktlint_test = rule(
    _ktlint_test_impl,
    attrs = {
        "srcs": attr.label_list(
            allow_files = [".kt", ".kts"],
            doc = "Source files to lint",
            mandatory = True,
            allow_empty = False,
        ),
        "editorconfig": attr.label(
            doc = "Editor config file to use",
            mandatory = False,
            allow_single_file = True,
        ),
        "_ktlint_tool": attr.label(
            default = "@com_github_pinterest_ktlint//file",
            executable = True,
            cfg = "target",
        ),
    },
    doc = "Lint Kotlin files, and fail if the linter raises errors.",
    test = True,
)

</code></pre>
We have two different parts here - the definition of the interface, with the
call to rule</code>, and the implementation of that rule, defined as
_ktlint_test_impl</code>.</p>
The call to rule</code> define how this rule can be invoked. We define that it
requires a list of .kt</code> and/or .kts</code> files named srcs</code>, an optional file
named editorconfig</code>, as well as a hidden argument named _ktlint_tool</code>, which
is just a helper for us to reference the Ktlint binary - to which we pass the
file we defined in the WORKSPACE</code> earlier.</p>
The actual implementation is working in multiple steps:</p>

It invokes the ktlint</code> action we defined earlier, to generate the script
that will be invoked.</li>
It generates an action to write that script, in a file referred as
ctx.outputs.executable</code> (which Bazel knows how to handle and what to do with
it, we don't need to worry about where it is or anything, it won't be in the
source tree anyway).</li>
It computes a list of files that are needed to run this target. This is what
allows Bazel to ensure hermeticity - it will know that this rule needs to be
re-run if any of those files are changed. If the target runs in a sandboxed
environment (which is the default on most platforms, as far as I'm aware), only
those files will be available.</li>
It returns a Provider</code>, responsible for holding a description of what this
target needs.</li>
</ol>
Finally, we write a file that only exposes the bits users should care about.
It's not mandatory, but makes a clear delimitation between what is an
implementation detail and what users can actually rely on. In
//tools/ktlint/defs.bzl</code>:</p>
load(
    "//tools/ktlint/internal:rules.bzl",
    _ktlint_test = "ktlint_test",
)

ktlint_test = _ktlint_test
</code></pre>
We just expose the rule we wrote in rules.bzl</code> as ktlint_test</code>.</p>
Once this is done, we can use this ktlint_test</code> rule where we needed it, in our
_common_tests</code> macro for Kotlin targets:</p>
def _common_tests(name, srcs):
    ktlint_test(
        name = "%s_ktlint_test" % name,
        srcs = srcs,
        editorconfig = "//:.editorconfig",
    )

    detekt(
        name = "%s_detekt_report" % name,
        srcs = srcs,
        config = "//java/rules/internal:detekt-config.yml",
    )

    build_test(
        name = "%s_detekt_test" % name,
        targets = [":%s_detekt_report" % name],
    )
</code></pre>
And there we have it - all our Kotlin targets have both Detekt and Ktlint test
targets. Because we're exposing those as Bazel targets, we automatically benefit
from its caching and remote execution capabilities - those linters won't re-run
if the inputs didn't change, and can run remotely, with Bazel being aware of
which files are needed on the remote machine.</p>
Closing thoughts</h2>
But what's the link between generating a blog with Bazel and compiling a Kotlin
application? Well, almost none, but there is one. The class diagram included
earlier in this article is generated with a tool called
PlantUML</a>, which generates images from a text
representation of a graph. The next article in this series will talk about
integrating this tool into Bazel (in a similar way as we did with Ktlint), but
also how to test the Bazel rule. And to have some integration tests, Phosphorus
will come in handy!</p>

Compat libraries incompatibilities

2017-05-06T19:52:26+02:00

Compat libraries are great. They allow us to work with the newest Android APIs, without thinking (much) about your minimum API level. Instead of thousands of devices, you can reach billions. With nearly no changes in your code.</p>

But sometimes, they're not so great…</p>

Use Apache HTTP Client on Android SDK 23

2015-10-01T15:46:00+02:00

With the Android M SDK (API 23), Google removed the Apache HTTP Client library. It was deprecated since API 22, and Google recommanded to use

HttpURLConnection</code> instead since API 9. While the classes are still bundled in
Android 6 ROMs, it won't be long until we see them completely go away.</p>

new AlertDialog.Builder(MyActivity.this) .setTitle("Dialog title") .setMessage("Dialog message") .setPositiveButton(android.R.string.yes, new DialogInterface.OnClickListener() { public void onClick(DialogInterface dialog, int which) { // Handle a positive answer } }) .setNegativeButton(android.R.string.no, new DialogInterface.OnClickListener() { public void onClick(DialogInterface dialog, int which) { // Handle a negative answer } }) .setIcon(R.drawable.ic_dialog_alert) .show(); </code></pre> Okay, that's a pretty usual code sample. But what about displaying it from an app-widget?</p>

Actions</h2>

What is Bazel?</h2>
Bazel is a build-system released by Google in 2015. It actually is derived from the internal build-system Google uses internally for most of its own code-base, called Blaze</a>.</p>

A three-steps build</h3>
Bazel runs in three distinct phases</a>. Each of them has a specific role, and specific capabilities.</p>

Execution</h4>
This is the phase that checks for any out-of-date output (either non-existent, or its inputs changed), and runs the matching actions.</p>

enoent.fr

Ktor: Altering served content

Serving static files with Ktor

Writing a Bazel rule set

Compiling a Kotlin application with Bazel

Why Bazel?

A new beginning

Compat libraries incompatibilities

Android Things: first look

Use Apache HTTP Client on Android SDK 23

Share code across multiple platforms

RecyclerView basics

Hosting different kinds of apps on nginx

Using an API - Trakt.tv example

Compile FFmpeg for Android

Arduino Leonardo fully-featured keyboard

Android: display a Dialog from an AppWidget

enoent.fr

Ktor: Altering served content

Serving static files with Ktor

Writing a Bazel rule set

PlantUML usage</h2> PlantUML is a tool that takes a text input looking like this:</p>

Actions</h2>

Public interface</h2> As we only have a single rule, and nothing else specific to do, the public interface is dead simple:</p>

Compiling a Kotlin application with Bazel

Why Bazel?

What is Bazel?</h2> Bazel is a build-system released by Google in 2015. It actually is derived from the internal build-system Google uses internally for most of its own code-base, called Blaze</a>.</p>

A three-steps build</h3> Bazel runs in three distinct phases</a>. Each of them has a specific role, and specific capabilities.</p>

Execution</h4> This is the phase that checks for any out-of-date output (either non-existent, or its inputs changed), and runs the matching actions.</p>

Great tooling</h3> Bazel comes with some really cool tools. Without spending too much time on that, here's a list of useful things:</p>

A new beginning

Compat libraries incompatibilities

Android Things: first look

Use Apache HTTP Client on Android SDK 23

Share code across multiple platforms

RecyclerView basics

Hosting different kinds of apps on nginx

Using an API - Trakt.tv example

Compile FFmpeg for Android

Arduino Leonardo fully-featured keyboard

Android: display a Dialog from an AppWidget

What is Bazel?</h2>
Bazel is a build-system released by Google in 2015. It actually is derived from the internal build-system Google uses internally for most of its own code-base, called Blaze</a>.</p>

A three-steps build</h3>
Bazel runs in three distinct phases</a>. Each of them has a specific role, and specific capabilities.</p>

Execution</h4>
This is the phase that checks for any out-of-date output (either non-existent, or its inputs changed), and runs the matching actions.</p>