Step-by-step tutorial

Estimated time to complete this tutorial (reading and exercising): an hour.

FYI there’s the Git repository which contains all example source codes from this tutorial: https://github.com/nirum-lang/nirum-tutorial-examples.

What Nirum is for

Today we live in the age of microservices. Traditional function calls in a single process and a single language have been replaced by remote procedure calls through multiple languages. It means we cannot call f(x) anymore but instead need to serialize arguments into JSON, make an HTTP request, await a response from the server, parse the response, and validate the result if it is in the allowed range. Function definitions become even more complicated than function calls – you need to make a server! How about unit tests? Hundreds of test cases about malformed JSON input data are not uncommon. Unit tests easily turn indistinguishable from functional tests.

Nirum is both an IDL which stands for interface description language and the compiler implementation of the language at a time. The language purposes to define signatures of remote procedures and data types used for parameters and return values. The point is that it defines only an interface; an implementation for it is written in a general-purpose language like Python. The interface code looks like the following:

service greet-service (
    text greet (text name)
);

The compiler takes an interface written in Nirum language and generates a stub code for the implementation in a general-purpose language. If the target language is Python it looks like the following code:

class GreetService:
    def greet(name: str) -> str:
        raise NotImplementedError

Alternatively, it could be Java:

public interface GreetService {
    @Nonnull
    public String greet(@Nonnull String name);
}

All that remain are to fill the body of greet() with actual implementation and to call the function. Let us imagine that an implementation is like the following Java code:

public class GreetServiceImpl implements GreetService {
    @Nonnull
    public String greet(@Nonnull String name) {
        // Don't have to check (name == null) here.
        return "Hello, " + name + "!";
    }
}

How is it like to call from a client code? Let’s try with a Python interactive shell (run python command without any arguments):

>>> c = GreetService_Client(...)
>>> c.greet("world")
'Hello, world!'

That’s all! All other things like making an HTTP request, serialization, receiving a request, deserialization, data validation, and making errors don’t have to be written by you but are generated by the compiler. Note that the name argument in the above implementation code does not check if it is null. Since the parameter type is not defined as an optional type (i.e., text name instead of text? name), it is guaranteed not null. Argument values are validated first.

To sum up, Nirum helps microservices architecture to be easily achieved and robust for the following reasons:

The rest of this article illustrates how you can utilize Nirum for your microservices through the steps by examples.

Installing compiler

The Nirum compiler works on the most major platforms like Linux, macOS, and Windows. The official releases page provides the prebuilt executable binaries for these three platforms. Note that there is no stable version (i.e., v1.0.0 or higher) as of May 2018.

This tutorial assumes that you have:

so that nirum in the command-line invokes the compiler ($ indicates a command-line prompt and it’s consistent within this tutorial):

$ nirum -v
0.5.0

Service interface

In order to show the elementary features of Nirum, this section supposes a simple artificial example: a counter. It has a state of a natural number, and the state starts with zero. Client code can increment the state by 1 and get the current state (after incremented) at a time. Long story short, it’s merely like a microservice version of ++state.

How does the interface of the counter service look like?

service counter (
    bigint increment ()
);

A service keyword in the above code is for a service definition, and counter which follows is the name of the service. It has one method named increment which takes no arguments and returns an arbitrary precision integer (that’s what bigint means). If it had any parameters it would be between the parentheses, but it’s empty in the above method.

There are other built-in number types as well: two signed integer types (int32, int64), two signed floating-point number types (float32, float64), and an arbitrary real number type (decimal).

In Python, all built-in integer types correspond to int and floating-point numbers correspond to float. The decimal is mapped to decimal.Decimal.

Save this as a file named counter.nrm. Nirum source files end with the suffix .nrm. To make the compiler compile an interface to a target language, a source tree need to be a Nirum package. A minimal example of Nirum package look like the below directory:

Every Nirum package has its own package.toml manifest. As the suffix implies, it is written in TOML:

version = "1.0.0"
description = "A useless counter service"

Wait, have we decided what language we target to? In this tutorial, we choose Python as a target language, because it’s the most mature target language of the Nirum compiler at this moment.

In order to add a target language, we need a corresponding targets.LANGUAGE section as well. In case of Python, it’s targets.python:

version = "1.0.0"
description = "A useless counter service"

[targets.python]
name = "counter-schema"

What does targets.python.name field mean? Not only the compiler builds a single Python source file, but it generates a Python package which consists of Python source files that define interfaces and setup.py file. So a Python package can be installed (and should be done) with pip and also can be submitted to PyPI (whichever it’s official or private). If we configure name = "counter-schema" the name field of setup.py file is set to 'counter-schema' so that you can uninstall it by pip uninstall counter-schema command after you’ve installed it. The name also can be appeared in the URL of the package on PyPI (e.g., https://pypi.python.org/pypi/counter-schema) if it’s submitted.

Compiling

Now we have the most minimal Nirum package that can be compiled. Let’s compile it to a corresponding Python package:

$ nirum --target=python --output-dir=out/ .

The above command can be broken down into two options and an argument:

--target=python : It configures the language we target to Python and can be shorten as -t python.

--output-dir=out/ : It configures a path of the directory that the generated Python files go into. It is okay even if out/ directory does not exist yet. The compiler makes a directory if necessary. It can be shorten as -o out/.

. : It refers to a directory that contains the Nirum package we are going to compile. It must be a directory that contains package.toml at least; in the above example it means there must be ./package.toml file. (If it were src/ instead of ., there would have to be src/package.toml file.)

As a result, several files are made in the out/ directory:

Two text files in the top-level are for Python packaging. Python packaging tools like pip and setuptools recognize them.

The src/ directory contains the actual Python source files. You can observe that counter.nrm file we made corresponds to counter/__init__.py file.

The src-py2/ directory is merely a clone of src/ except it’s ported to Python 2.7[^1] as its name implies. You don’t have to be aware of the distinction of two major Python versions; setup.py automatically switches between them; even if you pack wheel files (i.e., *.whl) two version-tagged wheels are created (further reading: PEP 425).

As you can guess, the out/ directory itself is a complete Python package which can be installed using pip installer.

So how was counter.counter service compiled in Python? Although we can look at the generated out/src/counter/__init__.py source file, unfortunately it’s not that human-readable. Because it’s only for machines! Nevertheless, we can remove extrinsic details from it and reformat the code to scan its gist; it’s like the below:

from nirum.service import Service
from nirum.transport import Transport

class Counter(Service):
    def increment(self) -> int:
        raise NotImplementedError('Counter has to implement increment()')

class Counter_Client(Counter):
    """The client object of :class:`Counter`."""

    def __init__(self, transport: Transport) -> None:
        ...

    def increment(self) -> int:
        ...

To sum up, it defines two classes: Counter and Counter_Client. We can guess several things from this:

Since the detailed rule of naming in the Nirum language has not been explained, you may wonder about this. To be mostly portable and compatible with many programming languages, it has the very limited naming rule:

Even though you can code a name with foo-bar, Foo-Bar, or FOO_BAR, the suggested style is foo-bar.

[^1]: Wonder why the compiler does not generate a single Python source tree that works on both Python 3 and 2.7? Because the Nirum compiler aggressively annotates type hints using the standard typing module to leverage Nirum IDL’s well-typeness. As the annotation syntax used by type hint annotations is introduced since Python 3, we need to port the result code to Python 2 by removing these type annotations. Further reading for modern Python type hints: PEP 3107, PEP 483, and PEP 484.

Filling interface with implementation

Before implementing the increment() method, we need to setup a minimal environment for programming Python. In this tutorial, we’re going to keep assuming we use Python 3.4[^2] or higher. (Recommend Python 3.6 though.)

Also as more source trees are getting involved, this article assumes you place Nirum package files in the above into ~/counter-schema/ directory. So it looks like the following hierarchy:

It’s time to fill the interface we made in the above with an actual implementation. In this chapter, we work on ~/counter-server/, an other directory:

$ mkdir ~/counter-server/
$ cd ~/counter-server/

Although you probably are already aware unless you haven’t used Python before, mostly every Python project need a corresponding virtual environment, also known as “venv.” You can thought it as a project workspace; it consists of interpreter settings (e.g., Python version to use) and third-party dependencies. A new virtual environment can be created using pyvenv command:

$ pyvenv venv

As a result it makes a directory named venv (configured through the argument of pyvenv command):

While creating a virtual environment has to be executed only once, you need to activate it everytime you’re starting to work on the project:

$ . venv/bin/activate
(venv) $

Notice a (venv) indication in the above prompt. It shows that you’ve activated the virtual environment. A way to activate a virtual environment may vary on the platform or the shell. For example, if you’re on Windows the way to activate it is like the below:

C:\counter-server\> venv\Scripts\activate.bat

See also creating virtual environments on the official docs of pyvenv for details.

After activating the environment, pip becomes to install Python packages in the environment, not the system global. Let’s install the Python package we generated in the above[^3]:

$ pip install --editable ~/counter-schema/out/

You can utilize the Python interactive shell to check if the package is installed well:

>>> import counter
>>> counter

Finally it’s ready to implement the method. The possible smallest working implementation would be like the below:

from counter import Counter

class CounterImpl(Counter):
    def __init__(self) -> None:
        self.state = 0
    def increment(self) -> int:
        self.state += 1
        return self.state

What a dumb implementation! It has its own state which counts from 0. The increment() method just increments the state by 1 and returns the current state.

Save the below Python code to counter_server.py to try to run. Now ~/counter-server/ directory has more files:

[^2]: Because it is the least Python version with built-in both pip and pyvenv. If you’re on Debian/Ubuntu you need to install python3-pip and python3-venv packages besides python3 package.

[^3]: Notice --editable (-e) option. Without this option, you need to install the package again everytime any files in ~/counter-schema/out/ are modified.

Remote procedure call

It’s time to run the implementation. To run it on the server, we need nirum-wsgi package[^4] (ensure that you’ve activated the virtual environment):

$ pip install nirum-wsgi

This package provides the nirum-server command which runs a server of a given Nirum service:

$ nirum-server -d 'counter_server:CounterImpl()'
 * Running on http://0.0.0.0:9322/ (Press CTRL+C to quit)
 * Restarting with stat
 * Debugger is active!
 * Debugger PIN: 123-456-789

The -d option turns on debug mode, which means the server reloads every time any of related files changes.

The argument of nirum-service command is a small language that consists of an import path (i.e., Python module path) followed by a colon, and a following Python expression, evaluated to a Nirum service instance (not a class itself).

Notice the address http://0.0.0.0:9322/ (you can change the port using -p/--port option); the protocol used for communication between a service server and a service client is HTTP and it’s intended to be also possible to be used with any HTTP user-agents – it will be more explained in a later.

However, first of all, let’s try a Counter_Client generated by the compiler. As a server-side needs nirum-wsgi package, a client-side needs nirum-http package:

$ pip install nirum-http

Here’s a demo session:

>>> from counter import Counter_Client
>>> from nirum_http import HttpTransport
>>> transport = HttpTransport('http://0.0.0.0:9322/')
>>> client = Counter_Client(transport)
>>> client.increment()
1
>>> client.increment()
2
>>> client.increment()
3

One fun fact is that Counter_Client too actually implements Counter interface:

>>> from counter import Counter
>>> isinstance(client, Counter)
True
>>> issubclass(Counter_Client, Counter)
True

Despite you would be realized soon how useful is this interesting relationship between both types for unit testing, let’s deal with this later.

Anyway, it works nicely, as if it’s called in a single process! Under the hood, however, there’s a lot going on such as making an HTTP request. How does it work then? We can simulate the way Counter_Client and HttpTransport objects communicate with the server, using curl:

$ curl -iX POST http://0.0.0.0:9322/?method=increment
HTTP/1.0 200 OK
Content-type: application/json
Content-Length: 1
Vary: Origin
Access-Control-Allow-Methods: POST, OPTIONS

4

It’s quite dumb and old-fashioned. Nevertheless, it is so simple that in any languages we can call this with only writing few code.

Indeed the protocol is simple and adopts widely-used standards like HTTP and JSON regardless of inefficiency. One of Nirum’s goals is to make any language can easily interoperate with other systems that use Nirum, even if it is not an official target language of Nirum. Network or runtime efficiency is a non-goal, or not a top priority at least, of the Nirum protocol.

To work well with many platforms, Nirum provides a way to route “fancy URL paths” to corresponding service methods; on the surface, these URLs can look like “HTTP resources” of typical HTTP REST APIs. This feature is enabled through an @http-resource annotation:

service counter (
    @http-resource(method = "POST", path = "/count/")
    bigint increment ()
);

What do annotations do? They per se have no meaning, but can be discovered by target backends in compile-time or by runtime libraries in run-time, and may be processed. For @http-resource annotations, nirum-wsgi server scans them and route an HTTP request to a corresponding method.

Since we have updated ~/counter-schema/counter.nrm, it’s necessary to recompile them (-t is a shorthand of --target and -o is of --output-dir):

$ nirum -t python -o ~/counter-schema/out/ ~/counter-schema/

The above command replaces the existing files in ~/counter-schema/out/ directory with Python files generated anew, and it makes nirum-server command to reload the service.

Now the service has a slightly better URL path:

$ curl -iX POST http://0.0.0.0:9322/count/
HTTP/1.0 200 OK
Content-type: application/json
Content-Length: 1
Vary: Origin
Access-Control-Allow-Methods: POST, OPTIONS

1

You may think the number 1 in the response strange. It’s because nirum-server command reloads the service. As the current implementation of CountImpl does not save the state into a persistent store (e.g., filesystem, database) but in memory, reloading it makes the state starts from 0 again.

We can fix this by making state to be saved in a file:

from counter import Counter

class CounterImpl(Counter):
    def __init__(self, state_path: str) -> None:
        self.state_path = state_path
    def increment(self) -> int:
        try:
            with open(self.state_path, 'r+') as f:
                state = int(f.read() or '0')
                state += 1
                f.seek(0)
                f.write(str(state))
        except FileNotFoundError:
            with open(self.state_path, 'w') as f:
                f.write('1')
                state = 1
        return state

(Although it doesn’t work properly for concurrent calls, please ignore that now.)

As a result, the nirum-server process will be unexpectedly terminated with a Python traceback:

 * Detected change in '~/counter-server/counter_server.py', reloading
 * Restarting with stat
Traceback (most recent call last):
  File "~/counter-server/venv/bin/nirum-server", line 11, in 
  ...
  File "", line 1, in 
TypeError: __init__() missing 1 required positional argument: 'state_path'

It’s because we add a state_path parameter to the constructor of CounterImpl class. A Python expression in the argument of nirum-server command also need to be changed (notice "/tmp/state" in the argument):

$ nirum-server -d 'counter_server:CounterImpl("/tmp/state")'
 * Running on http://0.0.0.0:9322/ (Press CTRL+C to quit)
 * Restarting with stat
 * Debugger is active!
 * Debugger PIN: 123-456-789

We haven’t touched the interface but only the implementation, so the way to call the service method has not changed:

$ curl -X POST http://0.0.0.0:9322/count/
1
$ curl -X POST http://0.0.0.0:9322/count/
2

Now you can observe the persistent state from /tmp/state file:

$ cat /tmp/state
2

[^4]: If you are familiar with Python and Python web development, you could guess what nirum-wsgi package does as the name implies. It’s a WSGI adapter application taking a Nirum service.

Method parameters

The previous example was designed to minimize the complexity by intention. That’s why increment() method has no parameters.

In this chapter, the counter example slighly evolves. Previously, increment() method only can increment the state by 1. How about making it possible to increment the state by any delta? If the previous one is like a microservice version of ++state this one is like a microservice version of state += delta.

How would the interface be changed?

service counter (
    @http-resource(method = "POST", path = "/count/{delta}/")
    bigint increment (bigint delta)
);

Now the increment() method has a parameter named delta. It’s typed as bigint which is the same to the return type.

The path option of the @http-resource annotation also changed; a variable path component {delta} is appended. It means a POST /count/123/ request corresponds to a increment(123) call.

As the interface changed, we need to recompile it:

$ nirum -t python -o ~/counter-schema/out/ ~/counter-schema/

The implementation of increment() method also has to be changed:

def increment(self, delta: int) -> int:
    try:
        with open(self.state_path, 'r') as f:
            state = int(f.read() or '0')
    except FileNotFoundError:
        state = 0
    state += delta
    with open(self.state_path, 'w') as f:
        f.write(str(state))
    return state

Again, as we touched counter_server.py, the nirum-server also reloads the service.

Would it work well then?

>>> from counter import Counter_Client
>>> from nirum_http import HttpTransport
>>> transport = HttpTransport('http://0.0.0.0:9322/')
>>> client = Counter_Client(transport)
>>> client.increment(1)
3
>>> client.increment(123)
126

That’s what we’ve expected. How is it like under the hood?

$ curl -X POST http://0.0.0.0:9322/count/123/
249

That also work well.

Server-side unit tests

Although unit testing a microservice is less painful than unit testing a user-facing app, it is much more complicated than unit testing a library. If microservice and library are somewhat analogous in the respect that both are a programming interface, why do not write unit tests for a microservice instead of functional tests? If unit tests of a traditional library don’t deal with calling conventions like cdecl or stdcall, why wouldn’t unit tests of a microservice ignore network transportation like HTTP, data coding, or serialization like JSON?

Because, whereas such low-level details in traditional libraries are hidden by languages, usually still no language is adopted for microservices to hide networking or serialization. Fortunately, we’ve defined the interface for our microservice in Nirum, an IDL, which hides such low-level details.

The following tests.py source shows how is unit testing a microservice like where its interface is defined in Nirum:

import os
from tempfile import mkstemp
from unittest import TestCase
from counter_server import CounterImpl

class CounterImplTest(TestCase):
    def setUp(self):
        fd, self.path = mkstemp()
        os.close(fd)  # only path is necessary
        self.counter = CounterImpl(self.path)
    def tearDown(self):
        os.unlink(self.path)
    def test_initial_state(self):
        self.assertEqual(0, self.counter.increment(delta=0))
    def test_increment(self):
        self.assertEqual(1, self.counter.increment(delta=1))
        self.assertEqual(2, self.counter.increment(delta=1))
    def test_increment_by_5(self):
        self.assertEqual(5, self.counter.increment(delta=5))
        self.assertEqual(10, self.counter.increment(delta=5))

The above code uses a unit testing framework named unittest in the Python standard library, because it is built-in so that we don’t need to install it and has the most friendly interface which derives from xUnit.

Its setUp() method tries to make a fixture instance of CounterImpl. Since the constructor of CounterImpl takes a file path to store its state, setUp() also creates a temporary file, and then tearDown() method removes it after tests finish.

The rest test_initial_state(), test_increment(), and test_increment_by_5() methods test if the CounterImpl instance satisfies behaviors we expect. It’s quite straightforward and concise; it doesn’t have to test like what if the increment() method takes a string instead of an integer or what if the payload under the hood is a malformed JSON.

The following command runs tests:

$ python -m unittest tests.py
...
----------------------------------------------------------------------
Ran 3 tests in 0.004s

OK

Further reading

This tutorial covers only few features of Nirum. If you’ve become interested in Nirum try to see also the below resources: