Why Go for Redis when You Can Use Mnesia?

Mnesia is cool. It’s quite cryptic but worth the pain. Already present in the OTP framework, it’s easy to use in both simple cases and out-of-the-box replication inside a cluster. This article covers what happened when we implemented Mnesia, including the difficulties we encountered and the results we got.

Use case

At Welcome to the Jungle, we do a lot of things. We’ve built an applicant tracking system (ATS), and we have a Welcome Kit, a media platform that provides insights into employment, and a website that allows candidates to find their dream job. We’ve also released our first game: Welcome to the Jungle THE GAME.

Our game is composed of the game itself, created using Pixi.js, and a backend built using Elixir (more on that may be covered in another article). During a game session, the client sends to the server any event that changes the score. At the end of the session, the client ensures with the server that the score is consistent (fraud prevention) and then registers it.

Each game session uses a GenServer that holds the score in its state. Every event that is sent to the server is transferred to the GenServer with a callback. At first this was an OK solution: it was easy to set up, fast, and reliable. Then, as the development continued, we decided to put the server behind a load balancer. By default, load balancing was round-robin, so there was the chance a request could be redirected to the wrong server, thereby causing an error.

Database choices

We looked at 3 solutions. The first, which involved redirecting requests for a game session to the related GenServer through the load balancer, was quickly discarded because it was too much hassle. The second involved using a global registry that allowed access to the GenServers from any server. This was also abandoned, as Elixir can’t do that natively (or it needs to use “dirty” tricks to do it), and while some libraries seemed interesting (such as Swarm), using them would generate other problems: if a GenServer were to crash, we would lose all the data for current game sessions.

Therefore, the solution was obviously to store the game states in a database and ensure that every server has access to those. We also had to choose the database from, among others, PostgreSQL, Redis, and MongoDB. PostgreSQL was a good idea—it’s powerful and we were already using it to store users and final scores for leaderboards—but we were doubtful about its capacity to handle a large amount of operations in short space of time (such as performance or race conditions).

We then decided to use Mnesia, Erlang’s distributed database management system. The main reason for this was that it’s embedded. With Redis or MongoDB we would have had to set up a dedicated server to solve our problem, whereas Mnesia requires nothing more than the node the server is already running on. Another, more personal, reason behind the decision was that I wanted to try out a real project with Mnesia instead of just playing around with it. This situation offered the perfect opportunity to give it a go.

Mnesia

As mentioned, Mnesia is a distributed database management system written in Erlang and part of the OTP framework. At first look, it doesn’t appear very attractive: there’s no Elixir documentation, very few resources to read, and an unappealing page on Erlang’s documentation, but once you get used to it, you can do great things. It stores tuples inside tables, which can contain any Elixir element—even functions! To read a specific entry, you can use :mnesia.read/1, and if you want to access a bunch of records you can use :mnesia.select/2 or :mnesia.match_object/3. You can use query list comprehensions (QLCs), too—this is a bit harder, but it allows you to make complex requests on multiple tables.

Current code

Let’s take a tour of the code we were using before we turned to Mnesia. I’ve removed a lot of code to keep it simple and get straight to the point.

defmodule WttjGame.Server do  use GenServer  def init(args) do    {:ok, args}  end  def start_link(id, options \\ []) do    GenServer.start_link(__MODULE__, %{id: id, score: 0}, options)  end  def update(pid, kind, params) do    GenServer.call(pid, {:update, kind, params})  end  def handle_call({:update, kind, params}, _from, %{score: score} = state) do    {:ok, value} =  process_event(kind, params)    new_state = Map.put(state, :score, score + value)    {:reply, :ok, new_state}  end  defp process_event(`shoot`, %{`combo` => combo}), do: {:ok, combo * 20}end

This is the GenServer from WTTJ Game with one callback to increase your score according to received events. If a GenServer crashes, the state is lost forever and the player will not be able to save their score. Our solution was to insert a new state into the database when starting a session and then update it each time an event that changes the score occurs.

Mnesia bases

To start using Mnesia, we needed to create the database using :mnesia.create_schema/1. Without a schema, nothing is saved on disc. To create our GameState table, we first needed to start Mnesia using :mnesia.start/0 and then run :mnesia.create_table/2. When all had been put together, we ended up with the following function:

  def init do    :mnesia.create_schema([node()])    :mnesia.start()    :mnesia.create_table(GameState, attributes: [:id, :score], disc_copies: [node()])  end

Note the disc_copies: [node()]—this means that data is stored both on disc and in the memory.

Now we needed to insert, retrieve, and update our records. In Mnesia, every action needs to be wrapped within a transaction. If something goes wrong with executing a transaction, it will be rolled back and nothing will be saved on the database. We can also use locks to prevent race conditions, which will be released at the end of a transaction. To read and write, we can use :mnesia.read/3 and :mnesia.write/1.

But be careful! Functions such as :mnesia.read/3 need Mnesia to be started with :mnesia.start/0. If Mnesia is not started in this way, the functions will fail. The following code being an extract, you won’t see any use of the start function.

  def insert(id) do    :mnesia.transaction(fn ->      case :mnesia.read(GameState, id, :write) do        [] -> :mnesia.write({GameState, id, 0})        _ -> :ok      end    end)  end  def update_score(id, value) do    :mnesia.transaction(fn ->      [{GameState, ^id, score}] = :mnesia.read(GameState, id, :write)      :mnesia.write({GameState, id, score + value})    end)  end

Here we have used locks to ensure that there are no race conditions. There are roughly two kinds of lock: :write, which prevents other transactions from acquiring a lock on a resource, and :read, which allows other nodes to obtain only :read locks. Here, the :write lock ensures that if one process tries to acquire the record we are working on, it will wait until we are done.

Improve our WttjGame.Server

The function insert/1 is used inside start_link/2 before starting the GenServer, while update_score/2 is inserted at the end of the callback for the update before sending back the response.

defmodule WttjGame.Server do  use GenServer  alias WttjGame.GameStates  def init(args) do    {:ok, args}  end  def start_link(id, options \\ []) do    GameStates.insert(id)    GenServer.start_link(__MODULE__, %{id: id}, options)  end  def update(pid, kind, params) do    GenServer.call(pid, {:update, kind, params})  end  def handle_call({:update, kind, params}, _from, %{id: id} = state) do    {:ok, value} = process_event(kind, params)    GameStates.update_score(id, value)    {:reply, :ok, state}  end  defp process_event(`shoot`, %{`combo` => combo}), do: {:ok, combo * 20}end

Here, we have a solid system that allows multiple processes to update the game concurrently. Nothing will be lost, but at this point the system is only local: if two servers run at the same time, they won’t be able to communicate. Fortunately, Elixir and Mnesia give us the tools to help.

Let’s communicate

Setting up the cluster

To connect our two servers together, we needed to use the Node.connect/1 function. But it doesn’t happen by magic—some configuration is required to make it work. We had to open ports to allow Elixir processes to communicate, the first one is 4369 for epmd (Erlang Port Mapper Daemon), which keeps track of locally started nodes along with their ports and routes messages from one Erlang process to another.

We then needed to allow a range of ports, which can be customized by setting the kernel variables inet_dist_listen_min and inet_dist_listen_max (refer to Erlang documentation for more information).

To be sure your servers are from the same cluster, you then need to provide a way for them to be able to authenticate themselves. To do that, simply run your server with the same magic cookie (use --cookie on vm.args or when running iex). Once that’s done, you can open a console and use Node.connect/1 with the node name. The node name format to use is name@host, but be careful to run your server using --name and not --sname. The latter is a short name, which omits the host name and thus does not allow the node to communicate with the outer world. The function returns true if all is OK (connected or already connected), or false if it fails, and :ignored if the local node is not alive.

Connecting Mnesia

When both servers are up and connected, you need to reconfigure Mnesia in order to share its contents. First, inform Mnesia of other nodes that belong to the cluster. For this, we used :mnesia.change_config/2 to change the variable :extra_db_nodes to Node.list(). Then, to ensure that data can be stored on disc, we used :mnesia.change_table_copy_type(:schema, node(), :disc_copies). Finally, we used :mnesia.add_table_copy/3 to add our GameState table to the second server.

  def connect_mnesia do    :mnesia.start()    :mnesia.change_config(:extra_db_nodes, Node.list())    :mnesia.change_table_copy_type(:schema, node(), :disc_copies)    :mnesia.add_table_copy(GameState, node(), :disc_copies)  end

Using this function after connecting to another server allows you to retrieve and share everything about the GameState table.

Deploying our work

The last issue with this is… it’s manual. Yep, in an era where everything needs to be automated, our system depends on manually connecting both servers and running this function. As we don’t really like things that require human interaction to work, we needed to automate this workflow.

For this, we needed two things: the current server IP and at least one IP from the cluster. To define the node name, you can either generate a vm.args with the node name (for example, -name wttj-game@10.0.0.1), or you can use Node.start/3. Last but not least, we needed to connect to our cluster and then run our connect_mnesia/0 function. Distillery, which we use to build releases, allowed us to create commands to run using the release binary by executing a shell script. Thus, we run the below just after starting the server.

#!/bin/shwhile true; do  require_live_node  EXIT_CODE=$?  if [ $EXIT_CODE -eq 0 ]; then    echo `Application is responding!`    break  fidonerelease_ctl eval --mfa `WttjGame.ReleaseTasks.init_cluster/1` --argv -- `$1`

WttjGame.ReleaseTasks.init_cluster/1 simply parses the parameters, extracts IPs, tries to connect to each of them, and then runs connect_mnesia/0 to create shared tables with all other instances.

  @ip_regexp ~r/^\d+\.\d+\.\d+\.\d+$/  def init_cluster(str) do    str    |> String.trim()    |> String.split(`\t`)    |> Enum.filter(fn ip -> String.match?(ip, @ip_regexp) end)    |> Enum.each(fn ip ->      node = :`wttj-game@#{ip}`      Logger.info(`Trying to connect to node #{node}`)      Node.connect(node)    end)    GameStates.connect_mnesia()  end

And… we’re done!

Wrapping up

After some difficulties, we managed to set it up so that it distributed and deployed automatically. But this came with a cost: poor documentation, few resources to read on the web, and errors that were quite difficult to understand (gotta love the {:aborted, error} tuple).

It has been a long journey through the lands of Erlang. We experimented with an obscure database that is not used by many. The results were really interesting: Mnesia revealed itself to be quick, its basic concepts are easy to learn, and you can use it out of the box because it’s included in Erlang/OTP!

That’s really the main reason I love Mnesia. It costs less, you don’t need to bother setting up anything as it’s already there. There’s no SQL, no JavaScript, just plain Elixir, with its powerful pattern matching and query comprehension syntax. It’s fast and easy to use: create a table, wrap your reads, write in a transaction, and enjoy!

• http://erlang.org/doc/man/mnesia.html
• https://elixirschool.com/en/lessons/specifics/mnesia/
• https://learnyousomeerlang.com/mnesia
• https://gist.github.com/matthias-endler/5273951

This article is part of Behind the Code, the media for developers, by developers. Discover more articles and videos by visiting Behind the Code!

Want to contribute? Get published!

Follow us on Twitter to stay tuned!

Illustration by Blok