Why Go for Redis when You Can Use Mnesia?
25 feb 2019
8 min
Back-end developer @ WTTJ
Mnesia is cool. It’s quite cryptic but worth the pain. Already present in the OTP framework, it’s easy to use in both simple cases and out-of-the-box replication inside a cluster. This article covers what happened when we implemented Mnesia, including the difficulties we encountered and the results we got.
Use case
At Welcome to the Jungle, we do a lot of things. We’ve built an applicant tracking system (ATS), and we have a Welcome Kit, a media platform that provides insights into employment, and a website that allows candidates to find their dream job. We’ve also released our first game: Welcome to the Jungle THE GAME.
Our game is composed of the game itself, created using Pixi.js, and a backend built using Elixir (more on that may be covered in another article). During a game session, the client sends to the server any event that changes the score. At the end of the session, the client ensures with the server that the score is consistent (fraud prevention) and then registers it.
Each game session uses a GenServer that holds the score in its state. Every event that is sent to the server is transferred to the GenServer with a callback. At first this was an OK solution: it was easy to set up, fast, and reliable. Then, as the development continued, we decided to put the server behind a load balancer. By default, load balancing was round-robin, so there was the chance a request could be redirected to the wrong server, thereby causing an error.
Database choices
We looked at 3 solutions. The first, which involved redirecting requests for a game session to the related GenServer through the load balancer, was quickly discarded because it was too much hassle. The second involved using a global registry that allowed access to the GenServers from any server. This was also abandoned, as Elixir can’t do that natively (or it needs to use “dirty” tricks to do it), and while some libraries seemed interesting (such as Swarm), using them would generate other problems: if a GenServer were to crash, we would lose all the data for current game sessions.
Therefore, the solution was obviously to store the game states in a database and ensure that every server has access to those. We also had to choose the database from, among others, PostgreSQL, Redis, and MongoDB. PostgreSQL was a good idea—it’s powerful and we were already using it to store users and final scores for leaderboards—but we were doubtful about its capacity to handle a large amount of operations in short space of time (such as performance or race conditions).
We then decided to use Mnesia, Erlang’s distributed database management system. The main reason for this was that it’s embedded. With Redis or MongoDB we would have had to set up a dedicated server to solve our problem, whereas Mnesia requires nothing more than the node the server is already running on. Another, more personal, reason behind the decision was that I wanted to try out a real project with Mnesia instead of just playing around with it. This situation offered the perfect opportunity to give it a go.
Mnesia
As mentioned, Mnesia is a distributed database management system written in Erlang and part of the OTP framework. At first look, it doesn’t appear very attractive: there’s no Elixir documentation, very few resources to read, and an unappealing page on Erlang’s documentation, but once you get used to it, you can do great things. It stores tuples inside tables, which can contain any Elixir element—even functions! To read a specific entry, you can use :mnesia.read/1
, and if you want to access a bunch of records you can use :mnesia.select/2
or :mnesia.match_object/3
. You can use query list comprehensions (QLCs), too—this is a bit harder, but it allows you to make complex requests on multiple tables.
Current code
Let’s take a tour of the code we were using before we turned to Mnesia. I’ve removed a lot of code to keep it simple and get straight to the point.
defmodule WttjGame.Server do use GenServer def init(args) do {:ok, args} end def start_link(id, options \\ []) do GenServer.start_link(__MODULE__, %{id: id, score: 0}, options) end def update(pid, kind, params) do GenServer.call(pid, {:update, kind, params}) end def handle_call({:update, kind, params}, _from, %{score: score} = state) do {:ok, value} = process_event(kind, params) new_state = Map.put(state, :score, score + value) {:reply, :ok, new_state} end defp process_event(`shoot`, %{`combo` => combo}), do: {:ok, combo * 20}end
This is the GenServer from WTTJ Game with one callback to increase your score according to received events. If a GenServer crashes, the state is lost forever and the player will not be able to save their score. Our solution was to insert a new state into the database when starting a session and then update it each time an event that changes the score occurs.
Mnesia bases
To start using Mnesia, we needed to create the database using :mnesia.create_schema/1
. Without a schema, nothing is saved on disc. To create our GameState
table, we first needed to start Mnesia using :mnesia.start/0
and then run :mnesia.create_table/2
. When all had been put together, we ended up with the following function:
def init do :mnesia.create_schema([node()]) :mnesia.start() :mnesia.create_table(GameState, attributes: [:id, :score], disc_copies: [node()]) end
Note the disc_copies: [node()]
—this means that data is stored both on disc and in the memory.
Now we needed to insert, retrieve, and update our records. In Mnesia, every action needs to be wrapped within a transaction. If something goes wrong with executing a transaction, it will be rolled back and nothing will be saved on the database. We can also use locks to prevent race conditions, which will be released at the end of a transaction. To read and write, we can use :mnesia.read/3
and :mnesia.write/1
.
But be careful! Functions such as :mnesia.read/3
need Mnesia to be started with :mnesia.start/0
. If Mnesia is not started in this way, the functions will fail. The following code being an extract, you won’t see any use of the start function.
def insert(id) do :mnesia.transaction(fn -> case :mnesia.read(GameState, id, :write) do [] -> :mnesia.write({GameState, id, 0}) _ -> :ok end end) end def update_score(id, value) do :mnesia.transaction(fn -> [{GameState, ^id, score}] = :mnesia.read(GameState, id, :write) :mnesia.write({GameState, id, score + value}) end) end
Here we have used locks to ensure that there are no race conditions. There are roughly two kinds of lock: :write
, which prevents other transactions from acquiring a lock on a resource, and :read
, which allows other nodes to obtain only :read
locks. Here, the :write
lock ensures that if one process tries to acquire the record we are working on, it will wait until we are done.
Improve our WttjGame.Server
The function insert/1
is used inside start_link/2
before starting the GenServer, while update_score/2
is inserted at the end of the callback for the update before sending back the response.
defmodule WttjGame.Server do use GenServer alias WttjGame.GameStates def init(args) do {:ok, args} end def start_link(id, options \\ []) do GameStates.insert(id) GenServer.start_link(__MODULE__, %{id: id}, options) end def update(pid, kind, params) do GenServer.call(pid, {:update, kind, params}) end def handle_call({:update, kind, params}, _from, %{id: id} = state) do {:ok, value} = process_event(kind, params) GameStates.update_score(id, value) {:reply, :ok, state} end defp process_event(`shoot`, %{`combo` => combo}), do: {:ok, combo * 20}end
Here, we have a solid system that allows multiple processes to update the game concurrently. Nothing will be lost, but at this point the system is only local: if two servers run at the same time, they won’t be able to communicate. Fortunately, Elixir and Mnesia give us the tools to help.
Let’s communicate
Setting up the cluster
To connect our two servers together, we needed to use the Node.connect/1
function. But it doesn’t happen by magic—some configuration is required to make it work. We had to open ports to allow Elixir processes to communicate, the first one is 4369
for epmd (Erlang Port Mapper Daemon), which keeps track of locally started nodes along with their ports and routes messages from one Erlang process to another.
We then needed to allow a range of ports, which can be customized by setting the kernel variables inet_dist_listen_min
and inet_dist_listen_max
(refer to Erlang documentation for more information).
To be sure your servers are from the same cluster, you then need to provide a way for them to be able to authenticate themselves. To do that, simply run your server with the same magic cookie
(use --cookie
on vm.args or when running iex). Once that’s done, you can open a console and use Node.connect/1
with the node name. The node name format to use is name@host
, but be careful to run your server using --name
and not --sname
. The latter is a short name, which omits the host name and thus does not allow the node to communicate with the outer world. The function returns true
if all is OK (connected or already connected), or false
if it fails, and :ignored
if the local node is not alive.
Connecting Mnesia
When both servers are up and connected, you need to reconfigure Mnesia in order to share its contents. First, inform Mnesia of other nodes that belong to the cluster. For this, we used :mnesia.change_config/2
to change the variable :extra_db_nodes
to Node.list()
. Then, to ensure that data can be stored on disc, we used :mnesia.change_table_copy_type(:schema, node(), :disc_copies)
. Finally, we used :mnesia.add_table_copy/3
to add our GameState
table to the second server.
def connect_mnesia do :mnesia.start() :mnesia.change_config(:extra_db_nodes, Node.list()) :mnesia.change_table_copy_type(:schema, node(), :disc_copies) :mnesia.add_table_copy(GameState, node(), :disc_copies) end
Using this function after connecting to another server allows you to retrieve and share everything about the GameState
table.
Deploying our work
The last issue with this is… it’s manual. Yep, in an era where everything needs to be automated, our system depends on manually connecting both servers and running this function. As we don’t really like things that require human interaction to work, we needed to automate this workflow.
For this, we needed two things: the current server IP and at least one IP from the cluster. To define the node name, you can either generate a vm.args
with the node name (for example, -name wttj-game@10.0.0.1
), or you can use Node.start/3
. Last but not least, we needed to connect to our cluster and then run our connect_mnesia/0
function. Distillery, which we use to build releases, allowed us to create commands to run using the release binary by executing a shell script. Thus, we run the below just after starting the server.
#!/bin/shwhile true; do require_live_node EXIT_CODE=$? if [ $EXIT_CODE -eq 0 ]; then echo `Application is responding!` break fidonerelease_ctl eval --mfa `WttjGame.ReleaseTasks.init_cluster/1` --argv -- `$1`
WttjGame.ReleaseTasks.init_cluster/1
simply parses the parameters, extracts IPs, tries to connect to each of them, and then runs connect_mnesia/0
to create shared tables with all other instances.
@ip_regexp ~r/^\d+\.\d+\.\d+\.\d+$/ def init_cluster(str) do str |> String.trim() |> String.split(`\t`) |> Enum.filter(fn ip -> String.match?(ip, @ip_regexp) end) |> Enum.each(fn ip -> node = :`wttj-game@#{ip}` Logger.info(`Trying to connect to node #{node}`) Node.connect(node) end) GameStates.connect_mnesia() end
And… we’re done!
Wrapping up
After some difficulties, we managed to set it up so that it distributed and deployed automatically. But this came with a cost: poor documentation, few resources to read on the web, and errors that were quite difficult to understand (gotta love the {:aborted, error}
tuple).
It has been a long journey through the lands of Erlang. We experimented with an obscure database that is not used by many. The results were really interesting: Mnesia revealed itself to be quick, its basic concepts are easy to learn, and you can use it out of the box because it’s included in Erlang/OTP!
That’s really the main reason I love Mnesia. It costs less, you don’t need to bother setting up anything as it’s already there. There’s no SQL, no JavaScript, just plain Elixir, with its powerful pattern matching and query comprehension syntax. It’s fast and easy to use: create a table, wrap your reads, write in a transaction, and enjoy!
- http://erlang.org/doc/man/mnesia.html
- https://elixirschool.com/en/lessons/specifics/mnesia/
- https://learnyousomeerlang.com/mnesia
- https://gist.github.com/matthias-endler/5273951
This article is part of Behind the Code, the media for developers, by developers. Discover more articles and videos by visiting Behind the Code!
Want to contribute? Get published!
Follow us on Twitter to stay tuned!
Illustration by Blok
Más inspiración: Coder stories
We can learn a lot by listening to the tales of those that have already paved a path and by meeting people who are willing to share their thoughts and knowledge about programming and technologies.
Keeping up with Swift's latest evolutions
Daniel Steinberg was our guest for an Ask Me Anything session (AMA) dedicated to the evolutions of the Swift language since Swift 5 was released.
10 may 2021
"We like to think of Opstrace as open-source distribution for observability"
Discover the main insights gained from an AMA session with Sébastien Pahl about Opstrace, an open-source distribution for observability.
16 abr 2021
The One Who Co-created Siri
Co-creator of the voice assistant Siri, Luc Julia discusses how the back end for Siri was built at Apple, and shares his vision of the future of AI.
07 dic 2020
The Breaking Up of the Global Internet
Only 50 years since its birth, the Internet is undergoing some radical changes.
26 nov 2020
On the Importance of Understanding Memory Handling
One concept that can leave developers really scratching their heads is memory, and how programming languages interact with it.
27 oct 2020
¿Estás buscando tu próxima oportunidad laboral?
Más de 200.000 candidatos han encontrado trabajo en Welcome to the Jungle
Explorar ofertas