Ruby on Rails to Elixir: Testing APIs

I recently decided to port one of my Ruby on Rails applications to Elixir/Phoenix. Not because it needed more performance or anything but rather because the application was a learning experience and it turned out that I didn’t have that much left to learn about Rails. To keep things interesting I decided that I wanted to learn more about Elixir.

Introduction

I am by no means an experience Elixir developer so this will be a learning exercise. There is probably better ways to do this but this is what I learned while writing tests for this application.

The first part to rewrite

The first part of the Rails app is a wrapper around an API that syncs video games with an online database. It uses an HTTP client to do this.

The tests for this part of the application uses VCR. VCR is a gem that records responses from the real API and then uses that response for the next time the test runs. You can choose when to re-record those test fixtures to update the fake data. It all happens under the hood and quite magical. The simplest configuration of VCR auto-records and names the fixture files for you. All you have to do is set the :vcr tag on your tests and you are off to the races.

Attempt #1: ExVcr

When I started implementing the app in Elixir I was delighted when I saw that VCR existed for Elixir as well as the project ExVcr. It is very similar to the Ruby version so using this I got this up and running very quickly. The setup for the test looked a little something like this:

defmodule MyApp.SyncingGamesTest do
  use MyApp.DataCase
  use ExVCR.Mock

  import MyApp.Factory

  test "Syncing a game" do
    use_cassette "syncing_games" do
      game = insert(:game, api_id: 1942)
      MyApp.Api.sync_game(game)
      ...
    end
  end
end

Easy enough! It also uses ExMachina which is an Elixir version of FactoryBot to create database records to use in tests. The ExVCR version worked fine until it suddenty didn’t. For some reason a text field in an API response came back as binary and I couldn’t figure out why. I’m sure sure I did something horribly wrong though.

VCR also has a few other problems such as the lack of the control of the responses. You can edit the fixtures once they are saved on disk but if you ever want to re-record them, which you probably do to make sure the API works for real, you still have to change them again. This is important if the API returns a giant amount of data for example and you don’t want a slow test that processes all of that.

Another issue is that every test in the system that uses this API will have to use a VCR cassette to record the responses. You can reuse the same cassette but then you have to make sure that the same request is made by both tests otherwise ExVcr will throw an error. If you test suite is complicated then this may become quite the hassle to deal with.

While a VCR setup has problems it would probably still be my goto for tests because of how easy it is to get started with it. However, since I’m trying to learn something here I decided to find another solution.

Attempt #2: Bypass

For Rubyists: Bypass is similar to Webmock. It allows you to stub API endpoints for “real” by mounting Plugs at specific endpoints and then generating fake responses while providing the ability to make assertions on the request. For my app the setup looks a little something like this:

setup do
  bypass = Bypass.open()
  {:ok, bypass: bypass}
end

test "Syncing a game", %{bypass: bypass} do
  Bypass.expect(bypass, "GET", "/games", fn conn ->
    Plug.Conn.resp(conn, 200, File.read!("test/fixtures/game.json"))
  end)

  Bypass.expect(bypass, "GET", "/companies", fn conn ->
    Plug.Conn.resp(conn, 200, File.read!("test/fixtures/company.json"))
  end)

  ...

  MyApp.Api.sync_game(game, "http://localhost:#{bypass.port}")
end

The real test setup does a few assertions on the body of the request and such which I left out to make it easier to read. There are also several more Bypass setups here because the API does quite a few requests do its thing.

Now we have full control of the responses. You can use Postman or something similar to record responses from the actual API and then save them as these JSON fixtures. These can easily be reused in other tests as well and you don’t have to worry about the re-recording issue with VCR.

However, this comes with at least two issues: First of all, the setup is quite tedious. If you want to test something that also uses this API module you have to repeat the test setup in that test as well. You can of course write some helpers to do this for you but it is still a bit of a pain to use.

Second, Bypass works a bit differently than Webmock does in Ruby. Bypass will mock an endpoint on localhost and that url has to be given to the API class to use instead of the actual API url. This is fine, but if you are testing something that uses the API rather than the API directly that has to take this API url as a parameter as well. Then you have to pass this down through the system all the way down to the API. This is is a real pain. You can overwrite the main app configuration or something in the test setup to set the API url but that setup is global in the application and if you run your tests asynchronously this can lead to some very nasty errors.

Attempt #3: Configured mocks

In the end I came across this article about mocks written by the creator of Elixir himself: José Valim. This solution involves replacing the entire API client library in the test environment with a fake one which behaves like the real thing but reads rsponses from disk like the Bypass solution above. The difference is that this is globally configued in the configuration files for the test environment and you don’t have to do anything to make the tests work. The implementation of the fake client looks a little something like this:

defmodule MyApp.SomeApi.Fake do
  @behaviour MyApp.SomeApi

  @impl true
  def get_game_data(_) do
    "test/fixtures/api/game.json"
    |> File.read!()
    |> Poison.decode!()
    |> List.first()
  end

  @impl true
  def get_involved_companies(_) do
    "test/fixtures/api/involved_companies.json"
    |> File.read!()
    |> Poison.decode!()
  end
end

This also uses the Behaviour functionality from Elixir to make sure that the fake behaves like the real thing. The fake client can now be used both in tests and in the development environment if the API calls are expensive or rate limited for example.

Then you simply use the app configuration to get an API client that behaves the same way. The real one for the production app and the fake one for tests. You can also very easily build additional implementations for this and replace the client in the configuration without changing a single line of code in the app. You can for example build a caching client that you can use in dev mode that saves the responses from the API so that you don’t perform the same request twice. Thus you save a lot of requests for a rate limited API.

What remains now is dealing with the tests of the actual API. Since all we are doing above is testing the behavior against a fake client we still don’t know if the the API works as expected. José recommends testing against the actual API by using tags and only running the tests in a controlled dev environment and not as part of the regular CI process. I also think that it would be a good idea to use ExVCR in the actual API tests if the API is rate limited. You can also do an integration test with a tag on it that tests the entire chain but against the real API. I like to avoid having redundant tests however so I’ll avoid this for now.

In the end I had the tests for the client hit the API for real and used the tag method above. This seemed to be the best way to make sure that the tests actually made sure that the API client behaved correctly.