POST DIRECTORY
Category software development

In the last article we looked at using Ecto schemas to read records and associations from a database built according to Rails naming conventions. This time we will extend the examples used in that article to see how Ecto changesets can help us write records to the database in a safe and consistent way.

The example apps can be found at PlaygroundRails and PlaygroundPhoenix.

Because there may be significant changes to Phoenix in 1.3, I’ll stick to describing Ecto changesets in a general way. Hopefully the examples will apply to both current and future Phoenix versions.

The Ecto.Changeset module has three responsibilities that help protect data consistency and integrity. First they allow us to filter incoming parameters and cast those parameters according to the field types defined in our Ecto schemas. Second they allow us to validate fields according to business rules and provide error information about validation failures. Finally they capture database constraint errors during the database write and report them using the same mechanism as validations.

In ActiveRecord terms: Ecto changeset filtering and casting are like a combination of Rails strong params and ActiveRecord’s field type metadata. Changeset validation and error reporting is very similar to how ActiveRecord does it, but the consistent handling of database constraint failures is nicer in Ecto than in ActiveRecord.

In addition to these three responsibilities, Ecto changesets are sometimes used to programmatically set fields like you might in ActiveRecord before_* callbacks. A typical example (and one you can find in the Programming Phoenix book) is automatically building a slug and adding it to the changeset.

There is no way to perform behavior similar to after_* callbacks using a changeset, but you can use Ecto.Multi to compose multiple database operations and normal function calls into transactions that rollback cleanly on any failure. In the future when controllers directly call service modules, I expect that Ecto.Multi will become the preferred way to implement all callback-like behaviors.

Building and Using Changesets

We can build a changeset using the functions Ecto.Changeset.cast/3 or Ecto.Changeset.change/2. These functions return a changeset struct that is then extended by piping it to additional validations, constraints, and custom functions. The resulting changeset can be validated and written to the database using functions in Ecto.Repo.

We can build a changeset and insert a record in iex like this:

$ iex -S mix
iex(1)> changeset = Ecto.Changeset.change(%PhoenixApp.Project{name: "Project"})
#Ecto.Changeset<action: nil, changes: %{}, errors: [],
 data: #PhoenixApp.Project<>, valid?: true>

iex(2)> PhoenixApp.Repo.insert(changeset)
begin []
INSERT INTO "projects" ("name","created_at","updated_at") VALUES ($1,$2,$3) RETURNING "id" ["Project", {{2016, 11, 1}, {21, 19, 24, 509981}}, {{2016, 11, 1}, {21, 19, 24, 529299}}]
commit []
{:ok,
 %PhoenixApp.Project{__meta__: #Ecto.Schema.Metadata<:loaded, "projects">,
  created_at: ~N[2016-11-01 21:19:24.509981], id: 5,
  items: #Ecto.Association.NotLoaded<association :items is not loaded>,
  lists: #Ecto.Association.NotLoaded<association :lists is not loaded>,
  name: "Project", slug: nil, updated_at: ~N[2016-11-01 21:19:24.529299],
  users: #Ecto.Association.NotLoaded<association :users is not loaded>}}

Taking a look at the fields on the changeset struct, we can see that data is the Project struct we passed to change/2. The changes field is a map of changed fields and their new values, we’ll see later that it behaves something like ActiveRecord’s changes instance method. The errors field contains any validation errors or constraint violation errors, again behaving like ActiveRecord’s errors method. The last field to notice is the valid? boolean which is set according to the validations we impose on the changeset.

With the basics under our belt we’re ready to explore changesets in more detail. We’ll start by looking at how we use them to filter and cast parameters.

Filtering and Casting Parameters

The first step in writing data to a database record is to sanitize and cast the incoming data into a form that is safe to manipulate and persist. In Ecto we use Ecto.Changeset.cast/3 to do this work. Let’s start by seeing an example of how cast/3 can filter out params we don’t want to accept.

In PhoenixApp both the Project and Item schemas have slug fields that will be generated automatically when inserted. We don’t want this slug field to be over-ridable, so we can filter it out during the cast.

In this example we create a changeset based on a Project struct, and then apply cast/3 to it. We can see that by limiting the allowed fields to just [:name] we prevented the "slug" param from being accepted in the new changeset, while "name" is allowed. Another thing to note is that the changed "name" param is added to the changes map, but the original struct remains the same.

$ iex -S mix
iex(1)> changeset = Ecto.Changeset.change(%PhoenixApp.Project{name: "Project"})
#Ecto.Changeset<action: nil, changes: %{}, errors: [],
 data: #PhoenixApp.Project<>, valid?: true>

iex(2)> changeset = Ecto.Changeset.cast(changeset, %{"name" => "PROJECT!!!", "slug" => "slugslug"}, [:name])
#Ecto.Changeset<action: nil, changes: %{name: "PROJECT!!!"}, errors: [],
 data: #PhoenixApp.Project<>, valid?: true>

iex(3)> changeset.data.name
"Project"

iex(4)> changeset.data.slug
nil

iex(5)> changeset.changes.name
"PROJECT!!!"

Because filtering and casting incoming data is often the first step in persisting a record, you will commonly see Ecto.Changeset.cast/3 used to create the initial changeset based on a new or existing struct and external input params.

$ iex -S mix
iex(1)> changeset = Ecto.Changeset.cast(%PhoenixApp.Project{name: "Project"}, %{"name" => "PROJECT!!!", "slug" => "slugslug"}, [:name])
#Ecto.Changeset<action: nil, changes: %{name: "PROJECT!!!"}, errors: [],
 data: #PhoenixApp.Project<>, valid?: true>

iex(2)> changeset.data.name
"Project"

iex(3)> changeset.data.slug
nil

iex(4)> changeset.changes.name
"PROJECT!!!"

Let’s take a look at how cast/3 helps us cast field types according to the associated schema definition. In this case let’s add a virtual float field called rating to Project:

schema "projects" do
  . . .
  field :rating, :float, virtual: true
  . . .
end

Virtual fields behave like normal fields during changeset operations, but they aren’t actually written to the database during inserts and updates. In this case we just want to see type casting in action, but the behavior is the same for persisted fields.

$ iex -S mix
iex(1)> changeset = Ecto.Changeset.cast(%PhoenixApp.Project{name: "Project"}, %{"name" => "PROJECT!!!", "rating" => "4.75"}, [:name, :rating])
#Ecto.Changeset<action: nil, changes: %{name: "PROJECT!!!", rating: 4.75},
 errors: [], data: #PhoenixApp.Project<>, valid?: true>

iex(2)> changeset.data.rating
nil

iex(3)> changeset.changes.rating
4.75

iex(4)> is_float changeset.changes.rating
true

Just like ActiveRecord, Ecto changesets only track actual field changes. If we set a field to its existing value nothing is added to the changes map. (Here we use Map.get to access name in the changes map, otherwise we’ll get a KeyError for trying to access a non-existant key.)

$ iex -S mix

iex(1)> changeset = Ecto.Changeset.cast(%PhoenixApp.Project{name: "Project"}, %{"name" => "Project"}, [:name])
#Ecto.Changeset<action: nil, changes: %{}, errors: [],
 data: #PhoenixApp.Project<>, valid?: true>

iex(2)> Map.get(changeset.changes, :name)
nil

The last thing to be aware of is how cast/3 handles “empty” param values. If the value is nil or an empty string, cast will convert it to a nil in the changes map.

$ iex -S mix
iex(1)> changeset = Ecto.Changeset.cast(%PhoenixApp.Project{name: "Project"}, %{"name" => nil}, [:name])
#Ecto.Changeset<action: nil, changes: %{name: nil}, errors: [],
 data: #PhoenixApp.Project<>, valid?: true>

iex(2)> changeset.changes.name
nil

iex(3)> changeset = Ecto.Changeset.cast(%PhoenixApp.Project{name: "Project"}, %{"name" => ""}, [:name])
#Ecto.Changeset<action: nil, changes: %{name: nil}, errors: [],
 data: #PhoenixApp.Project<>, valid?: true>

iex(4)> changeset.changes.name
nil

Validating Fields

Field validations in Ecto changesets look quite different in form from ActiveRecord validations, but the types of validations you can enforce are almost identical. If you look at the available validate_* changeset functions they should look very familiar.

There are a few omissions due to the way Ecto handles associations and database constraints. There is no direct analogue to validates_associated, but you can use put_assoc with an already validated association changeset to get a similar effect. There is also no uniqueness validation in Ecto, instead it is handled by catching constraint violation exceptions from the database. This prevents the race condition implicit in how ActiveRecord validates uniqueness before saving the record.

With that out of the way, let’s see how Ecto changesets respond to validations. We can use our now familiar cast/3 call to create a Project based changeset, and then we’ll validate that the name and slug fields are not empty, and that the rating field is positive.

$ iex -S mix
iex(1)> changeset = Ecto.Changeset.cast(%PhoenixApp.Project{name: "Project"}, %{"name" => "", "rating" => "-1.25"}, [:name, :rating])
#Ecto.Changeset<action: nil, changes: %{name: nil, rating: -1.25}, errors: [],
 data: #PhoenixApp.Project<>, valid?: true>

iex(2)> changeset = Ecto.Changeset.validate_required(changeset, [:name, :slug])
#Ecto.Changeset<action: nil, changes: %{name: nil, rating: -1.25},
 errors: [name: {"can't be blank", [validation: :required]},
  slug: {"can't be blank", [validation: :required]}],
 data: #PhoenixApp.Project<>, valid?: false>

iex(3)> changeset = Ecto.Changeset.validate_number(changeset, :rating, greater_than_or_equal_to: 0)
#Ecto.Changeset<action: nil, changes: %{name: nil, rating: -1.25},
 errors: [rating: {"must be greater than or equal to %{number}",
   [validation: :number, number: 0]},
  name: {"can't be blank", [validation: :required]},
  slug: {"can't be blank", [validation: :required]}],
 data: #PhoenixApp.Project<>, valid?: false>

iex(4)> PhoenixApp.Repo.insert(changeset)
{:error,
 #Ecto.Changeset<action: :insert, changes: %{name: nil, rating: -1.25},
  errors: [rating: {"must be greater than or equal to %{number}",
    [validation: :number, number: 0]},
   name: {"can't be blank", [validation: :required]},
   slug: {"can't be blank", [validation: :required]}],
  data: #PhoenixApp.Project<>, valid?: false>}

As you can see, we apply validations to our changeset at each step and the validation functions return a new changeset with the validations applied. In the example, we attempted to set the name to "" and never set a value for slug. Both these failures were caught by the first validate_required application. We also added a check to ensure the rating was numeric and greater than 0.

You can see that the errors list includes these three errors along with metadata about the type of validation that failed and the validation options used. When we try to save the changeset, we receive an error response including the invalid changeset.

If you need to define your own custom validation in Ecto, you have a few options. If you just need to do a simple field level validation, you can use =validate_change. This function accepts a changeset, a field, and a validation function. If the validation function returns an empty list the validation passes, if a list of errors is returned they are included in the changeset’s errors and it is marked as invalid.

If you need to implement a validation that does more than single field validation, you can simply implement your own validate_* function that takes a changeset as the first argument and returns a transformed changeset. If we want to validate that the Project slug contains the name ignoring case, we can write our own validate_slug_contains_name function. If necessary, we can use additional function arguments to modify the behavior of our custom validations.

Let’s look at validate_slug_contains_name in the PhoenixApp.Project module, and then we can see how it behaves in iex. We’re using two additional function from the Ecto.Changeset module, get_field and add_error.

The get_field function tries to fetch the field value first from changes, then from data, and finally from any default values in the schema definition. The add_error function makes the changeset invalid, adds an entry to the changeset errors list, and returns the transformed changeset. When the validation passes we just return the original changeset.

def validate_slug_contains_name(changeset) do
  slug = get_field(changeset, :slug)
  name = get_field(changeset, :name)
  if Regex.match?(~r[#{name}]i, slug) do
    changeset
  else
    add_error(changeset, :slug, "must contain '#{name}'")
  end
end

Our first example passes the validation and returns a valid changeset. If we use a slug that doesn’t contain the name, the changeset is marked invalid and has appropriate errors.

iex(1)> changeset = Ecto.Changeset.cast(%PhoenixApp.Project{slug: "project-101"}, %{"name" => "Project"}, [:name])
#Ecto.Changeset<action: nil, changes: %{name: "Project"}, errors: [],
 data: #PhoenixApp.Project<>, valid?: true>

iex(2)> changeset = PhoenixApp.Project.validate_slug_contains_name(changeset)
#Ecto.Changeset<action: nil, changes: %{name: "Project"}, errors: [],
 data: #PhoenixApp.Project<>, valid?: true>

iex(3)> changeset = Ecto.Changeset.cast(%PhoenixApp.Project{slug: "proj-101"}, %{"name" => "Project"}, [:name])
#Ecto.Changeset<action: nil, changes: %{name: "Project"}, errors: [],
 data: #PhoenixApp.Project<>, valid?: true>

iex(4)> changeset = PhoenixApp.Project.validate_slug_contains_name(changeset)
#Ecto.Changeset<action: nil, changes: %{name: "Project"},
 errors: [slug: {"must contain 'Project'", []}], data: #PhoenixApp.Project<>,
 valid?: false>

Enforcing Constraints

So far filtering and casting input params and validating field values feels very similar to how ActiveRecord handles things. When we look at handling database constraint violations, however, Ecto behaves quite differently from ActiveRecord.

Since Rails added support for declaring constraints in migrations, it is now common for Rails generated databases to have foreign key and uniqueness constraints. We will see how Ecto handles these constraints consistently with validations, and how this is nicer than the way ActiveRecord handles them.

Ecto changesets use *_constraint =functions to enforce rules that can only be checked using a round trip to the database. There is a function for every constraint type supported by PostgreSQL except for not null constraints. The logic behind this is that not null is something that can be validated without requiring a database round trip. One thing to note is that constraint checks are skipped if a changeset is already invalid to avoid the database round trip.

Let’s take a look at a unique_constraint example. We’ll start by loading the first Project, wrapping it in a changeset, setting the slug to "project", and finally using Repo.update to save the new slug.

$ iex -S mix
iex(1)> project = PhoenixApp.Repo.get(PhoenixApp.Project, 1)
SELECT p0."id", p0."name", p0."slug", p0."created_at", p0."updated_at" FROM "projects" AS p0 WHERE (p0."id" = $1) [1]
%PhoenixApp.Project{ . . . }

iex(2)> changeset = Ecto.Changeset.change(project)
#Ecto.Changeset<action: nil, changes: %{}, errors: [],
 data: #PhoenixApp.Project<>, valid?: true>

iex(3)> changeset = Ecto.Changeset.put_change(changeset, :slug, "project")
#Ecto.Changeset<action: nil, changes: %{slug: "project"}, errors: [],
 data: #PhoenixApp.Project<>, valid?: true>

iex(4)> PhoenixApp.Repo.update(changeset)
begin []
UPDATE "projects" SET "slug" = $1, "updated_at" = $2 WHERE "id" = $3 ["project", {{2016, 11, 4}, {4, 52, 51, 916715}}, 1]
commit []
{:ok,
 %PhoenixApp.Project{ . . . }}

If we try to save another Project with the same slug and don’t handle the uniqueness constraint, the adapter will raise a constraint error identifying the violation. This is how ActiveRecord treats all constraint violations, catching these errors and handling them is your responsibility in Rails.

$ iex -S mix
iex(1)> changeset = Ecto.Changeset.change(%PhoenixApp.Project{name: "Other Project", slug: "project"})
#Ecto.Changeset<action: nil, changes: %{}, errors: [],
 data: #PhoenixApp.Project<>, valid?: true>

iex(2)> PhoenixApp.Repo.insert(changeset)
begin []
INSERT INTO "projects" ("name","slug","created_at","updated_at") VALUES ($1,$2,$3,$4) RETURNING "id" ["Other Project", "project", {{2016, 11, 4}, {5, 3, 46, 127464}}, {{2016, 11, 4}, {5, 3, 46, 137523}}]
rollback []
** (Ecto.ConstraintError) constraint error when attempting to insert struct:

    * unique: index_projects_on_slug

If you would like to convert this constraint into an error, please
call unique_constraint/3 in your changeset and define the proper
constraint name. The changeset has not defined any constraint.

             (ecto) lib/ecto/repo/schema.ex:483: anonymous fn/4 in Ecto.Repo.Schema.constraints_to_errors/3
           (elixir) lib/enum.ex:1184: Enum."-map/2-lists^map/1-0-"/2
             (ecto) lib/ecto/repo/schema.ex:469: Ecto.Repo.Schema.constraints_to_errors/3
             (ecto) lib/ecto/repo/schema.ex:206: anonymous fn/11 in Ecto.Repo.Schema.do_insert/4
             (ecto) lib/ecto/repo/schema.ex:674: anonymous fn/3 in Ecto.Repo.Schema.wrap_in_transaction/6
             (ecto) lib/ecto/adapters/sql.ex:494: anonymous fn/3 in Ecto.Adapters.SQL.do_transaction/3
    (db_connection) lib/db_connection.ex:1063: DBConnection.transaction_run/4
    (db_connection) lib/db_connection.ex:987: DBConnection.run_begin/3
    (db_connection) lib/db_connection.ex:667: DBConnection.transaction/3

The exception message tells us what to do, so lets apply a unique_constraint to the changeset. Passing the resulting changeset to Repo.insert returns an error response changeset with valid? set to false and errors describing the uniqueness failure. The nice thing is that constraint violations are treated in the same way as normal validation failures, so you are not required to catch constraint exceptions yourself.

$ iex -S mix
iex(1)> changeset = Ecto.Changeset.change(%PhoenixApp.Project{name: "Other Project", slug: "project"})
#Ecto.Changeset<action: nil, changes: %{}, errors: [],
 data: #PhoenixApp.Project<>, valid?: true>

iex(2)> changeset = Ecto.Changeset.unique_constraint(changeset, :slug, name: "index_projects_on_slug")
#Ecto.Changeset<action: nil, changes: %{}, errors: [],
 data: #PhoenixApp.Project<>, valid?: true>

iex(3)> PhoenixApp.Repo.insert(changeset)
begin []
INSERT INTO "projects" ("name","slug","created_at","updated_at") VALUES ($1,$2,$3,$4) RETURNING "id" ["Other Project", "project", {{2016, 11, 6}, {17, 11, 10, 14220}}, {{2016, 11, 6}, {17, 11, 10, 27275}}]
rollback []
{:error,
 #Ecto.Changeset<action: :insert,
  changes: %{name: "Other Project", slug: "project"},
  errors: [slug: {"has already been taken", []}], data: #PhoenixApp.Project<>,
  valid?: false>}

Let’s see another example that enforces a foreign key constraint. In this example we’ll try to insert a List that refers to a non-existent project_id. Again, if we attempt a Repo.insert the adapter raises an informative constraint error.

$ iex -S mix
iex(1)> changeset = Ecto.Changeset.change(%PhoenixApp.List{project_id: 9999})
#Ecto.Changeset<action: nil, changes: %{}, errors: [], data: #PhoenixApp.List<>,
 valid?: true>

iex(2)> PhoenixApp.Repo.insert(changeset)
begin []
INSERT INTO "lists" ("project_id","created_at","updated_at") VALUES ($1,$2,$3) RETURNING "id" [9999, {{2016, 11, 4}, {5, 22, 20, 935013}}, {{2016, 11, 4}, {5, 22, 20, 941750}}]
rollback []
** (Ecto.ConstraintError) constraint error when attempting to insert struct:

    * foreign_key: fk_rails_67f2498cc9

If you would like to convert this constraint into an error, please
call foreign_key_constraint/3 in your changeset and define the proper
constraint name. The changeset has not defined any constraint.

             (ecto) lib/ecto/repo/schema.ex:483: anonymous fn/4 in Ecto.Repo.Schema.constraints_to_errors/3
           (elixir) lib/enum.ex:1184: Enum."-map/2-lists^map/1-0-"/2
             (ecto) lib/ecto/repo/schema.ex:469: Ecto.Repo.Schema.constraints_to_errors/3
             (ecto) lib/ecto/repo/schema.ex:206: anonymous fn/11 in Ecto.Repo.Schema.do_insert/4
             (ecto) lib/ecto/repo/schema.ex:674: anonymous fn/3 in Ecto.Repo.Schema.wrap_in_transaction/6
             (ecto) lib/ecto/adapters/sql.ex:494: anonymous fn/3 in Ecto.Adapters.SQL.do_transaction/3
    (db_connection) lib/db_connection.ex:1063: DBConnection.transaction_run/4
    (db_connection) lib/db_connection.ex:987: DBConnection.run_begin/3
    (db_connection) lib/db_connection.ex:667: DBConnection.transaction/3

If we want to handle this constraint error, we follow the suggestion and add a foreign_key_constraint. The foreign key names generated by Rails look strange, but they are named consistently using a hash based on table and column names. Because the constraint name is consistent across databases, it’s easy to apply the foreign key constraint to our changeset. The resulting error response changeset is marked invalid and has appropriate errors.

$ iex -S mix
iex(1)> changeset = Ecto.Changeset.change(%PhoenixApp.List{project_id: 9999})
#Ecto.Changeset<action: nil, changes: %{}, errors: [], data: #PhoenixApp.List<>,
 valid?: true>

iex(2)> changeset = Ecto.Changeset.foreign_key_constraint(changeset, :project_id, name: "fk_rails_67f2498cc9")
#Ecto.Changeset<action: nil, changes: %{}, errors: [], data: #PhoenixApp.List<>,
 valid?: true>

iex(3)> PhoenixApp.Repo.insert(changeset)
begin []
INSERT INTO "lists" ("project_id","created_at","updated_at") VALUES ($1,$2,$3) RETURNING "id" [9999, {{2016, 11, 6}, {17, 34, 42, 123422}}, {{2016, 11, 6}, {17, 34, 42, 133599}}]
rollback []
{:error,
 #Ecto.Changeset<action: :insert, changes: %{project_id: 9999},
  errors: [project_id: {"does not exist", []}], data: #PhoenixApp.List<>,
  valid?: false>}

Piping Changeset Operations Together

We’ve looked at the features of Ecto.Changeset that help us write valid and consistent records to a database. The code examples we’ve looked at in iex are not typical of how changesets are normally constructed, however. The examples apply Ecto.Changeset functions one at a time to help illustrate how the changeset transforms as each new function is applied. In real code you aren’t interested in these intermediate forms of the changeset, so they would normally be composed using Elixir’s pipe operator.

Our first validation example built the initial changeset using cast/3, and then applied validate_required and validate_number by passing a transformed changeset as the first argument.

changeset = Ecto.Changeset.cast(%PhoenixApp.Project{name: "Project"}, %{"name" => "", "rating" => "-1.25"}, [:name, :rating])
changeset = Ecto.Changeset.validate_required(changeset, [:name, :slug])
changeset = Ecto.Changeset.validate_number(changeset, :rating, greater_than_or_equal_to: 0)

The functions in Ecto.Changeset that take a changeset as their first argument are designed to be chained together with the pipe operator. Using this technique the same changeset can be written in a more readable form.

changeset =
  %PhoenixApp.Project{name: "Project"}
  |> Ecto.Changeset.cast(%{"name" => "", "rating" => "-1.25"}, [:name, :rating])
  |> Ecto.Changeset.validate_required([:name, :slug])
  |> Ecto.Changeset.validate_number(:rating, greater_than_or_equal_to: 0)

If the module you’re working with aliases PhoenixApp.Project and imports Ecto.Changeset the piped version looks even better.

changeset =
  %Project{name: "Project"}
  |> cast(%{"name" => "", "rating" => "-1.25"}, [:name, :rating])
  |> validate_required([:name, :slug])
  |> validate_number(:rating, greater_than_or_equal_to: 0)

Next Steps

Between this article and the previous one on Ecto schemas, we’ve looked at how to read records and associations from a Rails generated database and how to scrub, cast, and validate inputs when writing records to the database.

We still have a lot of missing pieces in order to replicate the behavior of ActiveRecord models. We need before_* callback behaviors to generate slugs or encrypt passwords. We need after_* callbacks that send emails or call external APIs.

We still need to look at Ecto’s query interface and see how to compose queries. We haven’t yet looked at Ecto migrations and how they relate to ActiveRecord migrations.

There’s a lot to cover, but our familiarity with ActiveRecord allows us to draw parallels and emphasize the differences between it and Ecto. Soon you’ll be as good at thinking in Ecto as you are at working with ActiveRecord.

''