Add support for many nested objects in SaveOperations

[jwoertink]

Original Issue: #43

I'm writing a new issue because the original one was created a long time ago and makes mention of a lot of things that no longer exist, though the sentiment is still the same.

We need a way to support creating and updating many nested records for a parent object.

Given a JSON payload that looks like this:

{
  "menu": {
    "title": "Main Menu",
    "active": true,
  }
  "items": [
    {
      "title": "First item",
      "active": true,
      "position": 0
    },
    {
      "title": "Second item",
      "active": false,
      "position": 1
    }
  ]
}

I should be able to use these params to create a new menu object as well as 2 new item objects. This must be able to support the ability to decide which of the nested keys are permitted and which are not similar to using permit_columns which helps to prevent updating unpermitted columns.

This is a rough idea of the intent:

class SaveMenu < Menu::SaveOperation
  permit_columns title, active

  has_many items : SaveItem
end

class SaveItem < Item::SaveOperation
  permit_columns title, active, position
end

# `params` here represents the above json structure
SaveMenu.create!(params)

Under the hood, we can use params.many_nested(:items) here. Ideally this should also update them in a single query. Each nested record could run through the operation to check all validations and run any data transformations that need to happen, then once all operations are valid? and have no errors, then it submits the single query.

Last thing to consider is if an Item requires a menu_id, then the items can't even begin to save until the menu has already been saved and created. This whole thing will need to be inside of a transaction so if the items fail to save, the transaction can be rolledback to remove the menu.

[akadusei]

This should also account for situations where some of the many nested records have to be deleted when the parent operation is updated. In a create operation, the many nested records will always be created. But in an update operation, some new records may be created, some existing ones updated and some deleted.

If you want to capture all many nested errors, it gets trickier. Somehow, many nested errors have to be ordered so that the UI/frontend knows which error belongs to which many nested operation.

This is how I handle it currently, if it helps:

# ->> src/operations/mixins/types.cr

alias OperationErrors = Hash(Symbol, Array(String))

# ->> src/operations/mixins/needs_items,cr

module Mixins::NeedsItems
  macro included
    getter save_items do
      Array(Union(
        {{ T }}Item::SaveOperation,
        {{ T }}Item::DeleteOperation
      )).new(
        keyed_items.size,
        {{ T }}Item::SaveOperation.new
      )
    end

    needs items : Array(Hash(String, String))

    def items_to_create
      items_to_save.reject(&.["id"]?)
    end

    def items_to_update
      items_to_save.select(&.["id"]?)
    end

    def items_to_delete
      keyed_items - items_to_save
    end

    # If `title` is explicitly set to an empty string, we take that to mean we should delete
    # the item. Everything else is saved (created or updated).
    #
    # Alternatively, we could check a `delete` key is set to `true` in the params.
    def items_to_save
      keyed_items.reject { |item| item["title"]?.try(&.empty?) }
    end

    def many_nested_errors : Hash(Symbol, Array(OperationErrors))
      {% if @type.methods.any?(&.name.== :many_nested_errors.id) %}
        errors = previous_def
      {% elsif @type.ancestors.any? do |ancestor|
        ancestor.methods.any?(&.name.== :many_nested_errors.id)
      end %}
        errors = super
      {% else %}
        errors = Hash(Symbol, Array(OperationErrors)).new
      {% end %}

      errors.tap do |_errors|
        # There may be other many nested operations in the parent operation, so
        # we save this particular one under its own hash key `:save_items`.
        _errors[:save_items] = save_items.map(&.errors)
      end
    end

    # Adds the order in which the items were received from
    # the request to each item.
    #
    # This should help us send back errors in the same order, so the
    # frontend can know which errors belong to which item.
    private getter keyed_items : Array(Hash(String, String)) do
      items.map_with_index do |item, i|
        item.tap  { |_item| _item["key"] = i.to_s }
      end
    end
  end
end

# ->> src/operations/mixins/create_menu_items.cr

module Mixins::CreateMenuItems
  macro included
    after_save :create_items

    include Mixins::NeedsItems
    # include Mixins::ValidateItems

    private def create_items(menu : Menu)
      create_menu_items(menu)

      rollback_failed_create_menu_items
    end

    private def create_menu_items(menu)
      items_to_create.each do |item|
        save_items[item["key"].to_i] = CreateMenuItem.new(
          Avram::Params.new(item),
          menu_id: menu.id
        )

        save_items[item["key"].to_i].as(MenuItem::SaveOperation).save
      end
    end

    private def rollback_failed_create_menu_items
      items_to_create.each do |item|
        write_database.rollback unless save_items[item["key"].to_i]
          .asMenuItem::SaveOperation)
          .saved?
      end
    end
  end
end

# ->> src/operations/mixins/update_menu_items.cr

module Mixins::UpdateMenuItems
  macro included
    after_save :update_items

    include Mixins::NeedsItems
    # include Mixins::ValidateItems

    private def update_items(menu : Menu)
      delete_menu_items(menu)
      update_menu_items(menu)
      create_menu_items(menu)

      rollback_failed_delete_menu_items
      rollback_failed_update_menu_items
      rollback_failed_create_menu_items
    end

    private def delete_menu_items(menu)
      items_to_delete.each do |item|
        menu_item_from_hash(item, menu).try do |menu_item|
          save_items[item["key"].to_i] = DeleteMenuItem.new(
            menu_item,
            Avram::Params.new(item),
            menu_id: menu.id
          )

          save_items[item["key"].to_i]
            .as(MenuItem::DeleteOperation)
            .delete
        end
      end
    end

    private def update_menu_items(menu)
      items_to_update.each do |item|
        menu_item_from_hash(item, menu).try do |menu_item|
          save_items[item["key"].to_i] = UpdateMenuItem.new(
            menu_item,
            Avram::Params.new(item),
            menu_id: menu.id
          )

          save_items[item["key"].to_i]
            .as(MenuItem::SaveOperation)
            .save
        end
      end
    end

    private def create_menu_items(menu)
      items_to_create.each do |item|
        save_items[item["key"].to_i] = CreateMenuItem.new(
          Avram::Params.new(item),
          menu_id: menu.id
        )

        save_items[item["key"].to_i]
          .as(MenuItem::SaveOperation)
          .save
      end
    end

    private def rollback_failed_delete_menu_items
      items_to_delete.each do |item|
        # If no record was found, this would still be a SaveOperation,
        # hence the nilable `#as?` call
        save_items[item["key"].to_i].as?(MenuItem::DeleteOperation)
          .try do |operation|

          write_database.rollback unless operation.deleted?
        end
      end
    end

    private def rollback_failed_update_menu_items
      items_to_update.each do |item|
        write_database.rollback unless save_items[item["key"].to_i]
          .as(MenuItem::SaveOperation)
          .saved?
      end
    end

    private def rollback_failed_create_menu_items
      items_to_create.each do |item|
        write_database.rollback unless save_items[item["key"].to_i]
          .as(MenuItem::SaveOperation)
          .saved?
      end
    end

    private def menu_item_from_hash(hash, menu)
      hash["id"]?.try do |id|
        MenuItemQuery.new.id(id).menu_id(menu.id).first?
      end
    end
  end
end

# ->> src/operations/create_menu.cr

class CreateMenu < Menu::SaveOperation
  # ...

  include Mixins::CreateMenuItems
  #include Mixins::ValidateMenu

  # ...
end

# ->> src/operations/update_menu.cr

class UpdateMenu < Menu::SaveOperation
  # ...

  include Mixins::UpdateMenuItems
  #include Mixins::ValidateMenu

  # ...
end

# ->> src/actions/menus/create.cr

class Menus::Create < PrivateApi
  post "/menus" do
    run_operation
  end

  def run_operation
    CreateMenu.create(
      params,
      items: params.many_nested?(:items),
    ) do |operation, menu|
     # ...
     # You may call `operation.many_nested_errors` here
    end
  end
end

# ->> src/actions/menus/update.cr

class Menus::Update < PrivateApi
  patch "/menus/:menu_id" do
    run_operation
  end

  def run_operation
    UpdateMenu.update(
      menu,
      params,
      items: params.many_nested?(:items),
    ) do |operation, updated_menu|
     # ...
     # You may call `operation.many_nested_errors` here
    end
  end
  
  getter menu : Menu do
    MenuQuery.new.preload_items.find(menu_id)
  end
end