Tiger & Dormouse

Rails fixtures and PostgreSQL referencial integrity

Mon, 15 Feb 2016 18:00:00 +0000

Recently, on a Rails project, I had the need to create a disposable Heroku app to show features in progress and get quick feedback from the stakeholders. Not a long time ago, I’d read about the Heroku Review App functionality and I though this would be a good opportunity to try it out.

Following the Heroku documentation I configured the Review apps for the project and, in no time, I had the ability to spin up a disposable Heroku app per pull request. The next step was populating these apps with demo data.

In this project we have a QA/UAT Heroku app with a database containing demo data. My first thought was cloning the UAT db in the review boxes, but Heroku is clear about this approach: “Copying full database contents from parent to Review Apps (similar to heroku fork) is not currently supported”. Instead, they recommend seeding the database with a post deploy script.

As this is a Rails project, I created some fixtures with demo data and loaded them using the command rake db:fixtures:load. Fixtures loaded fine in my local machine but in the review boxes I was getting a referential integrity violation error. What was happening?

We use PostgreSQL as database and PostgreSQL does integrity checks, like foreign key constraints. Rails fixtures are loaded in alphabetical order, so it’s common to find cases where a table that is being created has a reference to another table which hasn’t been created yet.

If we take a look at ActiveRecord code base we can see how it calls the method disable_referential_integrity from create_fixtures method. This method contains a warning which says “Rails needs superuser privileges to disable referential integrity”. That was the reason why it worked in my machine, I had super user privileges for the local database.

As expected, in Heroku it’s not possible to change PostreSQL database privileges, so I had to search for a different approach. After a bit of researching about PostgreSQL and referencial integrity, I found a post which shows how to drop and recreate PostgreSQL constraints.

Using the scripts in the mentioned post I created a new Rake task called populate_sample_data. This task resets the database, drops the PostgreSQL constraints, loads the fixtures and then recreates the previous constraints.

Finally, I added the populate_sample_data rake task to the Review Apps post deploy script and everything started to work as expected.

Rails contributors are working on a solution for this problem, but for now it’s a work in progress and it doesn’t cover PostgreSQL versions older than 9.4.

Hopefully this post will help other people facing the same problem, or someone can come up with a better solution.

Persisting Value Objects in Rails - Denormalised database

Sun, 10 Jan 2016 18:00:00 +0000

In my previous post, I talked about some of the options we have to persist value objects in a relational database. In this one, I’m going to show an example using Rails and the denormalised database approach.

Let’s say we want to model a Client entity which has an Address. In the Entities and Value Objects post, I already used an Address value object as an example, let’s bring it back

  class Address
    attr_reader :street, :postcode
    def initialize(street, postcode)
      @street = street
      @postcode = postcode
      freeze
    end
    def ==(other)
      other.is_a?(Address) && street == other.street && postcode == other.postcode
    end
    def hash
      [street, postcode].hash
    end
    alias_method :eql?, :==
  end

Following the denormalised database approach, we’ll put together in the same database table the Client and the Address attributes. In Rails this would result in a migration similar to this

  class CreateClient < ActiveRecord::Migration
    def change
      t.string :name
      t.string :address_street
      t.string :address_postcode
    end
  end

In Rails we can say that ActiveRecord models are entities where the identity is provided by the ActiveRecord id. If we model the Client as an entity, the Client class could be something like

  class Client < ActiveRecord::Base
    def address
      @address ||= Address.new(address_street, address_postcode)
    end

    def address=(address)
      self[:address_street] = address.street
      self[:address_postcode] = address.postcode

      @address = address
    end

    private
    attr_writter :address_street, :address_postcode
  end

Since one of the value objects’ main characteristics is immutability, in order to prevent unwanted modification attempts from the Client, this implementation restricts the access of Rails attribute writers to private.

It’s also worth mentioning that this denormalised approach can be handy when you are trying to extract behaviour from a convoluted Rails model. Imagine a Client model in which the Address attributes are already part of it, extracting a value object is trivial since there is no need to touch the database scheme. In fact, this refactor is a common one in articles that talk about refactoring fat ActiveRecord models.

And that’s it, we have a Client entity that has a value object Address where all its attributes (Client and Address attributes) are persisted in the same database table.

References:

Persisting Value Objects in a Relational Database

Sun, 13 Dec 2015 11:30:00 +0000

How a VO persists has nothing to do with Entity/VO semantics, but it’s something that has to be considered since business data frequently has to be persistent. When you want to persist a value object in a relational database, there are tree main options. As always, there is not a correct one and all of them have pros and cons:

Use a normalised database approach
- How:
  - Use the auto generated id to identify the table but hiding it in the VO objects.
- Pros:
  - Efficient database access. All the benefits of normalised databases: easily query, merge and update data.
  - Provides an easy way to solve one to many relations between VO objects.
- Cons:
  - Difficult to hide/manage the id at class/object level.
  - Disconnection of the object from its database id. Easy to end with more than one database row for the same object.
  - Not easy to clean a destroyed VO from the database. Can lead to stale data in the DB.
  - Slow global update of entities with value objects in the DB.
Use a denormalised database
- How:
  - Since VO objects don’t make sense without their Entity, at least at DB level, we can merge all the attributes in the same table.
- Pros
  - True value objects without database ids.
  - This approach makes it easy to move out a value object from an entity. Only the objects and the ORM mapping would need to change, there is no need to change the database scheme.
- Cons
  - Can be very tricky to implement properly with distributed systems.
  - Difficult to implement when the relation is not one to one.
Serialise representation next to the entities
- How:
  - Store the object graph in a column. This can be accomplished using serialisation.
- Pros
  - Efficient database loading.
- Cons
  - Can be very tricky to implement properly with distributed systems.
  - Deserialisation.
  - Query limitations.

This is my overview/index on persistence and value objects. I’ll try to dig more and illustrate with some examples each of these alternatives in future posts.

Related posts:

Entities and Value Objects

Thu, 22 Oct 2015 20:30:00 +0000

Entities and value objects are key concepts when talking about Domain Driven Design.

Entities are objects represented by its identity. For example, in our system we can have a User object with multiple attributes like name, password, etc. This object, regardless of its attributes, is going to be represented by an id. This means we can change any of the object’s attributes (name, password) and it will remain the same object. Example:

class User
  attr_accessor :name, :password
  def initialize(name, password)
    @name = name
    @password = password
  end
end
user = User.new('pablo', 'password')
# <User:0x007fc0199c5028 @name="pablo", @password="password">
user.name = 'peter'
# <User:0x007fc0199c5028 @name="peter", @password="password">
second_user = User.new('peter', 'password')
# <User:0x007fc01a171c90 @name="peter", @password="password">
second_user == user # false

On the other hand we have value objects. These objects represent something that measures, quantifies or describes an entity, and they don’t have an identity. Value objects are described by its attributes. Following the previous example, a User can have an Address where the address is a characteristic of this user. Address is characterised by its street and postal code attributes. If we change any of these attributes, we get a different Address.

Since any change in its attributes turn value objects into different objects, value objects should be treated as immutable objects. When you change the Address of a User, you don’t change the attributes of the Address object, instead you create a new Address and assign it to the User.

As we can have multiple objects representing the same Address, a common practice when working with value objects is to override the equality and hash methods.

Value object example:

class Address
  attr_reader :street, :postcode
  def initialize(street, postcode)
    @street = street
    @postcode = postcode
    freeze
  end
  def ==(other)
    other.is_a?(Address) && street == other.street && postcode == other.postcode
  end
  def hash
    [street, postcode].hash
  end
  alias_method :eql?, :==
end
address = Address.new('my address', 'NW18NH')
#<Address:0x007fc35205aba0 @street="my address", @post_code="NW18NH">
second_address = Address.new('second address', 'NW28JH')
#<Address:0x007fc352051578 @street="second address", @post_code="NW28JH">
copy_address = Address.new('my address', 'NW18NH')
#<Address:0x007fc352049008 @street="my address", @post_code="NW18NH">
address == second_address # false
address == copy_address # true

This was my introduction to entities and value objects. I’m currently reviewing my knowledge about Domain Driven Design and I mean to write a series of posts about it. If you want to read more about value objects and entities, here are some resources:

ROSSConf Berlin

Wed, 30 Sep 2015 20:00:00 +0000

Last weekend, I attended the ROSSConf in Berlin. This is not a classic conference where speakers talk about a subject and you listen to them, but a gathering of Ruby open source maintainers with Ruby developers who are interested in collaborating in their projects. The conference was divided into two parts: firstly the mantainers gave an overview of their projects and the areas where they wanted help; the second part was a hands-on session where attendees could choose one of the projects and work on it.

One thing that I really liked is the fact that the mantainers were right there, available all the time. It was easy to have a chat with them, get the overall idea of their projects, solve any doubts quickly and, in general, get that kind of context that it’s difficult to grasp from emails, irc channels and similar. And, of course, it’s also nice to see and know the person behind the project.

The project I chose was ROM which, to be honest, I’d decided to join before the conference. I’d read some blog posts and watched some talks by his mantainer, Piotr Solnica, before RossConf, and I liked and agreed with some of the ideas he showed on them.

So, what did I manage to do in this project? Well, not as much as I would have liked. I didn’t spend enough time before the conference reviewing and understanding the code base, so I dedicated most of the event to read the documentation. Despite the lack of context, I managed to do a small pull request and, more importantly, I saw mutation tests running for the first time in my life. I still don’t know much about this technique, but thanks to this conference I think I’m going to dig a bit more into it.

What could be improved from my point of view? If I have to complain about anything, it would be the lack of time to write code. I know that we can now continue to contribute from home, but I’d have liked to have used the motivation you get when you are surrounded by other highly involved developers to do a bit more. It would have been nice to have a second full day of coding.

Aside from looking at the ROM project, it was really nice to chat with other Ruby developers about the Berlin scene and the state of the art in general.

Thanks a lot to the RossConf organisers and sponsors, it was a great event!

Remote work

Mon, 31 Aug 2015 10:00:00 +0000

In my current company, upmysport, the dev team is distributed half in London (UK) and half in Chamonix (France). This is my first experience with a fully remote work. I’d worked from home some days before, but I wouldn’t consider it to amount to a remote job. These are my thoughts after nine months working remotely.

Lifestyle

My lifestyle has experienced a 360 degrees change. Even if I couldn’t complain of my previous life in London, Chamonix fits better with my interests and I can practise the sports I love. In addition, working remotely makes it easier for me to travel to other countries to visit family and friends while keeping my workflow going.

The lifestyle factor can seem to benefit only the employee, but I think that thanks to my new way of living my levels of stress and burnout have droped massively. I now feel fresher, happier and more focused at work, so my productivity is higher than before.

Team interaction

Chats and videoconference are not as good as a face to face conversation or a coffee together, which usually means less brainstorming and less informal feedback. On the other hand, the lack of personal communication makes the time we spend face to face more special, with everybody involved trying to get the most of it.

One thing I’ve noticed is the reduction of interruptions. When you are in an open-plan office, quite often people just come to your desk with a comment on a football match, last night’s TV show and so on. In a remote environment this kind of interruption is easier to filter, so you can “plan” your procrastination time.

The lack of human interaction also makes it more difficult to create a company culture and a group of work-friends, but this effect can be mitigated with a bit of discipline and company effort. For example, among other practices we have company events from time to time to gather toguether and get to know each other better. We are also always available via chat or videoconference, and we share the progress inside teams and among them then quite often. All these small things keep the team united.

Code practices

Apart from some face to face pairing, I’ve not found any major differences between my previous onsite jobs and my current remote code practices. We (developers) already work with “ready to remote” tools: remote repositories, pull requests, CI systems, bug trackers, online boards, management tools, and a long etc.

To sum up, I believe that remote work is working quite well for upmysport, and I’m really enjoying it. Of course their context makes it easy to work remotely: it’s a startup with a small number of employees, there’s not a big time difference between both offices and the hierarchy is quite flat. It would be interesting to hear other people’s experience with remote work in big enterprises.

Rails polymorphic associations

Mon, 27 Jul 2015 20:12:00 +0000

Working on a Rails project, I faced a problem similar to this: Given an ActiveRecordModel called Booking which can have multiple bills:

class Bill < ActiveRecord::Base
  belongs_to :booking
end

class Booking < ActiveRecord::Base
  has_many :bills
end

The amount of the Bill comes from the Booking’s amount:

class Bill < ActiveRecord::Base
  belongs_to :booking
  delegate :amount, to: :booking
end

Now we need a new model, Subscription, and a Subscription can also have a Bill so:

class Bill < ActiveRecord::Base
  belongs_to :subscription
  belongs_to :booking
  delegate :amount, to: :booking
end

class Subscription < ActiveRecord::Base
  has_many :bills
end

And here is the problem, it isn’t possible to delegate to both Booking and Subscription.

Ok, let’s stop thinking about Rails and try to solve this with OO and plain Ruby. In plain Ruby, the Bill class would probably look like this:

class Bill
  def initializer(booking)
    @booking = booking
  end

  def amount
    @booking.amount
  end
end

Then, when the Subscription entity enters the game, the only thing we need to take care of is the fact that Booking and Subscription have a common interface with one method amount. This can be written as follows:

class Bill
  def initializer(billable)
    @billable = billable
  end

  def amount
    @billable.amount
  end
end

Now, the Bill class doesn’t have to mind about Bookings or Subscriptions anymore.

Good, back to Rails. How do I model this in Rails? I would have to change the previous Rails code to something like this:

class Bill < ActiveRecord::Base
  belongs_to :billable
end

The answer is polymorphic associations. This type of association allows Bill to belong_to an interface. The only thing we have to do is mark this relation as polymorphic, and the Booking and Subscription models as billable

class Bill < ActiveRecord::Base
  belongs_to :billable, polymorphic: true
end

class Booking < ActiveRecord::Base
  has_many :bills, as: :billable
end

class Subscription < ActiveRecord::Base
  has_many :bills, as: :billable
end

Maybe in a future post I dig into how Rails maps these models to the database, but that’s it for today. Here I leave other posts about Rails polymorphic associations that I found interesting:

Heroku Rake tasks II

Sat, 04 Jul 2015 14:41:00 +0000

In the previous post, I wrote a couple of Rake tasks to manage Heroku db backups. Once I had this tasks in place I realised that it would be handy to have another one to load them into my local Postgres database, so I wrote this in my database.rake file:

namespace :db do
  desc 'Load remote database backup to local database'
  task :load_backup do
    fail 'latest.dump does not exist' unless File.exist?('latest.dump')
    Bundler.with_clean_env do
      `pg_restore --verbose --clean --no-acl --no-owner \
      -h localhost -U postgres_dev -d development latest.dump`
    end
    puts 'Load completed.'
  end
end

This worked fine, but if you take a look at the Rake documentation it’s easy to spot that Rake uses a dependency based style of computation rather than the usual imperative style.This means you can add dependent tasks and Rake will evaluate and run them first. Thanks to this Rake feature I rewrote my task to look like this:

namespace :db do
  desc 'Load remote database backup to local database'
  task :load_backup => 'heroku:download_backup' do
    Bundler.with_clean_env do
      `pg_restore --verbose --clean --no-acl --no-owner \
      -h localhost -U postgres_dev -d development latest.dump`
    end
    puts 'Load completed.'
  end
end

With these changes, the task heroku:download_backup, which was created in the previous post, runs before the load_backup task, so I can ensure that the lastest.dump file is going to be in place. As a side effect I don’t need to check for the dump file existence anymore.

It looks good, but after using these tasks a couple of times I felt the inconvenience of having to wait for the db dump to download. The solution to this problem was using the Rake file taks. It’s like a normal Rake task, but it checks for the file timestamps. This is how I rewrote my Heroku Rake task

desc 'Download database backup'
task :download_backup, [:app] =>  latest.dump

file 'latest.dump' do |task, args|
  args.with_defaults(app: DEFAULT_APP)
  Bundler.with_clean_env do
    url = `heroku pg:backups public-url --app #{args[:app]}`
    `curl -o latest.dump "#{url}"`
  end
  puts $?.exitstatus == 0 ? 'Database downloaded.' : 'Download failed.'
  end
end

The Rake file task only runs if the file doesn’t exist, so I can now run rake db:load_backup and it’ll only download the database dump file the first time.

It would also be nice if the heroku:download_backup ran when there is a new backup available in Heroku, but I’ll leave that for the future since this post is long enough.

Heroku Rake tasks

Sun, 07 Jun 2015 14:41:00 +0000

In the project I’m currently working on, we use Heroku as infrasture provider. Heroku has its own command line operations to execute tasks like db backups, managing addons, etc.

Last March, Heroku deprecated its PG Backups addon, so the CLI syntax changed. The project I’m working on is a Rails project, so by default we already use Rake to run tasks like database migrations. Thus, instead of learning the new Heroku syntax, I’ve decide to write a couple of Rake tasks to wrap the new Heroku commands.

namespace :heroku do
  DEFAULT_APP = 'my_app'

  desc 'Create database backup'
  task :backup, :app do |_task, args|
    args.with_defaults(app: DEFAULT_APP)
    Bundler.with_clean_env do
      `heroku pg:backups capture --app #{args[:app]}`
    end

    puts $?.exitstatus == 0 ? 'Backup created.' : 'Backup failed.'
  end

  desc 'Download latest database backup'
  task :download_backup, :app do |_task, args|
    args.with_defaults(app: DEFAULT_APP)
    Bundler.with_clean_env do
      url = `heroku pg:backups public-url --app #{args[:app]}`
      `curl -o latest.dump "#{url}"`
    end

    puts $?.exitstatus == 0 ? 'Database downloaded.' : 'Download failed.'
  end
end

Next time Heroku updates its command line syntax, I’ll only need to update the commands in my Rake tasks and I won’t have to learn the nex syntax.

Hub commands

Sat, 23 May 2015 20:15:00 +0000

During these last weeks, I’ve been trying out some Hub commands and finally decided to include some of them in my day to day workflow. Hub is a command line wrapper for Git that adds some GitHub specific functionality.

To install it

$ brew install hub

To mix the classic Git commands with the new Hub commands, add to the bash profile

eval "$(hub alias -s)"

All the commands available are in the project page, but the ones I’ve incorporated into my workflow are:

git compare
git pull-request -o -b
git ci-status -v

This is the way I use them:

Given that I’m in a feature, bug, etc. branch, when at some point I want to review my changes and compare them with the master branch, I use the command git compare. It launches the default browser with the GitHub diff view, comparing the current branch and master.

When I’m happy with my branch, I push my changes to remote with git push origin my_branch. This action triggers the CI server which runs the test suite and confirms that tests work not only on my machine. While the CI server is running, I use the Hub command git ci-status which returns a string with the current status of the CI server. Sometimes I run the command with the option -v, which also returns the CI server status plus the URL to CI build results.

Once the CI server runs ssuccessfully, I proceed to open a pull request. In order to do that, I use the command git pull-request, usually with the option -o which automatically opens the GitHub pull request page in the default browser. Another option I sometimes use is -b, which allows opening a pull request to a specific branch.

And that’s it. These are the three Hub commands I’ve incoporated to my git workflow.