Safe data migrations in Rails
Afraid to run data migrations in production? Don't be. There are a few easy tricks which can help you feel secure.
… what data migrations?
There are times when data in your application has to be explicitly adjusted. You might have changed the data model or found an error in the data consistency. It happens.
Those changes, except for the smallest of them, shouldn't be performed within
db:migrate tasks, around the deployment process. They may be lengthy and CI is usually not prepared for such operations.
Basically, sometimes we need to run a custom script to change production data. Let's see how we can make it less scary.
Create a folder, for example
db/data_migrations and put every data migration definition inside.
Keep them separated from the application
db/data_migrations folder we've created? It's not auto loaded into your application by default. Each time you want to use it, you have to explicitly load it using the require method.
You will never run it by accident.
Keep it consistent
All data migrations classes should follow the same pattern, so that they are predictable and safe to use.
# db/data_migrations/perform_an_important_change_data_migration.rb module DataMigrations class PerformAnImportantChangeDataMigration def self.call ... end end end
Then, run it in rails console
> require_relative 'db/data_migrations/perform_an_important_change_data_migration' => true > DataMigrations::PerformAnImportantChangeDataMigration.call
or wrap in a rake task
# lib/tasks/data_migration.rake require_relative '../../db/data_migrations/perform_an_important_change_data_migration' namespace :data_migration do desc 'Performs an Important Data Migration Change' task perform_an_important_change: :environment do DataMigrations::PerformAnImportantChangeDataMigration.call end end
Add a safety switch
This is a game changer. When you run a data migration in the production environment you usually cannot be 100% certain what comes out of it. A random invalid record can ruin your day in the middle of the run. Use the following safety mechanism
# db/data_migrations/perform_an_important_change_data_migration module DataMigrations CommitForbidden = Class.new(StandardError) class PerformAnImportantChangeDataMigration def self.call(commit_changes: false) ActiveRecord::Base.transaction do ... raise CommitForbidden unless commit_changes end end end end
The data migration can be performed as a dry run to check if it performs correctly.
commit_changes argument is set to false, it automatically rolls back the transaction block. It allows you to check the performance and the outcome of the migration just as if you used sandbox mode. Once you are ready and certain - add
commit_changes: true and run it for real.
I have run it in a multi-tenant (by schema) setup with ~30000 updates in the run. Worked like a charm. It also allows you to print out every change to the console. You may show it to your client and double-check if it matches expectations.
Write a few tests. There is no migration too small or too big for testing. A test helps you to think in the context of real data and gives a chance to realize that edge cases exist.
Ask somebody to check it
Even if you are at the super-pro-master level, you may still make a mistake. Even if you use continuous deployment / no code review approach, data migrations are tricky as they escape the classic flow. The best tests won't fully help if you don't think of an exception from a rule. Ask a colleague to give it a quick read. It may save you a week of fixing the mess.
Clean after yourself
Data migrations tend to become obsolete quite quickly. Once they are used, you may remove the test and the migration itself. All the files will forever remain in your repository - no need to pollute the codebase.
Let's sum it up!
Data migrations are needed in almost every application. We change schema and we change business approach. It has to be reflected in the data. There are ways to do it in a safe way. Don't rush, use safety mechanisms and... good luck :)