Remove stale resque workers using bash

I run a Rails application that uses Resque to manage background jobs.  It works beautifully.  Unfortunately, sometimes resque workers become stuck while working on a task.  This might happen for a number of reasons, but the end result is that you’re missing a worker and you might not realize it.  

Fortunately, when resque starts a job it forks a child process and places the time it began the job in the process description.  You can tell if your worker is stale by using this timestamp to compare it to the current time.  Epoch Converter is one of a number of websites to help you convert the timestamp manually.

If you’re using god to monitor your processes, resque provides an example to eliminate these “stale” workers.  I don’t use god and instead use monit to monitor my processes.  Resque comes with a monit example too, but it only handles workers that consume too much memory.  If you happen to find yourself with stale workers, try using the following bash script I made.

#!/bin/bash 

# Simple script to kill stale resque workers
# Adam St. John [astjohn] [@] [gmail]
# 2011-06-24

# Timeout set to 10 minutes
TIMEOUT=600

# 21028 resque-1.16.1: Forked 17674 at 1308879414 
output=$(ps -e -o pid,command | grep -e [r]esque.*Forked | awk '{printf "%i#%i#%i\n", $1, $6, $4}')
now=`date +%s`

echo "`date` - Looking for stale workers."
# output should produce something like: 21028#1308879414#17674
if [ -z "$output" ] ; then
  echo "Stale workers were not found."
else
  echo "$output" | while read -r line ; do
  
    pid=$(echo $line | cut -d\# -f1)
    ftime=$(echo $line | cut -d\# -f2)
    fpid=$(echo $line | cut -d\# -f3)

    if ! [[ "$pid" =~ ^[0-9]+$ ]] ; then
      exec >&2; echo "Unable to find PID of stale resque worker"; exit 1
    fi

    if ! [[ "$ftime" =~ ^[0-9]+$ ]] ; then
      exec >&2; echo "Unable to find Forked time for stale resque worker"; exit 1
    fi

    difference=$(($now - $ftime))
    if [ "$difference" -ge "$TIMEOUT" ] ; then
      echo "Found a stale worker!"
      kill -s USR1 $pid
      echo "Kill signal USR1 sent to worker $pid so that it will shut down forked child $fpid"
    fi  
  done
  echo "Finished dealing with stale workers."
fi
exit 0

The script checks the timestamp that each worker leaves and kills a worker that has exceeded the TIMEOUT value. I run this script from a cron job.

Rails 3.1 Engines, Rspec, and Cucumber

I thought I’d make a quick post about my first encounter with Rails 3.1 and with engines in general.  Rails engines are essentially applications or a subset of functionality that can be run in or shared with other applications.  In Rails 3.1, creating an engine is as easy as:
rails plugin new engine_name [options]

where several options are available, including full and mountable. Run with
--help to find out more.

Generating a new engine produces a file structure similar to a regular rails application, but with a few notable differences which have been explained in several great articles.

Setting up Rspec

I enjoy using rspec in my testing and instead of using the test folder which was generated automatically, I decided to rename it to spec. Inside this spec folder, I set up the usual rspec environment. For example, inside my spec folder, I have additional folders named models, controllers, views, support, etc. Rspec needs to load the dummy application’s environment in order to function. To allow it to do so, a few changes are necessary to spec_helper.rb.

# /spec/spec_helper.rb

# This file is copied to spec/ when you
# run 'rails generate rspec:install'
ENV["RAILS_ENV"] ||= 'test'
require File.expand_path("../dummy/config/environment", __FILE__)
require 'rspec/rails'

ENGINE_RAILS_ROOT = File.join(File.dirname(__FILE__), '../')

# Requires supporting ruby files with custom matchers and macros, etc,
# in spec/support/ and its subdirectories.
#Dir[Rails.root.join("spec/support/**/*.rb")].each {|f| require f}
Dir[File.join(ENGINE_RAILS_ROOT, "spec/support/**/*.rb")].each {|f| require f }

RSpec.configure do |config|
  # == Mock Framework
  #
  # If you prefer to use mocha, flexmock or RR, uncomment
  # the appropriate line:
  #
  # config.mock_with :mocha
  # config.mock_with :flexmock
  # config.mock_with :rr
  config.mock_with :rspec

  # Remove this line if you're not using ActiveRecord
  # or ActiveRecord fixtures
  config.fixture_path = "#{::Rails.root}/spec/fixtures"

  # If you're not using ActiveRecord, or you'd prefer not to run
  # each of your examples within a transaction, remove
  # the following line or assign false instead of true.
  config.use_transactional_fixtures = true

  config.include Cornerstone::Engine.routes.url_helpers
end

Of course, renaming the test folder to spec did have some unintended consequences. For one, running rake tasks did not work. In order to fix that, I had to change Rakefile to load the dummy application’s environment from the new location.

# /Rakefile

#!/usr/bin/env rake
begin
  require 'bundler/setup'
rescue LoadError
  puts 'You must `gem install bundler` and `bundle install` to run rake tasks'
end
begin
  require 'rdoc/task'
rescue LoadError
  require 'rdoc/rdoc'
  require 'rake/rdoctask'
  RDoc::Task = Rake::RDocTask
end

RDoc::Task.new(:rdoc) do |rdoc|
  rdoc.rdoc_dir = 'rdoc'
  rdoc.title    = 'EngineName'
  rdoc.options << '--line-numbers' << '--inline-source'
  rdoc.rdoc_files.include('README.rdoc')
  rdoc.rdoc_files.include('lib/**/*.rb')
end

# notice the path change in the following line
APP_RAKEFILE = File.expand_path("../spec/dummy/Rakefile", __FILE__)
load 'rails/tasks/engine.rake'

require 'rake/testtask'

Rake::TestTask.new(:test) do |t|
  t.libs << 'lib'
  t.libs << 'test'
  t.pattern = 'test/**/*_test.rb'
  t.verbose = false
end

task :default => :test

Bundler::GemHelper.install_tasks

Finally, running rails on the console will not work until you change the dummy application’s environment reference in the rails script.

# /script/rails

#!/usr/bin/env ruby
#!/usr/bin/env ruby
# This command will automatically be run when you run "rails" with Rails 3 gems installed from the root of your application.

ENGINE_PATH = File.expand_path('../..',  __FILE__)
# notice the path change on the following line
load File.expand_path('../../spec/dummy/script/rails',  __FILE__)

I believe that’s all it took to get rspec working again after renaming the test folder to spec. Of course, since I’m using rspec, I no longer needed test_helper.rb and so I removed it. If there’s something I missed, or if you can’t get rspec to work, please let me know.

Preparing Cucumber

As with Rspec, Cucumber also needs to load the dummy application’s environment in order to function. After adding cucumber-rails to your gemspec or Gemfile, running rails generate cucumber:install will generate the necessary files and folder structure for cucumber to function properly. Unfortunately, the files generated are geared toward normal rails applications and not rails engines. To make it work for a rails engine, I had to make the following changes to env.rb.

# /features/support/env.rb

# IMPORTANT: This file is generated by cucumber-rails - edit at your own peril.
# It is recommended to regenerate this file in the future when you upgrade to a
# newer version of cucumber-rails. Consider adding your own code to a new file
# instead of editing this one. Cucumber will automatically load all features/**/*.rb
# files.

ENV["RAILS_ENV"] = "test" # had to explicity state the test environment (or cucumber env.)
ENV["RAILS_ROOT"] = File.expand_path(File.dirname(__FILE__) + '/../../spec/dummy/')
require File.expand_path(File.dirname(__FILE__) + '/../../spec/dummy/config/environment')

require 'cucumber/rails'

# If you use factory girl, I had to add the following...
require 'factory_girl_rails'
require File.expand_path(File.dirname(__FILE__) + '/../../spec/support/factories')

# Capybara defaults to XPath selectors rather than Webrat's default of CSS3. In
# order to ease the transition to Capybara we set the default here. If you'd
# prefer to use XPath just remove this line and adjust any selectors in your
# steps to use the XPath syntax.
Capybara.default_selector = :css

# By default, any exception happening in your Rails application will bubble up
# to Cucumber so that your scenario will fail. This is a different from how
# your application behaves in the production environment, where an error page will
# be rendered instead.
#
# Sometimes we want to override this default behaviour and allow Rails to rescue
# exceptions and display an error page (just like when the app is running in production).
# Typical scenarios where you want to do this is when you test your error pages.
# There are two ways to allow Rails to rescue exceptions:
#
# 1) Tag your scenario (or feature) with @allow-rescue
#
# 2) Set the value below to true. Beware that doing this globally is not
# recommended as it will mask a lot of errors for you!
#
ActionController::Base.allow_rescue = false

# Remove/comment out the lines below if your app doesn't have a database.
# For some databases (like MongoDB and CouchDB) you may need to use :truncation instead.
begin
  DatabaseCleaner.strategy = :transaction
rescue NameError
  raise "You need to add database_cleaner to your Gemfile (in the :test group) if you wish to use it."
end

# You may also want to configure DatabaseCleaner to use different strategies for certain features and scenarios.
# See the DatabaseCleaner documentation for details. Example:
#
#   Before('@no-txn,@selenium,@culerity,@celerity,@javascript') do
#     DatabaseCleaner.strategy = :truncation, {:except => %w[widgets]}
#   end
#
#   Before('~@no-txn', '~@selenium', '~@culerity', '~@celerity', '~@javascript') do
#     DatabaseCleaner.strategy = :transaction
#   end
#

Note the two lines at the top which reference the dummy application. These lines are required because cucumber attempts to boot a rails application from the normal environment.rb which is usually located in the config folder. In a rails engine, this file doesn’t exist. It only exists within the dummy application. Adding the additional lines at the top of env.rb forces cucumber to bootstrap the dummy application’s environment. With these changes, cucumber should be fully functional.

One further change that had me stumped for a while was using Cucumber’s path_to helper located in paths.rb. I just could not make cucumber return the correct path when referencing using a path helper such as some_controller_method_path. In order to use these helpers (thanks Robin – see below) you can add the engines’ url_helpers to the World object of Cucumber like this:

module EngineRoutesHelper
  include MyEngine::Engine.routes_url_helper
end
World(EngineRoutesHelper)

Now, you will be able to rewrite the path_to method like this:

def path_to(page_name)
  case page_name
    when /^the discussions page$/
      discussions_path
    # etc…
  end
end

I hope my stumblings have helped others out there. Give me a shout if you run into any problems.

Converting a MySQL database to PostgreSQL using Rails

I recently decided to move one of my projects from MySQL to PostgreSQL for a number of reasons. I was looking for the easiest and quickest way to migrate the data. However, after playing with a number of solutions and tools, nothing was working well. Then I found this lovely rake task. The task uses rails to convert the data.

To convert your data, you will need to set up two databases. The rake task is designed to move data from the production database to the development database. To kick start things, download a backup of your production database and move it to your local computer. Next, establish a production MySQL database and then build the schema. rake db:schema:load RAILS_ENV=production. After the database is built, import the data to the local production database. mysql -u USER -p -D data_prod < backup.sql

Now, because the rake task moves data from a production database to a development database, you must configure your project’s database.yml file. This is what it could look like:

production:
  adapter: mysql2
  encoding: utf8
  username: rails_dev
  password: password
  host: localhost
  database: db_prod

development:
  adapter: postgresql
  encoding: utf8
  username: rails_dev
  password: password
  host: localhost
  database: db_dev

Note that the development database is set up for PostgreSQL. Therefore, before running the rake task you must establish this database as well. Build the database in PostgreSQL and then load the schema as above: rake db:schema:load RAILS_ENV=development. Also, do not forget to add a PostgreSQL adapter to your project’s Gemfile. I prefer the pg gem: gem "pg"

With the databases all set up, converting your data to the PostgreSQL database is as simple as:

rake db:convert:prod2dev

This worked great using ruby 1.9.2 and rails 3.0.8.

Rails 3.1 Engines – Mountable or Full? – Part 2

This is a continuation of my post Rails 3.1 Engines – Mountable or Full? – Part 1 which highlights some key features of a full engine. There are a few differences with a mountable engine, which I will try to explain below.

Mountable Engines

Engine Details

To generate a mountable engine, run rails plugin new MyEngine --mountable on the console. This will generate the engine’s basic structure.

The first difference I noted with a mountable engine is that, by default, the engine is namespaced. The engine’s starting config is:

module MyEngine
# my_engine/lib/my_engine/engine.rb

  class Engine < Rails::Engine
    isolate_namespace MyEngine
  end
end

Since the engine has an isolated namespace, the controllers, models, helpers, etc. are all bundled within the module MyEngine. The folder structure reflects this change as well. Unlike a full engine, the mountable engine generated the typical assets that are normally associated with a Rails application, namely application.js, application.css, and application_controller.rb. Note the namespacing that occurs with the application controller:

# my_engine/app/controllers/my_engine/application_controller.rb

module MyEngine
  class ApplicationController < ActionController::Base
  end
end

This namespacing occurs throughout the engine. There are subtle differences in terms of routing as well. With a mountable engine, the engine’s route file looks like this:

# my_engine/config/routes.rb

MyEngine::Engine.routes.draw do
  # route stuff goes here
end

Remember, the full application’s routes start with Rails.application.routes.draw. The dummy application’s routes file was generated as:

# my_engine/test/dummy/config/routes.rb

Rails.application.routes.draw do

  mount MyEngine::Engine => "/my_engine"
end

Notice that the engine is bundled under a “mounted” route at /my_engine. This means that all of the engine’s functionality will be located under the engine’s root location of /my_engine within the dummy (parent) application. This type of structure suggests that a mountable engine is best suited for situations when a developer wants the engine to behave as its own application operating in parallel with the parent application. Most of the engine’s features are isolated and independent of the parent application. To test this, I plugged the mountable engine into a test application.

Routing

I used a model within the engine called “Post”, along with the migration, controller, and helper to match (rails generate scaffold post title:string body:text). Now my engine’s routes file looks like this:

# my_engine/config/routes.rb

MyEngine::Engine.routes.draw do
  resources :posts
end

With a full engine, the Post routes were included directly into the parent application automatically. This time, using the mountable engine and running rake routes within the parent application, nothing happened! That’s because I did not mount the engine in a similar manner to what the dummy application in our engine is using. The parent application’s route file had to be adjusted like so:

# parent_app/config/routes.rb

ParentApp::Application.routes.draw do
 mount MyEngine::Engine => "/my_engine"
end

rake routes now produced the following:

my_engine  /my_engine {:to=>MyEngine::Engine}

…and that’s it. All of the engine’s functionality is bundled into the parent application underneath the route /my_engine.

After running rake my_engine:install:migrations and rake db:migrate in the parent application, I was ready to test how the engine’s controllers and helpers would integrate. [note: running migrations from an isolated engine will prefix your database tables with the engine name]. As with the full engine, I established a simple view template:

# my_engine/app/views/my_engine/posts/index.html.erb

<p>Hi There!</p>

Of course, navigating to /my_engine/posts in the parent application showed an almost blank page with the words “Hi There!” The controller actions and views from the engine were incorporated into the parent application, but only under the namespaced route /my_engine.

Next, I tested if the view could be overridden by the parent application. I created a new view template for the Post index action within the parent application that looked like this:

# parent_app/app/views/posts/index.html.erb

<p>Good Bye!</p>

No change and of course not! Our engine is namespaced, remember? Instead, I moved the template to parent_app/app/views/my_engine/posts/index.html.erb. Now visiting /my_engine/posts in the parent application showed “Good Bye!” That’s better. Next I decided to test out the engine’s helpers. I added a method to the Post helper inside the engine.

# my_engine/app/helpers/my_engine/posts_helper.rb

module MyEngine
  module PostsHelper
    def test
      raw("<p>hello world</p>")
    end
  end
end

I then changed the parent application’s view template to use the helper.

# parent_app/app/views/posts/index.html.erb

<p>Good Bye!</p>
<%= test %>

Visiting the page resulted in the words “Good Bye!” and “hello world.” Therefore, the engine’s helper methods were directly exposed to the parent application, allowing the parent to use them. 

 To be honest, I did not expect this to work. There is a section in the Rails comments about sharing an isolated engine’s helpers with the parent application by using helper MyEngine::Engine.helpers in the application controller of the parent. It states that in doing so “[the parent application] will include all of the helpers from engine’s directory.” Perhaps this is a bug in the release candidates of Rails 3.1? Maybe my interpretation of the instructions is way off? I’m not sure. If this does change by the time you use it, and you can’t access the engine’s helpers, I suggest trying the method described.

Summary

It seems as though mountable engines are best suited as complete applications in themselves which run along side a parent application. The engine routes, controllers, models, helpers, etc. are namespaced and bundled into the mounted route within the parent application, thus avoiding namespace conflicts. If I’m way off in my assessment or there are other things which I have missed, drop me a line and I’ll add them!

Rails 3.1 Engines – Mountable or Full? – Part 1

I decided to create my first rails engine about a week or so ago.  I figured this was a great way to learn Rails 3.1 and the new engine generator; however, I quickly ran into a problem.  Do I want a mountable or full engine?  What’s the difference anyway?  At the time, a quick google search showed that not many people really knew. The Rails code comments are great, but after reading them, I was still a bit confused.  For those of you out there in the same boat, here’s my attempt at highlighting some of the differences.

Full Engines

Engine Details

To generate a full engine, run rails plugin new MyEngine --full on the console. This will generate the engine’s basic structure.

The first thing I noticed was that the typical assets associated with a normal Rails application were not generated; application.js and application.css were not created. In fact, the application controller was completely omitted, alluding that perhaps a full engine is feared toward supplementing a parent application only. As far as routing goes, the engine’s route looked like this:

# my_engine/config/routes.rb

Rails.application.routes.draw do
end

… and the dummy application’s routes file looked like this:

# my_engine/test/dummy/config/routes.rb

Dummy::Application.routes.draw
  # route stuff
end

Another item that was missing from the basic application structure was the database folder. For engine configuration, a full engine is not namespaced, so the engine.rb file was essentially blank.

# my_engine/lib/my_engine/engine.rb

module MyEngine
  class Engine < Rails::Engine
  end
end

Now, the rails comments state that engines allow you to add a “subset of functionality to an existing Rails application.” Since the full engine is missing an application controller, it seems that its purpose is to plug into an existing application without namespacing to add functionality to the parent application. To test this theory, I generated a new application and referenced my engine in the parent application’s Gemfile.

Routing

Since the engine is not namespaced, the engine’s routes are incorporated into the parent application’s routes directly. To test this, I created a model within the engine called “Post”, along with the migration, controller, and helper to match. To create the migration, I also had to build the my_engine/db/migrate/ folder, which was not created when I generated the engine. Running rake routes within the parent application resulted in the following:

    posts GET    /posts(.:format)          {:action=>"index", :controller=>"posts"}
          POST   /posts(.:format)          {:action=>"create", :controller=>"posts"}
 new_post GET    /posts/new(.:format)      {:action=>"new", :controller=>"posts"}
edit_post GET    /posts/:id/edit(.:format) {:action=>"edit", :controller=>"posts"}
     post GET    /posts/:id(.:format)      {:action=>"show", :controller=>"posts"}
          PUT    /posts/:id(.:format)      {:action=>"update", :controller=>"posts"}
          DELETE /posts/:id(.:format)      {:action=>"destroy", :controller=>"posts"}

As you can see, the Post routes from the engine were included directly into the parent application.

After running rake my_engine:install:migrations and rake db:migrate in the parent application, I was ready to test how the engine’s controllers and helpers would integrate. In my engine, I had a simple view template:

# my_engine/app/views/posts/index.html.erb

<p>Hi There!</p>

Of course, navigating to /posts in the parent application showed an almost blank page with the words “Hi There!” The controller actions and views from the engine were incorporated into the parent application.

One helpful feature of engines is the fact that their controller methods and views can be overridden simply by placing similar files within the parent application. For example, I created a new view template for the Post index action within the parent application that looked like this:

# parent_app/app/views/posts/index.html.erb

<p>Good Bye!</p>

Now visiting /posts in the parent application shows “Good Bye!” That’s great, but what about the engine’s helpers? I tested that functionality by adding a method to the Post helper back inside the engine.

# my_engine/app/helpers/posts_helper.rb

module PostsHelper

  def test
    raw("<p>hello world</p>")
  end

end

I then changed the parent application’s view template to use the helper.

# parent_app/app/views/posts/index.html.erb

<p>Good Bye!</p>
<%= test %>

This time, visiting the page again in the parent application resulted in the words “Good Bye!” and “hello world.” Therefore, the engine’s helper methods were directly exposed to the parent application, allowing the parent to use them.

Summary

In summary, it seems as though full engines are best suited to augment a parent application. The engine routes, controllers, models, helpers, etc. are exposed to the parent application which allows for easy access, but could result in namespace conflicts.

It looks like this post is getting a bit lengthy so I’ll stop it here and talk about mountable engines soon.