Merged redis branch changes

HISTORY.md

@@ -1,3 +1,26 @@
+## 1.11.0 (2010-08-23)
+
+* Web UI: Group /workers page by hostnames
+
+## 1.10.0 (2010-08-23)
+
+* Support redis:// string format in `Resque.redis=`
+* Using new cross-platform JSON gem.
+* Added `after_enqueue` plugin hook.
+* Added `shutdown?` method which can be overridden.
+* Added support for the "leftright" gem when running tests.
+* Grammarfix: In the README
+
+## 1.9.10 (2010-08-06)
+
+* Bugfix: before_fork should get passed the job
+
+## 1.9.9 (2010-07-26)
+
+* Depend on redis-namespace 0.8.0
+* Depend on json_pure instead of json (for JRuby compat)
+* Bugfix: rails_env display in stats view
+
 ## 1.9.8 (2010-07-20)
 
 * Bugfix: Worker.all should never return nil

README.markdown

@@ -9,8 +9,8 @@ Background jobs can be any Ruby class or module that responds to
 jobs or you can create new classes specifically to do work. Or, you
 can do both.
 
-Resque is heavily inspired by DelayedJob (which rocks) and is
-comprised of three parts:
+Resque is heavily inspired by DelayedJob (which rocks) and comprises
+three parts:
 
 1. A Ruby library for creating, querying, and processing jobs
 2. A Rake task for starting a worker which processes jobs
@@ -30,6 +30,67 @@ not doing, what queues you're using, what's in those queues, provides
 general usage stats, and helps you track failures.
 
 
+What did you guys do to Resque?
+===============================
+
+This version of Resque uses mongo as the back end.  It also has some neat
+new features that you may enjoy
+
+Features
+--------
+
+It stores each queue in its own collection.
+
+For testing purposes. Resque.bypass_queues will call your job's
+perform method AT ONCE, and not touch the queue at all.  This can be
+useful when writing unit tests against things that would normally
+happen partially in your request and partially in a worker.
+
+If your job class has a variable called @unique_jobs = true, you
+can queue unique jobs - jobs that must only be processed once.  In 
+order to take advantage of this feature, the arguments to your enqueue
+call must include a hash containing a key called _id in the first
+position. 
+
+If your job class indicates that @delayed_jobs = true, you can queue
+delayed jobs.  These jobs will not be popped off the queue until the
+Time indicated in arg[0][:delay_until] has come.  Note that you must
+call Resque.enable_delay(:queue) before enququing any delayed jobs, to
+ensure that the performance impact on other queues is minimal.
+
+Stern Warnings
+--------------
+
+Sometimes, Resque-Mongo will drop a queue collection, or create some 
+indexes, or otherwise manipulate its database.  For this reason, it is
+STRONGLY recommended that you give it its own database in mongo.
+
+All jobs should be queued via Resque.enqueue.  All arguments passed to
+this method must be BSON-encodable.  Resque-Mongo does not serialize
+your objects for you.  Arrays, Hashes, Strings, Numbers, and Times
+are all ok, so don't worry.
+
+Many of the new queue-level features require the first argument of
+your perform method to be an options hash.  In fact, if you just start
+making all your perform()s take one param, that is an options hash,
+you'll probably save yourself some pain.
+
+Resque-Mongo will not create any indexes on your queues, only on its
+meta-data.  You will need to create any indexes you want.  Normally,
+This is not a problem, because you aren't querying by keys, but you may
+want to create indexes on the class key in some circumstances.  If you 
+use the unique or delay features, you may want some additional indexes, 
+depending on the nature of your workload.  Paranoid?  Test enqueuing and 
+processing all your jobs, and run with --notablescans.  Learn the profiler,
+and use it often.
+
+Specifically, a queue with many long-delayed jobs will result in slower queue pops
+for all jobs using that queue.  Index delay_until in the case of
+thousands of delayed jobs.
+
+Back to the original README
+===========================
+
 The Blog Post
 -------------
 
@@ -764,7 +825,7 @@ To join the list simply send an email to <resque@librelist.com>. This
 will subscribe you and send you information about your subscription,
 including unsubscribe information.
 
-The archive can be found at <http://librelist.com/browser/>.
+The archive can be found at <http://librelist.com/browser/resque/>.
 
 
 Meta

Rakefile

@@ -1,8 +1,6 @@
 #
 # Setup
 #
-
-load 'tasks/redis.rake'
 require 'rake/testtask'
 
 $LOAD_PATH.unshift 'lib'
@@ -23,7 +21,8 @@ desc "Run the test suite"
 task :test do
   rg = command?(:rg)
   Dir['test/**/*_test.rb'].each do |f|
-    rg ? sh("rg #{f}") : ruby(f)
+    #rg ? sh("rg #{f}") : ruby(f)
+    ruby(f)
   end
 end
 
@@ -40,7 +39,7 @@ end
 # Install
 #
 
-task :install => [ 'redis:install', 'dtach:install' ]
+task :install => [ 'dtach:install' ]
 
 
 #

bin/resque

@@ -1,6 +1,7 @@
 #!/usr/bin/env ruby
 
 $LOAD_PATH.unshift File.dirname(__FILE__) + '/../lib'
+require 'rubygems'
 require 'resque'
 
 def kill(worker)
@@ -34,6 +35,12 @@ def list
   end
 end
 
+def queues
+  Resque.queues.each do |queue|
+    puts"Queue #{queue}: #{Resque.size(queue)}"
+  end
+end
+
 if (i = ARGV.index('-r')) && ARGV[i+1]
   Resque.redis = ARGV[i+1]
   ARGV.delete_at(i)
@@ -47,11 +54,14 @@ when 'remove'
   remove ARGV[1]
 when 'list'
   list
+when 'queues'
+  queues
 else
   puts "Usage: resque [-r redis_host:redis_port] COMMAND [option]"
   puts
   puts "Commands:"
   puts "  remove WORKER   Removes a worker"
   puts "  kill WORKER     Kills a worker"
   puts "  list            Lists known workers"
+  puts "  queues          checks queue depth"
 end

bin/resque-web

@@ -16,8 +16,8 @@ Vegas::Runner.new(Resque::Server, 'resque-web', {
     load path.to_s.strip if path
   }
 }) do |runner, opts, app|
-  opts.on('-N NAMESPACE', "--namespace NAMESPACE", "set the Redis namespace") {|namespace|
-    runner.logger.info "Using Redis namespace '#{namespace}'"
-    Resque.redis.namespace = namespace
+  opts.on('-N NAMESPACE', "--namespace NAMESPACE", "set the Mongo database") {|namespace|
+    runner.logger.info "Using Mongo database '#{namespace}'"
+    Resque.mongo_db= namespace
   }
 end

docs/HOOKS.md

@@ -66,6 +66,9 @@ An unnamed hook (`before_perform`) will be executed first.
 
 The available hooks are:
 
+* `after_enqueue`: Called with the job args after a job is placed on the queue.
+  Any exception raised propagates up to the code which queued the job.
+
 * `before_perform`: Called with the job args before perform. If it raises
   `Resque::Job::DontPerform`, the job is aborted. If other exceptions
   are raised, they will be propagated up the the `Resque::Failure`
@@ -99,6 +102,13 @@ look something like this.
 
 Modules are even better because jobs can use many of them.
 
+    module ScaledJob
+      def after_enqueue_scale_workers(*args)
+        Logger.info "Scaling worker count up"
+        Scaler.up! if Redis.info[:pending].to_i > 25
+      end
+    end
+
     module LoggedJob
       def before_perform_log_job(*args)
         Logger.info "About to perform #{self} with #{args.inspect}"
@@ -115,6 +125,7 @@ Modules are even better because jobs can use many of them.
     class MyJob
       extend LoggedJob
       extend RetriedJob
+      extend ScaledJob
       def self.perform(*args)
         ...
       end