diff --git a/.travis.yml b/.travis.yml new file mode 100644 index 0000000..ee94e82 --- /dev/null +++ b/.travis.yml @@ -0,0 +1,6 @@ +language: ruby +rvm: + - 1.9.3 + - 2.0 + - 2.1 + - 2.2 diff --git a/README.md b/README.md index 34cf957..cae1569 100644 --- a/README.md +++ b/README.md @@ -1,14 +1,37 @@ +[![Build Status](https://travis-ci.org/stackbuilders/campfire_export.png)](https://travis-ci.org/stackbuilders/campfire_export) + # campfire_export # ## Quick Start ## - $ sudo gem install campfire_export + $ gem install stackbuilders-campfire_export $ campfire_export +## Note ## + +This gem is a fork from the original version of this project by +[Marc Hedlund](https://github.com/precipice). While almost all of the work is +still his, this fork contains fixes and updates for user-facing issues, +including the following: + +* Upgrade gem dependencies and add support for Ruby versions >= 1.9.3 +* Retry up to 5 times before giving up on a resource. Makes campfire_export usable for large exports. +* Fixes file size validation when file contains Unicode. +* Allow export of a specific room instead of exporting all rooms on a Campfire account. + +When you install the gem following the instructions above (i.e., with the name +stackbuilders-campfire-export) you will get these fixes, and the project will run +on modern Ruby versions. + +We will try to respond to issues and pull requests on the +[Stack Builders](http://www.stackbuilders.com) fork of this repository +(https://github.com/stackbuilders/campfire_export), and to push new releases in +a timely manner to https://rubygems.org/gems/stackbuilders-campfire_export. + ## Intro ## I had an old, defunct [Campfire](http://campfirenow.com/) account with five -years' worth of transcripts in it, some of them hilarious, others just +years' worth of transcripts in it, some of them hilarious, others just memorable. Unfortunately, Campfire doesn't currently have an export function; instead it provides pages of individual transcripts. I wanted a script to export everything from all five years, using the Campfire API. @@ -29,27 +52,27 @@ repo and a [Ruby gem](http://docs.rubygems.org/read/chapter/1). ## Installing ## -[Ruby 1.8.7](http://www.ruby-lang.org/en/downloads/) or later is required. +[Ruby 1.9.3](http://www.ruby-lang.org/en/downloads/) or later is required. [RubyGems](https://rubygems.org/pages/download) is also required -- I'd recommend having the latest version of RubyGems installed before starting. Once you are set up, to install, run the following: - $ sudo gem install campfire_export + $ gem install campfire_export ## Configuring ## There are a number of configuration variables required to run the export. The export script will prompt you for these; just run it and away you go. If you want to run the script repeatedly or want to control the start and end date of -the export, you can create a `.campfire_export.yaml` file in your home +the export, you can create a `.campfire_export.yml` file in your home directory using this template: # Your Campfire subdomain (for 'https://myco.campfirenow.com', use 'myco'). subdomain: myco # Your Campfire API token (see "My Info" on your Campfire site). - api_token: abababababababababababababababababababab + api_token: your-campfire-token # OPTIONAL: Export start date - the first transcript you want exported. # Uncomment to set. Defaults to the date each room was created. @@ -59,6 +82,11 @@ directory using this template: # Uncomment to set. Defaults to the date of the last comment in each room. #end_date: 2010/12/31 + # OPTIONAL: Campfire room - the room you want exported. All other rooms + # will be skipped. + # Uncomment to set. Defaults to all rooms. + #room_name: Stack Builders Website Dev + The `start_date` and `end_date` variables are inclusive (that is, if your end date is Dec 31, 2010, a transcript for that date will be downloaded), and both are optional. If they are omitted, export will run from the date each @@ -98,6 +126,7 @@ Also, thanks much for all the help, comments and contributions: * [Junya Ogura](https://github.com/juno) * [Chase Lee](https://github.com/chaselee) * [Alex Hofsteede](https://github.com/alex-hofsteede) +* [Justin Leitgeb](https://github.com/jsl) As mentioned above, some of the work on this was done by other people. The Gist I forked had contributions from: diff --git a/Rakefile b/Rakefile index ff06ea1..7773011 100644 --- a/Rakefile +++ b/Rakefile @@ -1,7 +1,12 @@ -require 'rubygems' -require 'bundler' -Bundler::GemHelper.install_tasks +require "bundler/gem_tasks" -require 'rspec/core/rake_task' -rspec = RSpec::Core::RakeTask.new -rspec.rspec_opts = '-f Fuubar --color' + +begin + require 'rspec/core/rake_task' + + RSpec::Core::RakeTask.new(:spec) + + task :default => :spec +rescue LoadError + # no rspec available +end diff --git a/bin/campfire_export b/bin/campfire_export index 98c6a7a..4bca11f 100755 --- a/bin/campfire_export +++ b/bin/campfire_export @@ -42,23 +42,27 @@ def ensure_config_for(config, key, prompt) end config = {} -config_file = File.join(ENV['HOME'], '.campfire_export.yaml') +config_file = File.join(ENV['HOME'], '.campfire_export.yml') if File.exists?(config_file) and File.readable?(config_file) - config = YAML.load_file(config_file) + config = YAML.load_file(config_file) end -ensure_config_for(config, 'subdomain', +ensure_config_for(config, 'subdomain', "Your Campfire subdomain (for 'https://myco.campfirenow.com', use 'myco')") ensure_config_for(config, 'api_token', "Your Campfire API token (see 'My Info' on your Campfire site)") begin - account = CampfireExport::Account.new(config['subdomain'], + account = CampfireExport::Account.new(config['subdomain'], config['api_token']) account.find_timezone account.rooms.each do |room| - room.export(config_date(config, 'start_date'), + if config["room_name"] && room.name != config["room_name"] + next + end + + room.export(config_date(config, 'start_date'), config_date(config, 'end_date')) end rescue => e diff --git a/campfire_export.gemspec b/campfire_export.gemspec index 8038105..baff7f5 100644 --- a/campfire_export.gemspec +++ b/campfire_export.gemspec @@ -3,25 +3,31 @@ $:.push File.expand_path("../lib", __FILE__) require "campfire_export/version" Gem::Specification.new do |s| - s.name = "campfire_export" + s.name = "stackbuilders-campfire_export" s.version = CampfireExport::VERSION s.platform = Gem::Platform::RUBY - s.authors = ["Marc Hedlund"] - s.email = ["marc@precipice.org"] + s.authors = ["Marc Hedlund", "Justin Leitgeb"] + s.email = ["marc@precipice.org", "justin@stackbuilders.com"] s.license = "Apache 2.0" - s.homepage = "https://github.com/precipice/campfire_export" - s.summary = %q{Export transcripts and uploaded files from your 37signals' Campfire account.} - s.description = s.summary + s.homepage = "https://github.com/stackbuilders/campfire_export" + s.summary = "Export transcripts and uploaded files from your 37signals' Campfire account." + + s.description = %{Exports content from all rooms in a + 37signals Campfire account. Creates a directory containing transcripts + and content uploaded to each one of your rooms. Can be configured to + recognize start and end date of content export.} s.rubyforge_project = "campfire_export" - s.required_ruby_version = '>= 1.8.7' - - s.add_development_dependency "bundler", "> 1.0.15" - s.add_development_dependency "fuubar", "~> 0.0.5" - s.add_development_dependency "rspec", "~> 2.6.0" - s.add_dependency "tzinfo", "~> 0.3.29" - s.add_dependency "httparty", "~> 0.7.8" - s.add_dependency "nokogiri", "~> 1.5.6" + s.required_ruby_version = '>= 1.9.3' + + s.add_development_dependency "bundler", "~> 1" + s.add_development_dependency "rake", "~> 10" + s.add_development_dependency "rspec", "~> 2.6" + + s.add_dependency "tzinfo", "~> 1.2" + s.add_dependency "httparty", "~> 0.13" + s.add_dependency "nokogiri", "~> 1.6" + s.add_dependency "retryable", "~> 2.0" s.files = `git ls-files`.split("\n") s.test_files = `git ls-files -- {test,spec,features}/*`.split("\n") diff --git a/lib/campfire_export.rb b/lib/campfire_export.rb index b26c4b4..2629373 100644 --- a/lib/campfire_export.rb +++ b/lib/campfire_export.rb @@ -22,32 +22,42 @@ require 'rubygems' -require 'campfire_export/timezone' - require 'cgi' require 'fileutils' require 'httparty' require 'nokogiri' +require 'retryable' require 'time' require 'yaml' module CampfireExport module IO + MAX_RETRIES = 5 + def api_url(path) "#{CampfireExport::Account.base_url}#{path}" end def get(path, params = {}) url = api_url(path) - response = HTTParty.get(url, :query => params, :basic_auth => - {:username => CampfireExport::Account.api_token, :password => 'X'}) + + response = Retryable.retryable(:tries => MAX_RETRIES) do |retries, exception| + if retries > 0 + msg = "Attempt ##{retries} to fetch #{path} failed, " + + "#{MAX_RETRIES - retries} attempts remaining" + log :error, msg, exception + end + + HTTParty.get(url, :query => params, :basic_auth => + {:username => CampfireExport::Account.api_token, :password => 'X'}) + end if response.code >= 400 raise CampfireExport::Exception.new(url, response.message, response.code) end response end - + def zero_pad(number) "%02d" % number end @@ -58,17 +68,17 @@ def export_dir "#{date.year}/#{zero_pad(date.mon)}/#{zero_pad(date.day)}" end - # Requires that room_name and date be defined in the calling object. + # Requires that room_name and date be defined in the calling object. def export_file(content, filename, mode='w') # Check to make sure we're writing into the target directory tree. true_path = File.expand_path(File.join(export_dir, filename)) - + unless true_path.start_with?(File.expand_path(export_dir)) raise CampfireExport::Exception.new("#{export_dir}/#{filename}", "can't export file to a directory higher than target directory; " + "expected: #{File.expand_path(export_dir)}, actual: #{true_path}.") end - + if File.exists?("#{export_dir}/#{filename}") log(:error, "#{export_dir}/#{filename} failed: file already exists") else @@ -77,20 +87,20 @@ def export_file(content, filename, mode='w') end end end - + def verify_export(filename, expected_size) full_path = "#{export_dir}/#{filename}" unless File.exists?(full_path) - raise CampfireExport::Exception.new(full_path, + raise CampfireExport::Exception.new(full_path, "file should have been exported but did not make it to disk") end unless File.size(full_path) == expected_size - raise CampfireExport::Exception.new(full_path, + raise CampfireExport::Exception.new(full_path, "exported file exists but is not the right size " + "(expected: #{expected_size}, actual: #{File.size(full_path)})") end end - + def log(level, message, exception=nil) case level when :error @@ -109,361 +119,15 @@ def log(level, message, exception=nil) end end end - - class Exception < StandardError - attr_accessor :resource, :message, :code - def initialize(resource, message, code=nil) - @resource = resource - @message = message - @code = code - end - - def to_s - "<#{resource}>: #{message}" + (code ? " (#{code})" : "") - end - end - - class Account - include CampfireExport::IO - include CampfireExport::TimeZone - - @subdomain = "" - @api_token = "" - @base_url = "" - @timezone = nil - - class << self - attr_accessor :subdomain, :api_token, :base_url, :timezone - end - - def initialize(subdomain, api_token) - Account.subdomain = subdomain - Account.api_token = api_token - Account.base_url = "https://#{subdomain}.campfirenow.com" - end - - def find_timezone - settings = Nokogiri::XML get('/account.xml').body - selected_zone = settings.xpath('/account/time-zone') - Account.timezone = find_tzinfo(selected_zone.text) - end - - def rooms - doc = Nokogiri::XML get('/rooms.xml').body - doc.xpath('/rooms/room').map {|room_xml| Room.new(room_xml) } - end - end - - class Room - include CampfireExport::IO - attr_accessor :id, :name, :created_at, :last_update - - def initialize(room_xml) - @id = room_xml.xpath('id').text - @name = room_xml.xpath('name').text - created_utc = DateTime.parse(room_xml.xpath('created-at').text) - @created_at = Account.timezone.utc_to_local(created_utc) - end - - def export(start_date=nil, end_date=nil) - # Figure out how to do the least amount of work while still conforming - # to the requester's boundary dates. - find_last_update - start_date.nil? ? date = created_at : date = [start_date, created_at].max - end_date.nil? ? end_date = last_update : end_date = [end_date, last_update].min - - while date <= end_date - transcript = Transcript.new(self, date) - transcript.export - - # Ensure that we stay well below the 37signals API limits. - sleep(1.0/10.0) - date = date.next - end - end - - private - def find_last_update - begin - last_message = Nokogiri::XML get("/room/#{id}/recent.xml?limit=1").body - update_utc = DateTime.parse(last_message.xpath('/messages/message[1]/created-at').text) - @last_update = Account.timezone.utc_to_local(update_utc) - rescue Exception => e - log(:error, - "couldn't get last update in #{room} (defaulting to today)", - e) - @last_update = Time.now - end - end - end - - class Transcript - include CampfireExport::IO - attr_accessor :room, :date, :xml, :messages - - def initialize(room, date) - @room = room - @date = date - end - - def transcript_path - "/room/#{room.id}/transcript/#{date.year}/#{date.mon}/#{date.mday}" - end - - def export - begin - log(:info, "#{export_dir} ... ") - @xml = Nokogiri::XML get("#{transcript_path}.xml").body - rescue Exception => e - log(:error, "transcript export for #{export_dir} failed", e) - else - @messages = xml.xpath('/messages/message').map do |message| - CampfireExport::Message.new(message, room, date) - end - - # Only export transcripts that contain at least one message. - if messages.length > 0 - log(:info, "exporting transcripts\n") - begin - FileUtils.mkdir_p export_dir - rescue Exception => e - log(:error, "Unable to create #{export_dir}", e) - else - export_xml - export_plaintext - export_html - export_uploads - end - else - log(:info, "no messages\n") - end - end - end - - def export_xml - begin - export_file(xml, 'transcript.xml') - verify_export('transcript.xml', xml.to_s.length) - rescue Exception => e - log(:error, "XML transcript export for #{export_dir} failed", e) - end - end - - def export_plaintext - begin - date_header = date.strftime('%A, %B %e, %Y').squeeze(" ") - plaintext = "#{CampfireExport::Account.subdomain.upcase} CAMPFIRE\n" - plaintext << "#{room.name}: #{date_header}\n\n" - messages.each {|message| plaintext << message.to_s } - export_file(plaintext, 'transcript.txt') - verify_export('transcript.txt', plaintext.length) - rescue Exception => e - log(:error, "Plaintext transcript export for #{export_dir} failed", e) - end - end - - def export_html - begin - transcript_html = get(transcript_path) - - # Make the upload links in the transcript clickable from the exported - # directory layout. - transcript_html.gsub!(%Q{href="/room/#{room.id}/uploads/}, - %Q{href="uploads/}) - # Likewise, make the image thumbnails embeddable from the exported - # directory layout. - transcript_html.gsub!(%Q{src="/room/#{room.id}/thumb/}, - %Q{src="thumbs/}) - - export_file(transcript_html, 'transcript.html') - verify_export('transcript.html', transcript_html.length) - rescue Exception => e - log(:error, "HTML transcript export for #{export_dir} failed", e) - end - end - - def export_uploads - messages.each do |message| - if message.is_upload? - begin - message.upload.export - rescue Exception => e - path = "#{message.upload.export_dir}/#{message.upload.filename}" - log(:error, "Upload export for #{path} failed", e) - end - end - end - end - end - - class Message - include CampfireExport::IO - attr_accessor :id, :room, :body, :type, :user, :date, :timestamp, :upload - - def initialize(message, room, date) - @id = message.xpath('id').text - @room = room - @date = date - @body = message.xpath('body').text - @type = message.xpath('type').text - - time = Time.parse message.xpath('created-at').text - localtime = CampfireExport::Account.timezone.utc_to_local(time) - @timestamp = localtime.strftime '%I:%M %p' - - no_user = ['TimestampMessage', 'SystemMessage', 'AdvertisementMessage'] - unless no_user.include?(@type) - @user = username(message.xpath('user-id').text) - end - - @upload = CampfireExport::Upload.new(self) if is_upload? - end - - def username(user_id) - @@usernames ||= {} - @@usernames[user_id] ||= begin - doc = Nokogiri::XML get("/users/#{user_id}.xml").body - rescue Exception => e - "[unknown user]" - else - # Take the first name and last initial, if there is more than one name. - name_parts = doc.xpath('/user/name').text.split - if name_parts.length > 1 - name_parts[-1] = "#{name_parts.last[0,1]}." - name_parts.join(" ") - else - name_parts[0] - end - end - end - def is_upload? - @type == 'UploadMessage' - end - - def indent(string, count) - (' ' * count) + string.gsub(/(\n+)/) { $1 + (' ' * count) } - end - - def to_s - case type - when 'EnterMessage' - "[#{user} has entered the room]\n" - when 'KickMessage', 'LeaveMessage' - "[#{user} has left the room]\n" - when 'TextMessage' - "[#{user.rjust(12)}:] #{body}\n" - when 'UploadMessage' - "[#{user} uploaded: #{body}]\n" - when 'PasteMessage' - "[" + "#{user} pasted:]".rjust(14) + "\n#{indent(body, 16)}\n" - when 'TopicChangeMessage' - "[#{user} changed the topic to: #{body}]\n" - when 'ConferenceCreatedMessage' - "[#{user} created conference: #{body}]\n" - when 'AllowGuestsMessage' - "[#{user} opened the room to guests]\n" - when 'DisallowGuestsMessage' - "[#{user} closed the room to guests]\n" - when 'LockMessage' - "[#{user} locked the room]\n" - when 'UnlockMessage' - "[#{user} unlocked the room]\n" - when 'IdleMessage' - "[#{user} became idle]\n" - when 'UnidleMessage' - "[#{user} became active]\n" - when 'TweetMessage' - "[#{user} tweeted:] #{body}\n" - when 'SoundMessage' - "[#{user} played a sound:] #{body}\n" - when 'TimestampMessage' - "--- #{timestamp} ---\n" - when 'SystemMessage' - "" - when 'AdvertisementMessage' - "" - else - log(:error, "unknown message type: #{type} - '#{body}'") - "" - end - end - end - - class Upload - include CampfireExport::IO - attr_accessor :message, :room, :date, :id, :filename, :content_type, :byte_size, :full_url - - def initialize(message) - @message = message - @room = message.room - @date = message.date - @deleted = false - end - - def deleted? - @deleted - end - - def is_image? - content_type.start_with?("image/") - end - - def upload_dir - "uploads/#{id}" - end - - # Image thumbnails are used to inline image uploads in HTML transcripts. - def thumb_dir - "thumbs/#{id}" - end - - def export - begin - log(:info, " #{message.body} ... ") +end - # Get the upload object corresponding to this message. - upload_path = "/room/#{room.id}/messages/#{message.id}/upload.xml" - upload = Nokogiri::XML get(upload_path).body - - # Get the upload itself and export it. - @id = upload.xpath('/upload/id').text - @byte_size = upload.xpath('/upload/byte-size').text.to_i - @content_type = upload.xpath('/upload/content-type').text - @filename = upload.xpath('/upload/name').text - @full_url = upload.xpath('/upload/full-url').text +require 'campfire_export/timezone' - export_content(upload_dir) - export_content(thumb_dir, path_component="thumb/#{id}", verify=false) if is_image? - - log(:info, "ok\n") - rescue CampfireExport::Exception => e - if e.code == 404 - # If the upload 404s, that should mean it was subsequently deleted. - @deleted = true - log(:info, "deleted\n") - else - raise e - end - end - end - - def export_content(content_dir, path_component=nil, verify=true) - # If the export directory name is different than the URL path component, - # the caller can define the path_component separately. - path_component ||= content_dir - - # Write uploads to a subdirectory, using the upload ID as a directory - # name to avoid overwriting multiple uploads of the same file within - # the same day (for instance, if 'Picture 1.png' is uploaded twice - # in a day, this will preserve both copies). This path pattern also - # matches the tail of the upload path in the HTML transcript, making - # it easier to make downloads functional from the HTML transcripts. - content_path = "/room/#{room.id}/#{path_component}/#{CGI.escape(filename)}" - content = get(content_path).body - FileUtils.mkdir_p(File.join(export_dir, content_dir)) - export_file(content, "#{content_dir}/#{filename}", 'wb') - verify_export("#{content_dir}/#{filename}", byte_size) if verify - end - end -end +require 'campfire_export/account' +require 'campfire_export/exception' +require 'campfire_export/message' +require 'campfire_export/room' +require 'campfire_export/transcript' +require 'campfire_export/upload' +require 'campfire_export/version' diff --git a/lib/campfire_export/account.rb b/lib/campfire_export/account.rb new file mode 100644 index 0000000..4c616b0 --- /dev/null +++ b/lib/campfire_export/account.rb @@ -0,0 +1,32 @@ +module CampfireExport + class Account + include CampfireExport::IO + include CampfireExport::TimeZone + + @subdomain = "" + @api_token = "" + @base_url = "" + @timezone = nil + + class << self + attr_accessor :subdomain, :api_token, :base_url, :timezone + end + + def initialize(subdomain, api_token) + Account.subdomain = subdomain + Account.api_token = api_token + Account.base_url = "https://#{subdomain}.campfirenow.com" + end + + def find_timezone + settings = Nokogiri::XML get('/account.xml').body + selected_zone = settings.xpath('/account/time-zone') + Account.timezone = find_tzinfo(selected_zone.text) + end + + def rooms + doc = Nokogiri::XML get('/rooms.xml').body + doc.xpath('/rooms/room').map {|room_xml| Room.new(room_xml) } + end + end +end diff --git a/lib/campfire_export/exception.rb b/lib/campfire_export/exception.rb new file mode 100644 index 0000000..a2b499f --- /dev/null +++ b/lib/campfire_export/exception.rb @@ -0,0 +1,16 @@ +module CampfireExport + class Exception < StandardError + + attr_reader :resource, :message, :code + + def initialize(resource, message, code=nil) + @resource = resource + @message = message + @code = code + end + + def to_s + "<#{resource}>: #{message}" + (code ? " (#{code})" : "") + end + end +end diff --git a/lib/campfire_export/message.rb b/lib/campfire_export/message.rb new file mode 100644 index 0000000..82a84a5 --- /dev/null +++ b/lib/campfire_export/message.rb @@ -0,0 +1,95 @@ +module CampfireExport + class Message + include CampfireExport::IO + attr_accessor :id, :room, :body, :type, :user, :date, :timestamp, :upload + + def initialize(message, room, date) + @id = message.xpath('id').text + @room = room + @date = date + @body = message.xpath('body').text + @type = message.xpath('type').text + + time = Time.parse message.xpath('created-at').text + localtime = CampfireExport::Account.timezone.utc_to_local(time) + @timestamp = localtime.strftime '%I:%M %p' + + no_user = ['TimestampMessage', 'SystemMessage', 'AdvertisementMessage'] + unless no_user.include?(@type) + @user = username(message.xpath('user-id').text) + end + + @upload = CampfireExport::Upload.new(self) if is_upload? + end + + def username(user_id) + @@usernames ||= {} + @@usernames[user_id] ||= begin + doc = Nokogiri::XML get("/users/#{user_id}.xml").body + rescue => e + "[unknown user]" + else + # Take the first name and last initial, if there is more than one name. + name_parts = doc.xpath('/user/name').text.split + if name_parts.length > 1 + name_parts[-1] = "#{name_parts.last[0,1]}." + name_parts.join(" ") + else + name_parts[0] + end + end + end + + def is_upload? + @type == 'UploadMessage' + end + + def indent(string, count) + (' ' * count) + string.gsub(/(\n+)/) { $1 + (' ' * count) } + end + + def to_s + case type + when 'EnterMessage' + "[#{user} has entered the room]\n" + when 'KickMessage', 'LeaveMessage' + "[#{user} has left the room]\n" + when 'TextMessage' + "[#{user.rjust(12)}:] #{body}\n" + when 'UploadMessage' + "[#{user} uploaded: #{body}]\n" + when 'PasteMessage' + "[" + "#{user} pasted:]".rjust(14) + "\n#{indent(body, 16)}\n" + when 'TopicChangeMessage' + "[#{user} changed the topic to: #{body}]\n" + when 'ConferenceCreatedMessage' + "[#{user} created conference: #{body}]\n" + when 'AllowGuestsMessage' + "[#{user} opened the room to guests]\n" + when 'DisallowGuestsMessage' + "[#{user} closed the room to guests]\n" + when 'LockMessage' + "[#{user} locked the room]\n" + when 'UnlockMessage' + "[#{user} unlocked the room]\n" + when 'IdleMessage' + "[#{user} became idle]\n" + when 'UnidleMessage' + "[#{user} became active]\n" + when 'TweetMessage' + "[#{user} tweeted:] #{body}\n" + when 'SoundMessage' + "[#{user} played a sound:] #{body}\n" + when 'TimestampMessage' + "--- #{timestamp} ---\n" + when 'SystemMessage' + "" + when 'AdvertisementMessage' + "" + else + log(:error, "unknown message type: #{type} - '#{body}'") + "" + end + end + end +end diff --git a/lib/campfire_export/room.rb b/lib/campfire_export/room.rb new file mode 100644 index 0000000..3457dfe --- /dev/null +++ b/lib/campfire_export/room.rb @@ -0,0 +1,44 @@ +module CampfireExport + class Room + include CampfireExport::IO + attr_accessor :id, :name, :created_at, :last_update + + def initialize(room_xml) + @id = room_xml.xpath('id').text + @name = room_xml.xpath('name').text + created_utc = DateTime.parse(room_xml.xpath('created-at').text) + @created_at = Account.timezone.utc_to_local(created_utc) + end + + def export(start_date=nil, end_date=nil) + # Figure out how to do the least amount of work while still conforming + # to the requester's boundary dates. + find_last_update + start_date.nil? ? date = created_at : date = [start_date, created_at].max + end_date.nil? ? end_date = last_update : end_date = [end_date, last_update].min + + while date <= end_date + transcript = Transcript.new(self, date) + transcript.export + + # Ensure that we stay well below the 37signals API limits. + sleep(1.0/10.0) + date = date.next + end + end + + private + def find_last_update + begin + last_message = Nokogiri::XML get("/room/#{id}/recent.xml?limit=1").body + update_utc = DateTime.parse(last_message.xpath('/messages/message[1]/created-at').text) + @last_update = Account.timezone.utc_to_local(update_utc) + rescue => e + log(:error, + "couldn't get last update in #{name} (defaulting to today)", + e) + @last_update = Time.now + end + end + end +end diff --git a/lib/campfire_export/timezone.rb b/lib/campfire_export/timezone.rb index 6415d28..aa6156e 100644 --- a/lib/campfire_export/timezone.rb +++ b/lib/campfire_export/timezone.rb @@ -9,7 +9,7 @@ module CampfireExport # https://github.com/rails/rails/blob/master/activesupport/lib/active_support/values/time_zone.rb # I'm copying it here to avoid bugs in the current active_support gem, to # avoid having a dependency on active_support that might freak out Rails - # users, and to avoid fighting with RubyGems about threads and deprecation. + # users, and to avoid fighting with RubyGems about threads and deprecation. # See for background: # https://github.com/rails/rails/pull/1215 # http://stackoverflow.com/questions/5176782/uninitialized-constant-activesupportdependenciesmutex-nameerror @@ -160,7 +160,7 @@ module TimeZone "Nuku'alofa" => "Pacific/Tongatapu" }.each { |name, zone| name.freeze; zone.freeze } MAPPING.freeze - + def find_tzinfo(name) TZInfo::Timezone.get(MAPPING[name] || name) end diff --git a/lib/campfire_export/transcript.rb b/lib/campfire_export/transcript.rb new file mode 100644 index 0000000..e35fb98 --- /dev/null +++ b/lib/campfire_export/transcript.rb @@ -0,0 +1,100 @@ +module CampfireExport + class Transcript + include CampfireExport::IO + attr_accessor :room, :date, :xml, :messages + + def initialize(room, date) + @room = room + @date = date + end + + def transcript_path + "/room/#{room.id}/transcript/#{date.year}/#{date.mon}/#{date.mday}" + end + + def export + begin + log(:info, "#{export_dir} ... ") + @xml = Nokogiri::XML get("#{transcript_path}.xml").body + rescue => e + log(:error, "transcript export for #{export_dir} failed", e) + else + @messages = xml.xpath('/messages/message').map do |message| + CampfireExport::Message.new(message, room, date) + end + + # Only export transcripts that contain at least one message. + if messages.length > 0 + log(:info, "exporting transcripts\n") + begin + FileUtils.mkdir_p export_dir + rescue => e + log(:error, "Unable to create #{export_dir}", e) + else + export_xml + export_plaintext + export_html + export_uploads + end + else + log(:info, "no messages\n") + end + end + end + + def export_xml + begin + export_file(xml, 'transcript.xml') + verify_export('transcript.xml', xml.to_s.bytesize) + rescue => e + log(:error, "XML transcript export for #{export_dir} failed", e) + end + end + + def export_plaintext + begin + date_header = date.strftime('%A, %B %e, %Y').squeeze(" ") + plaintext = "#{CampfireExport::Account.subdomain.upcase} CAMPFIRE\n" + plaintext << "#{room.name}: #{date_header}\n\n" + messages.each {|message| plaintext << message.to_s } + export_file(plaintext, 'transcript.txt') + verify_export('transcript.txt', plaintext.bytesize) + rescue => e + log(:error, "Plaintext transcript export for #{export_dir} failed", e) + end + end + + def export_html + begin + transcript_html = get(transcript_path).to_s + + # Make the upload links in the transcript clickable from the exported + # directory layout. + transcript_html.gsub!(%Q{href="/room/#{room.id}/uploads/}, + %Q{href="uploads/}) + # Likewise, make the image thumbnails embeddable from the exported + # directory layout. + transcript_html.gsub!(%Q{src="/room/#{room.id}/thumb/}, + %Q{src="thumbs/}) + + export_file(transcript_html, 'transcript.html') + verify_export('transcript.html', transcript_html.bytesize) + rescue => e + log(:error, "HTML transcript export for #{export_dir} failed", e) + end + end + + def export_uploads + messages.each do |message| + if message.is_upload? + begin + message.upload.export + rescue => e + path = "#{message.upload.export_dir}/#{message.upload.filename}" + log(:error, "Upload export for #{path} failed", e) + end + end + end + end + end +end diff --git a/lib/campfire_export/upload.rb b/lib/campfire_export/upload.rb new file mode 100644 index 0000000..48865ab --- /dev/null +++ b/lib/campfire_export/upload.rb @@ -0,0 +1,78 @@ +module CampfireExport + class Upload + include CampfireExport::IO + attr_accessor :message, :room, :date, :id, :filename, :content_type, :byte_size, :full_url + + def initialize(message) + @message = message + @room = message.room + @date = message.date + @deleted = false + end + + def deleted? + @deleted + end + + def is_image? + content_type.start_with?("image/") + end + + def upload_dir + "uploads/#{id}" + end + + # Image thumbnails are used to inline image uploads in HTML transcripts. + def thumb_dir + "thumbs/#{id}" + end + + def export + begin + log(:info, " #{message.body} ... ") + + # Get the upload object corresponding to this message. + upload_path = "/room/#{room.id}/messages/#{message.id}/upload.xml" + upload = Nokogiri::XML get(upload_path).body + + # Get the upload itself and export it. + @id = upload.xpath('/upload/id').text + @byte_size = upload.xpath('/upload/byte-size').text.to_i + @content_type = upload.xpath('/upload/content-type').text + @filename = upload.xpath('/upload/name').text + @full_url = upload.xpath('/upload/full-url').text + + export_content(upload_dir) + export_content(thumb_dir, path_component="thumb/#{id}", verify=false) if is_image? + + log(:info, "ok\n") + rescue CampfireExport::Exception => e + if e.code == 404 + # If the upload 404s, that should mean it was subsequently deleted. + @deleted = true + log(:info, "deleted\n") + else + raise e + end + end + end + + def export_content(content_dir, path_component=nil, verify=true) + # If the export directory name is different than the URL path component, + # the caller can define the path_component separately. + path_component ||= content_dir + + # Write uploads to a subdirectory, using the upload ID as a directory + # name to avoid overwriting multiple uploads of the same file within + # the same day (for instance, if 'Picture 1.png' is uploaded twice + # in a day, this will preserve both copies). This path pattern also + # matches the tail of the upload path in the HTML transcript, making + # it easier to make downloads functional from the HTML transcripts. + content_path = "/room/#{room.id}/#{path_component}/#{CGI.escape(filename)}" + content = get(content_path).body + FileUtils.mkdir_p(File.join(export_dir, content_dir)) + export_file(content, "#{content_dir}/#{filename}", 'wb') + verify_export("#{content_dir}/#{filename}", byte_size) if verify + end + end +end diff --git a/lib/campfire_export/version.rb b/lib/campfire_export/version.rb index 7424a29..15ed97e 100644 --- a/lib/campfire_export/version.rb +++ b/lib/campfire_export/version.rb @@ -1,3 +1,3 @@ module CampfireExport - VERSION = "0.3.0" + VERSION = "0.5.1" end diff --git a/spec/campfire_export/account_spec.rb b/spec/campfire_export/account_spec.rb index d6a6e0d..54cc2e3 100644 --- a/spec/campfire_export/account_spec.rb +++ b/spec/campfire_export/account_spec.rb @@ -1,5 +1,4 @@ -require 'campfire_export' -require 'tzinfo' +require 'spec_helper' module CampfireExport describe Account do @@ -7,7 +6,7 @@ module CampfireExport @subdomain = "test-subdomain" @api_token = "test-apikey" @account = Account.new(@subdomain, @api_token) - + @good_timezone = '' + '' + ' America/Los_Angeles' + @@ -21,12 +20,12 @@ module CampfireExport ' 999999' + '' - @bad_timezone = @good_timezone.gsub('America/Los_Angeles', + @bad_timezone = @good_timezone.gsub('America/Los_Angeles', 'No Such Timezone') - @account_xml = stub("Account XML") + @account_xml = double("Account XML") @account_xml.stub(:body).and_return(@good_timezone) end - + context "when it is created" do it "sets up the account config variables" do Account.subdomain.should equal(@subdomain) @@ -34,15 +33,15 @@ module CampfireExport Account.base_url.should == "https://#{@subdomain}.campfirenow.com" end end - + context "when timezone is loaded" do it "determines the user's timezone" do @account.should_receive(:get).with("/account.xml" ).and_return(@account_xml) @account.find_timezone - Account.timezone.to_s.should == "America - Los Angeles" + Account.timezone.to_s.should == "America - Los Angeles" end - + it "raises an error if it gets a bad time zone identifier" do @account_xml.stub(:body).and_return(@bad_timezone) @account.stub(:get).with("/account.xml" @@ -51,32 +50,31 @@ module CampfireExport @account.find_timezone }.to raise_error(TZInfo::InvalidTimezoneIdentifier) end - + it "raises an error if it can't get the account settings at all" do @account.stub(:get).with("/account.xml" - ).and_raise(CampfireExport::Exception.new("/account/settings", + ).and_raise(CampfireExport::Exception.new("/account/settings", "Not Found", 404)) expect { @account.find_timezone }.to raise_error(CampfireExport::Exception) end end - + context "when rooms are requested" do it "returns an array of rooms" do room_xml = "123" - room_doc = mock("room doc") + room_doc = double("room doc") room_doc.should_receive(:body).and_return(room_xml) @account.should_receive(:get).with('/rooms.xml').and_return(room_doc) - room = mock("room") + room = double("room") Room.should_receive(:new).exactly(3).times.and_return(room) - @account.rooms.should have(3).items + expect(@account.rooms.size).to eq(3) end it "raises an error if it can't get the room list" do - @account.stub(:get).with('/rooms.xml' - ).and_raise(CampfireExport::Exception.new('/rooms.xml', - "Not Found", 404)) + allow(@account).to receive(:get).with('/rooms.xml') { raise(CampfireExport::Exception.new('/rooms.xml', "Not Found", 404)) } + expect { @account.rooms }.to raise_error(CampfireExport::Exception) diff --git a/spec/campfire_export/message_spec.rb b/spec/campfire_export/message_spec.rb index 682603a..d252a52 100644 --- a/spec/campfire_export/message_spec.rb +++ b/spec/campfire_export/message_spec.rb @@ -1,12 +1,9 @@ -require 'campfire_export' -require 'campfire_export/timezone' +require 'spec_helper' -require 'nokogiri' - -module CampfireExport +module CampfireExport describe Message do include TimeZone - + before :each do @messages = Nokogiri::XML < @@ -44,7 +41,7 @@ module CampfireExport XML Account.timezone = find_tzinfo("America/Los_Angeles") end - + context "when it is created" do it "sets up basic properties" do message = Message.new(@messages.xpath('/messages/message[3]')[0], nil, nil) diff --git a/spec/campfire_export/room_spec.rb b/spec/campfire_export/room_spec.rb index 8667040..0158ade 100644 --- a/spec/campfire_export/room_spec.rb +++ b/spec/campfire_export/room_spec.rb @@ -1,19 +1,16 @@ -require 'campfire_export' -require 'campfire_export/timezone' +require 'spec_helper' -require 'nokogiri' - -module CampfireExport +module CampfireExport describe Room do include TimeZone - + before :each do doc = Nokogiri::XML "Test Room666" + "2009-11-17T19:41:38Z" - @room_xml = doc.xpath('/room') + @room_xml = doc.xpath('/room') Account.timezone = find_tzinfo("America/Los_Angeles") end - + context "when it is created" do it "sets up basic properties" do room = Room.new(@room_xml) @@ -22,9 +19,5 @@ module CampfireExport room.created_at.should == DateTime.parse("2009-11-17T11:41:38Z") end end - - context "when it finds the last update" do - it "loads the last update from the most recent message" - end end end diff --git a/spec/spec_helper.rb b/spec/spec_helper.rb new file mode 100644 index 0000000..1b3abf4 --- /dev/null +++ b/spec/spec_helper.rb @@ -0,0 +1,4 @@ +require 'rspec' + +require 'campfire_export' +require 'tzinfo'