Feb 1
Solr 1.4 Upgrade – Out with Faceting, In with MultiCore
I finally found some time, though it was a bit forced, to look into the new CoreAdmin functionality released in Solr 1.4. Our app was using the same index for multiple apps distinguished through Faceting. This worked fine for a while, but the larger our indexed data got, the slower the queries were returned. The responsiveness of every app was now dependent on the total indexed size. This was not ideal, so we upgraded to 1.4 to see how using multiple cores could help.
Upgrading was quite easy. See http://wiki.apache.org/solr/Solr1.4 for upgrading notes. What I will focus on are the details of using and automating the MultiCore functionality. To enable the dynamic SolrCore functionality, you just need to put the following XML in solr.xml inside your solr_home directory.
<solr persistent="true" sharedLib="lib"> <cores adminPath="/admin/cores"> </cores> </solr>
Don’t worry about creating any cores at this time. The next thing I did was create a directory called “default” in my solr_home directory to store all of my shared configs for each core. I then copied the conf/ directory from my solr_home into the newly created default dir.
The final step is to create the cores needed. You can automate this by posting the following request and checking the XML response, or by just pasting this into your browser.
http://{host}:{port}/{solr_home}/admin/cores?action=CREATE&name={core_name}&instanceDir=default&dataDir={/full/path/to/solr/home}/{core_name}
Here is how you automate the request using xml-simple and ruby
#set the url string equal to the above command response = Net::HTTP.get_response(URI.parse(url)).body xml = XmlSimple.xml_in(response, {'VarAttr' => 'name', 'ContentKey' => '-content', 'KeyAttr' => 'name'}) raise "Failed to create solr core" if xml["lst"]["responseHeader"]["int"]["status"].to_i != 0
No comments
Oct 15
iPhone 3g 3.1.2 jailbreak update PwnageTool
I updated my jailbroken iPhone to 3.1 when it first came out and had some major issues that required almost a daily hard reboot. I successfully downgraded it to 3.0.1, but still had some baseband issues with the service (phone and sms) not working. I was very excited to see the iPhone Dev Team came out with a 3.1.2 jailbreak using PwnageTool 3.1.4
Since I was already jailbroken, I simply built the custom IPSW, and clicked Option+Restore in iTunes. Everything went as expected, and I finally have a stable phone again.
I’ve been jailbreaking my phone since I first got it over a year ago, and I’m so glad to see the process getting easier and smoother every time. Big thanks to the iPhone Dev Team and all their hard work.
Here are the steps I took in case you are already jailbroken and just want the bug fixes and 3.1 update.
- Requirements
- Mac OS X
- iTunes 9.0.1
- Already jailbroken iPhone 3G on 3.0 firmware
- Backups!
- Steps
- Check for updates in iTunes
- Choose to download only for the update
- Download PwnageTool 3.1.4 Torrent Mirror
- Launch PwnageTool and go walk through the steps to create your custom IPSW
- Answer Yes to already jailbroken and exit the app when prompted
- Launch iTunes and hold the Option key and click Restore
- Select your custom IPSW file (probably on your destop)
- Enjoy your new 3.1.2 jailbreak
Jul 31
Aggregation with MySQL and ActiveRecord
I had some data that I wanted to aggregate by year for a flash graphing app called amCharts. Here was the SQL query I used to gather the data:
SELECT AVG(column1) AS column1, AVG(column2) AS column2 FROM table1 t1 JOIN table2 t2 ON t2.id = t1.tabel2_id GROUP BY YEAR(t2.datetime_field)
Rather than using this sql in ActiveRecord via find_by_sql, I figured I’d make it pretty and use the built in find method. Here is what I came up with:
Table1.find(:all, :select => 't2_id, AVG(column1) as column1, AVG(column2) as column2', :joins => :t2, :group => 'YEAR(t2.datetime_field)')
The :select option allows you to define the attributes you want returned with the AR object. I’m using MySQL to do the averaging of the data via the AVG function. I would normally use the :include option for joins, but I couldn’t get it to work with this query, so the :joins option seemed to work well by passing it the association definition from the model. The :group option simply works the same as the SQL group by. Just pass it a sql fragment, in my case just the year field portion of the datetime field.
No commentsJun 25
RJS redirect_to params contain &
I came across this today when I added a parameter to a RJS page.redirect_to call. It appears that prototype or rails is not properly handling the parameters when there are more than 2. I got around it by using a defined route in the routes.rb that looks like this:
map.route_name 'controller_name/action_name/:foo_id', :controller => 'controller_name', :action => 'action_name'
I had to change the redirect_to in my RJS file from this:
page.redirect_to page_recrawl_path(:foo_id => @foo_id, :bar_id => @bar_id, :bang => true)
To this:
page.redirect_to page_recrawl_path(@foo_id, {:bar_id => @bar_id, :bang => true})
Jun 20
Jailbroken iPhone 3g 3.0 firmware
Thanks to the iPhone Dev-Team for releasing the Pwnage Tool 3.0, I was able to upgrade to the jailbroken 3.0 firmware. The install process went very smoothly. I followed a similar procedure to what they demo’d on their youtube video. Some of my favorite jailbroken apps don’t work with the new firmware. Here are my findings:
Apps
- Cycorder: Yes
- OpenSSH: Yes
- SBS Settings: Yes
- Icy: Yes
- Cydia: Stability issues
- PdaNet: Yes, but possible stability issues
- Winterboard: Mostly, some themes weren’t stable
- Pwnplayer: no
- Snapture: no
- NES: no
Overall the new 3.0 features were well worth upgrading. The only app I will really miss is Pwn player and I hope there will be a similar replacement (it’s no longer maintained). As for the other apps, I’m sure they’ll be getting updates soon. Please let me know what apps you have working.
1 commentJun 2
Ruby on Rails background process: MySQL server has gone away
Using workling and starling to handle background tasks is very nice. Check out this link for info on setting it up. However, I encountered the error: “Mysql::Error: MySQL server has gone away” when accessing an external DB tables like users and roles. I found that someone recommended using verify_active_connections!, however the table the connection error occurred on was a HABTM table ‘user_roles’. I found that verify_active_connections will use the scope of the model/object you invoke it on. So the final solution was to do the following:
user.roles.verify_active_connections!
This will make sure users, user_roles, and roles all have valid connections. This error occurred using rails 2.0.1. It may be fixed in newer versions of ActiveRecord.
No commentsApr 3
Passing around arguments in rake tasks
So I had the need to pass arguments from one rake task to another. This is easy to do from the command line, but gets a bit tricking when its being called from within another rake task. So here is the task that is being called:
1 namespace :database do 2 desc %{Dump DB to create restore data in case of error} 3 task :dump, :tables, :needs => [:setup] do |t, args| 4 filename = "#{DB_NAME}#{'-'+args.tables.values.join('-') unless args.tables.nil?}-#{DB_DUMP_TIME}.sql.gz" 5 puts "Dumping the database to '#{filename}'" 6 7 cmd = "" 8 cmd << "echo 'SET AUTOCOMMIT=0;'|gzip> #{filename};" 9 cmd << "echo 'SET FOREIGN_KEY_CHECKS=0;'|gzip>> #{filename};" 10 cmd << "mysqldump -u #{DB_USER} -p'#{DB_PASSWORD}' " 11 cmd << "-h #{DB_HOST} #{DB_NAME} " 12 cmd << "--tables #{args.tables.values.join(' ')} " unless args.tables.nil? 13 cmd << "|gzip>> #{filename};" 14 cmd << "echo 'SET FOREIGN_KEY_CHECKS=1;'|gzip>> #{filename};" 15 cmd << "echo 'COMMIT;'|gzip>> #{filename};" 16 cmd << "echo 'SET AUTOCOMMIT=1;'|gzip>> #{filename};" 17 18 if system(cmd) 19 puts "Successfully ran databse:dump to restore run:" 20 puts "rake database:reload DB_DUMP_TIME='#{DB_DUMP_TIME}'" 21 else 22 puts "rake database:dump was unsuccessful" 23 end 24 end
So this task is used to dump the database for our app. I added the ability to pass in an optional hash to allow it to only dump the specified tables. Here are the important lines to note.
task :dump, :tables, :needs => [:setup] do |t, args|
The “tables” argument specifies what arguments are being passed in. The “t” and “args” block defines the task and the arguments. Your “tables” hash is now available through “args” values.
Now when I call this task from within another rake task, I can do something like this.
tables = {:table1 => 'users', :table2 => 'roles'} Rake::Task["database:dump"].invoke(tables)
Note, I define the hash outside of the invoke because it is used by two other tasks, but it could be directly passed to the task as an argument.
No commentsNov 15
Ubuntu Intrepid Sound Juicer MP3 support
After upgrading to Intrepid Kaudiocreator was no longer available, so I decided to give Sound Juicer a try. When I went to the preferences to change the encoding to MP3, I found that it was missing. So I clicked edit and found that the option was there for MP3 output. Turns out the option will not show up until you install this package – gstreamer0.10-plugins-ugly-multiverse. So just type the following and your set.
sudo apt-get install gstreamer0.10-plugins-ugly-multiverse
Nov 6
Rails, Time Zones, and Scheduling
The time zone support in rails has vastly improved since the integrated TZInfo into Rails 2.1. However if you have any scheduled tasks stored in your DB, you’ll notice it quickly breaks down. Here is a short and sweet explanation:
I have an app that delivers system notification to users at a specified time they set. I have a field in my “reminders” table called “deliver_at” which is a datetime field. The data is stored as a UTC timestamp. Everything works as expected until daylight savings time, or the user changes their time zone. This is because the “deliver_at” timestamp has no context of what offset was used when the record was created. So when daylight saving rolls around your messages are now delivered an hour off. Or if the user moves to a new timezone, it will be off by x hours. Here is the solution I came up with.
I added a field to the “reminders” table called “offset”.
add_column :reminders, :offset, :decimal, :precision =>; 4, :scale => 2, :null =>; false, :default => 0.0
This field is recorded when the record is first created. You can override the attribute using write_attribute and read_attribute, but I was already doing a seporate update call, so I just stuck it there. *Important, make sure you are setting the Time Zone on login.
self.update_attributes!(:delivered_at => Time.zone.now, :next_reminder_at => date, :offset => user_utc_offset)
Here is my user_utc_offset method:
def user_utc_offset TimeZone[self.user.timezone].utc_offset.to_f / 1.hour.to_f end
I’m using floats because not all the offsets are whole numbers. Same reason I used a decimal field in the DB. Then when I calculate my next reminder delivery, I basically do this:
offset_adjustment = adjust_for_difference_in_offsets case self.reminder.repeat when "daily" next_reminder = reminder.deliver_at + 1.day - offset_adjustment.hours when "weekly" next_reminder = reminder.deliver_at + 7.days - offset_adjustment.hours when "monthly" next_reminder = reminder.deliver_at + 1.month - offset_adjustment.hours end
I have a case statement to calculate daily, weekly, and monthly reminders. Here is the method to calculate the offset difference.
def adjust_for_difference_in_offsets user_utc_offset - self.offset.to_f end
I use the above for the mailer, but we need to do the same for displaying the datetime stamp to the user. So I created a helper method that does like so:
def reminder_repeat_datetime(reminder) offset_adjustment = reminder.adjust_for_difference_in_offsets case reminder.repeat when "monthly" repeat = "#{reminder.deliver_at.strftime('%d')}th" when "weekly" repeat = "#{reminder.deliver_at.strftime('%A')}s" when "daily" repeat = "Daily" when "once" repeat = "#{reminder.deliver_at.strftime('%B %d, %Y')}" end "#{repeat} at #{(reminder.deliver_at - offset_adjustment.hours).strftime('%I:%M %p')}" end
A lot of the above is just pretty formatting of the datetime for the user. You could most likely just get away with something like the last line. That is called from my index view in my reminders controller like so:
<%= reminder_repeat_datetime(reminder) %>
So now no matter if the offset changes because of DST or the user changes their timezone, the offset will be calculated and applied to the datetime field.
No commentsNov 4
My simple ruby backup
After using a Linux dev laptop for almost a year, I went back to a MacBook Pro for work. I had forgotten the pains of finding a good backup solution for a Mac. I needed something that would incrementally back up my work laptop and my home Linux box to the same external hard drive. After trying various solutions(Time Machine, CCC, iBackup, etc), they all seemed to either not work at all or hog up way too much room on the drive. So I decided to just write my own backup solution using Ruby and Rsync.
This initial release is very basic and only requires you have ruby and rsync installed. It supports Mac or Linux and can back up to any drive that supports hard links (ext2, ext3, hfs, hfs+). Future releases will include integration with s3 as well as Cron scheduling. For now I just wanted to share this with others having similar gripes.
Steps to getting up and running:
1. Download Ruby Backup Zip or Ruby Backup tarball.
2. Extract the contents of the above package.
3. Edit the backup_settings.yml.sample file and save it as backup_settings.yml
4. run the script via “ruby ruby_backup.rb” or “./ruby_backup.rb”

