Capistrano path and environment variable issues

We have been using Capistrano for awhile now and it has been working great. Recently, for reasons we are not sure of, migrations started failing. When we executed ‘cap deploy:migrate’, we would get an error “no such file to load — rcov/rcovtask”. After a lot of investigation, I found that this meant that the system, for whatever reason, couldn’t find our gems anymore.
Since Capistrano hardly gives any information when failing, it took awhile to figure out what was going on. sshing in as our Capistrano user, and running printenv, showed that the environment variables for the Capistrano user were all set as expected. After some investigation, we learned that since Capistrano runs a non-interactive user, the environment isn’t set up the same. To figure out the environment that Capistrano was running as, I created a task and added it to config/deploy.rb

desc "Echo environment vars"
namespace :env do
task :echo do
run "echo printing out cap info on remote server"
run "echo $PATH"
run "printenv"
end
end

After saving your deploy.rb file you should be able to execute that command on your servers by running “cap env:echo”.
This printed out a different path than sshing in as our Cap user and also was missing the RUBYOPT=rubygems environment variable. Obviously, not having RUBYOPT set was causing the problem of not finding out gems.
So next, we just had to set up the correct environment for the non-interactive Cap user. This was a little tricky since non-interactive users don’t load all the environment settings like normal users. This is done for extra security. Unfortunately, it means it is a little more annoying to set up these variables, and not as well documented.
If you need to add environment variables or change the PATH for a non-interactive user, you need to edit two files. First ssh in as the user under which the non-interactive scripts will be run. Then, in that user’s home directory, create a file ~/.ssh/environment (in our case /home/cap/.ssh/environment). Edit this file and add whatever environment variables you need (in our case RUBYOPT=rubygems), and save the file. Then edit /etc/ssh/sshd_config, and add the following line
PermitUserEnvironment yes
I would run ‘man sshd_config’ to make sure your version of sshd supports this option, as some older versions and distros (we are using Debian) do not. Then restart ssh by running “sudo /etc/init.d/ssh restart”. At this point, you should try running “cap env:echo” again, which should print the RUBYOPTS environment variable set correctly. This fixed our problems and migrations began running again without a problem.

Capistrano and PATH

Lately, we’ve been working on getting Capistrano set up so we can have easy, predictable deployments. If you’re not using Capistrano (or some other automated deployment solution), I highly suggest looking into it. It’s a no-brainer like automated tests or source control management. However, if you’re not using it, this post won’t be of any interest to you – it’s simply a little gotcha that we ran into.
One component of Capistrano that you need to provide is a so-called spinner script. This is the script that actually starts your app. In our case, it starts a Mongrel cluster like so:
./script/process/spawner -a 127.0.0.1 -p 8000 -i 3
This starts three instances of Mongrel (on ports 8000,8001, and 8002) and binds them to localhost.
When we would run Capistrano, we would see the following output:

...
...
* executing `deploy:start'
* executing "sh -c 'cd /var/www/seekler/current && nohup script/spin'"
servers: ["pretheory.com"]

But the actual Mongrel servers would not be running (i.e. ps aux | grep mongrel showed nothing). We would ssh into the server as the Capistrano user and run the same command and everything would work fine. Huh?
The problem turned out to be that spawner calls mongrel_rails (the program that actually starts the Mongrel instances) and that program was not in the PATH when Capistrano executed this code (although it was the in PATH if we remotely logged in and ran the spin script manually. Sudo on Debian can play some weird tricks with your PATH).
Compounding the problem was that the spawner code silently fails if it can’t find mongrel_rails – which made it much harder to diagnose the issue.
We solved the problem by creating a symbolic link to mongrel_rails into /usr/local/bin
ln -s /var/lib/gems/1.8/bin/mongrel_rails /usr/local/bin
and then everything worked.
However, your PATH may be different. To find out, temporarily edit spawner.rb (the path on my machine was /var/lib/gems/1.8/gems/rails-1.2.3/lib/commands/process/spawner.rb) and add the following line at the beginning
system('env') # DON'T LEAVE THIS IN! REMOVE WHEN YOU FIGURE THE PATH OUT!
That will print out all the environment variables as the script is run, including the PATH. Then you can create a symlink to mongrel_rails in the appropriate directory.
Hope that helps!

Well, that sucked

Last week was all kinds of fun. I had planned on refactoring a lot of code while Danny took a trip to meet his girlfriend in Honduras (she’s been volunteering there). But, just my luck, a mere twelve or so hours after he left, disaster struck.
For the past few years, Danny has hosted a number of sites on a Windows 2003 server hosted by ServerBeach. It’s gotten pretty slow with all the stuff we’ve had on it and we decided Linux would be a lot better for doing real hosting, so we decided about a month ago to switch to a Linux server. Unfortunately, at the time we were very busy with other stuff, so we didn’t get to it.
Well, fate took matters into its own hands. Just after Danny left, the Windows server totally ate it. The data was there, but booting was not happening. To make things just a little more fun, I knew practically nothing about how the server was set up, and all the accounts and passwords were under Danny’s name. Oh, and since phone calls to Honduras aren’t exactly cheap (and Danny had almost zero access to the internet), we could only communicate via text messages. Good times.
So I spent around four days 1) purchasing and setting up a new Linux server b) figuring out what data needed to be moved from the old Windows box and c) figuring out which sites were down and replicating them all on the new server. I’m not exactly an expert Windows or Linux sysadmin, so I was pretty much learning everything on the fly – while the clock was ticking since the sites were all down for a few days straight.
It was stressing and frustrating, since I didn’t accomplish what I had planned to, but on the plus side, we now have a brand-spanking new, way-faster Linux server up and running, which is nice.
Lessons learned:
1. As much as possible, have company accounts for servers and other services. Otherwise it’s a real pain if one person is unavailable.
2. Apparently, just installing automatic updates on W2K3 can bring a machine to it’s knees.
3. Debian is insanely easy to set up (apt-get rules!)
4. Writing code is a lot more satisfying than playing sysadmin.
Bonus Movable Type tip:
I decided to upgrade to a newer version of Movable Type, since our old Windows server was running 3.1. After importing the data from the database, the old passwords would not work. I tried looking up the encryption code MT uses and manually setting the fields in the DB, but no dice.
Then I tried resetting the passwords via MT itself. MT will mail you a new password if you know a user’s secret phrase. Luckily, that phrase is kept in plaintext in the DB, but unfortunately, I didn’t have sendmail or smtp working. So, I included the following line in my mt-config.cgi
MailTransfer debug
and then checked the Apache logs. Sure enough, the mail showed up in the logs – but MT wouldn’t accept the new generated passwords (even though it was changing the password field in the DB!).
I spent a good two hours banging my head against the wall trying to figure out what was going on. I finally just started looking around at the MT cgi scripts and trying various ones. Eventually I stumbled upon db2sql.cgi. Bingo! This fixed everything, which I don’t really understand, since my old DB was SQL, but whatever. Give it a shot if you’re having the same problem.

Capturing your startup-to-be

Wikis are brilliant. It’s one of those amazing technologies that is both so ingenious that it fundamentally changes age-old institutions (specifically, encyclopedias, but more generally, replacing expert knowledge with the wisdom of crowds) and is also so brain-dead simple that pretty much anyone can start using them within five minutes. If only more software could be that powerful and elegant.
Not surprisingly, wikis are great for startups. Here at Pretheory, we’ve been using our wiki* a lot recently for collecting technical howtos, links to business articles, task lists, product ideas, and a bunch of other stuff.
As great as the wiki is now, it doesn’t even compare to how useful it was in the twelve months before we officially got started. Without a doubt, the wiki was most important tool for making Pretheory a reality.
The most obvious benefit of a wiki is letting more than one person edit a set of documents, particularly if those people live far apart from each other. In our case, the wiki let my co-founder and I collaborate even though we were 1300+ miles away from each other. We could post business ideas, company names, books to read, funding ideas, and the pros/cons of starting up in Seattle vs. Denver – even when we were on different work schedules and couldn’t talk that often.
But collaboration over distance wasn’t the greatest benefit. You can share ideas remotely over email too. The wiki was great because it kept all our ideas in one place, let us link ideas that related, and even let us edit with confidence, since we knew the wiki would track all of our edits.
Keeping ideas together isn’t just convenient, it’s an great progress indicator. If our ideas were only contained within various emails and phone conversations, it would be hard to get a feel for how far along we were. But with a wiki, I could just browse around and quickly get an idea of what we had done and what we still needed to do. For that first year, the wiki was our startup, and we could watch it grow before our eyes. And when we thought we were ready, we had the documentation at our fingertips.
Even if you’re just considering starting a company sometime in the future, do yourself a favor. Create a startup wiki. Do it today. Put every good, bad, and ugly idea you have on it, and keep updating it. At worst, you’ll spend a little time and have a record of ideas for possible future use. At best, in a few months or years you’ll look at your wiki and realize you’ve what you need to take the leap.
—————————
* We use Instiki because its ridiculously easy to set up and it’s written in Ruby so we can easily hack on it if we need to.

LastTab: Making tabbed browsing safe for (Alt|Cmd)-Tabbers

I’m a big fan of Alt-Tab on Windows (or, if you will, Cmd-Tab on OS X). But I’m also a big fan of tabbed browsing in Firefox. Until recently, these two passions collided head-on, since Ctrl-Tab in Firefox simply selects the tab to the right of the currently selected tab. No fun.
Enter LastTab: an excellent plugin for Firefox that makes Ctrl+Tab behave like (Alt|Cmd)-Tab. My productivity while working in Firefox has skyrocketed. Check it out.