Archive for the ‘Development’ Category

Finding and Fixing Strange Rails Bugs (or, How I Wasted My Afternoon)

March 13, 2008

Rails is a great framework, but sometimes bugs can be incredibly tricky to track down. Case in point, I spent a significant portion of yesterday tracking down a very strange bug. I’ll post the resolution – but perhaps of more general interest, I’ll outline some tips that you can use when tracking down Rails issues (or really, any issues). These tips may be obvious, but sometimes in my desire to quickly fix a bug, I forget the most basic rules of debugging.
Short Story: Ruby Classifier requires ‘mathn’. As it turns out, requiring ‘mathn’ breaks redirect behavior in Rails. Check out Rails ticket 5433 for more info and a workaround.
(Very) Long Story:
The initial behavior I noticed was that one integration test was failing. The integration test was doing the following

post_via_redirect '/account/login', options
assert_response :success

And was failing with the following error:
Expected response to be a , but was <302>
Not so weird in itself, but the bug only appeared when I ran the tests using autotest. When I ran the test via rake test:integration, everything worked. I looked at the test and the controller and everything seemed normal (and I hadn’t changed anything in the test or in the controller lately), so I started to dig in deeper.
Tip 1: Isolate the changes. My first step was to use svn to track down the exact revision that had caused the bug (I hadn’t caught the bug before checking in because I had been testing using rake, not autotest). Luckily, this is pretty simple – just keep performing svn update -r REVISION_NUM and running the test until you find the revision that causes the test to fail (I wish I had a tool to do this automatically…).
However, in my laziness, I’d forgotten to …
Tip 2: Minimize the Repro. This really should be tip 1, but I’m telling this story chronologically, so you can learn from my mistakes. As I said, this bug only reproed when using autotest. Each time I wanted to test for the bug, I had to wait almost a minute for autotest to run the tests. Due to this long repro, I ended up spending probably ten times the amount of time I needed to figure out which checkin caused the bug.
Usually, when a test fails, you can just use ruby to run a single test. But if you ever have a bug that only seems to repro in autotest, remember that autotest prints out all the commands it runs. In this case, it was simply a case of copying and pasting the code (which ran a ton of test files) from the output of autotest onto the command line. It looked something like this:

ruby -I.:lib:test -rtest/unit
-e "%w[test/functional/resources_controller_test.rb
test/functional/account_controller_test.rb test/unit/merge_test.rb
test/unit/render_state_test.rb test/integration/filter_clears_test.rb
test/unit/item_test.rb test/functional/admin_items_controller_test.rb
test/unit/comment_test.rb test/unit/lists_helper_test.rb
test/functional/lists_controller_test.rb test/functional/items_controller_test.rb
test/unit/list_test.rb test/unit/memory_list_test.rb
test/integration/contact_form_test.rb test/unit/xpath_test.rb
test/functional/contact_controller_test.rb test/unit/resource_test.rb
test/unit/membership_test.rb test/unit/list_classifier_test.rb
test/functional/listgroups_controller_test.rb test/integration/resources_test.rb].each
{ |f| require f }" | unit_diff -u

Yikes. That successfully reproed the bug, but it took way too long. The next step was manually pulling out test files until the bug stopped reproing. That left me with a simpler (and much faster) repro:
ruby -I.:lib:test -rtest/unit -e "%w[test/unit/list_classifier_test.rb test/integration/resources_test.rb].each { |f| require f }"
OK, so it seemed that there is some weird interaction between list_classifier_test and resources_test.
Tip 3: Simplify, Simplify This is really an extension of tip 2, but I’m repeating it because most of debugging is systematically simplifying the problem by removing complicating factors. So, I went to list_classifier and commented out all the tests. The bug disappeared. I put a single test back in – the bug reappeared.
Now I was onto something. I then realized that loading the file under test (list_classifier.rb) was the problem. That code is based on the awesome Ruby Classifier library. As a result, the file begins with require 'classifier'. Interesting. So I tried just requiring ‘classifier’ in the integration test. Bingo. Somehow, using the Classifier library was breaking my integration test.
Tip 4: Follow your gut, but validate. For nasty bugs like this, the interactions are so complicated it would take forever to just step through the code and find the bug. So, you have to develop a gut feeling for what might be causing the bug and follow that first. Of course, you have to develop tricks to quickly confirm or reject your hunches so you don’t go down the wrong path for an hour.
For instance, my initial feeling was that monkey patching was to blame. Specifically, I noticed that both Classifier and Rails had defined a ‘sum’ method on Array (actually, Rails defines ‘sum’ on Enumerable, but Array includes Enumerable). A quick check confirmed that Classifier’s ‘sum’ had overwritten Rails’ ‘sum’ by the time my integration test ran. But I wasn’t sure if the Rails code actually ever called this method.
To check this, I used ruby-debug. I first set a breakpoint right before the troublesome line in the integration test. At the debugger prompt, I turned on trace mode (trace on all) and then continued (c). Trace mode prints out every single line of code that is executed, including the file in which the line appears. It was simple enough to copy/paste the trace into Emacs and search for any line executed in the ‘classifier’ library.
As it turns out, I didn’t find anything. My hunch was wrong – Classifier’s ‘sum’ method wasn’t to blame. But luckily, it didn’t take me too long figure it out.
So, I went back to simplifying the repro. Instead of doing require 'classifier' (which includes the whole gem), I started including specific files from the gem. After doing this for awhile, I found that only requiring the file ‘classifier-1.3.1/lib/classifier/extensions/vector.rb’ caused the bug.
I tried to simplify even more. That file starts with both require 'matrix and require 'mathn'. Could one of those be the problem? I tried requiring each of those files at the top of the integration test and sure enough, require 'mathn' caused the problem.
Tip 5: Google is your friend. All throughout this long process, I’d been Googling to see if anyone had solved this issue. Of course, until the end I didn’t really know what the root cause was, so my searches turned up nothing useful. However, after narrowing it down to integration tests and mathn, I found a this Rails ticket (search ‘rails integration test mathn’). As it turns out, mathn changes the behavior of integer division, which makes the code return the wrong HTTP status code. And the reporter includes a simple, one-line patch which is easy to apply.
In retrospect, it all seems to simple (and in fact, it’s frustrating that it took me so long to solve). When tracking down a bug, it’s very easy to get frustrated or overwhelmed with the sheer size of the search space and forget basic debugging practices. Although tracking down bugs will always remain somewhere between science and art, these tips should help you the next time you encounter a real head-scratcher.


Rails: Completing Requested Action After Authentication

February 26, 2008

Within Seekler, we have several actions that are only available to logged-in users. We wanted a general solution that would first redirect the user to login or create an account and then complete the action the user initially requested. The primary motivator for this was non-logged-in users clicking ‘create your own list’ on a Seekler list. If a user clicked this link, we wanted to allow them to login or register and then have the list creation action take place and return them to their newly created list. We had other cases of site actions, that also required being logged in, so I was hoping to find a simple way to do this across the site.
We use the acts_as_authenticated plugin (I know most people are now using restful_authentication, we haven’t switched yet), which relies on before_filters to verify users are logged in before using logged-in-only actions. I ended up just adding one line into that system that makes it simple to return the user to his or her previous action after logging in.
In a controller using acts_as_authenticated, there is normally a before_filter requiring authentication before any actions except the actions available to all visitors

before_filter :login_required, :except => [ :show, :list]

This means that on an action like :create the user will end up calling the login_required method in /lib/authenticated_system.rb. If you add the following line to that method, it will record the information you need to redirect the user back to the action they intended after they login.

def login_required
username, passwd = get_auth_data
self.current_user ||= User.authenticate(username, passwd) || :false if username && passwd
session[:last_action] = request.env['REQUEST_URI'] unless logged_in? && authorized? # add this line!
logged_in? && authorized? ? true : access_denied

At this point, the filter will direct all non-logged-in users to your associated login failure action, in our case /account/login
In the login action, (after doing any other login specific code) you just have to redirect the user if they have :last_action set

def login
# login stuff
if session[:last_action]
redirect_to session[:last_action]
redirect_to :controller => 'account', :action => 'index'

You can add a bunch of other things like ignoring certain actions you don’t want to forward the user to, or also return the user to wherever they were before clicking a login link, but this simple version gives the basic idea. Obviously this code is specifically for the acts_as_authenticated plugin, but it should be relatively easy to generalize this same sort of solution to work for any type of authentication that is in use on your Rails project.
If you want to see this in action, just visit a list like Best Charities on Seekler and click the ‘create your own list’ link in the top right.

The Maybe Monad in Ruby

February 14, 2008

| [tests]
I’m a little behind on my goal to learn a new programming language each year – I’m still working on really understanding last year’s language, Haskell. Judging from the number of tutorials online, understanding monads is one of the hardest parts of learning Haskell.
I won’t even try to explain monads to you, because (as this post will reveal), I’m still a bit shaky on them myself. However, I will say my attempts to build a Maybe monad in Ruby have helped me learn more about Monads – and have yielded a pretty useful piece of code.
Why should you care about the Maybe monad?
Have you ever written code like this?

if(customer && customer.order &&
# ... do something with customer

Don’t you wish there was just a way to call without all those intermediate checks for nil objects?
One solution is to obey the Law of Demeter – in other words, to change the call to customer.order_id. Two problems – one, sometimes obeying the LoD is more work than its worth (especially for quick and dirty solutions) and two, it still doesn’t help you if customer is nil.
Still not convinced there is a problem worth solving here? Check out better explanations by Oliver Steele or Reg Braithwaite – they explain the problem better than I did (and present some interesting solutions).
Anyway, here is my solution – a Maybe monad in Ruby. You would change the above code to:

# ... do something with customer

How does this work? Without going into too much theory about monads, the basic idea is that Maybe is a container object. It stores (or ‘wraps’) any value and then controls how methods are called on the wrapped object.
More concretely, the call to wraps a value in a Maybe object. Whenever you call methods on the Maybe object, it does a simple check: if the wrapped value is nil, then it returns another Maybe object that wraps nil. If the wrapped object is not nil, it calls the method on that object, then wraps it back up in a Maybe object.
Confused? Here’s a few examples:

>>"10")            # => #<Maybe:0x34cbd5c @value="10">
>>"10").to_i       #=> #<Maybe:0x34c919c @value=10>
>>"10").to_i.value #=> 10
>>             #=> <Maybe:0x34c4408 @value=nil>
>>        #=> #<Maybe:0x34c2644 @value=nil>
>>  #=> nil

I’ve started to use this in my code on Seekler and it’s definitely helped me clean up some messy conditional code quite a bit. Try it out and please let me know if there is any way I can improve it. You can find the source code here. You can also check out the tests.
All in all, this project was a lot of fun, helped me understand monads better, and showed me how a powerful concept from Haskell can be a very useful concept in Ruby.
Update: Thanks to the help from people commenting on this post (as well as some help from James Iry), I think I’ve solved the problem I describe below. It turns out I was confusing two different methods that monads should have: fmap (which takes any function and applies it in the monad) and pass (which takes a function that returns a Maybe and applies the wrapped value to it). I’m leaving the issue below in the hopes that others who may be confused can learn from it.
There is one big problem with my code (and it reveals my ignorance regarding monads): this monad doesn’t seem to meet the third monad law. In other words, the following test fails

  def test_monad_rule_3
f = {|x| x*2}
g = {|x| x+1}
m =
assert_equal m.pass{|x| f[x]}.pass{|x|g[x]}.value, m.pass{|x| f[x].pass{|y|g[y]}}.value

I learned about the monad laws from ‘Monads on Ruby’ on Moonbase, but I don’t yet understand how the third monad law works (in his examples at the bottom of this page). If any wise Ruby hacker (or Haskell hacker) can help me bridge this gap in my understanding, I’d really appreciate it.
For the interested reader, there’s some great information out there about this problem in general and Ruby monads in particular.
MenTaLguY has a great tutorial on Monads in Ruby over at Moonbase
Oliver Steele explores the problem in depth and looks at a number of different solutions
Reg Braithwaite explores this same problem and comes up with a different, but very cool solution in Ruby
Weave Jester has another solution, inspired by the Maybe monad
Update: James Iry has a great explanation of monads from the Scala perspective. His explanation of the monad laws (and the difference between map and flatMap, as they are called in Scala, really helped me out)

Featuritis, Tyranny and Web Applications

December 13, 2007

I’ve been excited about the possibilities of web apps ever since Paul Graham sung their praises back in 2001 (I’m sure other people were talking about them in 2001 and before, but Paul’s essay was especially influential for me). What’s not to like? With web apps you get to release early and often, get real-time feedback on which features your users actually use, there is no clunky installation, and you get to write your app in something other than C++ (in my case, we got to write Seekler in Ruby, which is sweet).
One of the most promising properties of web apps is that they aren’t as susceptible to featuritis. If you’re not familiar with featuritis, just think back to the earliest version of Winamp you ever installed. It was a lean, simple, just-plain-great mp3 player. You know how you stopped using Winamp because they added all these insane features that made it slower, bigger, and less usable? That’s featuritis – the tendency of successive versions of software to add more and more features, most of which you never use.
Featuritis isn’t completely the product of over-eager developers (although that’s part of it). It’s mostly a result of the economics of desktop software – if you want someone to buy/download the newest version of your software, you better add some new stuff to justify it. No one is going to pay $200 for a new version of Windows that adds no new features, but is simpler, faster, more stable, and more secure.
Web apps don’t have this same pressure, because there is a different business model. With web apps, developers don’t have to justify the next version. Instead, we just have to make sure the current version is awesome enough that you’ll keep coming back (and either click on our advertisements or pay a monthly fee, depending on the app).
Web apps can shift their focus away from features and compete on things like performance, usability, simplicity, community and content. Of course, features are still important – although you don’t have to convince users to get that next upgrade, you still have to prevent them from moving to your competitor. But whereas before the focus was almost entirely on features, it’s now a little more balanced, so hopefully that means less featuritis and less bloat.
Jeff Atwood got me thinking about all this when he wrote about featuritis over at Coding Horror. I was reading his thoughts on upgrading to the newest version of Winamp, when this caught my eye:
“I don’t have to upgrade, of course, and there’s nothing forcing me to upgrade.”
And there lies one of the key advantages of desktop apps – even though there is considerable pressure for user to upgrade their desktop software, they do have the freedom not upgrade. And many users don’t upgrade. I know many users happily plugging away with old versions of word processors or graphics programs, some of them quite content running Windows 98. In fact, there are whole sites devoted to making sure people can find and use old versions of desktop apps.
The vast majority of web apps simply take this freedom away from their users. And yet, in all the talk I hear about the greatness of web apps, I rarely hear this problem discussed.
I think as we move out of the early days of web apps, this is going to be a bigger and bigger problem. The sizable number of users who don’t upgrade their desktop applications regularly are going to be pretty annoyed every time we force some new, shiny toy down their throats. Are we developers going to be (hopefully benevolent) dictators, forcing each individual to endure what we think is best for the masses? Or, on the other hand, will we be so terrified of angering our users that we stop delivering new features altogether? Neither option sounds that pleasant to me.
Clearly, I’m not the only one concerned about this: check out the latest version of Gmail. You’ll notice that they have a new link in the right-hand corner: ‘Older Version’. And I’m glad they do, because GTDInbox doesn’t work with the new version. But how long will they keep that up? Will they keep a link to all past versions?
It would seem the ideal situation would be to let users opt-out of updates but this presents some pretty big techincal headaches. Maybe you can just skin the old UI on new functionality to hide some of the new features. But when a new feature requires changes to your database, how are you going to make sure all current and old versions play nicely together? Are you really going to test creating and viewing data on every pairwise combination of app versions? Or are you going to sandbox old and new users into different DBs? Any way you slice it, it’s going to get real complicated, real fast.
Perhaps this is just a fundamental trade-off for web applications: you get some great benefits, but on the downside, you’re at the whim of the developer. But I suspect it’s solvable. I doubt there is a silver bullet, but if we start talking seriously about the trade-offs and possible solutions now, we might be able to get the best of both worlds.

Pattern Matching in Ruby

November 11, 2007

for the impatient:[tests]
To paraphrase Alan Perlis, learning a good programming language should affect the way you think about programming. The only problem: since it’s such a good language, it’ll probably give you a bad case of feature envy when you use other languages.
Case in point: Pattern matching in Haskell (Yeah, I know, other languages have pattern matching, but Haskell introduced me to the concept). Pattern matching let’s you write very clear code by choosing an execution path based on the structure of a function’s arguments. So, for instance, you can write:

fib :: Integer -> Integer
fib 0 = 0
fib 1 = 1
fib n = fib (n-1) + fib (n-2)

Instead of checking the argument within the function, you essentially write three versions of the function ‘fib’ – one for when the argument is 0, one for when the argument is 1, and one for all other cases.
Now this may just seem like a fancy switch statement, but it’s much more powerful than that. For instance, pattern matmching can break down lists. Check out the Haskell implementation of map.

map _ []     = []
map f (x:xs) = f x : map f xs

What’s going on here? Remember, map is a function that takes a function and a list and applies the function to each member of the list. The _ is like a wild card – it matches anything. The [] stands for the empty list. So, if the arguments to ‘map’ are an empty list and ANY function, it just returns an empty list.
Otherwise, Haskell binds the first element of the list to the name ‘x’ and the remainder of the list to the name ‘xs’. It then applies the function to x and recursively calls ‘map’ on the remainder of the list. Very cool!
As you can see, pattern matching can really simplify methods, especially when the arguments to the method have complex structures (if you’re still not convinced, Steve Yegge motivates pattern matching much better than I have). The trouble is, I don’t use Haskell on a regular basis – I use Ruby. And Ruby doesn’t have pattern matching.
Luckily, Ruby is a remarkably flexible language. So much so that a number of people have implemented various versions of pattern matching in Ruby. But while those implemenations are very cool and taught me a lot, none of them quite felt right. Plus, I’d been looking for an excuse to play around with Ruby’s metaprogramming features and this seemed like a good opportunity.
So, I wrote my own version of pattern matching in Ruby. The Fibonacci sequence looks like this:

def fib(n)
match n do
with(0) {0}
with(1) {1}
otherwise {fib(n-1) + fib(n-2)}

(Although this isn’t really a great example because a case statement would suffice). And map looks like this:

def map(proc,arr)
match([proc,arr]) do
with([:_,[]]) {[]}
with([:f,:x % :xs]) { [] + map(f,xs) }

It’s not quite as succinct as the Haskell code, but it’s reasonably close (note that I can’t use Haskell’s : operator to destructure the list, so I use the % operator instead).
Here’s a more interesting, if contrived, example – let’s say you have a tree structure (represented by nested arrays). Each node has a left and right branch as well as an operator. So a tree might look like this:
[[5,'*',10],'+',[6, '/', 3]]
Now let’s write a method that will give us a string representation of any tree. So, for the example above, we’d want “((5 * 10) + (6 / 3))”. Without pattern matching, we’d write something like this:

def disp(tree)
elsif(tree.is_a?(Array) && tree.length==3 && tree[1].is_a?(String))
"(#{disp(tree[0])} #{tree[1]} #{disp(tree[2])})"
"Invalid tree structure"

but with pattern matching, we can write:

def disp(tree)
match tree do
with(:num & Fixnum) {num}
with([:left, :op & String, :right]){"(#{disp(left)} #{op} #{disp(right)})"}
otherwise {"Invalid tree structure"}

Let’s go through this step by step. If ‘tree’ is a Fixnum, we assign bind it to the name ‘num’ and return it (the & operator binds the matching argument to the symbol on its left – in this case, :num). If ‘tree’ is actually a tree node, we give a name (‘left’, ‘op’, and ‘right’) to the various parts of the node and recurse. Once you get past the strange looking syntax, the code is much cleaner and easier to understand than the version without pattern matching.
We can also destructure arrays. Here’s an example that drops the first three elements of an array:

def drop_three(array)
match array do
with(:_ % :_ % :_ % :xs) { xs }

If you’d like more examples, the tests are a good place to start. Or you can just check out the source code. Feel free to use it however you want – I’d love it if a better Ruby hacker than I improved this stuff…
A few caveats:

  • I really can’t take much credit for this, my code is heavily depedent on some very nice DSL and let code written by Reginald Braithwaite. It’s cool stuff, check it out.
  • This was my first attempt at some serious Ruby metaprogramming. I probably did some dumb stuff – so if you have any tips, please let me know.
  • It’s pretty darn slow right now. If anyone wants to take a crack at speeding it up, be my guest. I’ll probably look into it as time permits.

Thanks and enjoy!


April 21, 2007

After reading a great tutorial on debugging in Rails with ruby-debug, I inserted the following line at the end of my config/environment.rb

SCRIPT_LINES__ = {} if (ENV['RAILS_ENV'] == 'development' || ENV['RAILS_ENV'] == 'test')

which had been working just fine for me. Ruby-debug is a great debugger and has helped me track down a lot of tricky issues (and works more consistently for me than the built-in debugger within RadRails).
However, today I ran into a weird problem – tried to install rcov and the rails rcov plugin. But when I tried

$ rake test:units:rcov

I got the following:

Loaded suite /opt/local/bin/rcov
Finished in 4.366043 seconds.
36 tests, 114 assertions, 0 failures, 0 errors
| File | Lines | LOC | COV |
|Total | 0 | 0 | 0.0% |
0.0% 0 file(s) 0 Lines 0 LOC
No file to analyze was found. All the files loaded by rcov matched one of the
following expressions, and were thus ignored:

What’s going on here? After playing around for awhile, I tried commenting out the SCRIPT_LINES line above and bingo! everything worked.
After a bit of searching, I found a comment on this ruby-debug tutorial that states that the SCRIPT_LINES constant no longer needs to be defined for ruby-debug to work. I briefly tried ruby-debug without defining the constant and everything seemed fine.
So if you find yourself facing some weirdness with rcov or any other Rails plugin, send that SCRIPT_LINES constant packing and see if that helps.

We have a winner

April 18, 2007

For better or worse, we’ve decided to go with Ruby on Rails for our prototype. Why? In short, we think RoR will let us finish a version 1.0 very quickly – and that means we can start getting real feedback from early users sooner rather than later.
Django had some very nice features, but ultimately, it didn’t seem like a clear winner over Rails (both had pros and cons) and, probably most importantly, we know and prefer Ruby over Python. That’s not flame-bait, it’s just a personal preference (I heart blocks). YMMV.
Of course, the scaling issue is in the front of our minds, especially with the recent high-profile case of Twitter (and certain responses). Are we concerned? Yeah. But, scaling would be a nice problem to have, since it would imply that we actually have users. In all likelihood, scaling problems are still many months out.
But, when it comes up, our plan for scaling is four-fold – first of all, we’re hoping the Twitter experience motivates the RoR community to look more into scaling (particularly using multiple databases effectively), and we can reap the benefit of that work in few months (hey, maybe when we understand the issues better we can write something ourselves).
Also, JRuby is looking more and more promising. Not only does it look like it’s going to be as fast (or maybe a bit faster) than regular Ruby, but, since it runs on the JVM, we’ll be able to integrate it with faster, statically-typed languages on the backend (Scala, anyone?).
We might also try to use faster, statically typed language on separate machines and connect it to the front end using web services. I’m sure there is some overhead here, but on the plus side, it’d force us to develop a nice web API sooner rather than later, which is something we want to do anyway.
Failing that, we can always rewrite the damn thing in another language. It would suck, but hey, it didn’t kill Reddit.

Of prototypes and tracer bullets

April 10, 2007

While we’ve been waiting to hear back about funding, I’ve been spending my time working on a prototype of our app. At least that was the plan.
My initial intent was to do a proper prototype. In other words, a throw-away version that I could learn from. In one sense, I was successful – the code I wrote let us get a good understanding of potential problems and sparked some great discussions. I feel like I understand the project a lot better now than when I started.
But, as I wrote the code, a funny thing happened. I got a few features done, but I wanted to flesh out the app a little more. So, I wrote a few tests to ensure I didn’t break core functionality and then added a few more features. Then I noticed some bugs that broke main scenarios, so I fixed those. Then I added another test, another feature, and fixed another bug. And somehow, little by little, the code became less like a prototype and more like a tracer bullet (the distinction is described in The Pragmatic Programmer).
Like any startup, our goal has always been to release early, and the iterate quickly. I started the prototype with that goal in mind. However, at this point, it’s surprising to me that the role of the prototype changed out from under me. We’ve had to ask ourself – should we throw away the code and write the production version from scratch, we we initially planned? Or use this code as a foundation for our real implementation?
It wasn’t immediately clear to us what to do – going down either path requires some work to get to a code base that we will be comfortable working with. The more we thought about it, the more we found that the answer lies somewhere in the following questions.
How much code have you written? Clearly, if you’ve written a lot of features that took a long time to write, you’ll prefer working with the existing code.
How good is the code? Although Joel famously stated that rewrites are a terrible mistake , he’s talking about mature software that already has a lot of bug fixes. If your code doesn’t have many fixes (and therefore still has a lot of bugs in all likelihood), has an architecture you know is flawed, or is just fully of hacky code, then perhaps you should lean towards a rewrite (you’re going to have to do a lot of refactoring just to get it in a good state anyway).
How many tests have you written? If you don’t have any tests, chances are you’ll have to refactor the code just to get the code under test. Then you’ll have to refactor more to get the code clean. It will probably be less work just to rewrite from scratch, baking in tests from the beginning. On the other hand, if you already have tests, refactoring will likely be a lot cheaper than a full rewrite.
What language/platform are you using? Clearly, if you wrote your prototype in a different language/platform than your real app, you’ll need to do a rewrite. We wrote our prototype in Ruby on Rails, because I know Ruby pretty well and knew that we could get something running quickly with Rails. Although RoR was a great system for writing a prototype quickly, we’re not sure if it’s going to be the fastest way to get a version 1.0 release (any advice on this is appreciated).
The lesson I’ve learned is to have specific goals in mind when writing a prototype. What exactly are you trying to learn from the prototype? How many features do you need to write? Which features can be faked? What can you avoid testing? If you don’t have some answers beforehand (both for yourself and others), you may end up writing something entirely different than you intended.