Yep, that's the best way to handle it. I've been asked several times and that has ended the conversation right there, each time.
I have done work for two customers before, and in both cases the model didn't last a month. They don't seem to place the same value on them as we would after they are done, even after they paid me. In both cases, I'd rather have the models back on my shelf than have their money.