Sources: Fairy Tales, J. R. R. Tolkien's The Hobbit
Why We Love 'em
Riddles are miniature word-based puzzles. They usually contain some kind of paradox that you can only solve by looking at the world in a non-literal way. And they're fun. I remember collecting riddles as a kid, first trying to solve them and then passing them off as my own to stump the adults. It made me feel smart. Whether I solved the riddle or peeked at the answer sheet, there was always a wonderful "Oh!" moment when the code was cracked and the solution revealed itself like a crane from a sheet of origami.
How to Build 'em
Bear with me, because I'm no expert. I can only analyze what makes a riddle work and speculate how to create one. From what I can tell, there are two different methods, plus a cheat.
Start with a common household object. Something everyone knows but no one really looks at. Pencils, nails, tables, chairs, tea kettles, earrings, and radishes. The less people think of them the better. Then apply a solid dose of figurative language: either a metaphor, personification, or pun. Tables have legs, potatoes have eyes, refrigerators run, and newspapers are read. If you can't find a pun, don't despair. You can always create an elaborate metaphor, which is the more sophisticated route, anyway. So turn canes into legs and teeth into horses--anything goes.
It's not enough, however, to imply one thing is something else. If possible, try to focus on the paradoxical element of it. Things are suppose to work a certain way. Water flows downhill, the sun sets in the west. If something breaks the rule, we want to know why. And so we have riddles, like this one, from The Hobbit:
"A box without hinges, key, or lid,
Yet golden treasure inside is hid."
The paradox is that there is no way to open the box to put the treasure inside or take the treasure out. But if you start thinking of how to get the treasure--ie, by breaking the box--the metaphor starts to unfold. The box is a shell, the treasure is a yolk, and the answer is an egg.
"Little Nancy Etticoat
In a white petticoat
And a red nose.
The longer she stands
The shorter she grows."
In both instances, concrete details hint at how to solve the riddle: the golden treasure, the white petticoat, the red nose. Even the paradox itself is a clue. The briefer the riddle the better. Bonus points if you can work in a rhyme. If you can't memorize the riddle in 5 minutes, chances are it's too complicated.
Rather than start with a common object, start with a broad, abstract concept, though still something any elementary school kid would understand. So no Theory of Relativity allowed. You can either construct a metaphor as before or simply list off ways in which it interacts with the world. Then choose the most interesting ones and put them together in a list that befuddles the reader or seems to contradict itself.
"This thing all thing devours:
Birds, beasts, trees, flowers;
Gnaws iron, bites steel;
Grinds hard stones to meal;
Slays king, ruins town,
And beats high mountain down."
Notice that this riddle doesn't really have a paradox--it's just so all-encompassing that you have to stretch your imagination to figure out what it is. The answer, by the way, is time. Though the riddle uses a bit of personification to make it seem like some kind of monster, most of the riddle is simply a list of the things time does.
Here's a riddle a tour guide told my family a while back.
"What's greater than God,
More evil than the Devil.
Rich man wants it.
Poor man has it.
If you eat it, you die."
When you compile your list, don't go for the most obvious descriptors. ("The friend of clocks." "It lives between stars.") Look at the concept from all different angles. Go for both huge mind-blowing concepts ("all things devours" "greater than God") and smaller, more concrete examples ("gnaws iron, bites steel" "if you eat it, you die"). That's what create the paradox: what object could be so big and so small at the same time?
When people try to solve riddles, they usually start by compiling a list of things they can see, either with their eyes or with their imaginations. By choosing something they can't see, you'll be able to stump them, even if the answer is right there in front of their nose.
Even though I've analyzed what makes a good riddle tick, actually writing one requires more time, patience, and work than I'm usually willing to devoted. Fortunately, I write fantasy, and that means I'm not limited to the real world. Anything goes. And that means I can think up a paradox that will stump my readers and then make up an answer out of thin air. This is the cheat method and it's so much easier.
For example, in a story I'm currently working on called Three Floating Coffins, Princess Sophia asks her suitors to bring her "a gift worth my kingdom that can fit in the palm of my hand." The answer turns out to be an amulet containing magic. In the past, a similar amulet was used by Sophia's ancestor to win the kingdom in the first place.
Of course, even if the audience had guessed "Magic," they couldn't have known the specifics. That is the information the author (me) conceals until the last moment.
I'm hardly alone in this. In the fairy tale "What the Rose did to the Cypress," (The Brown Fairy Book, edited by Andrew Lang--no relation) a princess asks her suitors "What did the rose do to the cypress?" and executes them if they guess wrong. The answer winds up being a complicated story of how a woman (Rose) cheated on her husband (Cypress) and would have killed him but for the intervention of his dog. Thereafter Cypress decreed that his wife should be treated like his dog, his dog treated like his wife. (It's back before the days of political correctness.)
Normal riddles work because they test your brilliance. Cheat riddles work because they promise you a story. The author leads you on a journey into the heart of the world; seeking the answer to the riddle, you come to understand all aspects of the land's beauty, cruelty, and wonder. That's how you can build a cheat riddle and not have the audience feel cheated. The riddle is an excuse to explore the mysteries of your world.
First off, you need to know why the riddle exists. A normal riddle exists for its own sake; a cheat riddle is built into and dependent upon the plot. Usually, it spurs on a quest. The classic scenario, as I've shown, is that of a suitor who must solve the riddle to win the hand of a princess. But it can be anything.
I, personally, like to come up with the solution before I come up with a riddle. In Three Floating Coffins, the amulet plays an important role in the story, as the magic eventually falls into the hands of an evil man who plots to take over the kingdom. Magic, therefore, is the focal point, and I made up a riddle for it by drawing on the paradox of the amulet being tiny but invaluable.
However, the opposite could work just as well. You could come up with an interesting riddle first and spend the rest of the book trying to figure out what it is. It all depends on how you write. The main thing is to tie the riddle into the plot. The tighter you tie it, the better the cheat riddle works.
The Prize and the Penalty
Let's go back to the non-cheat riddles. Suppose you spend days or weeks or months crafting and refining one only for the reader to spend less than thirty seconds reading it. It almost feels like a waste. You need to stretch the riddle by attaching it to an important plot point.
The hero must solve a riddle (or series of riddles) in order to get something he needs or wants. It might be a magical sword or the princess' hand in marriage. Or, like Bilbo Baggins, it might be something as simple as being shown the way out. The riddle becomes an obstacle; solve it and you get a prize.
Of course, if you answer wrongly, there may be consequences. This traditionally involves some manner of gruesome death, with bonus points if the asker of the riddle is entitled to eat you. Then again maybe the penalty is small--instead of gaining an item you need, you lose an item. Maybe the penalty is not against you, but a loved one--Rumpelstiltskin claiming your firstborn. Or maybe you can guess as much as you want without cost, but the time it takes to solve the riddle will severely hinder your quest.
When you think about prize and penalty, think not just about your hero, but the person asking. Why do they bother with riddles? Princesses who want to avoid getting married often use it as a screening process; on the surface, it seems civilized, but it can tally up a high body count. And unlike, say, a joust, it can go on indefinitely with no winner. Is the riddler a monster or a human? Weak or strong? Cocky or bored? As you discover their personality, the prize and the penalty becomes clear.
The other major problem with riddles is that they're passive. The hero listens and then sits around and thinks. Don't get me wrong; their thinking process can be quite fascinating. But for cheat riddles especially, it's not enough to think. Your hero has to get up and do something.
Going back to my Three Floating Coffins, the princess didn't just want an answer to her riddle--she wanted the suitor to present her with the magical amulet. Which led to a whole new set of problems--how do you go about obtaining it? The riddle, in other words, sets off a larger test. You must first solve it to know where you're aiming; get it wrong and you're on a wild goose chase. But even if you get it right, there are still many obstacles between you and your prize.
Even if your hero is in an enclosed space, he can still take action. He can goad the riddle-master into giving him multiple guesses. He can try to bargain or bribe the riddler for hints. If it were me, I'd try to find out all the wrong guesses other people tried. Maybe even observing the riddle-master helps. Poker players learn each other's tells--maybe the hero can learn the riddler's "tell" and use it to his advantage. Conversely, maybe the asker doesn't let the hero sit and think. Maybe, he attacks him. Maybe the hero is in hostile territory and must fight, swim, climb, or run for his life while he thinks.
Riddles are, in the end, problems. Think of how you solve problems. Do you just sit quietly and think? Or do you do something--anything--write, rant to a friend, take a walk, take a bath, play with a toy, eat, beat your head with your hands--until a flash of brilliance comes. So don't discount action, whether big or small. Even Bilbo Baggins who was basically trapped in a dark cave with nothing to do but think fidgeted and pinched himself and gripped his sword, all in an effort to solve riddles.