Devlin's "The Math Gene" is a wonderful book, well worth reading if you've an interest in how we think, and absolutely essential if your interest extends further to why we can do mathematics.
This is an intriguing question. After all, it's a fairly new part of human behavior - having been around maybe 10,000 years - that we all can do, at least a bit, and the rest of the animal kingdom can't, at least as far as we know.
Devlin's the first mathematician I know of who's looked deeply into this subject using recent research in the area; he's done a great job fitting the available data to a theory that starts to answer the question, how it is we can do mathematics?
First, though, you have to understand what mathematics really is. Devlin's definition is the "science of patterns" and he explains clearly and convincingly why it's the right one.
His premise, roughly, is that however we acquired language, and he stays mostly on the sidelines of the heated debates about that, mathematical ability came along for the ride. His reasoning is that "off-line reasoning" is an essentially equivalent to language, as you can't have one without the other, and that this plus some other abilities, such as a number sense and spatial reasoning, give us the ability to do mathematics.
He then explains why so many of us find the subject difficult. A simplified version is that we use language mainly to talk about interpersonal relationships. In a word, gossip. Note he's not claiming this to have been the purpose for it's development, just that it's what we mostly do with it now. And we're very good at gossiping. In fact, it's so easy we consider it to be a form of relaxation. To Devlin, you need to have the same kind of relationship with mathematical objects in order to be able to work with them.
The book's greatest strength, to my mind, is its gathering of results in cognitive psychology into a coherently developed thesis regarding the origins of mathematical ability. It's a worthy contribution to the discussion, even if the theory proposed is completely wrong, as it may well be. Devlin's open and clear about it being highly speculative.
I do have quibbles, but they're just that. Its major weakness, if the book can be said to have any, is that it doesn't make much by the way of predictions based on his theory, which would make it far more convincing. But this is a terrific starting point for other work.