The third prevalent complaint I found in one-star reviews applies most specifically to writers who are forging into subsequent books in a series, and the word of warning one-star reviewers send your way is this: don't ramble. No matter how fascinating you might think the minutia of your world may be, if it doesn't serve to advance the plot in some meaningful way, don't include it.
To be fair, for every one-star review that complains about self-indulgent world building detail, there are ten reviews that rave about the depth of the story teller's world. The majority of loyal fantasy fans eat world details like Edmund Pevensie shovels Turkish Delight, but my study of one-star reviews shows there is a significant percentage of those who don't. As with many things in life, it seems to me moderation is the key.
Another refrain I found in these one-star reviews that decried irrelevant detail was an across-the-board complaint about overly-detailed sex scenes in fantasy books that did nothing to develop anyone's character (for better or worse) and felt more like voyeuristic indulgence of the author's weird thought life than anything that should populate the pages of a book. Just by reading the way the reviewers framed these comments, it's obvious they aren't moral ultra-conservatives. They might not even object to the concept of casual sex in the real world. They just don't want to see geeky fantasies lived out for pages and pages for no reason. It's not just us prudish Christians who'd rather not hear detailed language about body parts and their interaction. I find it a relief to know there are readers across the spectrum with a sense of decency, and they're not afraid to speak up about it.
I'll be the first to admit that finding the exact level of world exposition that pleases both those who geek out over it and those who demand continual story momentum is a huge challenge for those of us who have built worlds and mentally live in them. That's why I think it's very valuable to come out of fantasy land every now and again, take a look around at how people are reacting to the unveiling of that world, and adjust accordingly.