I think one of the most important parts about world-building, more than even believability, is that the parts off your world fit together. It doesn’t matter that your sci-fi world breaks the laws of physics so long as it doesn’t break it’s own laws. And this is one of the major struggled in creating a fantasy world.
You have to make sure all of your magic is consistent, yes, but it’s more than that. You have to have a society that makes sense, not in relation to our world, but in relation to itself. If there is magic, and everyone can use it, the society of the world will reflect that. If the world you make does not have fossil fuels, it doesn’t matter so much that you explain why it doesn’t but what effects that would have on the people and nations that have to develop in a world without oil or coal.
The world much of my book takes place on, Ardún, has had access to magic for millennia. The culture of the empire my main character experiences first revolves around magic. It tends to be more important how well someone can cast an ice spell than how well they can carve ice sculptures. Companies hire researchers to develop better magic, different materials that effect spell casting, or better enchanted bandages. A whole division of the legal system in the empire is dedicated to spell patent law. My point is, adding something like magic to a world fundamentally alters the culture that will develop.
But, the thing that affects your world might not have to be as big as “people can use magic”. It could be as tiny as one country deciding to do something that a country here in our world decided not to do. Lemme give you an example:
In 1795, France switched to a decimal currency, and int he following years forced all of their measurement systems into decimal (even their clocks, for a while). The wake of this decision is one of the major reasons international metrics are base-10. However, if France had instead decided to convert all of their metrics to dozenal, or counting in twelves, (which most systems of measurement for shipping were at the time.) it is not unlikely that the world-wide metric system would also be dozenal.
Could you imagine a world that counts in twelves instead of tens? Yet with a fairly small change in our own world, it could happen. And that is why this part of world-building is so, so important. Every thing you decide to add or remove from your world will have some sort of effect. Not all of them will be large, but it is important that you think carefully about the rules and facts you base your world on, and decide how your world came to be.
(As an post script, I want to emphasize how much this is just my opinion. My opinion on how I think worlds should be. This in no way should discourage you from writing. If you don’t want to think about how cultures in your world relate, that’s fine. If you want things to be a certain way, than you are damn well within your rights to make them that way. The world is yours. These are solely my ranty thoughts.)