Everyone has a job, even in fantasy. Hard work is what makes a world run. From the servant girl in the kitchens of the castle to the tailor, there is something for everyone to do.
A castle runs on the commoners who work it. The question remains, where do these people go at the end of the day? Do they have homes outside of the castle or is the castle their home? If you think about the kitchen of the castle, the people who work it are generally women. And they start before the sun rises to make sure dinner is done. Breakfast and lunch are made along the way, but they begin with dinner. Meat has to cook over fires for quite a long time, depending on what it is and how many you are feeding. Breads and pies have to be baked and cooled. Servants need to be fed as much as the King and his family and any guests that are staying over. Think of "Downton Abbey" and all the bustle to get ready for one simple meal.
Outside of the castle, you have stable boys, groomsmen, blacksmiths, tanners, weapon-smiths and guards. In the city you have tailors and bakers and innkeepers. Farmers and merchants and people who maintain the streets and keep them free of debris and human excrement. People who light lanterns at night and patrol the docks. Fishermen and traders.
There is a lot to keep a world running. You don't have to show all of this in your stories, but you have to build up an image of the area. Make sure you are including everyone from the King and Queen all the way down to the people who empty the chamber pots to make a world turn.