Agreed. The inside of a barn looks like this:

The building where they live looks nothing like that on the inside. Exterior design aside, it's a house, not a barn.

I always figured their house just happened to look like a barn, and we never saw their actual one. We pretty much see the entire interior through just a few episodes. Then there's where ever the hell they keep the pigs, which I'd assume is the other barn, which doesn't seem like they could all occupy the same space.