In this paper, I investigate whether we can use a world-involving framework to model the epistemic states of non-ideal agents. The standard possible-world framework falters in this respect because of a commitment to logical omniscience. A familiar attempt to overcome this problem centers around the use of impossible worlds where the truths of logic can be false. As we shall see, if we admit impossible worlds where “anything goes” in modal space, it is easy to model extremely non-ideal agents that are incapable of performing even the most elementary logical deductions. A much harder, and considerably less investigated challenge is to ensure that the resulting modal space can also be used to model moderately ideal agents that are not logically omniscient but nevertheless logically competent. Intuitively, while such agents may fail to rule out subtly impossible worlds that verify complex logical falsehoods, they are nevertheless able to rule out blatantly impossible worlds that verify obvious logical falsehoods. To model moderately ideal agents, I argue, the job is to construct a modal space that contains only possible and non-trivially impossible worlds where it is not the case that “anything goes”. But I prove that it is impossible to develop an impossible-world framework that can do this job and that satisfies certain standard conditions. Effectively, I show that attempts to model moderately ideal agents in a world-involving framework collapse to modeling either logical omniscient agents, or extremely non-ideal agents.