The inferential distance between two people, the teacher and the student, with respect to some concept one is trying to explain is the set of prerequisite knowledge which is (1) necessary to understand the concept and (2) lacked by the student.
Examples
A discussion on gardening requires one be familiar with the concept of life growing, soil, sunlight, heat, etc.
A philosophy text defining a set of terms before discussing ideas.
Evolution
Explaining the evidence for the theory of evolution to a physicist would be easy; even if the physicist didn’t already know about evolution, they would understand the concepts of evidence, Occam’s razor, naturalistic explanations, and the general orderly nature of the universe. Explaining the evidence for the theory of evolution to someone without a science background would be much harder. Before even mentioning the specific evidence for evolution, you would have to explain the concept of evidence, why some kinds of evidence are more valuable than others, what does and doesn’t count as evidence, and so on. This would be unlikely to work during a short conversation.
There is a short inferential distance between you and the physicist; there is a very long inferential distance between you and the person without any science background.