The speaker explains that the essence of special relativity involves different reference frames moving at different speeds and how they assign space and time. They also discuss how a basic structure involving updating things in connected updates and analyzing causal relationships is enough for analyzing graphs.