The difficulty of returning to math and physics after taking a long break is compared to being a violinist without regular practice. The speaker questions their ability to keep their skills sharp and retain information.