Neural networks are just a small subset of a larger space of functions, and when creating a network architecture, you want to use an admissible set of functions that are useful for your specific problem. It is important to understand this larger space of functions in order to create better networks.