It’s complex, but essentially, the provider that participates in a certain network agrees to discount his/her/its rates (sometimes substantially) for those with that carrier’s insurance plan. The agreed-upon fee for a particular in-network service is called the “contracted rate”. The lower the contracted rate, the less the insurance carrier (and the insured consumer) pays whenever there is a claim. There’s a sort of dance between the providers and the insurance companies, to agree upon contracted rates. The insurance carriers drive a hard bargain, asking for lower and lower contracted rates. The providers (especially hospitals) sometimes agree to meet those rates in order to stay in-network with a particular carrier, but sometimes will balk and walk away from the negotiating table (thus becoming an ‘out-of-network provider’).
As to why they are necessary, it’s a great question. If you’re thinking within the box, then you’d say it’s part of the existing insurance model. But if you think outside the box, it starts to present interesting ideas. For example, one can always simply offer to pay cash to a provider and ask what the ‘cash rate’ is for a particular service. This would mean you couldn’t also use your insurance coverage for that service, of course, but often the ‘cash rate’ is significantly discounted. Providers don’t like to accept the contracted rates of carrier in many cases because those rates are too low, and in the provider’s eyes, it makes business unprofitable. As a result, providers will sometimes inflate their rates so that the ‘write off’ for insurance still leaves them with room for profit.
The biggest game-changer of the ACA has had to do with networks. Nowadays, the more narrow the network of your insurance plan, the fewer providers you can see and often the longer your wait time becomes before you get seen. However, narrower-network insurance plans also cost less and/or cover more benefits.
It’s all part of the tangled mess our US insurance system has become. I wish I knew how to untangle the mess, but for the most part, we concentrate on helping people navigate through the options to their best advantage on a case-by-case basis! The short answer to your question is probably that things simply devolved to this point as the industry manipulated the system in order to play the margins. Originally, I think networks were created as a way to help providers gain customers since the insurance would direct consumers to them.