For the most part, I agree. The quality of our primary education really isn't that bad. In international tests American 4th graders perform exceptionally well. The problem arises in middle school and only becomes worse with each grade.
Financially, our system is broken because of bureaucratic influence - mostly in the form of special education. We have some of the best special ed programs in the world, but it's literally a black hole. Across every district nearly 10-20% of all students are eligible for IEPs (Individual Education Plans) and another 10-20% qualify for other assistance. High costs endemic to special ed are both a blessing and curse. When special education services are delivered well, students are less dependent on government over their lifespan. But at the same time a district could be forking over four times as much money assisting these children.
It would be political suicide to bring this up. In fact I'd probably be fired if someone found out I'm suggesting that special ed needs to be completely redone.
I think "diversifying" education is really the only solution. Special education is no longer exclusive. There's kids with severe disabilities, handicaps, anger issues, emotional trauma, etc. If we created different paths (academics, vocational, military/rigorous), we could then bring down enrollment in special ed and redefine its purpose.