We used Saxon too, from 5th grade up. We started with Math-U-See
We looked into Math-U-See. It looked really good, but there just wasn’t any used curriculum out there.
We bought some workbooks from Sam’s club for $6 each that taught all the basics that more expensive private school curriculum taught and started with Saxon 54 in third grade and just went at the kid’s own pace.