1. Look into your *.map file and see exactly what symbols occupies .text and how big they are.
2. Link with NewLib (nano) and don't use float options.
3. Use -Os for entire project but with care (force some drivers with -O0)
Original drivers are buggy and linking with -Os sometimes will provide unexpected results. Also some of them are completely un-optimized for small embedded so indeed, will produce a lot of code, consuming both PGM and performances.