I seem to have solved it, have no idea why but this worked - but I used some hints from previous posts.
Instead of using external build I used eclipse's internal build. This did a full compile the first time, as expected, but did not create the elf file.
So, I reverted to the external build option, and voila! it is correctly doing incremental builds now. 20sec versus 5 or so minutes.