The SD ADC conversion rate depends on the modulator clock (fmclk) and selected oversampling ratio (OSR). Datasheet parameters of the SD ADC (for normal operation) are specified for fmclk=6.144MHz and OSR=2048, where such configuration results in 3 ksps output sample. You can go down with OSR to 64, which will increase output sample rate to 96 ksps still providing 24-bit result but with lesser noise free bits (~ -2.8 bits).