Hi,
Regarding your question, even if you do not use the on-chip Voltage Reference as ADC voltage reference, you have to configure the on-chip Voltage Reference.

This is the ADC code for the LPC5536:
/* Disable VREF power down */
POWER_DisablePD(kPDRUNCFG_PD_VREF);
VREF_GetDefaultConfig(&vrefConfig);
vrefConfig.bufferMode = kVREF_ModeBandgapOnly;
vrefConfig.enableInternalVoltageRegulator = false;
vrefConfig.enableVrefOut = false;
VREF_Init(VREF, &vrefConfig);
Regarding the ADC clock, I suppose that the 8MHz is too slow. This is the value you should set up:

Regarding the ADC result which is not stable, I suggest you set a big value for the g_LpadcCommandConfigStruct.sampleTimeMode = kLPADC_SampleTimeADCK131;
/* Set conversion CMD configuration. */
LPADC_GetDefaultConvCommandConfig(&g_LpadcCommandConfigStruct);
g_LpadcCommandConfigStruct.channelNumber = xxxx;
g_LpadcCommandConfigStruct.sampleChannelMode = DEMO_LPADC_SAMPLE_CHANNEL_MODE;
g_LpadcCommandConfigStruct.sampleTimeMode = kLPADC_SampleTimeADCK131;
g_LpadcCommandConfigStruct.hardwareAverageMode = kLPADC_HardwareAverageCount128;
#if defined(FSL_FEATURE_LPADC_TEMP_SENS_BUFFER_SIZE)
g_LpadcCommandConfigStruct.loopCount = FSL_FEATURE_LPADC_TEMP_SENS_BUFFER_SIZE - 1U;
#endif /* FSL_FEATURE_LPADC_TEMP_SENS_BUFFER_SIZE */
#if defined(FSL_FEATURE_LPADC_HAS_CMDL_MODE) && FSL_FEATURE_LPADC_HAS_CMDL_MODE
g_LpadcCommandConfigStruct.conversionResolutionMode = kLPADC_ConversionResolutionHigh;
#endif /* FSL_FEATURE_LPADC_HAS_CMDL_MODE */
LPADC_SetConvCommandConfig(DEMO_LPADC_BASE, DEMO_LPADC_USER_CMDID, &g_LpadcCommandConfigStruct);
Especially, when the analog channel source has large impedance.
Hope it can help you
BR
XiangJun Rong