This is just a possible (not sure it can work) step that you can try:
1. set up yocto build system so that download dir is outside of build dir
2. copy your git clone to download/git2
3. delete sstate-cache, tmp and cache in build dir
4. run bitbake core-image-minimal again
Other alternative option:
1. clone kernel repo
2. modify your kernel-recipes to fetch from local git repo instead