I used an NXP flight controller with a custom frame similar to Holybro QAV250. I installed the px4 firmware chose the QAV250 as the airframe and calibrated the sensors and ESC. It could fly but it was extremely hard to keep the height with the RC. So I changed the throttle curve in the Flysky controller and made it flatter in the middle part.

It is still hard to keep the throttle at the point where it hovers but it's not impossible anymore! The problem is however that it also flies to one side relatively fast. It makes it impossible to keep the height and horizontal position. I tested it in both stabilized and manual modes. I then switched to altitude mode to see if it can keep the height. The throttle is either too much or too little and it's not possible to hover.
Any suggestion on how I can fix this?