PC doesn't start. Power lights blink once

When I press the power button of my PC, it sometimes powers on.

Most of the time, it gives a short flash and nothing else.

Then I have to either press and hold the power button for about 20 seconds and try again or switch of the power supply and then try again after some time.

These is what I tried and doesn't work:

1. Tried different Power cords.
2. Tried different Wall sockets.
3. Tried replacing CMOS cell.

These didn't work.

I tried unplugging the 4 pin connector from the PSU to motherboard and now it starts instantly every time.

But I see nothing on screen. Maybe the CPU isn't running now.

I tried powering it up having the case in an angle. Sometimes at some angles it works.

So I can't figure out what exactly the problem is. I doubt the CPU and the PSU.

Can you suggest what could be the possible cause?