If your plan was to call down, you have to call the river bet. You've given him no reason to believe you don't have any air, so his bet could easily be something like ace high or a 2 or a 8. There's no reason to believe he has a 4 or J here.

That said, if you want to avoid a difficult decision, you should have bet the turn or check-raised. There's no reason for you not to raise him on the turn, there's $7.50 in the pot + the $4 he has, if you have the best hand, time to take it down.