Sounds like you're playing scared money and should drop down to a different stake. Losing $4 or losing $8 or even losing $40 shouldnt be an issue at all if you are playing within your bankroll and a stake you arent scared to play at. iirc, you play 20nl so id suggest dropping down.