I honestly couldn't agree more. I am as guilty as anyone (or more so!), when I started learning I did exactly this and even though I did go for lessons, I never paid much attention to rhythm until the last few years.
Now that I'm doing (remedial) studies improving my rhythm, I'm finding guitar interesting again.
For me, the goal was playing bass guitar in a band. For you it could be regularly practicing with a metronome, jamming regularly, joining a band, playing drums - we're all a little different, find what works for you, set a goal and play (and have fun, because it's play)