If people only watch a programme for one character then they are not fans of the show. There have been episodes where daryl hasn't been on screen or even mentioned. Did these people tune out because of daryl not having his air time? If so I think it's pathetic. TWD is a show not about one character (even though it is based around Rick) you gain an interest and a love for the group and even an interest in the enemies the group face.
I think it will be daryl who meets the wrong end of Lucille and if I'm right I'll be happy because that would be a great storyline and instigate all out war