How old is your baby? What is your child's temperment? All of this can affect your sex drive IMO. My first child was easy so I was more relaxed in general. My second was a reflux baby and cried almost 8 hours a day AND I had an almost 2 year old at the same time. At the end of the day I just wanted to collapse.
Now things are fairly normal and we have a pretty predictable routine so those stressful days are fewer and far between. Not as stressed during the day = more sex for hubby after the kids go to bed.