I was 3 when my great-grandmother died. I remember her a little. I remember a sick little old lady, who was a little crazy. (she had altztimer's) I can't remember what my parents told me at the time but I'm sure it was something about going to Heaven, possibly about going to be with her husband, mother, etc. which was something she wanted to do. (in her demented state, she was always trying to go see them "down the road")
I also remember that it wasn't a great shock that she'd died, because I knew she was very sick and I'd been told that that's what happens when really old people get really sick.
Don't worry too much about your kid. They are pretty understanding about that kinda stuff. Tell him/her whatever fits your religious beliefs. Tell them "Grandpa's not in pain anymore." Wherever he's gone, he's not sick anymore.