Although it's not specifically drawing for me, I have found that if I have some sort of stimulation other than just sitting and listening, it dramatically helps my thoughts. Sometimes I'll play games simple games on my phone while listening to a speaker and I come away with a greater recollection than times where I just sat and listened.
You're right though, it does come across as being rude/distracted.
I've been doing this for a few years now and only a few times have people commented and it's never been a big deal even then. It's a nice way to create associations with what's being said and it actually helps keep me awake and attentive when there's a lull in whatever is going on.
I'm certainly glad I am not alone! It was odd yesterday when I was doing this - some around me were smiling at what I was doing but others were very confused. Especially when I asked an insightful question. I guess that's the trick…