Back in episode 1 or 2 (can't remember which) both King Baratheon & Catelyn Stark (Ned's wife)
alluded to the fact that Jon Snow's mother was....
Either....
1. just some 'chic from a brothel' (that Ned fell for)
2. or just an 'average jane', not born from nobility like Catelyn (that Ned fell for), from some region they conquered during the War....
If I remember correctly, both the King & Catelyn had a totally different perspective on her....
This is why Catelyn never wanted Ned to be "The Hand", in the first place
Because she said something like.... "The last time you followed your ol' buddy (the King) into War.... after a year or 2... you came back with a son."

(i.e. Jon Snow)
At the end of the day... I think that Ned had some 'real feelings' towards Jon's mother (whoever she is)... but is 'too ashamed' by his infidelity to admit it to anyone.
Plus, he has to 'keep his gameface on' (at all times) when it comes to the subject of having 'a bastard son'.
Cuz he has a Kingdom to run.
(Plus a castle in Winterfell)
Just my theory.
But I haven't read the books, so I dunno for sure.