well to me...baseball at its highest level..(not a game in june)..playoff game...elimination game....nothing has more drama....no clock to run out, just tension and very high focus / execution. The execution of baseball is probably the most difficult of all pro sports, but its very subtle and hard to see the drop on a breaking ball, or how hard it is to hit a 100mph fastball......I mean I dont understand why people love watching a 0-0 soccer game to its conclusion...I dont understand how something with so little offense is so popular...BUt that has nothing to do with the athletic abilities of the players.....I didnt grow up playing soccer and Im sure like baseball...it comes from growing up with it, playing it, and understanding the subtle things that are not seen on tv easily.
100%, if it means anything I can't stand soccer either pretty much for the same reason. I can't even be bother to follow the national team or the World Cup anymore.
As you rightly said (or I think you meant), 'insiders' can better appreciate the subtleties of each sport and thus enjoy it better. For the 'masses' it's probably more about the food stand and the beer while yelling with friends at the opposite team.
I grew up playing basketball in a proper junior team, I knew everything about Dr.J, Kareem, Bird and Magic and religiously bought the local basketball magazine to update all the stats, went to all the home games of the local A1 league (the top division there) and I caused my father's fury every time (and it was a lot) that he had to come and get me at the court because I was late for lunch/dinner (even though he was my biggest fan, God bless his soul). The works.
So when I was working in Detroit and was invited to watch a Pistons' game at the Arena I was in shock as I believe I was the only person actually watching the game, pretty much everybody else was either getting drunk, stuffing their mouths with food or watching the tv screens for other game results (I was in a suite with catering, I almost collapsed when they told me the price of the ticket, recovered pretty quickly when I was told it was free as a guest of the Ford Motor company). So it's probably not just the sport itself I guess.
BTW, those are my feelings about baseball, I have no disrespect for the 'love of the game' which I find quite amazing!