I'll grant that Tolkien created a more impressive world in almost every regard, but the cinematic take on that, by comparison, falls incredibly flat in my eyes. I never was quite able to love the LOTR series. For me, a world like that hinges so, so much on special effects, and they just weren't great in that movie. I often wonder about a LOTR with a Hobbit Trilogy or GOT level of visual intrigue.
You're right that the first season isn't amazing. Most of what kept me hanging on until season 2 was the cliffhangers at the end of every episode. By by the time season 3 rolled around, it was pretty clear to me I'd stumbled onto one of the best shows ever made. At worst, the violence and sex can be a little gratuitous, but at best, it's delivered some of the most jaw-dropping and spectacularly punishing blows a TV show ever has for me. I think the hype is deserved, but it wouldn't probably be very clear why without going beyond the first season. Hope you ultimately give it a try! Plus you have the luxury of knowing to stop at season 6 and sparing yourself those awful final two seasons.