I'm going to age myself, but I remember when the NFL had replay the first time in the 80's, but then got rid of it because it was interrupting the game too much and wasn't consistent (sound familiar?). Then they brought it back in the late 90's because, frankly, when you have a televised product where people can see replays and see the screw ups, the viewers get frustrated if you don't fix the screw ups. They first just brought it back with the challenge system, which helped decrease interruptions, but more importantly they got a lot better at making replay calls more consistent (not perfect, but it was much better than before).
I really think the main problem is inconsistency.
I really think the main problem is inconsistency.