Even at a burger place, no one will care (or, most likely, know) if you eat the burger and discard the bun.

We go to Smashburger once in a while, and I just toss the bun from whatever I get. Husband gets some sweet potato fries, I have two or maybe three, then eat my burger. By the time I'm done, the fries are cold, and not nearly as good.