If you can't eat refined sugars, then you definitely shouldn't. Sugar is sugar and will raise your insulin either way.
In my opinion, unless you have a medical aversion, nothing will kill you if used in extreme moderation (ie: once a week).

take care, good luck
j