Will ingesting milk while lactose intolerant cause harm?

I am moderately diagnosed to be allergic to dairy products. But it's not so severe as to where I'll get extremely bloated and gas up the room. It just upsets my stomach but I love the taste and texture of 1% pure milk. I don't like the imitators and soy and rice based products make me gag. But am I doing any damage for longterm?