I relate to what you wrote (not the drunk mom part, but the drunk circle part, and coming of age in alcoholic environments). In the first ten years after my decision to not drink — because it wasn’t for me — and to not drug— because it wasn’t for me either —, I found it hard to explain myself at social gatherings, even provide a way to decline that I could live with, a way that was nonconfrontational, firm and true to myself.
That was a long time ago. Guess what: people hardly care, hardly even noticed, except here and there, except those who (routinely) take offense at those who make different choices. In other words, it was a bigger deal to me how I could say no than it was to the majority of people I gathered with.
Self-trust and trusting your own memories is healthier than worrying about popularity. After some time it will get easier.
Do you. Be you.
…
In the coming decade or two a lot of people all over the world will change their attitude to drinking, because alcohol has been confirmed as a carcinogen, on top of destroying whole communities.
Then you will be a trendsetter. ☺