Guilt and shame could be commonly used on you to always stay and never leave. And if you wish to be a good or exemplary person, you might believe that putting aside your own personal desires for those of others is the right and noble thing to do. In most cases, it seems that way, but it isn’t. How likely is it to be twisted or merely mistaken?
Yes, community is important. Yes, cooperation too. But at what cost? You must ask yourself and think critically about what is happening and what is expected of you. How virtuous is it, really? What are you sacrificing and what for? Is it worth it?
Pleasing is both hard and easy. It is hard in the sense that it forces you to give of yourself to satisfy others and easy in the sense that it saves you from the repercussions of not doing so. Why do you do it? Can you justify it? In what instances?
If you care about yourself and what will become of you, you must trace a path and carve out a space that allows you to be okay.
If you can't or would rather not completely cut out or go zero contact with a destructive, toxic, draining or just not right for you person or group of people, remember that you can also simply put some distance between you and/or separate. But if they're concerningly awful, or just not worth another second of your time, the further out of your life they are the better (so they don't keep messing with you and sabotaging you).