When you make stuff nobody hates, nobody loves it either.
You also don't need the masses to unanimously love you to be successful and/or make a good living.
It's good to fully commit to what you think is right and build that. Even in the face of doubters. Be the doer.
If we water down everything we do in an attempt to appease all people, we make dispassionate art with no soul.
Don't let the critical ones dictate your decisions.