Isn't that just basically saying you're fighting yourself, are afraid of yourself or want to get rid of yourself?

Thats exactly it. Fear of yourself and what you think, of what you are capable of and aspects of yourself. Also some other may have memories you would rather not face so of course you will hate them, you hate the memories and what they bring you.