Toji doesn't want to marry you. Not because it's you, just because, well, he doesn't want to marry anyone. He doesn't want to listen to the Zenin clan at all. Marrying him off to the highest bidder, your parents, because he wasn't a sorcerer. He hates you by default, even though it's not your fault.