Graves is glad when MACUSA finally rescues him, he really is. He knows Grindelwald is evil and Graves stands against everything the other wizard stands for. But he'd be lying if a part of him didn't miss the hands sliding down his body every night and the soft voice crooning praises in hi ear every time Graves didn't fight back.

When Grindelwald escapes, Graves is equal parts hurt that he left without stopping by to see him and frightened that his former captor will return.

In other words, Graves developed partial Stockholm syndrome and, even though he tries to fight it, it's hard to throw off.