In Rottenburg, the grandparents did not want to leave Germany. The grandfather had fought in World War I and believed his service record would protect him. In August 1942, Albert and Rosa Horkheimer were deported to the Theresienstadt ghetto. Five weeks after their arrival, they were taken to the Treblinka extermination camp and murdered.

Lilian's great uncle Ferdinand and great aunt Jenny Horkheimer were also deported to Theresienstadt. Both died there.