Nussbaum was born in Osnabreck, Province of Hanover, as the son of Rahel and Phillip Nussbaum. Phillip was a World War I veteran and German patriot before the rise of the Nazis. He was an amateur painter when he was younger, but was forced to pursue other means of work for financial reasons. He therefore encouraged his son's artwork passionately. Nussbaum was a lifelong student, beginning his formal studies in 1920 in Hamburg and Berlin and continuing as long as the current political situation allowed him. In his earlier works, Felix was heavily influenced by Vincent Van Gogh and Henri Rousseau and he eventually pays homage to Giorgio de Chirico and Carlo Carre as well. Carl Hofer's expressionist paineng influenced Felix's careful approach color. In 1933, Nussbaum was studying on scholarship in Rome at the Berlin Academy of the Arts when the Nazis gained control of Germany.