William Cook watches a new box set from the BFI that reveals the full extent of the German director’s genius – and insanity