"I don't think Alfonso was a big expert on Potter either. He was feeling his way through it more than I was."