November is our favorite time to go to WDW and the weather is usually warm enough for swimming. There may be a few cool days and one year, it got downright cold for a few days, but that's been the exception.
The crowds aren't bad until you get to Thanksgiving week.
November is also one of the months with the least rain so it's a good balance all around.