Well - according to The Le Mans 24-Hour Race 1949-1973, publ. 1974, in October 1922, Charles Faroux and Emile Coquille met with Georges Durand "to discuss a new form of motoring event based on endurance rather than speed, which was to demonstrate the reliability of the cars, and bring about essential improvements.  Electrical systems then left much to be desired and tyres were still far from robust.  Faroux proposed a race for touring cars - as catalogued - lasting 8 hours, 4 in daylight, 4 in darkness, whereupon Durand cried "Why not 24 Hours?" ".
Although that doesn't really answer your question, I suspect a 4 pm start allows for marshalls etc to meet and get into position in good daylight, and allows time for people to get home afterwards.  It gives a fair time in daylight and night.  The race is not always held close to the longest day - so this would give even time to day and night driving regardless of when held.

 
Hubby and I can't be there this year!! Miss the atmosphere already!  Am sitting at the computer listenting to Radio LeMans.  *sigh* Hopefully we'll make it in two years for our 35th anniversary - and a big 75 for LeMans!!
Yes some women do get the 24 hours - but don't worry about trying to explain it to those who don't! Tell her to enjoy her weekend to herself!  (as if the prompting is needed! )
Hoist a few beers for Edmonton Alberta (Canada eh!)