New environment minister Nicolas Hulot revealed the plans on Thursday at a climate conference.
The minister did not specify if the police would ban the sale of petrol or diesels but said it would place a “heavy” burden on car makers.
French manufacturers Renault, Peugeot and Citroen are all partially owned by the French government and could force them to pursue hybrid and electric models.
Hulot said: “We are announcing the end of the sale of gasoline and diesel cars by 2040.”
It comes days after Volvo announced it will scrap petrol and diesel-only vehicles from its lineup by 2019.
The Swedish maker will replace internal combustion engine cars with hybrids and all-electric models.
Other car makers – including Renault – are already ditching smaller diesels from their cars, too, as the cost of meeting stringent emissions targets becomes too expensive.