Each of the territories are counted for the collapse of Vichy France. If the Allies invade NA - all four territitories go German - but for rolling for Vichy collapse purposes - each is counted individually. I will add an example to make this more clear. Per the rules:
Vichy France is fragile in the same manner as Italy and France. At the end of every Allied turn in which one or more of the following territories are Allied occupied, the German player rolls one die for Vichy French capitulation. The territories that trigger a Vichy die roll are: Madagascar, Syria, Corsica, Tunisia, Oran, Algiers, and Morocco. If the die roll is equal or less than the number of Allied occupied Vichy territories, Vichy France surrenders. (e.g. If the Allies control 2 Vichy French territories, on a "2" or less, Vichy France surrenders). Upon Vichy surrender, all Vichy infantry units are removed from play.