Okay, let's start with a typical late game situation - http://i.imgur.com/AW1ntdw.png
Late game as in "you start steamrolling".
There are several factors that lead to it.
First, the food production model is quite complex.
On a first glance we've got several tiers of planets: paradise and coral have base output 5, island and native for the race have 3, glaciers have 2, arid has 1, barrens, ice and inferno have 0.
We've got 3 tiers of farms, each tier of farms gives planet_base_output+2, +4 and +6 respectively.
Then we run into exceptions or variances depending on the temperature (it's not exactly clear), however, a third tier farm on a paradise planet gives 10 food, not 11 and on a coral it gives 9.
But the food consumption model is fairly simple - you sum your base output and all farms and substract the population.
Ok, so what do we have here?
Planet size 5 and lower have 4 improvement slots.
Planets 6-10 have 5 improvement slots.
Planets 11-18 have 6 improvement slots
And planets with size of 19 and more have 7 improvement slots.
As a rule of thumb (with bionomics), all planets with cap less than 5 can't have farming, most of planets capped on 6 can't too.
To sustain itself a 6-10 colony has to spend 20-40% of its capacity on farms (keep in mind that low cap means low base output).
A 11-18 colony needs to spend 33-50% of its capacity on farms.
A 19+ colony needs to spend 28-72% of its capacity on farms. In the worst case (Gaia, 51 cap and no farm techs) you'll have to cover it all with farms.
Ok, we've got an empire, we got freighters, we got the technology and it's clear that the planets should be specialized.
Well, it works while you're sitting in your corner and don't start invasions.
And you'll need invasions as bombing colonies into oblivion without a "drop all" button is pretty exhausting, starting a war with anyone who'll decide to settle on the freed space is exhausting too.
And then you get unsustainable planets falling into your lap, by twos and by threes. And while you can buy the first farm for 150 gold, they aren't enough and you need the money for the actual war.
The starvation starts and it works in an interesting way.
The population decreases in a proportion to the colony's food balance, so the specialized worlds are hit instantly and hit the hardest. Your largely useless worlds get out unscatched, but your prized worlds suffer.
Ironically, the population increase speed does not depend on the food surplus (if you count out that special project you won't be starting on your best worlds, because they are researching or churning out ships all the time).
So, yes, it's avoidable by painstakingly preparing for a war, as it doesn't make sense to have a huge food surplus all the time, you'd better use that space on labs.
But whether you can prepare depends on the map and not on your skill.
Sure, in the late game you can terraform.
But the terraformation rules are quite complex (why I can convert some inferno worlds into gardens, but not every one of them, and why can't I do the same with arids?) and it's a long or costly ordeal.
It would be better with an project "terraform and colonize" using the resources of a developed planet, not 1500 or 3000 out of your pocket you may not have.
Freigthers are nice in the early game.
However in the midgame you start yearning for a "produce straight to pool" option.
In the late game you also have stargates and can transport population instantly...
...but the food is still distributed by freighters and is a subject to freighters shortage or blockades.
As in the very late game you can easily have 600-800 food transported around, well, it's easy to do the math and the number of "send to trade" clicks.
So it goes.
After a crisis or two you just can't force yourself to take it all into account and just hate the mechanics.
Interestingly, the sweet spot for a specialized world is about 19 or 20 population cap.
Top-tier factories and labs give +20 which is roughly equivalent to 10 pop.
However not bringing in risks of a sudden -3 pop is far better than expanding the planet to 30-35 cap.