Another thing to consider besides fruits are tubers (starchy, edible roots like potatoes and carrots). Here's an article that discusses that issue:

Did Cooked Tubers Spur the Evolution of Big Brains?

The debate in this article is whether it was the consumption of meat or tubers that allowed larger brain development. However, no one really seems to object to the idea that some prehistoric people may have consumed small to moderate amounts of starchy roots. The one thing that virtually everyone agrees to is that grain consumption wasn't much until about 40,000 - 10,000 years ago.