That thing about the framerate makes a lot of sense. Too many polygons decreases the framerate because of the collision testing I guess.
I don't think the file size will make a big difference, but it makes sense also. A list of polygon indexes is stored apart, divided by sectors, for those polygons which collide, I think.