You don't ever need to add or remove divs when you're not resizing the window. You just need one more than your window is able to display and move the divs up and down with a tiny offset.
The other times you can just change their content.
In our framework, we have an option to yank complex components out of the DOM and replace them with empty containers. And when they scroll back into view, we put them back (and activate them if they are newly rendered).
