It's interesting to hear that has worked well, obviously this wasn't a small project. Your point about knowing how to use your tool definitely rings true. Also interesting that you had a use case where data loss and integrity actually mattered and in real time, unlike a social network or most start ups operating today. Going with a heavy oracle system instead of trying to roll your own creative distributed architecture definitely seems to make sense in that scenario. Just out curiosity, was this Java/Hibernate?
On one system we used Java/Oracle/Hibernate and went with the big single cluster. The other system was a .NET stack, using NHibernate and a large number of SQLServer instances. We also worked with Microsoft on integrating their latest (at the time beta) caching servers. We did indeed have to roll our own distributed architecture in that case, but it's not like we had to drop ORM to do it.
