Because HA, simple to use solutions are very expensive, closed source, or operationally costly.
if you think you've got the secret sauce, and you've actually had to put it into action and still hit five nines, awesome. But IME doing that with something like PostgreSQL is non-trivial (read: costly). That's why FoundationDB looked so appealing (to me).
Biased opinion here (Aerospike CTO and Founder), but you might want to check out open-source Aerospike DB. Like Foundation, it's clustered by default, very very good at SSD / Flash, runs great in cloud deployments, etc.
It's been used in production by big ad houses like AppNexus as well as retailers like SnapDeal. Lots of miles on the code.
Was closed source for years, but went open source about 9 months ago.
No high availability on open-source. Meant to be used with partitioning in mind so if you need cross-partition transactions most of the time, it's slow.
VoltDB has fantastic HA, but yes, not in the OSS version.
You might be surprised how many apps deploy on VoltDB with cross-partition transactions making up a solid chunk of their workload. Yes, they're slower than partitioned operations, but they're still faster than MySQL much of the time.
Most of the apps we see partition very well, especially for writes. The fact that they can run 10k distributed aggregates a second to get a global view is something few other systems can touch.
If you don't need SQL, there are a few pretty compelling options out there, each not without their faults... just the same.
MongoDB, RethinkDB, ElasticSearch, Cassandra and I'm sure a number of others.. each of them have HA options (though RethinkDB is still a few months out for auto-failover iirc), and Cassandra has pretty close to linear scalability in production for some very large data loads. I really like each for different reasons, and would lean towards one or another depending on load.
Not to mention, my speculation is PostgreSQL will likely have an in-the-box replication with failover and/or multi-master solution in place within a few versions.
PostgreSQL has been speculated to get "it'll be all better RSN" failover for as long as I've been using it (around 8.4 IIRC). ;-)
But yeah, what made FoundationDB's SQL-Layer exciting (for me) was:
- For small clusters it was free.
- Automatic HA
- Operationally Inexpensive (talking admin time/effort/training; far cheaper than PostgreSQL)
- Horizontal Scalability
The things that didn't matter for the 99%:
- It wasn't very fast.
I don't have "big data" problems (I could invent some). Most small shops (I suspect) don't.
The problem I do have is 3AM pagers, availability, wearing too many hats, putting dozens of hours into learning and experimentation to get PostgreSQL to use the hardware it's put on effectively, coming up with complex CARP+REPLICATION+FAILOVER plans, ZFS snapshotting because PostgreSQL still can't match the backup/restore process any commercial database had nailed down two decades ago, backing up the snapshots, figuring out how to partition clients into different table-spaces, blah blah blah.
You sacrifice some single-client performance with FoundationDB, but you solve almost every other problem you've got. And you now have the option of deploying a couple extra nodes to exceed your previously fairly intractable TPS milestones.
It's so easy in fact, you can now autoscale your database with your application servers.
And for your 80% of smaller clients it's absolutely free.
Your description of what matters to many customers doesn't get enough appreciation. Its faster too often trumps it lets me sleep at night in the battle for attention.
Looks very different in terms of the sort of performance you'd likely expect out of it, though. My initial reading is that if you want acceptable performance you're likely forced into thinking about sharding - and even then you're going to be punished by SQLite's poor concurrency support.
SQLite's concurrency support is irrelevant. ActorDB is a sharded by default type of database. The point of ActorDB is to split your dataset into "actors". Actor is an individual unit of storage (a sqlite database). As such your entire database is scalable across any number of nodes.
A natural fit is something like dropbox, evernote, feedly, etc. Every user has his own actor and it is a full SQL database. Which makes the kind of queries dropbox would need very easy to do.
It also has a KV store option. This is basically a large table across all your servers. This table can have foreign key sub tables. From my understanding of FoundationDB this part is closer to what they were offering.
It's not irrelevant. It means your ability to perform concurrent transactions depends entirely upon your ability to decompose your data into a very large number of different actors, otherwise you're bound to be hampered by SQLite's global lock. If you decompose too far you'll end up doing cross-actor transactions, which per the documentation has a substantial performance impact.
This is not to rubbish it - I've not used it after all - but the claims being made for ActorDB are pretty far away from the claims made for Foundation.
> All distributed databases shard data. If you hammer at only a specific shard area, performance will be limited to the speed of that shard.
Agreed. And what I'm saying is that it appears that ActorDB's per-shard area concurrency is limited to one writer. And that means that SQLite's concurrency support is (contrary to your earlier post) extremely relevant: not just in terms of pure performance, but also ability to perform concurrent operations. If you need more concurrency, your only choice is to shard extremely heavily (which might mean you require more cross-shard operations, which are apparently slow).
As you say, some data models fit the actor model well, but this is still a far cry from the capabilities that were promised by FoundationDB.
> FoundationDB was single process. They had no per-node writer concurrency.
Interesting - I didn't know that. Even so, it depends on what kind of writer concurrency we're talking about, I guess - I presume that ActorDB is limited not just by having to run requests one at a time per-process (which is a legitimate tactic to avoid latching overheads and so on), but by also not being able to run any new transactions against an actor that's received a write until that write commits?
> The reason why I said sqlite concurrency support is irrelevant is because ActorDB serializes requests anyway. It must do so for safe replication.
Do you mean by this that the entire cluster can only perform one request at a time? Or am I misreading you?
Individual actors are individual units of replication. What one actor is writing is concurrent to what another is doing.
Read/write calls to an actor are sequential. I'm quite sure this is how other KV stores like Riak do as well. They have X units per server and those process requests sequentially. Their actual concurrency is basically how many units per server are running. They may interleave reads/writes per node or they may not.
ActorDB does not allow reads while a write is in progress. It is quite possible we will optimize this part in the future as it is quite doable.
In FoundationDB you never had to think about splitting your dataset into something like your "actors". All transactions are independent and parallelizable, unless when they touch a common key - in which case one of the transactions is retried (optimistic concurrency).
if you think you've got the secret sauce, and you've actually had to put it into action and still hit five nines, awesome. But IME doing that with something like PostgreSQL is non-trivial (read: costly). That's why FoundationDB looked so appealing (to me).