"document stores", "nosql databases" , ODBMSs.

I was asked to compare & contrast odbms systems to the new nosql datastores out there. So I have asked several people in the last few weeks....
Below, you`ll find a few selected replies - I do not pretend this list to be complete, but hopefully it will help triggering a discussion....

RVZ

Michael Stonebraker (MIT):
"Greene`s reply is perfectly reasonable. I think the "one size does not fit all" mantra -- which I have been espousing for some time -- is a good way to think about things. After all, the no SQL folks are themselves pretty diverse."

Mårten Gustaf Mickos (previously CEO of MySQL AB):
" I think Kaj had a great response. Generally, in the early days of a new term, it doesn't in my mind make sense to try to define it exactly or narrowly. It's only a term, and it will take years before we know how significant it is and whether it is justified as a category of its own. For instance, it took a long time for Web2.0 to become an acknowledged term with a useful and reasonably well defined meaning."

Eric Falsken (db4o):
"NoSQL database (like Google's BigTable data behind their Gears API) is an awkward sort of "almost-sql" or "sql-like".
But it ends up being a columnar-database. What you call a "document store" is a row-based database. Where records are stored together in a single clump, called the row. By eliminating strongly-typed columns, they can speed up i/o by many factors (data written to one place rather than many places) just in the insert/select operation. By intelligent use of indexes, they should be able to achieve some astounding benchmarks. The complexity of object relationships is their shared drawback. Being unable to handle things like inheritance and polymorphism is what stops them from becoming object databases. You can think of db4o as a "document-oriented database" which has been extended to support object-oriented principles. (each level of inheritance is a "document" that all gets related together.)"

Peter Neubauer (Neo Technology):
"We have not modeled an ODBMS on Neo4j yet, but if you look at e.g. the Ruby bindings , it fits very naturally into dynamic language paradigms, mapping the domain models almost directly onto nodes and relationships. We have written a couple of blogs on the topic, latest Emils classification of the NOSQL space,and there are approaches to turn Neo4j into something that resembles a OODBMS:
1. JRuby bindings , hiding Neo4j as the backend largely form the code, but still exposing the Traverser APIs baside the normal Ruby collection etc for deep graph traversals.
2. Jo4neo by Taylor Cowan , which is persisting objects via Java annotations.
3. Neo4j traversers, and Gremlin , for deep and fast graph traversals (really not OODBMS like, but VERY useful in todays data sets) .
It would be very interesting to have more conversation on these topics!"

Jan Lehnardt (CouchDB):
" For me, NoSQL is about choice. OODBs give users a choice. By that definition though, Excel is a NoSQL storage solution and I wouldn't support that idea :) I think, as usual, common sense needs to be applied. Prescriptive categorisation rarely helps but those who are in the business of categorising."

Miguel-Angel Sicilia (University of Alcalá):
"The NoSQL movement, according to Wikipedia today promotes "non-relational data stores that do not need a fixed schema". I do not believe ODBMS really fit with that view on data management. Also, other aspects of the NoSQL "philosophy" make ODBMS be far from them. However, NoSQL focuses on several problems of traditional relational databases for which ODBMS can be a good option, so that they can be considered to be "cousins" in some sense. I do not see NoSQL and ODBMS as overlapping, but as complementary solutions for non-traditional data management problems."

Manfred Jeusfeld (Tilburg University):
" I am a bit frightened by this development. Basically, people step back to the time prior to database systems. I heard similar ideas at a Dagstuhl seminar from IT experts of STATOIL (the Norwegian oil giant). They experienced that non-database data stores are much faster, and they appear to be willing to dump ACID for increased performance of some of their systems.
This is typical for a programmer's attitude. They want the data storage & access to be optimized for their application and do not care too much about interoperability and reliability. If every data items can be referenced by a 64bit memory address, why would we need a query language to access the data. Following memory pointers is certainly much faster. OODBs can serve as a bridge between the pure programming view and the database view."

Peter Norvig (Director of Research at Google Inc.):
"You should probably use what works for you and not worry about what people call it."

Floyd Marinescu (Chief Editor InfoQ ):
"I think that web development has become so old and mature now that people are discovering that certain types of applications are better off with different solutions than your standard doctrine 3 tier system with SQL database. Plus the whole web services phenomenon has opened people`s minds to the notion of infrastructure as a service and that is driving this as well. This is my anthropological view of things. ;) "

Matthew Barker (Versant Corporation):
"As Robert (Greene) mentioned, when everyone thought "one size fits all", SQL evolved from a "simple query language" to a "do-it-all" tool including creating, modifying, and reorganizing data, not just querying. Many object databases such as Versant have an "SQL-like" query language but the capabilities are limited to actually querying the database, not updates, creates, deletes, etc. When you use SQL to modify data, you break the OO paradigm of safety and encapsulation; in a large application, it very easily becomes a monster that is difficult if not impossible to control. If we reign it and use SQL for it's original purpose, querying data, then SQL can fit in nicely with object databases - but the monster it has become does not fit into object database technologies."

Martin F. Kraft (Software Architect):
" NoSQL is a very interesting topic, though a standardized API like the W3C proposal would be challenging to adopt, and even more to outperform native legacy OODBMS (non-sql) queries.
I see the need to improve SQL performance in object oriented use as with J2EE and understand that some of the SQL performance implementations for OO like GemFire are doing great, but don't solve the underlying root-cause: the SQL
overhead in non-sql data queries like object traversal and key/value lookup..
So far I would rather use RDBMS' and OODMBS' where they perform best, as Mr. Greene said "one size does not fit all"....
Some object databases provide (slow) SQL interfaces, and NoSQL should not mean no-QL"

Kenneth Rugg: (Progress):
" I was at the "Boston Big Data Summit" a few weeks ago and the NoSQL movement came up as a topic. The moderator, Curt Monash whose is on dbms2.com, had a pretty funny quote on the topic. He said "The NoSQL movement is a lot like the Ron Paul campaign - it consists of people who are dissatisfied with the status quo, whose dissatisfaction has a lot to do with insufficient liberty and/or excessive expenditure, and who otherwise don't have a whole lot in common with each other."

Andy Riebs (Software Engineer at HP):
"Interesting blog! (Stonebraker sounds a bit too much like he's defending his baby ) Greene's comments are sensible. Following through on the "one size doesn't fit all" theme, just how many simple databases are best implemented as flat text files? The "NoSQL" discussions are reminiscent of the old "RISC vs. CISC" arguments. While people usually understood the notion of a simpler instruction set, no one noticed that pipelining and
huge register sets were introduced in the same package. Now you can find all those elements in most architectures.
In the same sense, one presumes that many of the good ideas that survive the "NoSQL" debates will, in fact, end up in relational databases. Will that make the resulting products any more or less relational? Some problems will best be resolved with non-relational, non-SQL tools. Best bet for a NoSQL technology that will survive? Harvesting meaningful data from the log files of data centers with 20.000 servers! With a proper MapReduce implementation, it will be a thousand times more effective to distribute the processing to the source of the data, and return only pre-processed results to the consuming application, rather than hundreds of gigabytes of raw data. Of course, the real winner will be the one who can implement SQL on top of a MapReduce engine! "

Tobias Downer (MckoiDDB):
" Interesting read. I don't think you can really nail down exactly what a 'NoSQL' technology is. It's a rejection of the prevailing popular opinion that's been around for the last decade, which is that a data management system that doesn't support SQL is no way to manage 'real' data, and a forum for advocates of certain scalable data technologies to promote themselves.
It's been successful at getting people interested and excited about data systems outside the world of SQL which I think is really what the aim was. I wouldn't count on the word being in our lexicon for very long though or any products seriously branding themselves using this label, because these systems are likely to eventually come back around and support SQL like query languages in the future."

Warren Davidson (Objectivity):
" This is of interest to me, and should be to anyone interested in market change. Change is always difficult, but in the database world it seems especially so when you see how often an RDBMS is used even though it might be a technically poor decision. Since change is risky, in order for Objectivity, or Versant or Progress to get people to adopt its technology, you need momentum and corroborating support. The NoSQL movement lends credence to the first notion of change; one size does not fit all. So to have multiple technologies saying the same thing establishes credibility for change to begin, and from there people can start making the right technical choice, which may or may not be ODBMS. But this is ok, the industry needs the market to embrace a very simple concept of 'the right database for the right application'. They don't do that today, but the cloud movement is going this direction. And the NoSQL movement may help everyone.
As they say, a rising tide lifts all boats. :) "

Stefan Edlich (Beuth Hochschule):
" There are some databases which are document oriented and nosql as CouchDB and SimpleDB. And there are document oriented ones which are not nosql as Lotus or Jackrabbit (a really weird system I think). I think the interesting tool and user group is the nosql group which excludes the latter group (hopefully). So the article you mentioned describes nosql with products storing documents as attribute data and not documents as pure byte / data documents (which Jackrabbit does)."

Daniel Weinreb (previously co-founder Object Design):
"They're being used not just as caches but as primary stores of data. There's one called Tokyo Tiger (and Tokyo Cabinet) that I've heard is particularly good."

William Cook (University of Texas at Austin):
" I think it is important! Facebook is using this style of data store. I'm not sure about the performance implications, but it needs to be studied carefully."

Raphael Jolly (Databeans):
"That a database could be designed without SQL is not a surprise to me since Databeans has no query language and is meant to be queried in Java (native queries). In addition, I'll happily believe that "relational databases are tricky to scale".
However, the subject of extending Databeans with distributed computing capability has been on my mind for a long time and I presently have no idea how it could be done. What is interesting about NoSQL is how they mean to perform queries, i.e. through MapReduce. I don't know whether everything that can be expressed in SQL is amenable to MapReduce (this is probably not the case), but obviously a fair amount of what is done today on the internet is, the killer app being... search engines.
In summary, I tend to agree with this comment by alphadogg: "The main reason this [relegating to nich usage] will happen with the various key-value stores that are now in vogue is that they, like the predecessors, are built to solve a very, very specific issue on high-volume, low-complexity data, which is not the norm. The NoSQL moniker is indicative of a misplaced focus. NO structured query language? Really? We want to go back to chasing pointers and recreating wrappers for access? ".
My perception is that even when they literally have no SQL based queries, object databases are very different from NoSQL technologies as currently understood because, as is clearly explained in your references, there is more to ODBMS than just the query language. Specifically : ACID transaction constraints, which "NoSQL" seem to relax quite a bit.
These constraints are difficult to manage in a distributed setting. One has to consider advanced concurrency control techniques. But with a careful design, nothing seems to prevent a fully structured approach.
In this respect, DHTs are clearly limited compared to classical object databases. Recently, I was reading about such an attempt a distributed design, yet with "strict" transactions: (download the book "XSTM: Object Replication for Java, .NET & GWT " as .pdf). "

Steve Graves (McObject):
" I thought alphadogg had good (comments), although he has a relational/SQL bias."

Jonathan Gennick (Apress):
" it is an interesting discussion. I have heard the term "NoSQL". I did find the comment about relational databases not supporting key/value stores amusing: "...and index key/value based data, another key characteristic of "NoSQL" technology. The schema seems very simple but may be challenging to implement in a relational database because the value type is arbitrary."
In Oracle, one simply needs a table as follows:

CREATE TABLE key_value (
the_key NUMBER,
the_value BLOB);

There you go! Key/value. How much simpler can you get? "

##

Data Stores vs. ODBMSs

I asked Anat Gafni, VP of Engineering at db4objects, on her opinion on how ODBMSs compare with respect to new "data stores", such as "document stores", and "nosql databases".

RVZ: Anat, systems such as CouchDB, MongoDB, SimpleDB, Voldemort, Scalaris, etc. provide less functionality than OODBs but a distributed "object" cache over multiple machines. How do they compare with respect to ODBMSs?

Anat: We can categorize "stores" and see if they are more similar or different than OODBs, in a couple of ways:

By each dimension of the purpose of OODBS:

1. persistent (could be accomplished by other methods like: replicating to other machines, using non-volatile caches, etc.)
2. Being queriable
3. Scalable (beyond what can be in cache, but could be distributed instead)
4. Objects vs.Relations

Arguable properties:
5. can express and query based on complex relationships among data items
6. can be shared among multiple "clients"

Many of these other database are similar to oodbs in item 1, 3 and 4.
I am not sure they have capabilities in 3, 5 and 6 above.

There is a lot of interest in the USA in particular, w.r.t. to internet based applications and cloud computing.
Big_table, and such look to me more like an algorithm based on traditional stores, rather than a db.
##

Anat Gafni has over 20 years of experience in managing software development and product strategy.
At db4objects she is responsible for managing engineering and support.
Anat earned her Ph.D. in Computer Science from the University of Southern California, and a MA degree from Boston University in Math and Computer Science.

Call for Contributions: 3rd International Conference on Objects and Databases 2010 (ICOODB)

I`d like to inform you that the 3rd International Conference on Objects and Databases (ICOODB), will take place September 28-30, 2010, at the Goethe University Frankfurt, in Frankfurt am Main, Germany.

ICOODB 2010 is the third in a series of international conferences aimed at promoting the exchange of information and ideas between members of the objects and databases communities.

A key feature of the conference is its goal to bring together developers, users and researchers. At the same time, the conference aims to meet the needs of the different sub-communities. The conference therefore consists of three different tracks offered as a tutorial & workshop day, an industry day and a research day.

The call for contributions is open. For 2010 we extended the scope of the conference. We recognize that the world of data management is changing. The linkage to service platforms, operation within scalable (cloud) platforms, object-relational bindings, NoSQL databases, and new approaches to concurrency control are all becoming hot topics both in academia and industry. We therefore also encourage papers in any of these areas.

Therefore the general topics of interest include, but are not restricted to:

Object Database Modeling and Design
object data models
design of object databases
semantics of object databases

Object Database Frameworks and Software Engineering
application development and application frameworks
software engineering issues
constraint models and mechanisms
event models and mechanisms
object-oriented frameworks for data management
ontologies and object stores

Object Databases
object storage systems
object query languages
transaction management for object databases
access structures and indexing in object databases
distributed object databases
architecture and engineering of object database engines
evaluation of object databases
benchmarks
novel applications of object databases
object databases in education
Integration of Object-Oriented Programming Languages with databases

Integrating objects and databases
Novel design of Programming Languages for objects and database

Object/Relational mappings
Novel design and/or Implementation of O/R Systems.
Benchmarks

Cloud Data Stores
Novel design and/or Implementation of Cloud Data Stores
Novel Application of Cloud Data Stores
Benchmarks

Document Stores
Novel design and/or Implementation of Document Stores
Novel Application of Document Stores

NoSQL Databases
Novel design and/or Implementation of NoSQL Databases
Novel Application of NoSQL Databases
Benchmarks

Here are the links to the call for contributions:

Call for Research Papers.

Call for Industry Presentations.

Call for Tutorial Proposals.

##

Rick Cattell on "Relational Databases, Object Databases, Key-Value Stores, Document Stores, and Extensible Record Stores: A Comparison."

What`s new at ODBMS.ORG in 2010?

I have extended the focus of ODBMS.ORG to include, besides object database technologies, new developments in data management, such as the linkage to service platforms, operation within scalable (cloud) platforms, object-relational bindings, NoSQL databases and new approaches to concurrency control.

ODBMS.ORG will offer in 2010 educational resources in all of these areas.

I have just published a new expert article by Rick Cattell on this topic:
"Relational Databases, Object Databases, Key-Value Stores, Document Stores, and Extensible Record Stores: A Comparison".

Rick Cattell, formerly at Sun Microsystems, and co-creator of JDBC, and chair of the Object Data Management Group (ODMG), explains: "Traditionally, the obvious platform for most database applications has been a relational DBMS. You might use a specialized parallel relational DBMS if you required high throughput for “data warehousing”, or an object database system if your application had unusual functionality or performance requirements, e.g. for in-memory caching or fast relationship traversal. However, an RDBMS like Oracle or MySQL has usually been the answer. This has changed somewhat recently.

There is now recognition in database research that “one size does not fit all”, for example in the widely-referenced paper by Stonebraker and colleagues.

And in the Web 2.0 industry, many companies have abandoned traditional RDBMSs for so-called “NoSQL” data stores that provide much higher scalability, or they have built a distributed caching layer on top of RDBMSs. More scalable RDBMSs are also coming to market" so Cattell.

Rick`s article ""Relational Databases, Object Databases, Key-Value Stores, Document Stores, and Extensible Record Stores: A Comparison" is available for free download as (PDF)..
Worth reading it...

In fact, we already started last year to look at new developments in data management such as NoSQL databases and Cloud Stores.
An example is the article "On NoSQL technologies. Part I" (PDF) which presents interviews on new data stores with Patrick Linskey, Robert Charles Greene, Kaj Arno and Giuseppe Maxia.

RVZ

"NoSQL technologies" interview with John Clapperton

Happy New Year!

We start in the new year with a topic we covered already in 2009: "NoSQL technologies".

I asked this time John Clapperton for his opinion. John Clapperton BSc CEng MBCS CITP is proprietor and author of the 'VOSS' virtual object storage system, which extends Smalltalk with integrated database management, providing transparent access and transaction processing of persistent, versioned, Smalltalk objects. Previously, John has worked on database applications and research at Standard Telephones & Cables, Unilever Research, Acorn Computers and Deductive Systems.

RVZ: John, are object databases "NoSQL" technologies?:

John Clapperton:: The absence of SQL in "NoSQL" databases is less an a priori choice than a consequence of their simplified schema capability, imposed in the interests of higher performance, being unable to support the full set of SQL language constructs. It does not follow, therefore, that object databases from which SQL has been excluded for the opposite reason, as a language unable to address their more general representational capabilities, should be automatically included in the NoSQL classification.

A person thinking of adopting a NoSQL database will have certain capabilities in mind, so the question is really "Might an object database have 'NoSQL' capabilities?"

These include:

1) Data partition (which is application dependent).
2) Optimistic locking (which helps only if most accesses are read-only).
3) Relaxation of ACID transaction rules by:

a) Data replication with eventual consistency, and/or

b) Suppression of transaction logging and/or flushing, and/or

c) Data storage in fast but volatile memory, sacrificing durability.
4) Fast navigational access to arbitrary data structures.

and in principle, an object database is capable of any or all of these.

The characteristics of an object database are its ability to manage arbitrarily complex object structures and to represent relationships by explicit named references. These have the potential for better performance by, respectively, reducing the required number of file writes for (de-normalised) data structures, and fast navigation of direct references instead of relational joins. However, against that must be set the cost of serialising arbitrary object structures for durable storage and instantiating the same on retrieval, compared with the simpler handling of pre-defined rows in a relational database.

Normalisation of behaviour, encapsulated in class definitions in language persistence odbms such as Logic Arts' VOSS for Smalltalk, reducing implicit replication in application programs and queries, may have an advantage in NoSQL applications, but it's not clear to me how significant that might be, given that its benefit is in managing complexity whereas NoSQL applications tend to be simpler.

"Nonschematic" databases.

Carl Olofson, Research Vice President, Database Management and Data Integration Software Research, at IDC, sent me this note, where he argues about the term "NoSQL" in relation with object databases.

RVZ

Carl Olofson:
I would shy away from this term. A number of analysts (including myself) consider it a somewhat sloppy term intended to convey a certain spirit of rebellion. It actually derives from the core idea that the so-called "No-SQL" databases do not require schemas, and since most DBMSs are relational, it is simpler to say "NoSQL" than the more obscure "NoDDL".
In fact, OODBMS does require a schema, and the data structure, which is tied to the application object model, is key to how it operates, and especially to its transparent operational nature. The so-called "NoSQL" database, which I call a "nonschematic" database, is one that requires no schema to be defined before data is loaded. One usually does define a schema afterward, through a process of data discovery and definition. If you know of a OODBMS that can accept undefined data, and allow schema definition after the fact, that could qualify. Otherwise, I would shy away from the term altogether.

Are object databases "NoSQL" technologies? Part II

I asked the opinion of another ODBMS vendor on the topic of "NoSQL databases": Luis Ramos who is Principal Systems Engineer at Progress Software.

RVZ: Luis, how do you position yourself with respect to the so called "NoSQL" databases?:

Luis Ramos: We view many of the characteristics of the growing "NoSQL" movement as a market reaction to the realities of present day cloud-based data requirements, where ACID properties are not as important as performance, the bulk of the data's schema is not as complex, and the corresponding queries are relatively simple. Gone could be the days of complex relational schemas and the DBAs that are needed to maintain and administer them. Similar phenomena have been seen in other areas. For example in programming languages, the reaction against the very complex and error prone C++ led to the popularity of Java.

In many respects, object databases can be classified as "NoSQL" technology. It satisfies many of the pivotal characteristics of today's "NoSQL" data stores. Object databases have been around since the late 1980s in response to the needs and requirements initially of the CAD market. At that time, the CAD practitioners needed an approach to data management that was fundamentally different than that provided by the relational databases. Consequently, a whole new breed of non relational (object-oriented) databases emerged. Customers from other markets, whose requirements could not be met by SQL databases, followed. Call it the original "NoSQL" movement? We certainly agree with Robert Greene's stipulation that "one size does not fit all."
An alternative way to put it is "You can put lipstick on a 'relational table' but its still a 'relational table'".

The schema-free characteristic that one finds in many "NoSQL" technologies is not entirely new. This is a requirement of many eCommerce applications developed in the 90s. There are object databases that support this nicely, enabling applications to store, manage, and index key/value based data, another key characteristic of "NoSQL" technology. The schema seems very simple but may be challenging to implement in a relational database because the value type is arbitrary.

The horizontal scaling characteristic is another key requirement that object databases more easily supports. Multiple terabytes databases have been successfully deployed. These object database systems have a client-centric (rather than a server-centric) architecture. Data is distributed to the client and queries are performed on the client instead of on monolithic servers. Consequently, the data can be partitioned, replicated, and scaled much more easily without being tied down to the hardware limitations of a single server computer.

So indeed, object database systems could be considered "NoSQL" technologies. They can be utilized either as a persistent store for data as well as a cache.