I'd go with MarkLogic for that. If the large documents are segmentable. I've run upwards of 100 million XML document files comfortably on a three node cluster.

Kurt Cagle
Principle Evangelist, Semantic Technologies

Avalon Consulting, LLC

kurt.cagle@g..., personal

caglek@a..., business

443-837-8725

On Thu, Mar 5, 2015 at 5:08 PM, liam <liam@w...> wrote:

On 2015-03-05 19:05, Gareth Oakes wrote:

Querying 2.4PB (2400TB) of data is going to take a
long time regardless of how you do it.

The best way is probably not doing it, where possible - e.g. project a subset and query that.

[...]

MarkLogic is marketed as petabyte-scale,
but XML may not be the most optimal representation of the data, it is
worth considering the alternatives.

Luckily neither MarkLogic nor most other XML-native databases store XML; rather, they store an internal representation that can be mapped to the XPath and XQuery Data Model (the XDM).

A common mistake people make (I'm not saying you're making it, but people reading your post might jump to an incorrect conclusion) is thinking that XML databases store XML (e.g. store pointy brackets), which is a bit like thinking that relational databases operate on CSV files.

Distributed storage and query are probably the way forward,

Depending on the nature of the queries, that does seem likely.



--
Liam Quin, W3C
XML Activity Lead;
Digital publishing; HTML Accessibility

_______________________________________________________________________

XML-DEV is a publicly archived, unmoderated list hosted by OASIS
to support XML implementation and development. To minimize
spam in the archives, you must subscribe before posting.

[Un]Subscribe/change address: http://www.oasis-open.org/mlmanage/
Or unsubscribe: xml-dev-unsubscribe@l....org
subscribe: xml-dev-subscribe@l....org
List archive: http://lists.xml.org/archives/xml-dev/
List Guidelines: http://www.oasis-open.org/maillists/guidelines.php