Cart
[Home] [By Thread] [By Date] [Recent Entries]
Last time someone told me they had a big XML file it turned out to be only 500Kb, so my advice was all wrong. What do you mean by "big"? Michael Kay http://www.saxonica.com/ > -----Original Message----- > From: william.j.boxall@a... > [mailto:william.j.boxall@a...] > Sent: 07 January 2005 17:01 > To: xml-dev@l... > Subject: Large xml processing into SQL Server > > Hello, all.. I just joined this list and have a question that > may have been answered by someone before, but here it is. > > At work, we're working on a BEA Weblogic project (Java), and > part of it is I wrote an interface that takes an xml file, > processes it with a validation routine, and then stores it in > a database. I have found, though, that when they send me a > large xml file, I can't handle it. I get a transaction > timeout, which is set by Weblogic, but that can be increased. > But then I run into out of memory issues. I've done a lot > of research, and the best solution appears to be to figure > out a way to bulk load the xml document into SQL Server and > then do my data manipulation. For some reason, even when I > trim down the operations so that all I'm doing is a straight > insert into the database, it still takes way too long and times out. > > I am not an xml guru at all, and am getting a bit of an > education in the process of trying to find a workable > solution. Is there a bulk loader I can use? If so, where > and how do I use it? Should I use an embedded xml database > for processing? If so, are there detailed instructions, and > will that alleviate in any way storing the xml records into > SQL Server? > > I cannot load the entire xml file into memory, it's too big. > Is there some sort of streaming operation I can take advantage of? > > One solution I was messing with was to take the schema and > map it to the database fields through using sql:relation, > etc.. But the problem there is the schema is built from the > xml file, which is hierarchical in nature. I need to flatten > out the elements in the hierarchy in order to do the mapping > to database fields. I've searched all over the net for a way > to do this, and have found some cryptic solutions using xslt, > which I am totally unfamiliar with. Again, any help with > that would be gratefully appreciated. > > Thank you in advance! > Bill > > > This message is for the designated recipient only and may > contain privileged, proprietary, or otherwise private > information. If you have received it in error, please notify > the sender immediately and delete the original. Any other > use of the email by you is prohibited. > > ----------------------------------------------------------------- > The xml-dev list is sponsored by XML.org <http://www.xml.org>, an > initiative of OASIS <http://www.oasis-open.org> > > The list archives are at http://lists.xml.org/archives/xml-dev/ > > To subscribe or unsubscribe from this list use the subscription > manager: <http://www.oasis-open.org/mlmanage/index.php> > >
|
