OutOfmemory exception when processing a large number of documents
Hello,
I want to process a large number of documents (around 3 million). The process consists of updating a metadata of each document from another metadata (so I will need to do a read and write from the database).
For that, I was thinking of using elasticsearch's SCROLL API.
The problem is that I will have a “java.lang.OutOfMemoryError: GC overhead limit exceeded” exception in the middle of processing (knowing that I have xmx = xms = 24g in JAVA_OPTS)
I have tried different configuration for the garbage collector but no great effect noticed.
Can someone help me or give me an idea how to process a large batch of documents in nuxeo.
Thank you in advance.
 |
Ahmad Ben Maallem
Member (1.9K) 2 4 7 |
04/15/2021 ( History) |
Hello,
I have used the bulkAction mechanism in nuxeo and it looks good.
Thank you
|
Ahmad Ben Maallem
Member (1.9K) 2 4 7 |
04/16/2021 ( History) |
Can you help me how to use bulk upload with custom metadata?
 |
sujoy debnath
Member (80) 1 0 0 |
04/21/2021 ( History) |
What type of bulk upload do you use ?
for example, the following doc https://doc.nuxeo.com/nxdoc/nuxeo-bulk-document-importer/ explains how to use Nuxeo Bulk Document Importer.
there are other types like for example the csv importer…