java - Activemq went out of memory when sending large files over queue -

when sending 100mb size messages using queue, activemq runs out of memory error, using file cursor queue -

queue detail - our producer sending non-persistent messages @ size of 100mb per messages , producer keep producing through while loop of same 100mb messages.

we use default heap size came activemq 1gb max.

we have activemq config setting follows:

<policyentry queue=">" producerflowcontrol="false" memorylimit="512mb" maxpagesize="1000000">                <pendingqueuepolicy>                             <filequeuecursor />  </pendingqueuepolicy> </policyentry>   

on consume side have async consumer keep listening on messages coming in , send auto-ack.

after program runs while activemq throws following error:

    2016-04-21 14:52:18,961 | error | error in thread 'activemq brokerservice.worker.1' | org.apache.activemq.broker.brokerservice | activemq brokerservice.worker.1 java.lang.outofmemoryerror: java heap space     @ org.apache.activemq.util.databytearrayoutputstream.ensureenoughbuffer(databytearrayoutputstream.java:249)[activemq-client-5.13.1.jar:5.13.1]     @ org.apache.activemq.util.databytearrayoutputstream.writeboolean(databytearrayoutputstream.java:140)[activemq-client-5.13.1.jar:5.13.1]     @ org.apache.activemq.openwire.v11.basedatastreammarshaller.loosemarshalbytesequence(basedatastreammarshaller.java:627)[activemq-client-5.13.1.jar:5.13.1]     @ org.apache.activemq.openwire.v11.messagemarshaller.loosemarshal(messagemarshaller.java:300)[activemq-client-5.13.1.jar:5.13.1]     @ org.apache.activemq.openwire.v11.activemqmessagemarshaller.loosemarshal(activemqmessagemarshaller.java:111)[activemq-client-5.13.1.jar:5.13.1]     @ org.apache.activemq.openwire.v11.activemqtextmessagemarshaller.loosemarshal(activemqtextmessagemarshaller.java:111)[activemq-client-5.13.1.jar:5.13.1]     @ org.apache.activemq.openwire.openwireformat.marshal(openwireformat.java:161)[activemq-client-5.13.1.jar:5.13.1]     @ org.apache.activemq.broker.region.cursors.filependingmessagecursor.getbytesequence(filependingmessagecursor.java:480)[activemq-broker-5.13.1.jar:5.13.1]     @ org.apache.activemq.broker.region.cursors.filependingmessagecursor.flushtodisk(filependingmessagecursor.java:440)[activemq-broker-5.13.1.jar:5.13.1]     @ org.apache.activemq.broker.region.cursors.filependingmessagecursor.onusagechanged(filependingmessagecursor.java:401)[activemq-broker-5.13.1.jar:5.13.1]     @ org.apache.activemq.usage.usage$1.run(usage.java:308)[activemq-client-5.13.1.jar:5.13.1]     @ java.util.concurrent.threadpoolexecutor.runworker(threadpoolexecutor.java:1142)[:1.8.0_74]     @ java.util.concurrent.threadpoolexecutor$worker.run(threadpoolexecutor.java:617)[:1.8.0_74]     @ java.lang.thread.run(thread.java:745)[:1.8.0_74] 

does knows how resolve this? doesn't seems happening when keep sending smaller size messages, example, messages less 10mb.

non-persistent messages going stored in memory rather persisted data store, described here. 1gb going disappear quickly, if can't consume fast produce.

of course, increase amount of memory allocated activemq in activemq.xml, might better off persisting, if don't need recover, , potentially expire messages after period of time emulate non-persistence (if necessary).

i'd suggest other solutions, such breaking messages more manageable or using shared file storage data , sending messages including pointers data. besides processing overhead activemq, believe have larger-than-normal network impact (for example, if have secure communications amq instance, you're encrypting/decrypting 100m messages, not cheap).