java - Activemq went out of memory when sending large files over queue -

-

when sending 100mb size messages using queue, activemq runs out of memory error, using file cursor queue -

queue detail - our producer sending non-persistent messages @ size of 100mb per messages , producer keep producing through while loop of same 100mb messages.

we use default heap size came activemq 1gb max.

we have activemq config setting follows:

<policyentry queue=">" producerflowcontrol="false" memorylimit="512mb" maxpagesize="1000000">                <pendingqueuepolicy>                             <filequeuecursor />  </pendingqueuepolicy> </policyentry>

on consume side have async consumer keep listening on messages coming in , send auto-ack.

after program runs while activemq throws following error:

    2016-04-21 14:52:18,961 | error | error in thread 'activemq brokerservice.worker.1' | org.apache.activemq.broker.brokerservice | activemq brokerservice.worker.1 java.lang.outofmemoryerror: java heap space     @ org.apache.activemq.util.databytearrayoutputstream.ensureenoughbuffer(databytearrayoutputstream.java:249)[activemq-client-5.13.1.jar:5.13.1]     @ org.apache.activemq.util.databytearrayoutputstream.writeboolean(databytearrayoutputstream.java:140)[activemq-client-5.13.1.jar:5.13.1]     @ org.apache.activemq.openwire.v11.basedatastreammarshaller.loosemarshalbytesequence(basedatastreammarshaller.java:627)[activemq-client-5.13.1.jar:5.13.1]     @ org.apache.activemq.openwire.v11.messagemarshaller.loosemarshal(messagemarshaller.java:300)[activemq-client-5.13.1.jar:5.13.1]     @ org.apache.activemq.openwire.v11.activemqmessagemarshaller.loosemarshal(activemqmessagemarshaller.java:111)[activemq-client-5.13.1.jar:5.13.1]     @ org.apache.activemq.openwire.v11.activemqtextmessagemarshaller.loosemarshal(activemqtextmessagemarshaller.java:111)[activemq-client-5.13.1.jar:5.13.1]     @ org.apache.activemq.openwire.openwireformat.marshal(openwireformat.java:161)[activemq-client-5.13.1.jar:5.13.1]     @ org.apache.activemq.broker.region.cursors.filependingmessagecursor.getbytesequence(filependingmessagecursor.java:480)[activemq-broker-5.13.1.jar:5.13.1]     @ org.apache.activemq.broker.region.cursors.filependingmessagecursor.flushtodisk(filependingmessagecursor.java:440)[activemq-broker-5.13.1.jar:5.13.1]     @ org.apache.activemq.broker.region.cursors.filependingmessagecursor.onusagechanged(filependingmessagecursor.java:401)[activemq-broker-5.13.1.jar:5.13.1]     @ org.apache.activemq.usage.usage$1.run(usage.java:308)[activemq-client-5.13.1.jar:5.13.1]     @ java.util.concurrent.threadpoolexecutor.runworker(threadpoolexecutor.java:1142)[:1.8.0_74]     @ java.util.concurrent.threadpoolexecutor$worker.run(threadpoolexecutor.java:617)[:1.8.0_74]     @ java.lang.thread.run(thread.java:745)[:1.8.0_74]

does knows how resolve this? doesn't seems happening when keep sending smaller size messages, example, messages less 10mb.

non-persistent messages going stored in memory rather persisted data store, described here. 1gb going disappear quickly, if can't consume fast produce.

of course, increase amount of memory allocated activemq in activemq.xml, might better off persisting, if don't need recover, , potentially expire messages after period of time emulate non-persistence (if necessary).

i'd suggest other solutions, such breaking messages more manageable or using shared file storage data , sending messages including pointers data. besides processing overhead activemq, believe have larger-than-normal network impact (for example, if have secure communications amq instance, you're encrypting/decrypting 100m messages, not cheap).


Comments

Post a Comment

Popular posts from this blog

Django REST Framework perform_create: You cannot call `.save()` after accessing `serializer.data` -

-
i have following viewset: class artistprofileviewset(viewsets.modelviewset): queryset = artistprofile.objects.all() serializer_class = artistprofileserializer def perform_create(self, serializer): if serializer.is_valid(): serializer.save() with following serializers: class simpleartisttrackserializer(serializers.hyperlinkedmodelserializer): class meta: model = artisttrack fields = (...my fields...) class artistprofileserializer(serializers.hyperlinkedmodelserializer): owners = userserializer(many=true, required=false) tracks = simpleartisttrackserializer(many=true, required=false) class meta: model = artistprofile fields = (...my fields...) i getting following error: assertionerror: cannot call `.save()` after accessing `serializer.data`.if need access data before committing database inspect 'serializer.validated_data' instead. i don't see editing serializer.data object. th...
Read more

Why does Go error when trying to marshal this JSON? -

-
i'm trying rather simple example of trying decode json file structs using golang. however, when attempting error that json: cannot unmarshal object go value of type []main.track which don't understand. here's code in question, , json can found here . package main import ( "encoding/json" "fmt" "net/http" "os" ) // tracks group of track types type tracks struct { tracks []track } // track represents singular track type track struct { name string url string artist artist } // artist represents single artist type artist struct { name string mbid string url string } func main() { url := "http://ws.audioscrobbler.com/2.0/?method=geo.gettoptracks&api_key=c1572082105bd40d247836b5c1819623&format=json&country=netherlands" response, err := http.get(url) if err != nil { fmt.printf("%s\n", err) os.exit(1) } de...
Read more