Memory Being Consumed

classic Classic list List threaded Threaded
4 messages Options
Reply | Threaded
Open this post in threaded view
|

Memory Being Consumed

michaelbinary
This post has NOT been accepted by the mailing list yet.
HI,
I am using quickfix to send price updates at the rate of about 1000 per second, steady state.
The resident memory of my process is slowly increasing at the rate of 1MB every 17 secs and it seems pretty constant so its related to the message rate.
So it seems to be eating about 60 bytes per message.

Has anybody got any ideas as to what its doing or how to stop it ?.

Is it caching something in memory that I can stop ?.


regards

Michael
Reply | Threaded
Open this post in threaded view
|

Re: Memory Being Consumed

michaelbinary
This post has NOT been accepted by the mailing list yet.
I have discovered why so as a reference for anybody else who comes across this, the explanation is.
The QF message store seems to keep  in memory some form of reference to each outgoing message for recovery purposes so I have determined from google research.

This is fine if you send 20 messages a second, who cares, but I am sending 2000 price updates a second, and it matters, ......a lot, by the end of the day resident process size is over 4GB and growing.

What a effing stupid thing to do.

Oh yes, you could write your own file store, but why ?., Why should you have to rewrite part of a product just to get the file store to work properly on commercial sized data sets.

I am just lucky that replaying price updates after the fact is pointless, so we don't need the file store so set PersistMessages=N.

There may be other and better solutions to this problem, if so I would be interested to hear.

Hope this helps other readers scratching their heads.

regards

Michael
Reply | Threaded
Open this post in threaded view
|

Re: Memory Being Consumed

Viktor Pogrebnyak
hi, Michael.

If you use MemoryStore ( PersistMessages=N ) then you code suffers from missing MessageStore truncate functionality.

This issue is not so visible when you use FileStore since quickfix stored just file offsets in memory. But MemoryStore keeps ALL outgoing messages in memory which eats a lot of RAM when your outgoing message rate is high.

So to solve your troubles you might want to inherit from MemoryStore class and add truncate functionality into MessageStore::set() method e.g. keep in memory just N last sent messages.

Hope this helps,
Viktor
Reply | Threaded
Open this post in threaded view
|

Re: Memory Being Consumed

michaelbinary
This post has NOT been accepted by the mailing list yet.
Thanks for the advice I will look into it.