Memory Allocation

classic Classic list List threaded Threaded
8 messages Options
Reply | Threaded
Open this post in threaded view
|

Memory Allocation

aupadras
This post has NOT been accepted by the mailing list yet.
Hello,

I am using socket initiator for listening the execution reports from my counterparty.  The application logic is looped several times, after sometime the program is crashing due to in sufficient memory std::bad_allocator. We would like to know how to solve this issue and i.e.., run completely with out any break or initiating the initiator.stop() from morning to evening on a single session.

a) Does freeing the message store on timely fashion will fix the memory and program crashing issue ?

If the application cannot be ran continuously in a single day, how to determine where to initiate the log out or initiator.stop() logic ? As well as how can we re-establish the session connection. What application message should trigger an initiator stop and re-start the socket.
 
Below is the sample message log: we receive all execution reports promptly via FIX, but when the message traffic is low it is sending back and forth heart beat for couple of times and ultimately crashes.

Is this problem completely a memory related issue or mis sync in heart beats between me and counter-party Or combination of both ?

20130812-20:30:54.293 : 8=FIX.4.29=28935=849=beta56=alpha34=6452=20130812-20:30:3537=db751954-d1-1aaw17=16355=/NGV360=20130812-20:10:2875=2013081254=1167=FUT40=244=3.3310200=201309205=261=NEUTRAL11=Order0812220=039=1150=138=5031=3.331032=114=196=3.33100030=XCME151=31207=NYM15=USD29=4432=09/26/201310=182
20130812-20:30:54.353 : 8=FIX.4.29=28935=849=beta56=alpha34=6552=20130812-20:30:3537=db751954-d1-1aaw17=16455=/NGV360=20130812-20:10:2875=2013081254=1167=FUT40=244=3.3310200=201309205=261=NEUTRAL11=Order0812220=039=1150=138=5031=3.331032=114=206=3.33100030=XCME151=30207=NYM15=USD29=4432=09/26/201310=175
20130812-20:30:54.403 : 8=FIX.4.29=28935=849=beta56=alpha34=6652=20130812-20:30:3537=db751954-d1-1aaw17=16555=/NGV360=20130812-20:10:2875=2013081254=1167=FUT40=244=3.3310200=201309205=261=NEUTRAL11=Order0812220=039=1150=138=5031=3.331032=114=216=3.33100030=XCME151=29207=NYM15=USD29=4432=09/26/201310=186
20130812-20:30:54.443 : 8=FIX.4.29=28935=849=beta56=alpha34=6752=20130812-20:30:3537=db751954-d1-1aaw17=16655=/NGV360=20130812-20:10:2875=2013081254=1167=FUT40=244=3.3310200=201309205=261=NEUTRAL11=Order0812220=039=1150=138=5031=3.331032=114=226=3.33100030=XCME151=28207=NYM15=USD29=4432=09/26/201310=188
20130812-20:30:54.523 : 8=FIX.4.29=28935=849=beta56=alpha34=6852=20130812-20:30:3537=db751954-d1-1aaw17=16755=/NGV360=20130812-20:10:2975=2013081254=1167=FUT40=244=3.3310200=201309205=261=NEUTRAL11=Order0812220=039=1150=138=5031=3.331032=114=236=3.33100030=XCME151=27207=NYM15=USD29=4432=09/26/201310=191
20130812-20:30:54.573 : 8=FIX.4.29=28935=849=beta56=alpha34=6952=20130812-20:30:3537=db751954-d1-1aaw17=16855=/NGV360=20130812-20:10:2975=2013081254=1167=FUT40=244=3.3310200=201309205=261=NEUTRAL11=Order0812220=039=1150=138=5031=3.331032=114=246=3.33100030=XCME151=26207=NYM15=USD29=4432=09/26/201310=193
20130812-20:30:54.643 : 8=FIX.4.29=28935=849=beta56=alpha34=7052=20130812-20:30:3537=db751954-d1-1aaw17=16955=/NGV360=20130812-20:10:2975=2013081254=1167=FUT40=244=3.3310200=201309205=261=NEUTRAL11=Order0812220=039=1150=138=5031=3.331032=114=256=3.33100030=XCME151=25207=NYM15=USD29=4432=09/26/201310=186
20130812-20:30:54.693 : 8=FIX.4.29=28935=849=beta56=alpha34=7152=20130812-20:30:3537=db751954-d1-1aaw17=17055=/NGV360=20130812-20:10:2975=2013081254=1167=FUT40=244=3.3310200=201309205=261=NEUTRAL11=Order0812220=039=1150=138=5031=3.331032=114=266=3.33100030=XCME151=24207=NYM15=USD29=4432=09/26/201310=179
20130812-20:30:54.743 : 8=FIX.4.29=28935=849=beta56=alpha34=7252=20130812-20:30:3537=db751954-d1-1aaw17=17155=/NGV360=20130812-20:10:2975=2013081254=1167=FUT40=244=3.3310200=201309205=261=NEUTRAL11=Order0812220=039=1150=138=5031=3.331032=114=276=3.33100030=XCME151=23207=NYM15=USD29=4432=09/26/201310=181
20130812-20:30:54.793 : 8=FIX.4.29=28935=849=beta56=alpha34=7352=20130812-20:30:3537=db751954-d1-1aaw17=17255=/NGV360=20130812-20:10:2975=2013081254=1167=FUT40=244=3.3310200=201309205=261=NEUTRAL11=Order0812220=039=1150=138=5031=3.331032=114=286=3.33100030=XCME151=22207=NYM15=USD29=4432=09/26/201310=183
20130812-20:30:54.843 : 8=FIX.4.29=28935=849=beta56=alpha34=7452=20130812-20:30:3537=db751954-d1-1aaw17=17355=/NGV360=20130812-20:10:2975=2013081254=1167=FUT40=244=3.3310200=201309205=261=NEUTRAL11=Order0812220=039=1150=138=5031=3.331032=114=296=3.33100030=XCME151=21207=NYM15=USD29=4432=09/26/201310=185
20130812-20:30:54.883 : 8=FIX.4.29=28935=849=beta56=alpha34=7552=20130812-20:30:3537=db751954-d1-1aaw17=17455=/NGV360=20130812-20:10:2975=2013081254=1167=FUT40=244=3.3310200=201309205=261=NEUTRAL11=Order0812220=039=1150=138=5031=3.331032=114=306=3.33100030=XCME151=20207=NYM15=USD29=4432=09/26/201310=178
20130812-20:30:54.953 : 8=FIX.4.29=28935=849=beta56=alpha34=7652=20130812-20:30:3537=db751954-d1-1aaw17=17555=/NGV360=20130812-20:10:4775=2013081254=1167=FUT40=244=3.3330200=201309205=261=NEUTRAL11=Order0812220=039=1150=138=5031=3.333032=814=386=3.33142130=XCME151=12207=NYM15=USD29=4432=09/26/201310=207
20130812-20:30:55.003 : 8=FIX.4.29=28835=849=beta56=alpha34=7752=20130812-20:30:3537=db751954-d1-1aaw17=17655=/NGV360=20130812-20:10:4775=2013081254=1167=FUT40=244=3.3330200=201309205=261=NEUTRAL11=Order0812220=039=1150=138=5031=3.333032=814=466=3.33169630=XCME151=4207=NYM15=USD29=4432=09/26/201310=174
20130812-20:30:55.063 : 8=FIX.4.29=28835=849=beta56=alpha34=7852=20130812-20:30:3537=db751954-d1-1aaw17=17755=/NGV360=20130812-20:10:4775=2013081254=1167=FUT40=244=3.3330200=201309205=261=NEUTRAL11=Order0812220=039=2150=238=5031=3.333032=414=506=3.33180030=XCME151=0207=NYM15=USD29=4432=09/26/201310=152
20130812-20:31:04.523 : 8=FIX.4.29=4935=034=249=alpha52=20130812-20:31:04.52356=beta10=120
20130812-20:31:05.153 : 8=FIX.4.29=4635=049=beta56=alpha34=7952=20130812-20:31:0510=236
20130812-20:31:34.523 : 8=FIX.4.29=4935=034=349=alpha52=20130812-20:31:34.52356=beta10=124
20130812-20:31:35.153 : 8=FIX.4.29=4635=049=beta56=alpha34=8052=20130812-20:31:3510=231
20130812-20:32:04.556 : 8=FIX.4.29=4935=034=449=alpha52=20130812-20:32:04.55656=beta10=129
20130812-20:32:05.156 : 8=FIX.4.29=4635=049=beta56=alpha34=8152=20130812-20:32:0510=230
20130812-20:32:34.556 : 8=FIX.4.29=4935=034=549=alpha52=20130812-20:32:34.55656=beta10=133
20130812-20:32:35.156 : 8=FIX.4.29=4635=049=beta56=alpha34=8252=20130812-20:32:3510=234
20130812-20:33:04.718 : 8=FIX.4.29=4935=034=649=alpha52=20130812-20:33:04.71856=beta10=132
20130812-20:33:05.151 : 8=FIX.4.29=4635=049=beta56=alpha34=8352=20130812-20:33:0510=233
20130812-20:33:34.731 : 8=FIX.4.29=4935=034=749=alpha52=20130812-20:33:34.73156=beta10=131
20130812-20:33:35.151 : 8=FIX.4.29=4635=049=beta56=alpha34=8452=20130812-20:33:3510=237
20130812-20:34:04.731 : 8=FIX.4.29=4935=034=849=alpha52=20130812-20:34:04.73156=beta10=130
20130812-20:34:05.151 : 8=FIX.4.29=4635=049=beta56=alpha34=8552=20130812-20:34:0510=236


Please suggest some potential solution for the problem.


Thanks,
aupadras.
Reply | Threaded
Open this post in threaded view
|

Re: Memory Allocation

Viktor Pogrebnyak
Hi, aupadras.

What quickfix revision do you use? Do you use a MemoryStoreFactory or a FileStoreFactory for message persistance?

If you use a MemoryStoreFactory then the answer to your question you might find here: Memory Being Consumed

Regards,
Viktor
Reply | Threaded
Open this post in threaded view
|

Re: Memory Allocation

aupadras
This post has NOT been accepted by the mailing list yet.
Hi Viktor,

I am using "quickfix-1.13.3" version. I am using FileStoreFactory, please see declaration my below

FIX::SessionSettings settings(fileName);
FIX::FileLogFactory logFactory(settings);
FIX::FileStoreFactory storeFactory( settings );
FIX::SocketInitiator initiator(application, storeFactory, settings, logFactory);

Please let me know if you have potential solution or advise to the problem.


Thanks,
aupadras.





Reply | Threaded
Open this post in threaded view
|

Re: Memory Allocation

Viktor Pogrebnyak
Hi, aupadras.

So you are running a clean '1.13.3' release version from here? If so then I suggest you to run a memory leak detector ( IBM Purify, Intel Inspector XE etc ) to see what's leaking memory in your app.

We run the same clean '1.13.3' release ( Windows XP 32bit, VS2005 ) in Production with pretty heavy incoming market data traffic ( 2000+ MarketDataIncrementalRefresh msgs/sec ) and we have no issues with memory leaks.

Regards,
Viktor
Reply | Threaded
Open this post in threaded view
|

Re: Memory Allocation

aupadras
This post has NOT been accepted by the mailing list yet.
Hi Viktor,

Thanks for your reply. I will look into the memory leak application. But before that I have noticed following and let me know your thoughts on it. I think program is crashing or memory mis-allocation (may be)  due to debug vs release version or mixed version of platforms. Please find below the call stack and Out put from VS 2012 (c++) that I ran on Debug mode Win32 and let me know your thoughts on :

Call Stack:

  ntdll.dll!_NtRaiseException@12() Unknown
  ntdll.dll!_KiUserExceptionDispatcher@8() Unknown
  KernelBase.dll!_RaiseException@16() Unknown
> msvcr110d.dll!_CxxThrowException(void * pExceptionObject, const _s__ThrowInfo * pThrowInfo) Line 152 C++
  msvcr110d.dll!operator new(unsigned int size) Line 63 C++
  ExecutionReports.exe!std::_Allocate<std::_Tree_node<std::pair<int const ,std::vector<FIX::FieldMap *,std::allocator<FIX::FieldMap *> > >,void *> >(unsigned int _Count, std::_Tree_node<std::pair<int const ,std::vector<FIX::FieldMap *,std::allocator<FIX::FieldMap *> > >,void *> * __formal) Line 28 C++
  ExecutionReports.exe!std::allocator<std::_Tree_node<std::pair<int const ,std::vector<FIX::FieldMap *,std::allocator<FIX::FieldMap *> > >,void *> >::allocate(unsigned int _Count) Line 591 C++
  ExecutionReports.exe!std::_Wrap_alloc<std::allocator<std::_Tree_node<std::pair<int const ,std::vector<FIX::FieldMap *,std::allocator<FIX::FieldMap *> > >,void *> > >::allocate(unsigned int _Count) Line 877 C++
  ExecutionReports.exe!std::_Tree_alloc<0,std::_Tree_base_types<std::pair<int const ,std::vector<FIX::FieldMap *,std::allocator<FIX::FieldMap *> > >,std::allocator<std::pair<int const ,std::vector<FIX::FieldMap *,std::allocator<FIX::FieldMap *> > > > > >::_Buyheadnode() Line 842 C++
  ExecutionReports.exe!std::_Tree_alloc<0,std::_Tree_base_types<std::pair<int const ,std::vector<FIX::FieldMap *,std::allocator<FIX::FieldMap *> > >,std::allocator<std::pair<int const ,std::vector<FIX::FieldMap *,std::allocator<FIX::FieldMap *> > > > > >::_Tree_alloc<0,std::_Tree_base_types<std::pair<int const ,std::vector<FIX::FieldMap *,std::allocator<FIX::FieldMap *> > >,std::allocator<std::pair<int const ,std::vector<FIX::FieldMap *,std::allocator<FIX::FieldMap *> > > > > >(const std::allocator<std::pair<int const ,std::vector<FIX::FieldMap *,std::allocator<FIX::FieldMap *> > > > & __formal) Line 797 C++
  ExecutionReports.exe!std::_Tree_buy<std::pair<int const ,std::vector<FIX::FieldMap *,std::allocator<FIX::FieldMap *> > >,std::allocator<std::pair<int const ,std::vector<FIX::FieldMap *,std::allocator<FIX::FieldMap *> > > > >::_Tree_buy<std::pair<int const ,std::vector<FIX::FieldMap *,std::allocator<FIX::FieldMap *> > >,std::allocator<std::pair<int const ,std::vector<FIX::FieldMap *,std::allocator<FIX::FieldMap *> > > > >(const std::allocator<std::pair<int const ,std::vector<FIX::FieldMap *,std::allocator<FIX::FieldMap *> > > > & _Al) Line 894 C++
  ExecutionReports.exe!std::_Tree_comp<0,std::_Tmap_traits<int,std::vector<FIX::FieldMap *,std::allocator<FIX::FieldMap *> >,std::less<int>,std::allocator<std::pair<int const ,std::vector<FIX::FieldMap *,std::allocator<FIX::FieldMap *> > > >,0> >::_Tree_comp<0,std::_Tmap_traits<int,std::vector<FIX::FieldMap *,std::allocator<FIX::FieldMap *> >,std::less<int>,std::allocator<std::pair<int const ,std::vector<FIX::FieldMap *,std::allocator<FIX::FieldMap *> > > >,0> >(const std::less<int> & __formal, const std::allocator<std::pair<int const ,std::vector<FIX::FieldMap *,std::allocator<FIX::FieldMap *> > > > & _Al) Line 1008 C++
  ExecutionReports.exe!std::_Tree<std::_Tmap_traits<int,std::vector<FIX::FieldMap *,std::allocator<FIX::FieldMap *> >,std::less<int>,std::allocator<std::pair<int const ,std::vector<FIX::FieldMap *,std::allocator<FIX::FieldMap *> > > >,0> >::_Tree<std::_Tmap_traits<int,std::vector<FIX::FieldMap *,std::allocator<FIX::FieldMap *> >,std::less<int>,std::allocator<std::pair<int const ,std::vector<FIX::FieldMap *,std::allocator<FIX::FieldMap *> > > >,0> >(const std::less<int> & _Parg, const std::allocator<std::pair<int const ,std::vector<FIX::FieldMap *,std::allocator<FIX::FieldMap *> > > > & _Al) Line 1072 C++
  ExecutionReports.exe!std::map<int,std::vector<FIX::FieldMap *,std::allocator<FIX::FieldMap *> >,std::less<int>,std::allocator<std::pair<int const ,std::vector<FIX::FieldMap *,std::allocator<FIX::FieldMap *> > > > >::map<int,std::vector<FIX::FieldMap *,std::allocator<FIX::FieldMap *> >,std::less<int>,std::allocator<std::pair<int const ,std::vector<FIX::FieldMap *,std::allocator<FIX::FieldMap *> > > > >() Line 93 C++
  ExecutionReports.exe!FIX::FieldMap::FieldMap(const FIX::message_order & order) Line 65 C++
  ExecutionReports.exe!FIX::Message::Message(const std::basic_string<char,std::char_traits<char>,std::allocator<char> > & string, const FIX::DataDictionary & dataDictionary, bool validate) Line 57 C++
  ExecutionReports.exe!FIX::Session::next(const std::basic_string<char,std::char_traits<char>,std::allocator<char> > & msg, const FIX::UtcTimeStamp & timeStamp, bool queued) Line 1309 C++
  ExecutionReports.exe!FIX::SocketConnection::readMessages(FIX::SocketMonitor & s) Line 234 C++
  ExecutionReports.exe!FIX::SocketConnection::read(FIX::SocketConnector & s) Line 125 C++
  ExecutionReports.exe!FIX::SocketInitiator::onData(FIX::SocketConnector & connector, int s) Line 202 C++
  ExecutionReports.exe!FIX::ConnectorWrapper::onEvent(FIX::SocketMonitor & __formal, int socket) Line 67 C++
  ExecutionReports.exe!FIX::SocketMonitor::processReadSet(FIX::SocketMonitor::Strategy & strategy, fd_set & readSet) Line 268 C++
  ExecutionReports.exe!FIX::SocketMonitor::block(FIX::SocketMonitor::Strategy & strategy, bool poll, double timeout) Line 245 C++
  ExecutionReports.exe!FIX::SocketConnector::block(FIX::SocketConnector::Strategy & strategy, bool poll, double timeout) Line 147 C++
  ExecutionReports.exe!FIX::SocketInitiator::onStart() Line 96 C++
  ExecutionReports.exe!FIX::Initiator::startThread(void * p) Line 336 C++
  msvcr110d.dll!_callthreadstartex() Line 354 C
  msvcr110d.dll!_threadstartex(void * ptd) Line 337 C
  kernel32.dll!@BaseThreadInitThunk@12() Unknown
  ntdll.dll!___RtlUserThreadStart@8() Unknown
  ntdll.dll!__RtlUserThreadStart@8() Unknown

Out Put:

'ExecutionReports.exe' (Win32): Loaded 'C:\Windows\SysWOW64\ntdll.dll'. Symbols loaded.
'ExecutionReports.exe' (Win32): Loaded 'C:\Windows\SysWOW64\kernel32.dll'. Symbols loaded.
'ExecutionReports.exe' (Win32): Loaded 'C:\Windows\SysWOW64\KernelBase.dll'. Symbols loaded.
'ExecutionReports.exe' (Win32): Loaded 'C:\Windows\SysWOW64\sysfer.dll'. Cannot find or open the PDB file.
'ExecutionReports.exe' (Win32): Loaded 'C:\Windows\SysWOW64\ole32.dll'. Symbols loaded.
'ExecutionReports.exe' (Win32): Loaded 'C:\Windows\SysWOW64\msvcrt.dll'. Symbols loaded.
'ExecutionReports.exe' (Win32): Loaded 'C:\Windows\SysWOW64\gdi32.dll'. Symbols loaded.
'ExecutionReports.exe' (Win32): Loaded 'C:\Windows\SysWOW64\user32.dll'. Symbols loaded.
'ExecutionReports.exe' (Win32): Loaded 'C:\Windows\SysWOW64\advapi32.dll'. Symbols loaded.
'ExecutionReports.exe' (Win32): Loaded 'C:\Windows\SysWOW64\sechost.dll'. Symbols loaded.
'ExecutionReports.exe' (Win32): Loaded 'C:\Windows\SysWOW64\rpcrt4.dll'. Symbols loaded.
'ExecutionReports.exe' (Win32): Loaded 'C:\Windows\SysWOW64\sspicli.dll'. Symbols loaded.
'ExecutionReports.exe' (Win32): Loaded 'C:\Windows\SysWOW64\cryptbase.dll'. Symbols loaded.
'ExecutionReports.exe' (Win32): Loaded 'C:\Windows\SysWOW64\lpk.dll'. Symbols loaded.
'ExecutionReports.exe' (Win32): Loaded 'C:\Windows\SysWOW64\usp10.dll'. Symbols loaded.
'ExecutionReports.exe' (Win32): Loaded 'C:\Windows\SysWOW64\oleaut32.dll'. Symbols loaded.
'ExecutionReports.exe' (Win32): Loaded 'C:\Windows\SysWOW64\odbc32.dll'. Symbols loaded.
'ExecutionReports.exe' (Win32): Loaded 'C:\Windows\SysWOW64\ws2_32.dll'. Symbols loaded.
'ExecutionReports.exe' (Win32): Loaded 'C:\Windows\SysWOW64\nsi.dll'. Symbols loaded.
'ExecutionReports.exe' (Win32): Loaded 'C:\Windows\SysWOW64\msvcp110d.dll'. Symbols loaded.
'ExecutionReports.exe' (Win32): Loaded 'C:\Windows\SysWOW64\msvcr110d.dll'. Symbols loaded.
'ExecutionReports.exe' (Win32): Loaded 'C:\Windows\SysWOW64\imm32.dll'. Symbols loaded.
'ExecutionReports.exe' (Win32): Loaded 'C:\Windows\SysWOW64\msctf.dll'. Symbols loaded.
'ExecutionReports.exe' (Win32): Loaded 'C:\Windows\SysWOW64\odbcint.dll'. Module was built without symbols.
'ExecutionReports.exe' (Win32): Loaded 'C:\Windows\SysWOW64\clbcatq.dll'. Symbols loaded.
'ExecutionReports.exe' (Win32): Loaded 'C:\Windows\SysWOW64\msxml3.dll'. Symbols loaded.
'ExecutionReports.exe' (Win32): Loaded 'C:\Windows\SysWOW64\shlwapi.dll'. Symbols loaded.
'ExecutionReports.exe' (Win32): Unloaded 'C:\Windows\SysWOW64\msxml3.dll'
'ExecutionReports.exe' (Win32): Unloaded 'C:\Windows\SysWOW64\shlwapi.dll'

First-chance exception at 0x7652C41F in ExecutionReports.exe: Microsoft C++ exception: FIX::ConfigError at memory location 0x0028E314.
First-chance exception at 0x7652C41F in ExecutionReports.exe: Microsoft C++ exception: FIX::ConfigError at memory location 0x0028E310.
First-chance exception at 0x7652C41F in ExecutionReports.exe: Microsoft C++ exception: std::bad_alloc at memory location 0x0028E9D8.
First-chance exception at 0x7652C41F in ExecutionReports.exe: Microsoft C++ exception: std::bad_alloc at memory location 0x0379F354.
Unhandled exception at at 0x7652C41F in ExecutionReports.exe: Microsoft C++ exception: std::bad_alloc at memory location 0x0379F354.

Thank you,
aupadras.
Reply | Threaded
Open this post in threaded view
|

Re: Memory Allocation

Viktor Pogrebnyak
Hi, aupadras.

I can see that your application has crashed in Message class constructor. Thing is that std::bad_alloc() exception was thrown before Message::setString() method got invoked which means that message hasn't even started parsing the FIX string.

I can only suggest you to run a memory leak detector to see what the problem is.

Regards,
Viktor
Reply | Threaded
Open this post in threaded view
|

Re: Memory Allocation

aupadras
This post has NOT been accepted by the mailing list yet.
Hi Viktor,

I have found following summary from Intel Inspector and my under standing is memory allocated at one spot and try to delete from another spot. I am not sure why it is behaving this way, it will be appreciated if you can shed some light.


Thanks,
aupadras.Error_Report.txt
Reply | Threaded
Open this post in threaded view
|

Re: Memory Allocation

Viktor Pogrebnyak
Hi, aupadras.

I've checked your output but it contains nothing suspicious.

Are you sure you do not create an excessive amount of SocketInitiators in your main loop? Each SocketInitiator creates a DataDictionary instances which consume a lot of memory so if used without caution they can waste lots of RAM.

Regards,
Viktor