Persisting unprocessed messages

classic Classic list List threaded Threaded
7 messages Options
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Persisting unprocessed messages

Simeon Simeonov
QuickFIX Documentation: http://www.quickfixengine.org/quickfix/doc/html/


Hi, I have a quickfix initiator and I am trying to solve the following problem. If I have a burst of fills, quickfix will receive all the fills and start passing them to the application. The fill messages at this point have been received and are held in memory by quickfix. If the application crashes while processing the messages then what is in memory is lost. When the client restarts it has to re-request the unprocessed messages from the server which might not have them.

My question is this - is there a simple way to make quickfix dump messages to disk so that when the application crashes it can pick from where it left off? I have enabled 'PersistMessages=Y' and I have a disk store but it doesn't seem to be picking up from where it left off upon restart. I am not sure what config I need to get this functionality.

Kind regards,
Simeon

------------------------------------------------------------------------------
Mobile security can be enabling, not merely restricting. Employees who
bring their own devices (BYOD) to work are irked by the imposition of MDM
restrictions. Mobile Device Manager Plus allows you to control only the
apps on BYO-devices by containerizing them, leaving personal data untouched!
https://ad.doubleclick.net/ddm/clk/304595813;131938128;j
_______________________________________________
Quickfix-developers mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/quickfix-developers
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Persisting unprocessed messages

Viktor Pogrebnyak
Hi,

Quickfix extracts/processes messages from network buffer one-by-one and writes them to the disk if configured. There is no way to force behavior you want in the current engine.

Cheers,
Viktor
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Persisting unprocessed messages

Simeon Simeonov
QuickFIX Documentation: http://www.quickfixengine.org/quickfix/doc/html/


Thanks Viktor,

I imagine this is a common problem that one needs to guard against. What is the common solution? One way would be to take messages from quickfix with minimal processing and persist them at application level then process them in a separate thread/process - this offloads persisting responsibilities to the application. Another way would be to persist the SeqNum of the last processed message and then at login request from the acceptor all messages after this. However, I will be relying on the acceptor supporting this functionality (i.e. keeping a history of messages).

Do these methods sound like a reasonable way to deal with this?


On 23 May 2016 at 18:36, Viktor Pogrebnyak <[hidden email]> wrote:
QuickFIX Documentation: http://www.quickfixengine.org/quickfix/doc/html/

Hi,

Quickfix extracts/processes messages from network buffer one-by-one and
writes them to the disk if configured. There is no way to force behavior you
want in the current engine.

Cheers,
Viktor



--
View this message in context: http://quickfix.13857.n7.nabble.com/Persisting-unprocessed-messages-tp6807p6808.html
Sent from the QuickFIX - Dev mailing list archive at Nabble.com.

------------------------------------------------------------------------------
Mobile security can be enabling, not merely restricting. Employees who
bring their own devices (BYOD) to work are irked by the imposition of MDM
restrictions. Mobile Device Manager Plus allows you to control only the
apps on BYO-devices by containerizing them, leaving personal data untouched!
https://ad.doubleclick.net/ddm/clk/304595813;131938128;j
_______________________________________________
Quickfix-developers mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/quickfix-developers


------------------------------------------------------------------------------
Mobile security can be enabling, not merely restricting. Employees who
bring their own devices (BYOD) to work are irked by the imposition of MDM
restrictions. Mobile Device Manager Plus allows you to control only the
apps on BYO-devices by containerizing them, leaving personal data untouched!
https://ad.doubleclick.net/ddm/clk/304595813;131938128;j
_______________________________________________
Quickfix-developers mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/quickfix-developers
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Persisting unprocessed messages

Viktor Pogrebnyak
Hi,

The whole purpose of ResendRequest message is to retrieve messages that weren't processed by the client even when it received them. If service does not keep track of the messages being sent + lacks replay capability then you are in a big trouble, unfortunately. In your case client must not only remember which messages it received but also which were actually processed.

Cheers,
Viktor
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Persisting unprocessed messages

K. Frank
In reply to this post by Simeon Simeonov
QuickFIX Documentation: http://www.quickfixengine.org/quickfix/doc/html/

Hello Simeon!

On Mon, May 23, 2016 at 11:51 AM, Simeon Simeonov
<[hidden email]> wrote:
>
> Hi, I have a quickfix initiator and I am trying to solve the following problem. If I have a burst of fills, quickfix will receive all the fills and start passing them to the application. The fill messages at this point have been received and are held in memory by quickfix. If the application crashes while processing the messages then what is in memory is lost. When the client restarts it has to re-request the unprocessed messages from the server which might not have them.

To emphasize what Viktor said, you have a fundamental problem
here.  No matter how promptly you persist the messages you
receive, there is always the possibility that some messages will
be lost in flight.

As Viktor said, that is what ResendRequest is for, and for guaranteed
reliability, it is necessary for the server to remember (presumably by
persisting) the messages it has sent, and to honor the ResendRequest.

Let's say your client persists received messages "as soon as" it gets
them.  Let's say the server writes the messages it is sending to its
socket and immediately deletes them from memory (because they've
been sent).  Now let's say your server crashes (for example, a power
failure) before the network has actually delivered the messages to
your application.  Under these assumptions, you have an unrecoverable
error because the messages have been lost for good.  No matter how
quickly you persist the messages you receive, there will always be
such an in-flight window where messages can be lost, so you're better
off implementing a robust solution that solves the problem, rather than
"persisting really fast" in order to reduce the problem to a small, but
non-zero level.

The FIX solution is to require both sides of the FIX conversation to
remember the messages they have sent, and correctly honor the
ResendRequest.

(You could imagine a protocol where the client sends back to the
server an "I have received and persisted this message -- you can
delete it on your end now" message.  But you would still have to
have a way -- some ResendRequest, or equivalent -- to recover any
messages that were in flight when you crashed, before you had a
chance to persist them.)

To recap:  You say "When the client restarts it has to re-request the
unprocessed messages from the server which might not have them."
The fact that "the server ... might not have them" means that your
server is fundamentally broken, and there is no way that the FIX
protocol can fix that for you.

> My question is this - is there a simple way to make quickfix dump messages to disk

Yes, QuickFIX will persist messages to disk and/or a database.

> so that when the application crashes it can pick from where it left off?

Sort of.  It depends what you mean by "from where it left off."
It can pick up from where it left off successfully persisting
messages, but it can't necessarily pick up from where the server
left off writing messages to its socket because of the in-flight
issue described above.

> I have enabled 'PersistMessages=Y' and I have a disk store but it doesn't seem to be picking up from where it left off upon restart.

Presumably the server has written more messages than you have
persisted, as described above.  That is, the client's notion of "where
it left off" does not match that of the server.

> I am not sure what config I need to get this functionality.

There is no client-side configuration that will (or could) solve the
in-flight issue.  You need to have your server persist the messages
it sends and properly honor ResendRequest.  That's the FIX way
to implement this kind of fully legitimate requirement of being able to
recover after a crash.

> Kind regards,
> Simeon


Good luck.


K. Frank

------------------------------------------------------------------------------
Mobile security can be enabling, not merely restricting. Employees who
bring their own devices (BYOD) to work are irked by the imposition of MDM
restrictions. Mobile Device Manager Plus allows you to control only the
apps on BYO-devices by containerizing them, leaving personal data untouched!
https://ad.doubleclick.net/ddm/clk/304595813;131938128;j
_______________________________________________
Quickfix-developers mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/quickfix-developers
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Persisting unprocessed messages

Michael C. Starkie
In reply to this post by Viktor Pogrebnyak
QuickFIX Documentation: http://www.quickfixengine.org/quickfix/doc/html/


If the service can not be depended upon to honor a resend request it's not a FIX service and nothing you do on the client side will guarantee against lost data. Unless a TCP socket is closed cleanly you can still lose data. A network card fails for example.

What if you persisted the sequence number of the last message processed at the application layer? Then you know which messages to ignore on a replay. Having the application separately be able to detect a duplicate message would be a double safeguard in case the app crashed after processing but just before persistence.
On Mon, May 23, 2016 at 4:41 PM Viktor Pogrebnyak <[hidden email]> wrote:
QuickFIX Documentation: http://www.quickfixengine.org/quickfix/doc/html/

Hi,

The whole purpose of ResendRequest message is to retrieve messages that
weren't processed by the client even when it received them. If service does
not keep track of the messages being sent + lacks replay capability then you
are in a big trouble, unfortunately. In your case client must not only
remember which messages it received but also which were actually processed.

Cheers,
Viktor



--
View this message in context: http://quickfix.13857.n7.nabble.com/Persisting-unprocessed-messages-tp6807p6810.html
Sent from the QuickFIX - Dev mailing list archive at Nabble.com.

------------------------------------------------------------------------------
Mobile security can be enabling, not merely restricting. Employees who
bring their own devices (BYOD) to work are irked by the imposition of MDM
restrictions. Mobile Device Manager Plus allows you to control only the
apps on BYO-devices by containerizing them, leaving personal data untouched!
https://ad.doubleclick.net/ddm/clk/304595813;131938128;j
_______________________________________________
Quickfix-developers mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/quickfix-developers

------------------------------------------------------------------------------
Mobile security can be enabling, not merely restricting. Employees who
bring their own devices (BYOD) to work are irked by the imposition of MDM
restrictions. Mobile Device Manager Plus allows you to control only the
apps on BYO-devices by containerizing them, leaving personal data untouched!
https://ad.doubleclick.net/ddm/clk/304595813;131938128;j
_______________________________________________
Quickfix-developers mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/quickfix-developers
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Persisting unprocessed messages

Simeon Simeonov
QuickFIX Documentation: http://www.quickfixengine.org/quickfix/doc/html/


Yes, ResendRequest is the way to go forward here and even if quickfix had a mechanism of persisting unprocessed (wrt application layer) messages, it will not be a very useful feature. I'll pick the SeqNum from the last persisted message on disk. I know messages are delivered in-order due to TCP so I know if last message I've processed and stored to disk has SeqNum X than I've seen all messages up to and including X.

On 23 May 2016 at 23:08, Michael C. Starkie <[hidden email]> wrote:
QuickFIX Documentation: http://www.quickfixengine.org/quickfix/doc/html/


If the service can not be depended upon to honor a resend request it's not a FIX service and nothing you do on the client side will guarantee against lost data. Unless a TCP socket is closed cleanly you can still lose data. A network card fails for example.

What if you persisted the sequence number of the last message processed at the application layer? Then you know which messages to ignore on a replay. Having the application separately be able to detect a duplicate message would be a double safeguard in case the app crashed after processing but just before persistence.
On Mon, May 23, 2016 at 4:41 PM Viktor Pogrebnyak <[hidden email]> wrote:
QuickFIX Documentation: http://www.quickfixengine.org/quickfix/doc/html/

Hi,

The whole purpose of ResendRequest message is to retrieve messages that
weren't processed by the client even when it received them. If service does
not keep track of the messages being sent + lacks replay capability then you
are in a big trouble, unfortunately. In your case client must not only
remember which messages it received but also which were actually processed.

Cheers,
Viktor



--
View this message in context: http://quickfix.13857.n7.nabble.com/Persisting-unprocessed-messages-tp6807p6810.html
Sent from the QuickFIX - Dev mailing list archive at Nabble.com.

------------------------------------------------------------------------------
Mobile security can be enabling, not merely restricting. Employees who
bring their own devices (BYOD) to work are irked by the imposition of MDM
restrictions. Mobile Device Manager Plus allows you to control only the
apps on BYO-devices by containerizing them, leaving personal data untouched!
https://ad.doubleclick.net/ddm/clk/304595813;131938128;j
_______________________________________________
Quickfix-developers mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/quickfix-developers

------------------------------------------------------------------------------
Mobile security can be enabling, not merely restricting. Employees who
bring their own devices (BYOD) to work are irked by the imposition of MDM
restrictions. Mobile Device Manager Plus allows you to control only the
apps on BYO-devices by containerizing them, leaving personal data untouched!
https://ad.doubleclick.net/ddm/clk/304595813;131938128;j
_______________________________________________
Quickfix-developers mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/quickfix-developers


------------------------------------------------------------------------------
Mobile security can be enabling, not merely restricting. Employees who
bring their own devices (BYOD) to work are irked by the imposition of MDM
restrictions. Mobile Device Manager Plus allows you to control only the
apps on BYO-devices by containerizing them, leaving personal data untouched!
https://ad.doubleclick.net/ddm/clk/304595813;131938128;j
_______________________________________________
Quickfix-developers mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/quickfix-developers
Loading...