Quantcast

Using .seqnums file to trigger a full resend

classic Classic list List threaded Threaded
24 messages Options
12
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Using .seqnums file to trigger a full resend

Dermot
QuickFIX Documentation: http://www.quickfixengine.org/quickfix/doc/html/


Hi Guys,

I'm trying to trigger a resend of all the executor's messages for the day. I manually set the incoming seq no in the .seqnums file to 0, then login and this triggers a resend request for all messages as soon as my FIX engine receives the logon response message - all good so far. All the messages are resent and received correctly. The problem is when I log out I can see that the .seqnums file does not have the updated incoming seq no (it's just set to 1) so when I log in again it triggers another resend request.

This problem should be very easy to reproduce. The steps are:-
1. Log on
2. Send order
3. Receive execution
4. Log out
5. Manually change the .seqnums file from "0000000004 : 0000000004" to "0000000004 : 0000000000"
6. Log on (resend is triggered)
7. Log out.
8. Check the seqnums file again and you get "0000000007 : 0000000001" when it should be "0000000007 : 0000000006"

Any ideas?

Thanks a lot!
Dermot

------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, SlashDot.org! http://sdm.link/slashdot
_______________________________________________
Quickfix-developers mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/quickfix-developers
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Using .seqnums file to trigger a full resend

Grant Birchmeier
QuickFIX Documentation: http://www.quickfixengine.org/quickfix/doc/html/


I would call this a bad practice.  (You're not the first to invent it.)

You're hacking your app's state to basically lie to your counterparty about missing messages.  If you are asking for a *lot* of history, they'll probably truncate it.

Have you checked your counterparty's docs to see if they provide another way to get the historical data that you need?  Surely you aren't the first person to need historical data from this system.

On Wed, Feb 22, 2017 at 9:29 PM, <[hidden email]> wrote:
QuickFIX Documentation: http://www.quickfixengine.org/quickfix/doc/html/


Hi Guys,

I'm trying to trigger a resend of all the executor's messages for the day. I manually set the incoming seq no in the .seqnums file to 0, then login and this triggers a resend request for all messages as soon as my FIX engine receives the logon response message - all good so far. All the messages are resent and received correctly. The problem is when I log out I can see that the .seqnums file does not have the updated incoming seq no (it's just set to 1) so when I log in again it triggers another resend request.

This problem should be very easy to reproduce. The steps are:-
1. Log on
2. Send order
3. Receive execution
4. Log out
5. Manually change the .seqnums file from "0000000004 : 0000000004" to "0000000004 : 0000000000"
6. Log on (resend is triggered)
7. Log out.
8. Check the seqnums file again and you get "0000000007 : 0000000001" when it should be "0000000007 : 0000000006"

Any ideas?

Thanks a lot!
Dermot

------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, SlashDot.org! http://sdm.link/slashdot
_______________________________________________
Quickfix-developers mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/quickfix-developers



--
Grant Birchmeier
Connamara Systems, LLC
Made-To-Measure Trading Solutions.
Exactly what you need. No more. No less.

------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, SlashDot.org! http://sdm.link/slashdot
_______________________________________________
Quickfix-developers mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/quickfix-developers
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Using .seqnums file to trigger a full resend

Dermot
QuickFIX Documentation: http://www.quickfixengine.org/quickfix/doc/html/


Hi Grant,

Yes it sounded a bit 'hacky' to me too but this was the approach recommended to me by Asher Newcomer on this forum last October. Ok I will speak to the broker and ask what they recommend.

Thanks,
Dermot

----- Original Message -----
From: Grant Birchmeier <[hidden email]>
To: [hidden email]
Cc: "[hidden email]" <[hidden email]>
Date: 2017/2/23, Thu 13:07
Subject: Re: [Quickfix-developers] Using .seqnums file to trigger a full resend

I would call this a bad practice.  (You're not the first to invent it.)

You're hacking your app's state to basically lie to your counterparty about missing messages.  If you are asking for a *lot* of history, they'll probably truncate it.

Have you checked your counterparty's docs to see if they provide another way to get the historical data that you need?  Surely you aren't the first person to need historical data from this system.

On Wed, Feb 22, 2017 at 9:29 PM, <[hidden email]> wrote:
QuickFIX Documentation: http://www.quickfixengine.org/ quickfix/doc/html/


Hi Guys,

I'm trying to trigger a resend of all the executor's messages for the day. I manually set the incoming seq no in the .seqnums file to 0, then login and this triggers a resend request for all messages as soon as my FIX engine receives the logon response message - all good so far. All the messages are resent and received correctly. The problem is when I log out I can see that the .seqnums file does not have the updated incoming seq no (it's just set to 1) so when I log in again it triggers another resend request.

This problem should be very easy to reproduce. The steps are:-
1. Log on
2. Send order
3. Receive execution
4. Log out
5. Manually change the .seqnums file from "0000000004 : 0000000004" to "0000000004 : 0000000000"
6. Log on (resend is triggered)
7. Log out.
8. Check the seqnums file again and you get "0000000007 : 0000000001" when it should be "0000000007 : 0000000006"

Any ideas?

Thanks a lot!
Dermot

------------------------------ ------------------------------ ------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, SlashDot.org! http://sdm.link/slashdot
______________________________ _________________
Quickfix-developers mailing list
[hidden email]
https://lists.sourceforge.net/ lists/listinfo/quickfix- developers



--
Grant Birchmeier
Connamara Systems, LLC
Made-To-Measure Trading Solutions.
Exactly what you need. No more. No less.



------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, SlashDot.org! http://sdm.link/slashdot
_______________________________________________
Quickfix-developers mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/quickfix-developers
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Using .seqnums file to trigger a full resend

Grant Birchmeier
QuickFIX Documentation: http://www.quickfixengine.org/quickfix/doc/html/


Your broker should have some documentation for their FIX interface, which explains what messages and fields are allowed and expected.  If there is a history-retrieval feature, I'm sure it is documented.


On Wed, Feb 22, 2017 at 11:24 PM, <[hidden email]> wrote:
Hi Grant,

Yes it sounded a bit 'hacky' to me too but this was the approach recommended to me by Asher Newcomer on this forum last October. Ok I will speak to the broker and ask what they recommend.

Thanks,
Dermot

----- Original Message -----
From: Grant Birchmeier <[hidden email]>
To: [hidden email]
Cc: "[hidden email]" <[hidden email]>
Date: 2017/2/23, Thu 13:07
Subject: Re: [Quickfix-developers] Using .seqnums file to trigger a full resend

I would call this a bad practice.  (You're not the first to invent it.)

You're hacking your app's state to basically lie to your counterparty about missing messages.  If you are asking for a *lot* of history, they'll probably truncate it.

Have you checked your counterparty's docs to see if they provide another way to get the historical data that you need?  Surely you aren't the first person to need historical data from this system.

On Wed, Feb 22, 2017 at 9:29 PM, <[hidden email]> wrote:
QuickFIX Documentation: http://www.quickfixengine.org/ quickfix/doc/html/


Hi Guys,

I'm trying to trigger a resend of all the executor's messages for the day. I manually set the incoming seq no in the .seqnums file to 0, then login and this triggers a resend request for all messages as soon as my FIX engine receives the logon response message - all good so far. All the messages are resent and received correctly. The problem is when I log out I can see that the .seqnums file does not have the updated incoming seq no (it's just set to 1) so when I log in again it triggers another resend request.

This problem should be very easy to reproduce. The steps are:-
1. Log on
2. Send order
3. Receive execution
4. Log out
5. Manually change the .seqnums file from "0000000004 : 0000000004" to "0000000004 : 0000000000"
6. Log on (resend is triggered)
7. Log out.
8. Check the seqnums file again and you get "0000000007 : 0000000001" when it should be "0000000007 : 0000000006"

Any ideas?

Thanks a lot!
Dermot

------------------------------ ------------------------------ ------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, SlashDot.org! http://sdm.link/slashdot
______________________________ _________________
Quickfix-developers mailing list
[hidden email]
https://lists.sourceforge.net/ lists/listinfo/quickfix- developers



--
Grant Birchmeier
Connamara Systems, LLC
Made-To-Measure Trading Solutions.
Exactly what you need. No more. No less.





--
Grant Birchmeier
Connamara Systems, LLC
Made-To-Measure Trading Solutions.
Exactly what you need. No more. No less.

------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, SlashDot.org! http://sdm.link/slashdot
_______________________________________________
Quickfix-developers mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/quickfix-developers
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Using .seqnums file to trigger a full resend

Mike Gatny
In reply to this post by Dermot
QuickFIX Documentation: http://www.quickfixengine.org/quickfix/doc/html/


Fiddling with the seqnum file should definitely work if you completely shut down your app, edit the file, then restart your app.  
To do it with just a logout/edit/logon, try RefreshOnLogon=Y.

--
Mike Gatny
Connamara Systems, LLC

On Thu, Feb 23, 2017 at 12:24 AM, <[hidden email]> wrote:
QuickFIX Documentation: http://www.quickfixengine.org/quickfix/doc/html/


Hi Grant,

Yes it sounded a bit 'hacky' to me too but this was the approach recommended to me by Asher Newcomer on this forum last October. Ok I will speak to the broker and ask what they recommend.

Thanks,
Dermot

----- Original Message -----
From: Grant Birchmeier <[hidden email]>
To: [hidden email]
Cc: "[hidden email]" <[hidden email]>
Date: 2017/2/23, Thu 13:07
Subject: Re: [Quickfix-developers] Using .seqnums file to trigger a full resend

I would call this a bad practice.  (You're not the first to invent it.)

You're hacking your app's state to basically lie to your counterparty about missing messages.  If you are asking for a *lot* of history, they'll probably truncate it.

Have you checked your counterparty's docs to see if they provide another way to get the historical data that you need?  Surely you aren't the first person to need historical data from this system.

On Wed, Feb 22, 2017 at 9:29 PM, <[hidden email]> wrote:
QuickFIX Documentation: http://www.quickfixengine.org/ quickfix/doc/html/


Hi Guys,

I'm trying to trigger a resend of all the executor's messages for the day. I manually set the incoming seq no in the .seqnums file to 0, then login and this triggers a resend request for all messages as soon as my FIX engine receives the logon response message - all good so far. All the messages are resent and received correctly. The problem is when I log out I can see that the .seqnums file does not have the updated incoming seq no (it's just set to 1) so when I log in again it triggers another resend request.

This problem should be very easy to reproduce. The steps are:-
1. Log on
2. Send order
3. Receive execution
4. Log out
5. Manually change the .seqnums file from "0000000004 : 0000000004" to "0000000004 : 0000000000"
6. Log on (resend is triggered)
7. Log out.
8. Check the seqnums file again and you get "0000000007 : 0000000001" when it should be "0000000007 : 0000000006"

Any ideas?

Thanks a lot!
Dermot

------------------------------ ------------------------------ ------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, SlashDot.org! http://sdm.link/slashdot
______________________________ _________________
Quickfix-developers mailing list
[hidden email]
https://lists.sourceforge.net/ lists/listinfo/quickfix- developers



--
Grant Birchmeier
Connamara Systems, LLC
Made-To-Measure Trading Solutions.
Exactly what you need. No more. No less.



------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, SlashDot.org! http://sdm.link/slashdot
_______________________________________________
Quickfix-developers mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/quickfix-developers


------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, SlashDot.org! http://sdm.link/slashdot
_______________________________________________
Quickfix-developers mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/quickfix-developers
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Using .seqnums file to trigger a full resend

Dermot
QuickFIX Documentation: http://www.quickfixengine.org/quickfix/doc/html/


Hi Mike, Grant,

Ok thanks for your feedback. Let me see what the broker comes back with first. There's nothing specific in their ROE regarding history-retrieval.

As a test I did try sending a resend request message [BeginSeqNo=0,EndSeqNo=0] but my side then immediately logged out for some reason.

Dermot

----- Original Message -----
From: Mike Gatny <[hidden email]>
To: [hidden email]
Cc: Grant Birchmeier <[hidden email]>; "[hidden email]" <[hidden email]>
Date: 2017/2/24, Fri 00:50
Subject: Re: [Quickfix-developers] Using .seqnums file to trigger a full resend

Fiddling with the seqnum file should definitely work if you completely shut down your app, edit the file, then restart your app.  
To do it with just a logout/edit/logon, try RefreshOnLogon=Y.

--
Mike Gatny
Connamara Systems, LLC

On Thu, Feb 23, 2017 at 12:24 AM, <[hidden email]> wrote:
QuickFIX Documentation: http://www.quickfixengine.org/ quickfix/doc/html/


Hi Grant,

Yes it sounded a bit 'hacky' to me too but this was the approach recommended to me by Asher Newcomer on this forum last October. Ok I will speak to the broker and ask what they recommend.

Thanks,
Dermot

----- Original Message -----
From: Grant Birchmeier <[hidden email]>
To: [hidden email]
Cc: "[hidden email]" <[hidden email]>
Date: 2017/2/23, Thu 13:07
Subject: Re: [Quickfix-developers] Using .seqnums file to trigger a full resend

I would call this a bad practice.  (You're not the first to invent it.)

You're hacking your app's state to basically lie to your counterparty about missing messages.  If you are asking for a *lot* of history, they'll probably truncate it.

Have you checked your counterparty's docs to see if they provide another way to get the historical data that you need?  Surely you aren't the first person to need historical data from this system.

On Wed, Feb 22, 2017 at 9:29 PM, <[hidden email]> wrote:
QuickFIX Documentation: http://www.quickfixengine.org/ quickfix/doc/html/


Hi Guys,

I'm trying to trigger a resend of all the executor's messages for the day. I manually set the incoming seq no in the .seqnums file to 0, then login and this triggers a resend request for all messages as soon as my FIX engine receives the logon response message - all good so far. All the messages are resent and received correctly. The problem is when I log out I can see that the .seqnums file does not have the updated incoming seq no (it's just set to 1) so when I log in again it triggers another resend request.

This problem should be very easy to reproduce. The steps are:-
1. Log on
2. Send order
3. Receive execution
4. Log out
5. Manually change the .seqnums file from "0000000004 : 0000000004" to "0000000004 : 0000000000"
6. Log on (resend is triggered)
7. Log out.
8. Check the seqnums file again and you get "0000000007 : 0000000001" when it should be "0000000007 : 0000000006"

Any ideas?

Thanks a lot!
Dermot

------------------------------ ------------------------------ ------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, SlashDot.org! http://sdm.link/slashdot
______________________________ _________________
Quickfix-developers mailing list
[hidden email]
https://lists.sourceforge.net/ lists/listinfo/quickfix- developers



--
Grant Birchmeier
Connamara Systems, LLC
Made-To-Measure Trading Solutions.
Exactly what you need. No more. No less.



------------------------------ ------------------------------ ------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, SlashDot.org! http://sdm.link/slashdot
______________________________ _________________
Quickfix-developers mailing list
[hidden email]
https://lists.sourceforge.net/ lists/listinfo/quickfix- developers




------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, SlashDot.org! http://sdm.link/slashdot
_______________________________________________
Quickfix-developers mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/quickfix-developers
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Using .seqnums file to trigger a full resend

Mike Gatny
QuickFIX Documentation: http://www.quickfixengine.org/quickfix/doc/html/


Definitely don't send your own resend requests (or any other admin msg) -- that will mess with the QF state in unsafe ways.
Editing the seqnums file is definitely safe if you do it while your app is not running.  And it should be safe if you are running but logged out, and RefreshOnLogon=Y.
Also note that you can use QF's http interface to reset seqnums.

--
Mike Gatny
Connamara Systems, LLC

On Thu, Feb 23, 2017 at 11:29 AM, <[hidden email]> wrote:
Hi Mike, Grant,

Ok thanks for your feedback. Let me see what the broker comes back with first. There's nothing specific in their ROE regarding history-retrieval.

As a test I did try sending a resend request message [BeginSeqNo=0,EndSeqNo=0] but my side then immediately logged out for some reason.

Dermot

----- Original Message -----
From: Mike Gatny <[hidden email]>
To: [hidden email]
Cc: Grant Birchmeier <[hidden email]>; "[hidden email]" <[hidden email]>
Date: 2017/2/24, Fri 00:50
Subject: Re: [Quickfix-developers] Using .seqnums file to trigger a full resend

Fiddling with the seqnum file should definitely work if you completely shut down your app, edit the file, then restart your app.  
To do it with just a logout/edit/logon, try RefreshOnLogon=Y.

--
Mike Gatny
Connamara Systems, LLC

On Thu, Feb 23, 2017 at 12:24 AM, <[hidden email]> wrote:
QuickFIX Documentation: http://www.quickfixengine.org/ quickfix/doc/html/


Hi Grant,

Yes it sounded a bit 'hacky' to me too but this was the approach recommended to me by Asher Newcomer on this forum last October. Ok I will speak to the broker and ask what they recommend.

Thanks,
Dermot

----- Original Message -----
From: Grant Birchmeier <[hidden email]>
To: [hidden email]
Cc: "[hidden email]" <[hidden email]>
Date: 2017/2/23, Thu 13:07
Subject: Re: [Quickfix-developers] Using .seqnums file to trigger a full resend

I would call this a bad practice.  (You're not the first to invent it.)

You're hacking your app's state to basically lie to your counterparty about missing messages.  If you are asking for a *lot* of history, they'll probably truncate it.

Have you checked your counterparty's docs to see if they provide another way to get the historical data that you need?  Surely you aren't the first person to need historical data from this system.

On Wed, Feb 22, 2017 at 9:29 PM, <[hidden email]> wrote:
QuickFIX Documentation: http://www.quickfixengine.org/ quickfix/doc/html/


Hi Guys,

I'm trying to trigger a resend of all the executor's messages for the day. I manually set the incoming seq no in the .seqnums file to 0, then login and this triggers a resend request for all messages as soon as my FIX engine receives the logon response message - all good so far. All the messages are resent and received correctly. The problem is when I log out I can see that the .seqnums file does not have the updated incoming seq no (it's just set to 1) so when I log in again it triggers another resend request.

This problem should be very easy to reproduce. The steps are:-
1. Log on
2. Send order
3. Receive execution
4. Log out
5. Manually change the .seqnums file from "0000000004 : 0000000004" to "0000000004 : 0000000000"
6. Log on (resend is triggered)
7. Log out.
8. Check the seqnums file again and you get "0000000007 : 0000000001" when it should be "0000000007 : 0000000006"

Any ideas?

Thanks a lot!
Dermot

------------------------------ ------------------------------ ------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, SlashDot.org! http://sdm.link/slashdot
______________________________ _________________
Quickfix-developers mailing list
[hidden email]
https://lists.sourceforge.net/ lists/listinfo/quickfix- developers



--
Grant Birchmeier
Connamara Systems, LLC
Made-To-Measure Trading Solutions.
Exactly what you need. No more. No less.



------------------------------ ------------------------------ ------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, SlashDot.org! http://sdm.link/slashdot
______________________________ _________________
Quickfix-developers mailing list
[hidden email]
https://lists.sourceforge.net/ lists/listinfo/quickfix- developers





------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, SlashDot.org! http://sdm.link/slashdot
_______________________________________________
Quickfix-developers mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/quickfix-developers
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Using .seqnums file to trigger a full resend

Dermot
QuickFIX Documentation: http://www.quickfixengine.org/quickfix/doc/html/


Ok noted, thanks - don't send my own resend requests. I'm still waiting to hear back from the broker re: recommendations for getting a full day's messages resent. Their support is not great to be honest.

In the meantime I tried editing the seqnums file and that worked but with some problems:-

1. The resend included all heartbeats as well. Is there a way to specify not to include heartbeats/test requests?
2. The resend included the broker's response to the original logout message. This in turn triggers a new logout on my side. Is this normal behavior? Is there a way to exclude this message also? Basically I'm only interested in messages relating to orders/executions.

Thanks,
Dermot

----- Original Message -----
From: Mike Gatny <[hidden email]>
To: [hidden email]
Cc: Grant Birchmeier <[hidden email]>; "[hidden email]" <[hidden email]>
Date: 2017/2/24, Fri 01:41
Subject: Re: [Quickfix-developers] Using .seqnums file to trigger a full resend

Definitely don't send your own resend requests (or any other admin msg) -- that will mess with the QF state in unsafe ways.
Editing the seqnums file is definitely safe if you do it while your app is not running.  And it should be safe if you are running but logged out, and RefreshOnLogon=Y.
Also note that you can use QF's http interface to reset seqnums.

--
Mike Gatny
Connamara Systems, LLC

On Thu, Feb 23, 2017 at 11:29 AM, <[hidden email]> wrote:
Hi Mike, Grant,

Ok thanks for your feedback. Let me see what the broker comes back with first. There's nothing specific in their ROE regarding history-retrieval.

As a test I did try sending a resend request message [BeginSeqNo=0,EndSeqNo=0] but my side then immediately logged out for some reason.

Dermot

----- Original Message -----
From: Mike Gatny <[hidden email]>
To: [hidden email]
Cc: Grant Birchmeier <[hidden email]>; "[hidden email]" <[hidden email]>
Date: 2017/2/24, Fri 00:50
Subject: Re: [Quickfix-developers] Using .seqnums file to trigger a full resend

Fiddling with the seqnum file should definitely work if you completely shut down your app, edit the file, then restart your app.  
To do it with just a logout/edit/logon, try RefreshOnLogon=Y.

--
Mike Gatny
Connamara Systems, LLC

On Thu, Feb 23, 2017 at 12:24 AM, <[hidden email]> wrote:
QuickFIX Documentation: http://www.quickfixengine.org/ quickfix/doc/html/


Hi Grant,

Yes it sounded a bit 'hacky' to me too but this was the approach recommended to me by Asher Newcomer on this forum last October. Ok I will speak to the broker and ask what they recommend.

Thanks,
Dermot

----- Original Message -----
From: Grant Birchmeier <[hidden email]>
To: [hidden email]
Cc: "[hidden email]" <[hidden email]>
Date: 2017/2/23, Thu 13:07
Subject: Re: [Quickfix-developers] Using .seqnums file to trigger a full resend

I would call this a bad practice.  (You're not the first to invent it.)

You're hacking your app's state to basically lie to your counterparty about missing messages.  If you are asking for a *lot* of history, they'll probably truncate it.

Have you checked your counterparty's docs to see if they provide another way to get the historical data that you need?  Surely you aren't the first person to need historical data from this system.

On Wed, Feb 22, 2017 at 9:29 PM, <[hidden email]> wrote:
QuickFIX Documentation: http://www.quickfixengine.org/ quickfix/doc/html/


Hi Guys,

I'm trying to trigger a resend of all the executor's messages for the day. I manually set the incoming seq no in the .seqnums file to 0, then login and this triggers a resend request for all messages as soon as my FIX engine receives the logon response message - all good so far. All the messages are resent and received correctly. The problem is when I log out I can see that the .seqnums file does not have the updated incoming seq no (it's just set to 1) so when I log in again it triggers another resend request.

This problem should be very easy to reproduce. The steps are:-
1. Log on
2. Send order
3. Receive execution
4. Log out
5. Manually change the .seqnums file from "0000000004 : 0000000004" to "0000000004 : 0000000000"
6. Log on (resend is triggered)
7. Log out.
8. Check the seqnums file again and you get "0000000007 : 0000000001" when it should be "0000000007 : 0000000006"

Any ideas?

Thanks a lot!
Dermot

------------------------------ ------------------------------ ------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, SlashDot.org! http://sdm.link/slashdot
______________________________ _________________
Quickfix-developers mailing list
[hidden email]
https://lists.sourceforge.net/ lists/listinfo/quickfix- developers



--
Grant Birchmeier
Connamara Systems, LLC
Made-To-Measure Trading Solutions.
Exactly what you need. No more. No less.



------------------------------ ------------------------------ ------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, SlashDot.org! http://sdm.link/slashdot
______________________________ _________________
Quickfix-developers mailing list
[hidden email]
https://lists.sourceforge.net/ lists/listinfo/quickfix- developers







------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, SlashDot.org! http://sdm.link/slashdot
_______________________________________________
Quickfix-developers mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/quickfix-developers
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Using .seqnums file to trigger a full resend

Mike Gatny
QuickFIX Documentation: http://www.quickfixengine.org/quickfix/doc/html/


On Fri, Feb 24, 2017 at 12:21 PM, <[hidden email]> wrote:
Ok noted, thanks - don't send my own resend requests. I'm still waiting to hear back from the broker re: recommendations for getting a full day's messages resent. Their support is not great to be honest.

In the meantime I tried editing the seqnums file and that worked but with some problems:-

1. The resend included all heartbeats as well. Is there a way to specify not to include heartbeats/test requests?
2. The resend included the broker's response to the original logout message. This in turn triggers a new logout on my side. Is this normal behavior? Is there a way to exclude this message also? Basically I'm only interested in messages relating to orders/executions.

They should not be sending you any admin-type messages (htbeat/testrequest/logon/logout) at all during a resend.  Instead they should be replacing those msgs with sequence reset msgs with GapFillFlag field set to "Y".   That's just totally broken resend behavior I'm afraid -- broken enough that I don't see what we can do on your side to mitigate it.  And it could trip you up even if you aren't doing a contrived resend via the seqnums file.

--
Mike Gatny
Connamara Systems, LLC

 

Thanks,
Dermot

----- Original Message -----
From: Mike Gatny <[hidden email]>
To: [hidden email]
Cc: Grant Birchmeier <[hidden email]>; "[hidden email]" <[hidden email]>
Date: 2017/2/24, Fri 01:41
Subject: Re: [Quickfix-developers] Using .seqnums file to trigger a full resend

Definitely don't send your own resend requests (or any other admin msg) -- that will mess with the QF state in unsafe ways.
Editing the seqnums file is definitely safe if you do it while your app is not running.  And it should be safe if you are running but logged out, and RefreshOnLogon=Y.
Also note that you can use QF's http interface to reset seqnums.

--
Mike Gatny
Connamara Systems, LLC

On Thu, Feb 23, 2017 at 11:29 AM, <[hidden email]> wrote:
Hi Mike, Grant,

Ok thanks for your feedback. Let me see what the broker comes back with first. There's nothing specific in their ROE regarding history-retrieval.

As a test I did try sending a resend request message [BeginSeqNo=0,EndSeqNo=0] but my side then immediately logged out for some reason.

Dermot

----- Original Message -----
From: Mike Gatny <[hidden email]>
To: [hidden email]
Cc: Grant Birchmeier <[hidden email]>; "[hidden email]" <[hidden email]>
Date: 2017/2/24, Fri 00:50
Subject: Re: [Quickfix-developers] Using .seqnums file to trigger a full resend

Fiddling with the seqnum file should definitely work if you completely shut down your app, edit the file, then restart your app.  
To do it with just a logout/edit/logon, try RefreshOnLogon=Y.

--
Mike Gatny
Connamara Systems, LLC

On Thu, Feb 23, 2017 at 12:24 AM, <[hidden email]> wrote:
QuickFIX Documentation: http://www.quickfixengine.org/ quickfix/doc/html/


Hi Grant,

Yes it sounded a bit 'hacky' to me too but this was the approach recommended to me by Asher Newcomer on this forum last October. Ok I will speak to the broker and ask what they recommend.

Thanks,
Dermot

----- Original Message -----
From: Grant Birchmeier <[hidden email]>
To: [hidden email]
Cc: "[hidden email]" <[hidden email]>
Date: 2017/2/23, Thu 13:07
Subject: Re: [Quickfix-developers] Using .seqnums file to trigger a full resend

I would call this a bad practice.  (You're not the first to invent it.)

You're hacking your app's state to basically lie to your counterparty about missing messages.  If you are asking for a *lot* of history, they'll probably truncate it.

Have you checked your counterparty's docs to see if they provide another way to get the historical data that you need?  Surely you aren't the first person to need historical data from this system.

On Wed, Feb 22, 2017 at 9:29 PM, <[hidden email]> wrote:
QuickFIX Documentation: http://www.quickfixengine.org/ quickfix/doc/html/


Hi Guys,

I'm trying to trigger a resend of all the executor's messages for the day. I manually set the incoming seq no in the .seqnums file to 0, then login and this triggers a resend request for all messages as soon as my FIX engine receives the logon response message - all good so far. All the messages are resent and received correctly. The problem is when I log out I can see that the .seqnums file does not have the updated incoming seq no (it's just set to 1) so when I log in again it triggers another resend request.

This problem should be very easy to reproduce. The steps are:-
1. Log on
2. Send order
3. Receive execution
4. Log out
5. Manually change the .seqnums file from "0000000004 : 0000000004" to "0000000004 : 0000000000"
6. Log on (resend is triggered)
7. Log out.
8. Check the seqnums file again and you get "0000000007 : 0000000001" when it should be "0000000007 : 0000000006"

Any ideas?

Thanks a lot!
Dermot

------------------------------ ------------------------------ ------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, SlashDot.org! http://sdm.link/slashdot
______________________________ _________________
Quickfix-developers mailing list
[hidden email]
https://lists.sourceforge.net/ lists/listinfo/quickfix- developers



--
Grant Birchmeier
Connamara Systems, LLC
Made-To-Measure Trading Solutions.
Exactly what you need. No more. No less.



------------------------------ ------------------------------ ------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, SlashDot.org! http://sdm.link/slashdot
______________________________ _________________
Quickfix-developers mailing list
[hidden email]
https://lists.sourceforge.net/ lists/listinfo/quickfix- developers








------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, SlashDot.org! http://sdm.link/slashdot
_______________________________________________
Quickfix-developers mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/quickfix-developers
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Using .seqnums file to trigger a full resend

Dermot
QuickFIX Documentation: http://www.quickfixengine.org/quickfix/doc/html/


Ok I hear ya Mike. You're right - there's not much I can do with this as is. I'll just have to push the broker for a proper solution.
Thanks!
Dermot

----- Original Message -----
From: Mike Gatny <[hidden email]>
To: [hidden email]
Cc: Grant Birchmeier <[hidden email]>; "[hidden email]" <[hidden email]>
Date: 2017/2/25, Sat 03:05
Subject: Re: [Quickfix-developers] Using .seqnums file to trigger a full resend

On Fri, Feb 24, 2017 at 12:21 PM, <[hidden email]> wrote:
Ok noted, thanks - don't send my own resend requests. I'm still waiting to hear back from the broker re: recommendations for getting a full day's messages resent. Their support is not great to be honest.

In the meantime I tried editing the seqnums file and that worked but with some problems:-

1. The resend included all heartbeats as well. Is there a way to specify not to include heartbeats/test requests?
2. The resend included the broker's response to the original logout message. This in turn triggers a new logout on my side. Is this normal behavior? Is there a way to exclude this message also? Basically I'm only interested in messages relating to orders/executions.

They should not be sending you any admin-type messages (htbeat/testrequest/logon/logout) at all during a resend.  Instead they should be replacing those msgs with sequence reset msgs with GapFillFlag field set to "Y".   That's just totally broken resend behavior I'm afraid -- broken enough that I don't see what we can do on your side to mitigate it.  And it could trip you up even if you aren't doing a contrived resend via the seqnums file.

--
Mike Gatny
Connamara Systems, LLC

 

Thanks,
Dermot

----- Original Message -----
From: Mike Gatny <[hidden email]>
To: [hidden email]
Cc: Grant Birchmeier <[hidden email]>; "[hidden email]" <[hidden email]>
Date: 2017/2/24, Fri 01:41
Subject: Re: [Quickfix-developers] Using .seqnums file to trigger a full resend

Definitely don't send your own resend requests (or any other admin msg) -- that will mess with the QF state in unsafe ways.
Editing the seqnums file is definitely safe if you do it while your app is not running.  And it should be safe if you are running but logged out, and RefreshOnLogon=Y.
Also note that you can use QF's http interface to reset seqnums.

--
Mike Gatny
Connamara Systems, LLC

On Thu, Feb 23, 2017 at 11:29 AM, <[hidden email]> wrote:
Hi Mike, Grant,

Ok thanks for your feedback. Let me see what the broker comes back with first. There's nothing specific in their ROE regarding history-retrieval.

As a test I did try sending a resend request message [BeginSeqNo=0,EndSeqNo=0] but my side then immediately logged out for some reason.

Dermot

----- Original Message -----
From: Mike Gatny <[hidden email]>
To: [hidden email]
Cc: Grant Birchmeier <[hidden email]>; "[hidden email]" <[hidden email]>
Date: 2017/2/24, Fri 00:50
Subject: Re: [Quickfix-developers] Using .seqnums file to trigger a full resend

Fiddling with the seqnum file should definitely work if you completely shut down your app, edit the file, then restart your app.  
To do it with just a logout/edit/logon, try RefreshOnLogon=Y.

--
Mike Gatny
Connamara Systems, LLC

On Thu, Feb 23, 2017 at 12:24 AM, <[hidden email]> wrote:
QuickFIX Documentation: http://www.quickfixengine.org/ quickfix/doc/html/


Hi Grant,

Yes it sounded a bit 'hacky' to me too but this was the approach recommended to me by Asher Newcomer on this forum last October. Ok I will speak to the broker and ask what they recommend.

Thanks,
Dermot

----- Original Message -----
From: Grant Birchmeier <[hidden email]>
To: [hidden email]
Cc: "[hidden email]" <[hidden email]>
Date: 2017/2/23, Thu 13:07
Subject: Re: [Quickfix-developers] Using .seqnums file to trigger a full resend

I would call this a bad practice.  (You're not the first to invent it.)

You're hacking your app's state to basically lie to your counterparty about missing messages.  If you are asking for a *lot* of history, they'll probably truncate it.

Have you checked your counterparty's docs to see if they provide another way to get the historical data that you need?  Surely you aren't the first person to need historical data from this system.

On Wed, Feb 22, 2017 at 9:29 PM, <[hidden email]> wrote:
QuickFIX Documentation: http://www.quickfixengine.org/ quickfix/doc/html/


Hi Guys,

I'm trying to trigger a resend of all the executor's messages for the day. I manually set the incoming seq no in the .seqnums file to 0, then login and this triggers a resend request for all messages as soon as my FIX engine receives the logon response message - all good so far. All the messages are resent and received correctly. The problem is when I log out I can see that the .seqnums file does not have the updated incoming seq no (it's just set to 1) so when I log in again it triggers another resend request.

This problem should be very easy to reproduce. The steps are:-
1. Log on
2. Send order
3. Receive execution
4. Log out
5. Manually change the .seqnums file from "0000000004 : 0000000004" to "0000000004 : 0000000000"
6. Log on (resend is triggered)
7. Log out.
8. Check the seqnums file again and you get "0000000007 : 0000000001" when it should be "0000000007 : 0000000006"

Any ideas?

Thanks a lot!
Dermot

------------------------------ ------------------------------ ------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, SlashDot.org! http://sdm.link/slashdot
______________________________ _________________
Quickfix-developers mailing list
[hidden email]
https://lists.sourceforge.net/ lists/listinfo/quickfix- developers



--
Grant Birchmeier
Connamara Systems, LLC
Made-To-Measure Trading Solutions.
Exactly what you need. No more. No less.



------------------------------ ------------------------------ ------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, SlashDot.org! http://sdm.link/slashdot
______________________________ _________________
Quickfix-developers mailing list
[hidden email]
https://lists.sourceforge.net/ lists/listinfo/quickfix- developers










------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, SlashDot.org! http://sdm.link/slashdot
_______________________________________________
Quickfix-developers mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/quickfix-developers
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Using .seqnums file to trigger a full resend

Dermot
In reply to this post by Mike Gatny
QuickFIX Documentation: http://www.quickfixengine.org/quickfix/doc/html/


Hi again Mike, I spoke to the broker who gave me the following:-

1. They don't have any special procedure for sending historical messages and recommend using the regular resend (i.e. they don't truncate messages).
2. They agree there should not be any admin messages included - this is a bug and they are looking into it.

So this means I'll have to go with the contrived resend via the seqnums file. And this brings me all the way back to my original post regarding the seqnums file not containing the correct incoming seq no after resend. Do you have any inclination to take a look at this and perhaps "fix" it? I'm guessing that's a No given you don't really recommend this approach anyway so in that case what are my options? I guess I'll just have to manually update the seqnums file again with the correct number after resend. Thoughts?

Thanks,
Dermot



----- Original Message -----
From: Mike Gatny <[hidden email]>
To: [hidden email]
Cc: Grant Birchmeier <[hidden email]>; "[hidden email]" <[hidden email]>
Date: 2017/2/25, Sat 03:05
Subject: Re: [Quickfix-developers] Using .seqnums file to trigger a full resend

On Fri, Feb 24, 2017 at 12:21 PM, <[hidden email]> wrote:
Ok noted, thanks - don't send my own resend requests. I'm still waiting to hear back from the broker re: recommendations for getting a full day's messages resent. Their support is not great to be honest.

In the meantime I tried editing the seqnums file and that worked but with some problems:-

1. The resend included all heartbeats as well. Is there a way to specify not to include heartbeats/test requests?
2. The resend included the broker's response to the original logout message. This in turn triggers a new logout on my side. Is this normal behavior? Is there a way to exclude this message also? Basically I'm only interested in messages relating to orders/executions.

They should not be sending you any admin-type messages (htbeat/testrequest/logon/logout) at all during a resend.  Instead they should be replacing those msgs with sequence reset msgs with GapFillFlag field set to "Y".   That's just totally broken resend behavior I'm afraid -- broken enough that I don't see what we can do on your side to mitigate it.  And it could trip you up even if you aren't doing a contrived resend via the seqnums file.

--
Mike Gatny
Connamara Systems, LLC

 

Thanks,
Dermot

----- Original Message -----
From: Mike Gatny <[hidden email]>
To: [hidden email]
Cc: Grant Birchmeier <[hidden email]>; "[hidden email]" <[hidden email]>
Date: 2017/2/24, Fri 01:41
Subject: Re: [Quickfix-developers] Using .seqnums file to trigger a full resend

Definitely don't send your own resend requests (or any other admin msg) -- that will mess with the QF state in unsafe ways.
Editing the seqnums file is definitely safe if you do it while your app is not running.  And it should be safe if you are running but logged out, and RefreshOnLogon=Y.
Also note that you can use QF's http interface to reset seqnums.

--
Mike Gatny
Connamara Systems, LLC

On Thu, Feb 23, 2017 at 11:29 AM, <[hidden email]> wrote:
Hi Mike, Grant,

Ok thanks for your feedback. Let me see what the broker comes back with first. There's nothing specific in their ROE regarding history-retrieval.

As a test I did try sending a resend request message [BeginSeqNo=0,EndSeqNo=0] but my side then immediately logged out for some reason.

Dermot

----- Original Message -----
From: Mike Gatny <[hidden email]>
To: [hidden email]
Cc: Grant Birchmeier <[hidden email]>; "[hidden email]" <[hidden email]>
Date: 2017/2/24, Fri 00:50
Subject: Re: [Quickfix-developers] Using .seqnums file to trigger a full resend

Fiddling with the seqnum file should definitely work if you completely shut down your app, edit the file, then restart your app.  
To do it with just a logout/edit/logon, try RefreshOnLogon=Y.

--
Mike Gatny
Connamara Systems, LLC

On Thu, Feb 23, 2017 at 12:24 AM, <[hidden email]> wrote:
QuickFIX Documentation: http://www.quickfixengine.org/ quickfix/doc/html/


Hi Grant,

Yes it sounded a bit 'hacky' to me too but this was the approach recommended to me by Asher Newcomer on this forum last October. Ok I will speak to the broker and ask what they recommend.

Thanks,
Dermot

----- Original Message -----
From: Grant Birchmeier <[hidden email]>
To: [hidden email]
Cc: "[hidden email]" <[hidden email]>
Date: 2017/2/23, Thu 13:07
Subject: Re: [Quickfix-developers] Using .seqnums file to trigger a full resend

I would call this a bad practice.  (You're not the first to invent it.)

You're hacking your app's state to basically lie to your counterparty about missing messages.  If you are asking for a *lot* of history, they'll probably truncate it.

Have you checked your counterparty's docs to see if they provide another way to get the historical data that you need?  Surely you aren't the first person to need historical data from this system.

On Wed, Feb 22, 2017 at 9:29 PM, <[hidden email]> wrote:
QuickFIX Documentation: http://www.quickfixengine.org/ quickfix/doc/html/


Hi Guys,

I'm trying to trigger a resend of all the executor's messages for the day. I manually set the incoming seq no in the .seqnums file to 0, then login and this triggers a resend request for all messages as soon as my FIX engine receives the logon response message - all good so far. All the messages are resent and received correctly. The problem is when I log out I can see that the .seqnums file does not have the updated incoming seq no (it's just set to 1) so when I log in again it triggers another resend request.

This problem should be very easy to reproduce. The steps are:-
1. Log on
2. Send order
3. Receive execution
4. Log out
5. Manually change the .seqnums file from "0000000004 : 0000000004" to "0000000004 : 0000000000"
6. Log on (resend is triggered)
7. Log out.
8. Check the seqnums file again and you get "0000000007 : 0000000001" when it should be "0000000007 : 0000000006"

Any ideas?

Thanks a lot!
Dermot

------------------------------ ------------------------------ ------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, SlashDot.org! http://sdm.link/slashdot
______________________________ _________________
Quickfix-developers mailing list
[hidden email]
https://lists.sourceforge.net/ lists/listinfo/quickfix- developers



--
Grant Birchmeier
Connamara Systems, LLC
Made-To-Measure Trading Solutions.
Exactly what you need. No more. No less.



------------------------------ ------------------------------ ------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, SlashDot.org! http://sdm.link/slashdot
______________________________ _________________
Quickfix-developers mailing list
[hidden email]
https://lists.sourceforge.net/ lists/listinfo/quickfix- developers










------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, SlashDot.org! http://sdm.link/slashdot
_______________________________________________
Quickfix-developers mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/quickfix-developers
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Using .seqnums file to trigger a full resend

Mike Gatny
QuickFIX Documentation: http://www.quickfixengine.org/quickfix/doc/html/


Dermot,

Are you able to reproduce your original seqnums-file-not-updating issue using e.g. the 'tradeclient' and 'executor' example apps, with RefreshOnLogon=Y?  I'd like to rule out the weirdness of the resends they are sending you as a factor.

I don't see a problem with what you are trying to do -- I have used this technique many times when building systems that must failover to a remote site.  But as I mentioned, I've only ever done it by completely shutting down the app while editing the file.

Also, if it fits into your architecture, do consider trying the HttpAcceptPort setting, and using the http interface to edit your seqnums.

Finally, are you using the latest version of quickfix (1.14.3)?

--
Mike Gatny
Connamara Systems, LLC

On Tue, Feb 28, 2017 at 8:23 AM, <[hidden email]> wrote:
Hi again Mike, I spoke to the broker who gave me the following:-

1. They don't have any special procedure for sending historical messages and recommend using the regular resend (i.e. they don't truncate messages).
2. They agree there should not be any admin messages included - this is a bug and they are looking into it.

So this means I'll have to go with the contrived resend via the seqnums file. And this brings me all the way back to my original post regarding the seqnums file not containing the correct incoming seq no after resend. Do you have any inclination to take a look at this and perhaps "fix" it? I'm guessing that's a No given you don't really recommend this approach anyway so in that case what are my options? I guess I'll just have to manually update the seqnums file again with the correct number after resend. Thoughts?

Thanks,
Dermot



----- Original Message -----
From: Mike Gatny <[hidden email]>
To: [hidden email]
Cc: Grant Birchmeier <[hidden email]>; "[hidden email]" <[hidden email]>
Date: 2017/2/25, Sat 03:05
Subject: Re: [Quickfix-developers] Using .seqnums file to trigger a full resend

On Fri, Feb 24, 2017 at 12:21 PM, <[hidden email]> wrote:
Ok noted, thanks - don't send my own resend requests. I'm still waiting to hear back from the broker re: recommendations for getting a full day's messages resent. Their support is not great to be honest.

In the meantime I tried editing the seqnums file and that worked but with some problems:-

1. The resend included all heartbeats as well. Is there a way to specify not to include heartbeats/test requests?
2. The resend included the broker's response to the original logout message. This in turn triggers a new logout on my side. Is this normal behavior? Is there a way to exclude this message also? Basically I'm only interested in messages relating to orders/executions.

They should not be sending you any admin-type messages (htbeat/testrequest/logon/logout) at all during a resend.  Instead they should be replacing those msgs with sequence reset msgs with GapFillFlag field set to "Y".   That's just totally broken resend behavior I'm afraid -- broken enough that I don't see what we can do on your side to mitigate it.  And it could trip you up even if you aren't doing a contrived resend via the seqnums file.

--
Mike Gatny
Connamara Systems, LLC

 

Thanks,
Dermot

----- Original Message -----
From: Mike Gatny <[hidden email]>
To: [hidden email]
Cc: Grant Birchmeier <[hidden email]>; "[hidden email]" <[hidden email]>
Date: 2017/2/24, Fri 01:41
Subject: Re: [Quickfix-developers] Using .seqnums file to trigger a full resend

Definitely don't send your own resend requests (or any other admin msg) -- that will mess with the QF state in unsafe ways.
Editing the seqnums file is definitely safe if you do it while your app is not running.  And it should be safe if you are running but logged out, and RefreshOnLogon=Y.
Also note that you can use QF's http interface to reset seqnums.

--
Mike Gatny
Connamara Systems, LLC

On Thu, Feb 23, 2017 at 11:29 AM, <[hidden email]> wrote:
Hi Mike, Grant,

Ok thanks for your feedback. Let me see what the broker comes back with first. There's nothing specific in their ROE regarding history-retrieval.

As a test I did try sending a resend request message [BeginSeqNo=0,EndSeqNo=0] but my side then immediately logged out for some reason.

Dermot

----- Original Message -----
From: Mike Gatny <[hidden email]>
To: [hidden email]
Cc: Grant Birchmeier <[hidden email]>; "[hidden email]" <[hidden email]>
Date: 2017/2/24, Fri 00:50
Subject: Re: [Quickfix-developers] Using .seqnums file to trigger a full resend

Fiddling with the seqnum file should definitely work if you completely shut down your app, edit the file, then restart your app.  
To do it with just a logout/edit/logon, try RefreshOnLogon=Y.

--
Mike Gatny
Connamara Systems, LLC

On Thu, Feb 23, 2017 at 12:24 AM, <[hidden email]> wrote:
QuickFIX Documentation: http://www.quickfixengine.org/ quickfix/doc/html/


Hi Grant,

Yes it sounded a bit 'hacky' to me too but this was the approach recommended to me by Asher Newcomer on this forum last October. Ok I will speak to the broker and ask what they recommend.

Thanks,
Dermot

----- Original Message -----
From: Grant Birchmeier <[hidden email]>
To: [hidden email]
Cc: "[hidden email]" <[hidden email]>
Date: 2017/2/23, Thu 13:07
Subject: Re: [Quickfix-developers] Using .seqnums file to trigger a full resend

I would call this a bad practice.  (You're not the first to invent it.)

You're hacking your app's state to basically lie to your counterparty about missing messages.  If you are asking for a *lot* of history, they'll probably truncate it.

Have you checked your counterparty's docs to see if they provide another way to get the historical data that you need?  Surely you aren't the first person to need historical data from this system.

On Wed, Feb 22, 2017 at 9:29 PM, <[hidden email]> wrote:
QuickFIX Documentation: http://www.quickfixengine.org/ quickfix/doc/html/


Hi Guys,

I'm trying to trigger a resend of all the executor's messages for the day. I manually set the incoming seq no in the .seqnums file to 0, then login and this triggers a resend request for all messages as soon as my FIX engine receives the logon response message - all good so far. All the messages are resent and received correctly. The problem is when I log out I can see that the .seqnums file does not have the updated incoming seq no (it's just set to 1) so when I log in again it triggers another resend request.

This problem should be very easy to reproduce. The steps are:-
1. Log on
2. Send order
3. Receive execution
4. Log out
5. Manually change the .seqnums file from "0000000004 : 0000000004" to "0000000004 : 0000000000"
6. Log on (resend is triggered)
7. Log out.
8. Check the seqnums file again and you get "0000000007 : 0000000001" when it should be "0000000007 : 0000000006"

Any ideas?

Thanks a lot!
Dermot

------------------------------ ------------------------------ ------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, SlashDot.org! http://sdm.link/slashdot
______________________________ _________________
Quickfix-developers mailing list
[hidden email]
https://lists.sourceforge.net/ lists/listinfo/quickfix- developers



--
Grant Birchmeier
Connamara Systems, LLC
Made-To-Measure Trading Solutions.
Exactly what you need. No more. No less.



------------------------------ ------------------------------ ------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, SlashDot.org! http://sdm.link/slashdot
______________________________ _________________
Quickfix-developers mailing list
[hidden email]
https://lists.sourceforge.net/ lists/listinfo/quickfix- developers











------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, SlashDot.org! http://sdm.link/slashdot
_______________________________________________
Quickfix-developers mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/quickfix-developers
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Using .seqnums file to trigger a full resend

Dermot
QuickFIX Documentation: http://www.quickfixengine.org/quickfix/doc/html/


Hi Mike,

Yes I've reproduced it with 'tradeclient'<->'executor' both with no 'RefreshOnLogon' entry and with RefreshOnLogon=Y.

The steps are:-
1. Log on
2. Send order
3. Receive execution
4. Exit program
5. Manually change the .seqnums file from "0000000004 : 0000000004" to "0000000004 : 0000000000"
6. Log on (resend is triggered)
7. Exit program
8. Check the seqnums file again and you get "0000000007 : 0000000001" when it should be "0000000007 : 0000000006"

Actually I'm using MySql for the store and logs so it suits me better to just update using SQL.

Yes I'm using quickfix 1.14.3 C++ version.

Thanks,
Dermot


----- Original Message -----
From: Mike Gatny <[hidden email]>
To: [hidden email]
Cc: Grant Birchmeier <[hidden email]>; "[hidden email]" <[hidden email]>
Date: 2017/2/28, Tue 22:39
Subject: Re: [Quickfix-developers] Using .seqnums file to trigger a full resend

Dermot,

Are you able to reproduce your original seqnums-file-not-updating issue using e.g. the 'tradeclient' and 'executor' example apps, with RefreshOnLogon=Y?  I'd like to rule out the weirdness of the resends they are sending you as a factor.

I don't see a problem with what you are trying to do -- I have used this technique many times when building systems that must failover to a remote site.  But as I mentioned, I've only ever done it by completely shutting down the app while editing the file.

Also, if it fits into your architecture, do consider trying the HttpAcceptPort setting, and using the http interface to edit your seqnums.

Finally, are you using the latest version of quickfix (1.14.3)?

--
Mike Gatny
Connamara Systems, LLC

On Tue, Feb 28, 2017 at 8:23 AM, <[hidden email]> wrote:
Hi again Mike, I spoke to the broker who gave me the following:-

1. They don't have any special procedure for sending historical messages and recommend using the regular resend (i.e. they don't truncate messages).
2. They agree there should not be any admin messages included - this is a bug and they are looking into it.

So this means I'll have to go with the contrived resend via the seqnums file. And this brings me all the way back to my original post regarding the seqnums file not containing the correct incoming seq no after resend. Do you have any inclination to take a look at this and perhaps "fix" it? I'm guessing that's a No given you don't really recommend this approach anyway so in that case what are my options? I guess I'll just have to manually update the seqnums file again with the correct number after resend. Thoughts?

Thanks,
Dermot



----- Original Message -----
From: Mike Gatny <[hidden email]>
To: [hidden email]
Cc: Grant Birchmeier <[hidden email]>; "[hidden email]" <[hidden email]>
Date: 2017/2/25, Sat 03:05
Subject: Re: [Quickfix-developers] Using .seqnums file to trigger a full resend

On Fri, Feb 24, 2017 at 12:21 PM, <[hidden email]> wrote:
Ok noted, thanks - don't send my own resend requests. I'm still waiting to hear back from the broker re: recommendations for getting a full day's messages resent. Their support is not great to be honest.

In the meantime I tried editing the seqnums file and that worked but with some problems:-

1. The resend included all heartbeats as well. Is there a way to specify not to include heartbeats/test requests?
2. The resend included the broker's response to the originallogout message. This in turn triggers a new logout on my side. Is this normal behavior? Is there a way to exclude this message also? Basically I'm only interested in messages relating to orders/executions. 

They should not be sending you any admin-type messages (htbeat/testrequest/logon/ logout) at all during a resend.  Instead they should be replacing those msgs with sequence reset msgs with GapFillFlag field set to "Y".   That's just totally broken resend behavior I'm afraid -- broken enough that I don't see what we can do on your side to mitigate it.  And it could trip you up even if you aren't doing a contrived resend via the seqnums file.

--
Mike Gatny
Connamara Systems, LLC

 

Thanks,
Dermot

----- Original Message -----
From: Mike Gatny <[hidden email]>
To: [hidden email]
Cc: Grant Birchmeier <[hidden email]>; "[hidden email]" <[hidden email]>
Date: 2017/2/24, Fri 01:41
Subject: Re: [Quickfix-developers] Using .seqnums file to trigger a full resend

Definitely don't send your own resend requests (or any other admin msg) -- that will mess with the QF state in unsafe ways.
Editing the seqnums file is definitely safe if you do it while your app is not running.  And it should be safe if you are running but logged out, and RefreshOnLogon=Y.
Also note that you can use QF's http interface to reset seqnums.

--
Mike Gatny
Connamara Systems, LLC

On Thu, Feb 23, 2017 at 11:29 AM, <[hidden email]> wrote:
Hi Mike, Grant,

Ok thanks for your feedback. Let me see what the broker comes back with first. There's nothing specific in their ROE regarding history-retrieval.

As a test I did try sending a resend request message [BeginSeqNo=0,EndSeqNo=0] but my side then immediately logged out for some reason.

Dermot

----- Original Message -----
From: Mike Gatny <[hidden email]>
To: [hidden email]
Cc: Grant Birchmeier <[hidden email]>; "[hidden email]" <[hidden email]>
Date: 2017/2/24, Fri 00:50
Subject: Re: [Quickfix-developers] Using .seqnums file to trigger a full resend

Fiddling with the seqnum file should definitely work if you completely shut down your app, edit the file, then restart your app.  
To do it with just a logout/edit/logon, try RefreshOnLogon=Y.

--
Mike Gatny
Connamara Systems, LLC

On Thu, Feb 23, 2017 at 12:24 AM, <[hidden email]> wrote:
QuickFIX Documentation: http://www.quickfixengine.org/ quickfix/doc/html/


Hi Grant,

Yes it sounded a bit 'hacky' to me too but this was the approach recommended to me by Asher Newcomer on this forum last October. Ok I will speak to the broker and ask what they recommend.

Thanks,
Dermot

----- Original Message -----
From: Grant Birchmeier <[hidden email]>
To: [hidden email]
Cc: "[hidden email]" <[hidden email]>
Date: 2017/2/23, Thu 13:07
Subject: Re: [Quickfix-developers] Using .seqnums file to trigger a full resend

I would call this a bad practice.  (You're not the first to invent it.)

You're hacking your app's state to basically lie to your counterparty about missing messages.  If you are asking for a *lot* of history, they'll probably truncate it.

Have you checked your counterparty's docs to see if they provide another way to get the historical data that you need?  Surely you aren't the first person to need historical data from this system.

On Wed, Feb 22, 2017 at 9:29 PM, <[hidden email]> wrote:
QuickFIX Documentation: http://www.quickfixengine.org/ quickfix/doc/html/


Hi Guys,

I'm trying to trigger a resend of all the executor's messages for the day. I manually set the incoming seq no in the .seqnums file to 0, then login and this triggers a resend request for all messages as soon as my FIX engine receives the logon response message - all good so far. All the messages are resent and received correctly. The problem is when I log out I can see that the .seqnums file does not have the updated incoming seq no (it's just set to 1) so when I log in again it triggers another resend request.

This problem should be very easy to reproduce. The steps are:-
1. Log on
2. Send order
3. Receive execution
4. Log out
5. Manually change the .seqnums file from "0000000004 : 0000000004" to "0000000004 : 0000000000"
6. Log on (resend is triggered)
7. Log out.
8. Check the seqnums file again and you get "0000000007 : 0000000001" when it should be "0000000007 : 0000000006"

Any ideas?

Thanks a lot!
Dermot

------------------------------ ------------------------------ ------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, SlashDot.org! http://sdm.link/slashdot
______________________________ _________________
Quickfix-developers mailing list
[hidden email]
https://lists.sourceforge.net/ lists/listinfo/quickfix- developers



--
Grant Birchmeier
Connamara Systems, LLC
Made-To-Measure Trading Solutions.
Exactly what you need. No more. No less.



------------------------------ ------------------------------ ------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, SlashDot.org! http://sdm.link/slashdot
______________________________ _________________
Quickfix-developers mailing list
[hidden email]
https://lists.sourceforge.net/ lists/listinfo/quickfix- developers













------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, SlashDot.org! http://sdm.link/slashdot
_______________________________________________
Quickfix-developers mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/quickfix-developers
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Using .seqnums file to trigger a full resend

Mike Gatny
QuickFIX Documentation: http://www.quickfixengine.org/quickfix/doc/html/


Dermot,

Thanks for the accurate steps to repro.  I just tried it, and the problem turns out to be quite simple: you are resetting the target seqnum to zero instead of resetting it to 1 -- something that quickfix itself would never do.  When the executor sends a sequence reset message under MsgSeqNum=1, the tradeclient rejects it because it is expecting MsgSeqNum=0, which is an impossible condition.

So, long story short, just set your seqnum file (or database) to "0000000004 : 0000000001" instead of "0000000004 : 0000000000" and it should behave like you wanted.

--
Mike Gatny
Connamara Systems, LLC

On Tue, Feb 28, 2017 at 10:11 AM, <[hidden email]> wrote:
Hi Mike,

Yes I've reproduced it with 'tradeclient'<->'executor' both with no 'RefreshOnLogon' entry and with RefreshOnLogon=Y.

The steps are:-
1. Log on
2. Send order
3. Receive execution
4. Exit program
5. Manually change the .seqnums file from "0000000004 : 0000000004" to "0000000004 : 0000000000"
6. Log on (resend is triggered)
7. Exit program
8. Check the seqnums file again and you get "0000000007 : 0000000001" when it should be "0000000007 : 0000000006"

Actually I'm using MySql for the store and logs so it suits me better to just update using SQL.

Yes I'm using quickfix 1.14.3 C++ version.

Thanks,
Dermot


----- Original Message -----
From: Mike Gatny <[hidden email]>
To: [hidden email]
Cc: Grant Birchmeier <[hidden email]>; "[hidden email]" <[hidden email]>
Date: 2017/2/28, Tue 22:39
Subject: Re: [Quickfix-developers] Using .seqnums file to trigger a full resend

Dermot,

Are you able to reproduce your original seqnums-file-not-updating issue using e.g. the 'tradeclient' and 'executor' example apps, with RefreshOnLogon=Y?  I'd like to rule out the weirdness of the resends they are sending you as a factor.

I don't see a problem with what you are trying to do -- I have used this technique many times when building systems that must failover to a remote site.  But as I mentioned, I've only ever done it by completely shutting down the app while editing the file.

Also, if it fits into your architecture, do consider trying the HttpAcceptPort setting, and using the http interface to edit your seqnums.

Finally, are you using the latest version of quickfix (1.14.3)?

--
Mike Gatny
Connamara Systems, LLC

On Tue, Feb 28, 2017 at 8:23 AM, <[hidden email]> wrote:
Hi again Mike, I spoke to the broker who gave me the following:-

1. They don't have any special procedure for sending historical messages and recommend using the regular resend (i.e. they don't truncate messages).
2. They agree there should not be any admin messages included - this is a bug and they are looking into it.

So this means I'll have to go with the contrived resend via the seqnums file. And this brings me all the way back to my original post regarding the seqnums file not containing the correct incoming seq no after resend. Do you have any inclination to take a look at this and perhaps "fix" it? I'm guessing that's a No given you don't really recommend this approach anyway so in that case what are my options? I guess I'll just have to manually update the seqnums file again with the correct number after resend. Thoughts?

Thanks,
Dermot



----- Original Message -----
From: Mike Gatny <[hidden email]>
To: [hidden email]
Cc: Grant Birchmeier <[hidden email]>; "[hidden email]" <[hidden email]>
Date: 2017/2/25, Sat 03:05
Subject: Re: [Quickfix-developers] Using .seqnums file to trigger a full resend

On Fri, Feb 24, 2017 at 12:21 PM, <[hidden email]> wrote:
Ok noted, thanks - don't send my own resend requests. I'm still waiting to hear back from the broker re: recommendations for getting a full day's messages resent. Their support is not great to be honest.

In the meantime I tried editing the seqnums file and that worked but with some problems:-

1. The resend included all heartbeats as well. Is there a way to specify not to include heartbeats/test requests?
2. The resend included the broker's response to the originallogout message. This in turn triggers a new logout on my side. Is this normal behavior? Is there a way to exclude this message also? Basically I'm only interested in messages relating to orders/executions. 

They should not be sending you any admin-type messages (htbeat/testrequest/logon/ logout) at all during a resend.  Instead they should be replacing those msgs with sequence reset msgs with GapFillFlag field set to "Y".   That's just totally broken resend behavior I'm afraid -- broken enough that I don't see what we can do on your side to mitigate it.  And it could trip you up even if you aren't doing a contrived resend via the seqnums file.


--
Mike Gatny
Connamara Systems, LLC

 

Thanks,
Dermot

----- Original Message -----
From: Mike Gatny <[hidden email]>
To: [hidden email]
Cc: Grant Birchmeier <[hidden email]>; "[hidden email]" <[hidden email]>
Date: 2017/2/24, Fri 01:41
Subject: Re: [Quickfix-developers] Using .seqnums file to trigger a full resend

Definitely don't send your own resend requests (or any other admin msg) -- that will mess with the QF state in unsafe ways.
Editing the seqnums file is definitely safe if you do it while your app is not running.  And it should be safe if you are running but logged out, and RefreshOnLogon=Y.
Also note that you can use QF's http interface to reset seqnums.

--
Mike Gatny
Connamara Systems, LLC

On Thu, Feb 23, 2017 at 11:29 AM, <[hidden email]> wrote:
Hi Mike, Grant,

Ok thanks for your feedback. Let me see what the broker comes back with first. There's nothing specific in their ROE regarding history-retrieval.

As a test I did try sending a resend request message [BeginSeqNo=0,EndSeqNo=0] but my side then immediately logged out for some reason.

Dermot

----- Original Message -----
From: Mike Gatny <[hidden email]>
To: [hidden email]
Cc: Grant Birchmeier <[hidden email]>; "[hidden email]" <[hidden email]>
Date: 2017/2/24, Fri 00:50
Subject: Re: [Quickfix-developers] Using .seqnums file to trigger a full resend

Fiddling with the seqnum file should definitely work if you completely shut down your app, edit the file, then restart your app.  
To do it with just a logout/edit/logon, try RefreshOnLogon=Y.

--
Mike Gatny
Connamara Systems, LLC

On Thu, Feb 23, 2017 at 12:24 AM, <[hidden email]> wrote:
QuickFIX Documentation: http://www.quickfixengine.org/ quickfix/doc/html/


Hi Grant,

Yes it sounded a bit 'hacky' to me too but this was the approach recommended to me by Asher Newcomer on this forum last October. Ok I will speak to the broker and ask what they recommend.

Thanks,
Dermot

----- Original Message -----
From: Grant Birchmeier <[hidden email]>
To: [hidden email]
Cc: "[hidden email]" <[hidden email]>
Date: 2017/2/23, Thu 13:07
Subject: Re: [Quickfix-developers] Using .seqnums file to trigger a full resend

I would call this a bad practice.  (You're not the first to invent it.)

You're hacking your app's state to basically lie to your counterparty about missing messages.  If you are asking for a *lot* of history, they'll probably truncate it.

Have you checked your counterparty's docs to see if they provide another way to get the historical data that you need?  Surely you aren't the first person to need historical data from this system.

On Wed, Feb 22, 2017 at 9:29 PM, <[hidden email]> wrote:
QuickFIX Documentation: http://www.quickfixengine.org/ quickfix/doc/html/


Hi Guys,

I'm trying to trigger a resend of all the executor's messages for the day. I manually set the incoming seq no in the .seqnums file to 0, then login and this triggers a resend request for all messages as soon as my FIX engine receives the logon response message - all good so far. All the messages are resent and received correctly. The problem is when I log out I can see that the .seqnums file does not have the updated incoming seq no (it's just set to 1) so when I log in again it triggers another resend request.

This problem should be very easy to reproduce. The steps are:-
1. Log on
2. Send order
3. Receive execution
4. Log out
5. Manually change the .seqnums file from "0000000004 : 0000000004" to "0000000004 : 0000000000"
6. Log on (resend is triggered)
7. Log out.
8. Check the seqnums file again and you get "0000000007 : 0000000001" when it should be "0000000007 : 0000000006"

Any ideas?

Thanks a lot!
Dermot

------------------------------ ------------------------------ ------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, SlashDot.org! http://sdm.link/slashdot
______________________________ _________________
Quickfix-developers mailing list
[hidden email]
https://lists.sourceforge.net/ lists/listinfo/quickfix- developers



--
Grant Birchmeier
Connamara Systems, LLC
Made-To-Measure Trading Solutions.
Exactly what you need. No more. No less.



------------------------------ ------------------------------ ------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, SlashDot.org! http://sdm.link/slashdot
______________________________ _________________
Quickfix-developers mailing list
[hidden email]
https://lists.sourceforge.net/ lists/listinfo/quickfix- developers














------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, SlashDot.org! http://sdm.link/slashdot
_______________________________________________
Quickfix-developers mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/quickfix-developers
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Using .seqnums file to trigger a full resend

Dermot
QuickFIX Documentation: http://www.quickfixengine.org/quickfix/doc/html/


Hi Mike,

Fantastic.. you're right, that works! (Actually Asher told me to set it to 0 not 1 when I first asked about it back in Oct).

Really appreciate your time on this.. very impressive support from you guys!

Thanks,
Dermot

----- Original Message -----
From: Mike Gatny <[hidden email]>
To: [hidden email]
Cc: Grant Birchmeier <[hidden email]>; "[hidden email]" <[hidden email]>
Date: 2017/3/1, Wed 01:38
Subject: Re: [Quickfix-developers] Using .seqnums file to trigger a full resend

Dermot,

Thanks for the accurate steps to repro.  I just tried it, and the problem turns out to be quite simple: you are resetting the target seqnum to zero instead of resetting it to 1 -- something that quickfix itself would never do.  When the executor sends a sequence reset message under MsgSeqNum=1, the tradeclient rejects it because it is expecting MsgSeqNum=0, which is an impossible condition.

So, long story short, just set your seqnum file (or database) to "0000000004 : 0000000001" instead of "0000000004 : 0000000000" and it should behave like you wanted.

--
Mike Gatny
Connamara Systems, LLC

On Tue, Feb 28, 2017 at 10:11 AM, <[hidden email]> wrote:
Hi Mike,

Yes I've reproduced it with 'tradeclient'<->'executor' both with no 'RefreshOnLogon' entry and with RefreshOnLogon=Y.

The steps are:-
1. Log on
2. Send order
3. Receive execution
4. Exit program
5. Manually change the .seqnums file from "0000000004 : 0000000004" to "0000000004 : 0000000000"
6. Log on (resend is triggered)
7. Exit program
8. Check the seqnums file again and you get "0000000007 : 0000000001" when it should be "0000000007 : 0000000006"

Actually I'm using MySql for the store and logs so it suits me better to just update using SQL.

Yes I'm using quickfix 1.14.3 C++ version.

Thanks,
Dermot


----- Original Message -----
From: Mike Gatny <[hidden email]>
To: [hidden email]
Cc: Grant Birchmeier <[hidden email]>; "[hidden email]" <[hidden email]>
Date: 2017/2/28, Tue 22:39
Subject: Re: [Quickfix-developers] Using .seqnums file to trigger a full resend

Dermot,

Are you able to reproduce your original seqnums-file-not-updating issue using e.g. the 'tradeclient' and 'executor' example apps, with RefreshOnLogon=Y?  I'd like to rule out the weirdness of the resends they are sending you as a factor.

I don't see a problem with what you are trying to do -- I have used this technique many times when building systems that must failover to a remote site.  But as I mentioned, I've only ever done it by completely shutting down the app while editing the file.

Also, if it fits into your architecture, do consider trying the HttpAcceptPort setting, and using the http interface to edit your seqnums.

Finally, are you using the latest version of quickfix (1.14.3)?

--
Mike Gatny
Connamara Systems, LLC

On Tue, Feb 28, 2017 at 8:23 AM, <[hidden email]> wrote:
Hi again Mike, I spoke to the broker who gave me the following:-

1. They don't have any special procedure for sending historical messages and recommend using the regular resend (i.e. they don't truncate messages).
2. They agree there should not be any admin messages included - this is a bug and they are looking into it.

So this means I'll have to go with the contrived resend via the seqnums file. And this brings me all the way back to my original post regarding the seqnums file not containing the correct incoming seq no after resend. Do you have any inclination to take a look at this and perhaps "fix" it? I'm guessing that's a No given you don't really recommend this approach anyway so in that case what are my options? I guess I'll just have to manually update the seqnums file again with the correct number after resend. Thoughts?

Thanks,
Dermot



----- Original Message -----
From: Mike Gatny <[hidden email]>
To: [hidden email]
Cc: Grant Birchmeier <[hidden email]>; "[hidden email]" <[hidden email]>
Date: 2017/2/25, Sat 03:05
Subject: Re: [Quickfix-developers] Using .seqnums file to trigger a full resend

On Fri, Feb 24, 2017 at 12:21 PM, <[hidden email]> wrote:
Ok noted, thanks - don't send my own resend requests. I'm still waiting to hear back from the broker re: recommendations for getting a full day's messages resent. Their support is not great to be honest.

In the meantime I tried editing the seqnums file and that worked but with some problems:-

1. The resend included all heartbeats as well. Is there a way to specify not to include heartbeats/test requests?
2. The resend included the broker's response to the originallogout message. This in turn triggers a new logout on my side. Is this normal behavior? Is there a way to exclude this message also? Basically I'm only interested in messages relating to orders/executions. 

They should not be sending you any admin-type messages (htbeat/testrequest/logon/ logout) at all during a resend.  Instead they should be replacing those msgs with sequence reset msgs with GapFillFlag field set to "Y".   That's just totally broken resend behavior I'm afraid -- broken enough that I don't see what we can do on your side to mitigate it.  And it could trip you up even if you aren't doing a contrived resend via the seqnums file.


--
Mike Gatny
Connamara Systems, LLC

 

Thanks,
Dermot

----- Original Message -----
From: Mike Gatny <[hidden email]>
To: [hidden email]
Cc: Grant Birchmeier <[hidden email]>; "[hidden email]" <[hidden email]>
Date: 2017/2/24, Fri 01:41
Subject: Re: [Quickfix-developers] Using .seqnums file to trigger a full resend

Definitely don't send your own resend requests (or any other admin msg) -- that will mess with the QF state in unsafe ways.
Editing the seqnums file is definitely safe if you do it while your app is not running.  And it should be safe if you are running but logged out, and RefreshOnLogon=Y.
Also note that you can use QF's http interface to reset seqnums.

--
Mike Gatny
Connamara Systems, LLC

On Thu, Feb 23, 2017 at 11:29 AM, <[hidden email]> wrote:
Hi Mike, Grant,

Ok thanks for your feedback. Let me see what the broker comes back with first. There's nothing specific in their ROE regarding history-retrieval.

As a test I did try sending a resend request message [BeginSeqNo=0,EndSeqNo=0] but my side then immediately logged out for some reason.

Dermot

----- Original Message -----
From: Mike Gatny <[hidden email]>
To: [hidden email]
Cc: Grant Birchmeier <[hidden email]>; "[hidden email]" <[hidden email]>
Date: 2017/2/24, Fri 00:50
Subject: Re: [Quickfix-developers] Using .seqnums file to trigger a full resend

Fiddling with the seqnum file should definitely work if you completely shut down your app, edit the file, then restart your app.  
To do it with just a logout/edit/logon, try RefreshOnLogon=Y.

--
Mike Gatny
Connamara Systems, LLC

On Thu, Feb 23, 2017 at 12:24 AM, <[hidden email]> wrote:
QuickFIX Documentation: http://www.quickfixengine.org/ quickfix/doc/html/


Hi Grant,

Yes it sounded a bit 'hacky' to me too but this was the approach recommended to me by Asher Newcomer on this forum last October. Ok I will speak to the broker and ask what they recommend.

Thanks,
Dermot

----- Original Message -----
From: Grant Birchmeier <[hidden email]>
To: [hidden email]
Cc: "[hidden email]" <[hidden email]>
Date: 2017/2/23, Thu 13:07
Subject: Re: [Quickfix-developers] Using .seqnums file to trigger a full resend

I would call this a bad practice.  (You're not the first to invent it.)

You're hacking your app's state to basically lie to your counterparty about missing messages.  If you are asking for a *lot* of history, they'll probably truncate it.

Have you checked your counterparty's docs to see if they provide another way to get the historical data that you need?  Surely you aren't the first person to need historical data from this system.

On Wed, Feb 22, 2017 at 9:29 PM, <[hidden email]> wrote:
QuickFIX Documentation: http://www.quickfixengine.org/ quickfix/doc/html/


Hi Guys,

I'm trying to trigger a resend of all the executor's messages for the day. I manually set the incoming seq no in the .seqnums file to 0, then login and this triggers a resend request for all messages as soon as my FIX engine receives the logon response message - all good so far. All the messages are resent and received correctly. The problem is when I log out I can see that the .seqnums file does not have the updated incoming seq no (it's just set to 1) so when I log in again it triggers another resend request.

This problem should be very easy to reproduce. The steps are:-
1. Log on
2. Send order
3. Receive execution
4. Log out
5. Manually change the .seqnums file from "0000000004 : 0000000004" to "0000000004 : 0000000000"
6. Log on (resend is triggered)
7. Log out.
8. Check the seqnums file again and you get "0000000007 : 0000000001" when it should be "0000000007 : 0000000006"

Any ideas?

Thanks a lot!
Dermot

------------------------------ ------------------------------ ------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, SlashDot.org! http://sdm.link/slashdot
______________________________ _________________
Quickfix-developers mailing list
[hidden email]
https://lists.sourceforge.net/ lists/listinfo/quickfix- developers



--
Grant Birchmeier
Connamara Systems, LLC
Made-To-Measure Trading Solutions.
Exactly what you need. No more. No less.



------------------------------ ------------------------------ ------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, SlashDot.org! http://sdm.link/slashdot
______________________________ _________________
Quickfix-developers mailing list
[hidden email]
https://lists.sourceforge.net/ lists/listinfo/quickfix- developers
















------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, SlashDot.org! http://sdm.link/slashdot
_______________________________________________
Quickfix-developers mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/quickfix-developers
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Network disconnect recovery testing

Dermot
In reply to this post by Mike Gatny
QuickFIX Documentation: http://www.quickfixengine.org/quickfix/doc/html/


Hi Guys, I've been doing some recovery testing and I have a couple of questions.

Firstly, my setup is pretty simple - my trading application (initiator) and sell-side simulator (acceptor) are set up on two different networked machines, both are quickfix apps. Here's what I'm doing:- 

1. Establish a connection between initiator and acceptor.
2. Send some orders and ack them on the acceptor side.
2. Break the network connection on the initiator side.
3. Send some more orders from the initiator.
4. Fill existing orders on acceptor side.
5. Reconnect the initiator to the network.

At this point quickfix handles the reconnect and all the executions flow back to the initiator. However the new orders I sent from the initiator during the downtime are not resent to the executor [they are gap filled instead]. Is this the correct behaviour? Actually this is exactly the behaviour I want but I thought quickfix would resend them. So that's my first question.

My second question is what is the best way to determine a connection problem before sending any order?

Thanks!
Dermot

------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot
_______________________________________________
Quickfix-developers mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/quickfix-developers
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Network disconnect recovery testing

Mike Gatny
QuickFIX Documentation: http://www.quickfixengine.org/quickfix/doc/html/


Dermot,

Have you set PersistMessages=N in your SessionSettings? This is one way to prevent resend (i.e. cause GapFill over App-level msgs).

Another way is to throw DoNotSend in toApp for possdup msgs:

void App::toApp(FIX::Message& msg, const FIX::SessionID&) throw(FIX::DoNotSend)
{
  try 
  {
    FIX::PossDupFlag possDupFlag;
    msg.getHeader().getField(possDupFlag);
    if (possDupFlag)
      throw FIX::DoNotSend();
  }
  catch (FIX::FieldNotFound&)
  { }
}

As you say, this is almost certainly the behavior you want for sending orders, and that's the two main ways to achieve it.  If you are not doing either of those things, QF will indeed resend instead of sending GapFills.

If you want to detect a connection problem, you can use onLogon() and onLogout() to alert or set some state.  You can think of onLogout() as "onDisconnect" as well -- it will get called even if you didn't get a clean Logout.
Another way to do it on the fly is:

FIX::Session * pSession = FIX::Session::lookupSession(sessionID);
if (pSession->isLoggedOn()) { /* ... */  }

--
Mike Gatny
Connamara Systems, LLC

On Thu, Apr 13, 2017 at 10:37 AM, <[hidden email]> wrote:
Hi Guys, I've been doing some recovery testing and I have a couple of questions.

Firstly, my setup is pretty simple - my trading application (initiator) and sell-side simulator (acceptor) are set up on two different networked machines, both are quickfix apps. Here's what I'm doing:- 

1. Establish a connection between initiator and acceptor.
2. Send some orders and ack them on the acceptor side.
2. Break the network connection on the initiator side.
3. Send some more orders from the initiator.
4. Fill existing orders on acceptor side.
5. Reconnect the initiator to the network.

At this point quickfix handles the reconnect and all the executions flow back to the initiator. However the new orders I sent from the initiator during the downtime are not resent to the executor [they are gap filled instead]. Is this the correct behaviour? Actually this is exactly the behaviour I want but I thought quickfix would resend them. So that's my first question.

My second question is what is the best way to determine a connection problem before sending any order?

Thanks!
Dermot


------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot
_______________________________________________
Quickfix-developers mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/quickfix-developers
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Network disconnect recovery testing

Dermot
QuickFIX Documentation: http://www.quickfixengine.org/quickfix/doc/html/


Hi Mike,

Thanks for the quick response. I haven't included PersistMessages in the config file but I can see it defaults to 'N' so that makes sense.

Ok noted on the onLogon()/onLogout(). Up to now I've just been catching the event text "Received logon response" and "Disconnecting" which is a bit hacky but the result is the same. The only issue I have is I've been trying to keep HeartBtInt=120 so as not to flood MySql with too many messages - this means that it takes a minute or two for quickfix to realise there's a network problem and disconnect and in the meantime my app will still be sending orders. I guess I can't have my cake and eat it so if I want to catch it sooner I'll have to up the heartbeat frequency, right?

Dermot



----- Original Message -----
From: Mike Gatny <[hidden email]>
To: [hidden email]
Cc: Grant Birchmeier <[hidden email]>; "[hidden email]" <[hidden email]>
Date: 2017/4/13, Thu 23:55
Subject: Re: Network disconnect recovery testing

Dermot,

Have you set PersistMessages=N in your SessionSettings? This is one way to prevent resend (i.e. cause GapFill over App-level msgs).

Another way is to throw DoNotSend in toApp for possdup msgs:

void App::toApp(FIX::Message& msg, const FIX::SessionID&) throw(FIX::DoNotSend)
{
  try 
  {
    FIX::PossDupFlag possDupFlag;
    msg.getHeader().getField(possDupFlag);
    if (possDupFlag)
      throw FIX::DoNotSend();
  }
  catch (FIX::FieldNotFound&)
  { }
}

As you say, this is almost certainly the behavior you want for sending orders, and that's the two main ways to achieve it.  If you are not doing either of those things, QF will indeed resend instead of sending GapFills.

If you want to detect a connection problem, you can use onLogon() and onLogout() to alert or set some state.  You can think of onLogout() as "onDisconnect" as well -- it will get called even if you didn't get a clean Logout.
Another way to do it on the fly is:

FIX::Session * pSession = FIX::Session::lookupSession(sessionID);
if (pSession->isLoggedOn()) { /* ... */  }

--
Mike Gatny
Connamara Systems, LLC

On Thu, Apr 13, 2017 at 10:37 AM, <[hidden email]> wrote:
Hi Guys, I've been doing some recovery testing and I have a couple of questions.

Firstly, my setup is pretty simple - my trading application (initiator) and sell-side simulator (acceptor) are set up on two different networked machines, both are quickfix apps. Here's what I'm doing:- 

1. Establish a connection between initiator and acceptor.
2. Send some orders and ack them on the acceptor side.
2. Break the network connection on the initiator side.
3. Send some more orders from the initiator.
4. Fill existing orders on acceptor side.
5. Reconnect the initiator to the network.

At this point quickfix handles the reconnect and all the executions flow back to the initiator. However the new orders I sent from the initiator during the downtime are not resent to the executor [they are gap filled instead]. Is this the correct behaviour? Actually this is exactly the behaviour I want but I thought quickfix would resend them. So that's my first question.

My second question is what is the best way to determine a connection problem before sending any order?

Thanks!
Dermot




------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot
_______________________________________________
Quickfix-developers mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/quickfix-developers
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Network disconnect recovery testing

Mike Gatny
QuickFIX Documentation: http://www.quickfixengine.org/quickfix/doc/html/


Dermot,

This is one of the reasons the qf/j implementation has config options like JdbcLogHeartBeats/FileLogHeartbeats.
I don't think it would be too hard to add this to MySQLLog for qf/c++.

--
Mike Gatny
Connamara Systems, LLC

On Thu, Apr 13, 2017 at 11:18 AM, <[hidden email]> wrote:
Hi Mike,

Thanks for the quick response. I haven't included PersistMessages in the config file but I can see it defaults to 'N' so that makes sense.

Ok noted on the onLogon()/onLogout(). Up to now I've just been catching the event text "Received logon response" and "Disconnecting" which is a bit hacky but the result is the same. The only issue I have is I've been trying to keep HeartBtInt=120 so as not to flood MySql with too many messages - this means that it takes a minute or two for quickfix to realise there's a network problem and disconnect and in the meantime my app will still be sending orders. I guess I can't have my cake and eat it so if I want to catch it sooner I'll have to up the heartbeat frequency, right?

Dermot



----- Original Message -----
From: Mike Gatny <[hidden email]>
To: [hidden email]
Cc: Grant Birchmeier <[hidden email]>; "[hidden email]" <[hidden email]>
Date: 2017/4/13, Thu 23:55
Subject: Re: Network disconnect recovery testing

Dermot,

Have you set PersistMessages=N in your SessionSettings? This is one way to prevent resend (i.e. cause GapFill over App-level msgs).

Another way is to throw DoNotSend in toApp for possdup msgs:

void App::toApp(FIX::Message& msg, const FIX::SessionID&) throw(FIX::DoNotSend)
{
  try 
  {
    FIX::PossDupFlag possDupFlag;
    msg.getHeader().getField(possDupFlag);
    if (possDupFlag)
      throw FIX::DoNotSend();
  }
  catch (FIX::FieldNotFound&)
  { }
}

As you say, this is almost certainly the behavior you want for sending orders, and that's the two main ways to achieve it.  If you are not doing either of those things, QF will indeed resend instead of sending GapFills.

If you want to detect a connection problem, you can use onLogon() and onLogout() to alert or set some state.  You can think of onLogout() as "onDisconnect" as well -- it will get called even if you didn't get a clean Logout.
Another way to do it on the fly is:

FIX::Session * pSession = FIX::Session::lookupSession(sessionID);
if (pSession->isLoggedOn()) { /* ... */  }

--
Mike Gatny
Connamara Systems, LLC

On Thu, Apr 13, 2017 at 10:37 AM, <[hidden email]> wrote:
Hi Guys, I've been doing some recovery testing and I have a couple of questions.

Firstly, my setup is pretty simple - my trading application (initiator) and sell-side simulator (acceptor) are set up on two different networked machines, both are quickfix apps. Here's what I'm doing:- 

1. Establish a connection between initiator and acceptor.
2. Send some orders and ack them on the acceptor side.
2. Break the network connection on the initiator side.
3. Send some more orders from the initiator.
4. Fill existing orders on acceptor side.
5. Reconnect the initiator to the network.

At this point quickfix handles the reconnect and all the executions flow back to the initiator. However the new orders I sent from the initiator during the downtime are not resent to the executor [they are gap filled instead]. Is this the correct behaviour? Actually this is exactly the behaviour I want but I thought quickfix would resend them. So that's my first question.

My second question is what is the best way to determine a connection problem before sending any order?

Thanks!
Dermot





------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot
_______________________________________________
Quickfix-developers mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/quickfix-developers
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Network disconnect recovery testing

Dermot
QuickFIX Documentation: http://www.quickfixengine.org/quickfix/doc/html/


Got it. Will take a look at that. Thanks a lot.
Dermot

----- Original Message -----
From: Mike Gatny <[hidden email]>
To: [hidden email]
Cc: Grant Birchmeier <[hidden email]>; "[hidden email]" <[hidden email]>
Date: 2017/4/14, Fri 00:31
Subject: Re: Network disconnect recovery testing

Dermot,

This is one of the reasons the qf/j implementation has config options like JdbcLogHeartBeats/FileLogHeartbeats.
I don't think it would be too hard to add this to MySQLLog for qf/c++.

--
Mike Gatny
Connamara Systems, LLC

On Thu, Apr 13, 2017 at 11:18 AM, <[hidden email]> wrote:
Hi Mike,

Thanks for the quick response. I haven't included PersistMessages in the config file but I can see it defaults to 'N' so that makes sense.

Ok noted on the onLogon()/onLogout(). Up to now I've just been catching the event text "Received logon response" and "Disconnecting" which is a bit hacky but the result is the same. The only issue I have is I've been trying to keep HeartBtInt=120 so as not to flood MySql with too many messages - this means that it takes a minute or two for quickfix to realise there's a network problem and disconnect and in the meantime my app will still be sending orders. I guess I can't have my cake and eat it so if I want to catch it sooner I'll have to up the heartbeat frequency, right?

Dermot



----- Original Message -----
From: Mike Gatny <[hidden email]>
To: [hidden email]
Cc: Grant Birchmeier <[hidden email]>; "[hidden email]" <[hidden email]>
Date: 2017/4/13, Thu 23:55
Subject: Re: Network disconnect recovery testing

Dermot,

Have you set PersistMessages=N in your SessionSettings? This is one way to prevent resend (i.e. cause GapFill over App-level msgs).

Another way is to throw DoNotSend in toApp for possdup msgs:

void App::toApp(FIX::Message& msg, const FIX::SessionID&) throw(FIX:: DoNotSend)
{
  try 
  {
    FIX::PossDupFlag possDupFlag;
    msg.getHeader().getField( possDupFlag);
    if (possDupFlag)
      throw FIX::DoNotSend();
  }
  catch (FIX::FieldNotFound&)
  { }
}

As you say, this is almost certainly the behavior you want for sending orders, and that's the two main ways to achieve it.  If you are not doing either of those things, QF will indeed resend instead of sending GapFills.

If you want to detect a connection problem, you can use onLogon() and onLogout() to alert or set some state.  You can think of onLogout() as "onDisconnect" as well -- it will get called even if you didn't get a clean Logout.
Another way to do it on the fly is:

FIX::Session * pSession = FIX::Session::lookupSession( sessionID);
if (pSession->isLoggedOn()) { /* ... */  }

--
Mike Gatny
Connamara Systems, LLC

On Thu, Apr 13, 2017 at 10:37 AM, <[hidden email]> wrote:
Hi Guys, I've been doing some recovery testing and I have a couple of questions.

Firstly, my setup is pretty simple - my trading application (initiator) and sell-side simulator (acceptor) are set up on two different networked machines, both are quickfix apps. Here's what I'm doing:- 

1. Establish a connection between initiator and acceptor.
2. Send some orders and ack them on the acceptor side.
2. Break the network connection on the initiator side.
3. Send some more orders from the initiator.
4. Fill existing orders on acceptor side.
5. Reconnect the initiator to the network.

At this point quickfix handles the reconnect and all the executions flow back to the initiator. However the new orders I sent from the initiator during the downtime are not resent to the executor [they are gap filled instead]. Is this the correct behaviour? Actually this is exactly the behaviour I want but I thought quickfix would resend them. So that's my first question.

My second question is what is the best way to determine a connection problem before sending any order?

Thanks!
Dermot







------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot
_______________________________________________
Quickfix-developers mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/quickfix-developers
12
Loading...