Understanding IBAPI’s Session/ClientID and OrderID in a Multi-Host, Multi-Account Environment

Posted on June 17, 2024, by Peter Andersson

Introduction

Managing the client and order ID in a straightforward environment with a single IBAPI application and one account is simple. You just listen to the nextValidId() callback and increment the ID manually after each order. Easy-peasy.

But what happens when you have multiple TWS instances, Gateways, and possibly multiple accounts?

And, maybe more common, multiple applications, each with its own client ID?

In this post, I’ll clarify the details you might otherwise overlook. These details can become significant in more complex scenarios.

ClientID and OrderID Management

Reservations

This guide is based on my team’s experiments. We use Linux exclusively for servers and clients, so if you develop on Windows, this might not be entirely accurate. Its applicability to macOS is uncertain. Real developers use Linux—live with it.

We tested against TWS and Gateway version 10.26, with client programs in Python 3.12 using IBAPI version 10.26.

User Folders

Under your Jts directory (~/Jts on Linux), you’ll find one or more folders with long, unique names. Each folder corresponds to a different user/environment combination (e.g., paper trading and live).

For example:

ll Jts/
total 15M
<snip>
drwxrwxr-x 4 peter peter 4.0K Jun 14 03:00 eblcfcdgi<snip><snip><snip>pncofimngicff
drwxr-xr-x 4 peter peter 12K Jun 11 23:43 gpifkjocn<snip><snip><snip>ieffjbfpgminj
drwxr-xr-x 4 peter peter 4.0K Jun 10 06:00 iphcjdcci<snip><snip><snip>bbdolblcnogpo
<snip>

Inside each folder, you’ll find several encrypted files, including tws.xml. This file can be decrypted using TWS (details below) and contains the client IDs and their next valid order IDs.

Understanding ClientID

The client ID, sometimes called the session ID, is more persistent than a temporary ID like a socket or file handle. Since only one user can be logged in at a time, only one of these folders is “active,” with its own set of client IDs and order ID sequences.

Order ID Requirements:

1. It must be unique.
2. Each new order ID must be larger than all preceding orders. The sequence does not need to be continuous; gaps are allowed.

The ClientID/OrderID combination is stored in the tws.xml file within the active user’s directory. This data is saved when TWS exits normally, but may not be saved if TWS crashes.

So what happens if you use the same client ID, simultaneausly logging into different TWS/Gateways, with syncing enabled?

I didn’t even think of trying that. But it can’t be a great idea. I suggest you avoid it.

Locating ClientID and Next Valid Order ID

Access the tws.xml file via Help -> Troubleshooting -> Diagnostics -> “TWS Layout/Settings.” Copy the text and search for “ddeIdMap” in your editor of choice.

<MapOfStrings varName=”ddeIdMap”>
<Entry key=”1941″ val=”3″/>
<Entry key=”1632837453″ val=”2″/>
<Entry key=”3847″ val=”3″/>
<Entry key=”5873″ val=”14″/>
<Entry key=”123″ val=”6″/>
<Entry key=”1632840530″ val=”2″/>
<Entry key=”19430″ val=”12″/>
<Entry key=”12″ val=”17″/>
<Entry key=”9549″ val=”3″/>
<Entry key=”5909″ val=”8″/>
<Entry key=”10317″ val=”24″/>
<Entry key=”4221″ val=”3″/>
<Entry key=”1632823421″ val=”1″/>
<Entry key=”2951″ val=”14″/>
<Entry key=”2163″ val=”3″/>
<Entry key=”6436″ val=”14″/>
</MapOfStrings>

The four-figure entries are client IDs.

Special Client IDs

• Client ID 0: Retrieves orders placed using TWS.
• Client ID MASTER: Retrieves orders from all client IDs running on this TWS. The exact number is set in TWS Config -> API -> Settings.

Experiments and Common Use Cases

• reqOpenOrders(): Retrieves orders placed on your current client ID (except for 0 or MASTER).
• reqOpenAllOrders(): Retrieves orders placed on all client IDs.

Example Scenario:

• Synced Settings (Config-Lock and Exit-Use/store settings on server):
  • Client CA connects to TWS TA (paper trading) using clientID 111.
  • Client CA receives nextValidId 1.
  • Client CA places an order with orderID 10 (valid since it is >= 1).
  • Client CA logs out.
  • Client CB connects to Gateway GA (paper trading).
  • Client CB receives nextValidId 11.
• Unsynced Settings (TWS TA does not have “Use/store settings on server”):
  • Same steps as above for Client CA.
  • Client CB connects to Gateway GA (paper trading) and receives whatever was last used for this user on this gateway.

Conclusion

Remember that local data is primarily stored in the User folder. If you have multiple installations of TWS you might have multiple installation folders. And if you have multiple users (even one live and one for paper trading) you will have multiple subfolders to the installtion directory. This is where you user data is stored, and this is replicaded if you have chosen so.

Always listen to the nextValidId() callback, but remember it only applies to your own client ID. Order IDs do not need to be sequential but must always increase. So if you use reqAllOpenOrders you have to use the order.clientId in the index to keep the orders apart.

Always check the account field of orders/executions in a multi-account environment.

Order of importance

1. User folder. Possibly replicated!
2. Client ID.
3. Order ID.

Recommendations:

• Prefer reqOpenOrders(). Use reqOpenAllOrders() with caution.
• Check both client ID and account number in callbacks, even with a single client ID and account.
• Sync settings across multiple TWS/Gateway installations. However this last entry is 50/50…

Hopefully, this clarifies how TWS/IBGateway handles client and order ID sequences.