Equity Analytics Framework - Adding Custom Analytics

The Equity Analytics Framework calculates a wide number of Order Analytics. This framework is built with flexibility in mind and can easily be extended with custom analytics.

This page provides an example outlining how to add custom analytics with varying levels of complexity.

Step 1 - Add a patch to the ${PKG_NAME} Accelerator package

Patches can be used to customize your accelerator package. Refer to the documentation on customizing the FSI Accelerator for more information.

The Equity Analytics Framework generates order analytics using data from the Order, Trade and Quote tables. The result is then upserted to the OrderAnalytics table.

If you wish to add custom analytics, you must first ensure that these analytics exist as columns in the OrderAnalytics table.

You must use the kxi CLI to add a patch to your package.

The following example adds a patch called CustomAnalytics.

# Unpack the accelerator package
kxi package unpack ${PKG_NAME}-${VERSION}.kxi

# Create patch to add columns to the OrderAnalytics schema
kxi package add --to ${PKG_NAME} patch --name CustomAnalytics

This generates a YAML patch file in the path ${PKG_NAME}/patches/CustomAnalytics.yaml. We can use this patch file to add custom analytic columns to the OrderAnalytics table.

Update the file ${PKG_NAME}/patches/CustomAnalytics.yaml using your editor of choice.

The yaml snippet below contains all the details you need for this example:

kind: Package
apiVersion: pakx/v1
metadata:
  name: target
spec:  
  manifest: {}
  tables:
    schemas:
    - name: OrderAnalytics
      columns:   
      - name: reversionAskPrice_30
        type: float
      - name: reversionBidPrice_30
        type: float        
      - name: strikeToCompletionBidMidPrice
        type: float
      - name: strikeToCompletionAskMidPrice
        type: float
      - name: countTradesUnderLimitPrice
        type: int
      - name: sumVolumeUnderLimitPrice
        type: long                   
      - name: myArrivalTradePrice
        type: float        
      - name: myArrivalTradePrice_5
        type: float        
      - name: myArrivalTradePrice_10
        type: float

Here, we are adding several columns to the OrderAnalytics schema, defining the name and type of each column added.

After adding the required columns to your patch, overlay the patch on your accelerator package using the kxi CLI, as shown below.

# Apply the overlay
kxi package overlay ${PKG_NAME} ${PKG_NAME}/patches/CustomAnalytics.yaml

Step 2 - Define custom analytics

Custom Analytics can be added to the Equity Analytics Framework using a q file that matches the naming convention *eqeaCustomAnalytics.q.

This example creates a *eqeaCustomAnalytics.q file called example.eqeaCustomAnalytics.q.

The *eqeaCustomAnalytics.q file should contain:

• A configuration table named .eqea.config.custom.analytics which defines all custom analytics.

• Definitions of any custom functions that are used to calculate the custom analytics.

The .eqea.config.custom.analytics table must have the following structure:

In this example, we are defining several custom analytics of varying complexity.

Add the snippet below to your example.eqeaCustomAnalytics.q file:

// Example Custom Analytics
.eqea.config.custom.analytics:flip `analytic`analyticType`funcName`aggClause`marketDataTabName`joinTimeOffset! flip (
    
    // Examples of adding analytics to the pre-packaged analytic types.
    (`reversionAskPrice_30          ; `reversion        ; `.eqea.analytics.reversion                ; `askPrice                               ; `Quote   ; 00:00:30);    
    (`reversionBidPrice_30          ; `reversion        ; `.eqea.analytics.reversion                ; `bidPrice                               ; `Quote   ; 00:00:30);    
    
    // Examples of adding simple custom analytic types (simple aggregation on already calculated values in our OrderAnalytics table)
    (`strikeToCompletionBidMidPrice ;`simpleCustomAgg   ;`.example.orderExecution.simpleCustomAgg   ; (%;(+;`arrivalAskPrice;`endAskPrice);2) ; `       ; 0Nt );
    (`strikeToCompletionAskMidPrice ;`simpleCustomAgg   ;`.example.orderExecution.simpleCustomAgg   ; (%;(+;`arrivalAskPrice;`endAskPrice);2) ; `       ; 0Nt );
    
    // Examples of adding more complex custom analytics (aggregations on tick data with multiple where clauses)
    (`countTradesUnderLimitPrice             ;`tickDataAgg  ;`.example.orderExecution.tickDataAgg  ; (count;`i)                              ; `Trade   ; 0Nt );
    (`sumVolumeUnderLimitPrice               ;`tickDataAgg  ;`.example.orderExecution.tickDataAgg  ; (sum;`volume)                           ; `Trade   ; 0Nt );
    
    // Examples which use as of joins
    (`myArrivalTradePrice           ;`ajExample   ;`.example.orderExecution.ajExample                ; `price                                ; `Trade     ; 00:00:00 );
    (`myArrivalTradePrice_5         ;`ajExample   ;`.example.orderExecution.ajExample                ; `price                                ; `Trade     ; 00:00:05 );
    (`myArrivalTradePrice_10        ;`ajExample   ;`.example.orderExecution.ajExample                ; `price                                ; `Trade     ; 00:00:10 )

    );

From the above, see various analytics grouped - for the sake of this example - into four analytic types:

1. reversion

   • This is an extension of an existing analytic type that makes use of an existing function .eqea.analytics.reversion.

   • Here, you are extending .eqea.analytics.reversion to get the bid and ask price at orderCompletedTime + 30 seconds.

   • As this is an extension of an existing function, you do not need to add any custom functions for these analytics.

2. simpleCustomAgg

   • This is an example of analytics defined by standard operations on existing columns / analytics in the OrderAnalytics table.

   • You have specified that a custom function .example.orderExecution.simpleCustomAgg is used to calculate this analytic.

   • Define .example.orderExecution.simpleCustomAgg in your example.eqeaCustomAnalytics.q file using the below q code:

     q

     Copy

     // Function to be used to generate strikeToCompletionBidMidPrice & strikeToCompletionAskMidPrice
     .example.orderExecution.simpleCustomAgg:{[OrderAnalyticsRes]
     
         // Get config for this function
         cfg:select from .eqea.config.custom.analytics where analyticType=`simpleCustomAgg;
     
         // Define argument dictionary for .eqea.util.simpleAnalytics
         argDict:(`OrderAnalyticsRes`cfg)!(OrderAnalyticsRes;cfg);
     
         // Calculate analytics using .eqea.util.simpleAnalytics
         OrderAnalyticsRes:.eqea.util.simpleAnalytics[argDict];
     
         // Return results.
         OrderAnalyticsRes      
     
         };

3. tickDataAgg

   • This is an example of using market data in analytic calculations.

   • You have specified that a custom function .example.orderExecution.tickDataAgg should be used to calculate this analytic.

   • Define .example.orderExecution.tickDataAgg in your example.eqeaCustomAnalytics.q file using the below q code:

     q

     Copy

     // Function to be used to calculate countTradesUnderLimitPrice & sumVolumeUnderLimitPrice
     .example.orderExecution.tickDataAgg:{[OrderAnalyticsRes]
             
         // Get config required for this function.
         cfg:select from .eqea.config.custom.analytics where analyticType=`tickDataAgg;
             
         // Our where clause changes depending on whether the order is a buy or a sell.
         // By generating wcList from our input OrderAnalytics table we also ensure the limitPrice value relates to the Order we are analyzing.
         wcList:exec whereClause from select whereClause:(((((`BUY`SELL)!(<=;>=)) orderSideCode),\:(`price)),'limitPrice) from OrderAnalyticsRes;
             
         // Define argument dictionary for .eqea.util.tickData.getDataAndAggFromCfg
         argDict:`OrderAnalyticsRes`cfg`wcList`startTime`endTime`useQueryWindowAdjustmentFactor!(OrderAnalyticsRes;cfg;wcList;`strikeTime;`orderCompletedTime;0b);
         OrderAnalyticsRes:.eqea.util.tickData.getDataAndAggFromCfg[argDict];
             
         // Return results
         OrderAnalyticsRes
         
         };  

4. aj Example

   • This is an example of analytics using the 'as of join' functionality.

   • You have specified that a custom function .example.orderExecution.ajExample should be used to calculate this analytic.

   • Define .example.orderExecution.ajExample in your example.eqeaCustomAnalytics.q file using the below q code:

     q

     Copy

     // Function to be used to calculate myArrivalTradePrice, myArrivalTradePrice_5 & myArrivalTradePrice_10
     .example.orderExecution.ajExample:{[OrderAnalyticsRes]
             
         // Get config required for this function.
         cfg:select from .eqea.config.custom.analytics where analyticType=`ajExample;
           
         // Define argument dictionary for .eqea.util.asof.ajFromCfg
         argDict:(`OrderAnalyticsRes`cfg`joinOrderTimeCol`kCols`wc)!(OrderAnalyticsRes;cfg;`strikeTime;keys OrderAnalyticsRes;());
           
         // Use utility function .eqea.util.asof.ajFromCfg to run analytics that require as-of join functionality.
         OrderAnalyticsRes:.eqea.util.asof.ajFromCfg[argDict];
         OrderAnalyticsRes
     
         };  

Step 3 - Deploy package with custom analytics

When you are ready to test your custom analytics, do the following:

• Add your *eqeaCustomAnalytics.q file to your package.

# Copy our q file containing our analytics to the `src` directory in our Accelerator package
cp example.eqeaCustomAnalytics.q ${PKG_NAME}/src/

• First ensure original package has been torn down.

kxi pm teardown ${PKG_NAME}

• Push and deploy your updated package directory using the kxi CLI.

kxi pm push ./${PKG_NAME} --force --deploy

Troubleshooting

Analytic defined in .eqea.config.custom.analytics but not in the OrderAnalytics table

If you see the following error message:

"Error running .fsi.eqea.generateOrderAnalytics Analytic Defined in .eqea.config.custom.analytics but not in OrderAnalytics table: someAnalytic, someOtherAnalytic "

• This error occurs in the Analytic API if an analytic has been defined in the .eqea.config.custom.analytics but is not present in the OrderAnalytics table.

• The missing analytic(s) are called out by the error message. In this example, they are someAnalytic and someOtherAnalytic.

• Resolve by ensuring all analytics in .eqea.config.custom.analytics are also defined in your patch file, as shown by the file ${PKG_NAME}/patches/CustomAnalytics.yaml in step 1.

• Reapply the patch, repackage, and redeploy when you are confident you have aligned the analytics defined in .eqea.config.custom.analytics and your patch file.

Custom function failing

If you see the following error message:

"Error Encountered Running Order Execution Analytic Function: [ .my.custom.function ] - Error Message: [ rank ] "

• This error occurs when a function encounters an error.

• In this case the function .my.custom.function has encountered a rank error.

• To fix this issue debug your code, repackage, and redeploy using the steps in the above example.

• For more information, refer to debugging custom analytics.

Next steps

Now that your custom analytics have been deployed, test them to ensure they work as intended.

Further reading

• Order Execution Analytics Utility Functions

• generateOrderAnalytics API

• Test Custom Analytics

• Equity Analytics Results Generation