Equity Analytics Framework - Adding Custom Analytics

The Equity Analytics Framework calculates a wide number of Order Analytics. This framework is built with flexibility in mind and can easily be extended with custom analytics.

This page provides an example outlining how to add custom analytics with varying levels of complexity.

Step 1 - Add a patch to the fsi-app-eqea Accelerator package

Patches can be used to customize your accelerator package. Refer to the documentation on customizing the FSI Accelerator for more information.

The Equity Analytics Framework generates order analytics using data from the Order, Trade and Quote tables. The result is then upserted to the OrderAnalytics table.

If you wish to add custom analytics, you must first ensure that these analytics exist as columns in the OrderAnalytics table.

You must use the kxi CLI to add a patch to your package.

The following example adds a patch called CustomAnalytics.

# Unpack the accelerator package
kxi package unpack ${PKG_NAME}-${VERSION}.kxi

# Create patch to add columns to the OrderAnalytics schema
kxi package add --to ${PKG_NAME} patch --name CustomAnalytics

This generates a YAML patch file in the path ${PKG_NAME}/patches/CustomAnalytics.yaml. We can use this patch file to add custom analytic columns to the OrderAnalytics table.

Update the file ${PKG_NAME}/patches/CustomAnalytics.yaml using your editor of choice.

The yaml snippet below contains all the details you need for this example:

kind: Package
apiVersion: pakx/v1
metadata:
  name: target
spec:
  manifest: {}
  tables:
    schemas:
    - name: OrderAnalytics
      columns:   
      - name: reversionAskPrice_30
        type: float
      - name: reversionBidPrice_30
        type: float        
      - name: strikeToCompletionBidMidPrice
        type: float
      - name: strikeToCompletionAskMidPrice
        type: float
      - name: countPriceUnderLimitPrice
        type: int
      - name: sumVolumeUnderLimitPrice
        type: long                   
      - name: myArrivalTradePrice
        type: float        
      - name: myArrivalTradePrice_5
        type: float        
      - name: myArrivalTradePrice_10
        type: float

Here, we are adding several columns to the OrderAnalytics schema, defining the name and type of each column added.

After adding the required columns to your patch, overlay the patch on your accelerator package using the kxi CLI, as shown below.

# Apply the overlay
kxi package overlay ${PKG_NAME} ${PKG_NAME}/patches/CustomAnalytics.yaml

Step 2 - Define custom analytics

Custom Analytics can be added to the Equity Analytics Framework using a q file that matches the naming convention *eqeaCustomAnalytics*.q.

This example creates a *eqeaCustomAnalytics*.q file called example.eqeaCustomAnalytics.q.

The *eqeaCustomAnalytics*.q file should contain:

• A configuration table named .eqea.config.custom.analytics which defines all custom analytics.

• Definitions of any custom functions which are used to calculate the custom analytics.

The .eqea.config.custom.analytics table must have the following structure:

In this example, we are defining several custom analytics of varying complexity.

Add the snippet below to your example.eqeaCustomAnalytics.q file:

// Example Custom Analytics
.eqea.config.custom.analytics:flip `analytic`analyticType`funcName`aggClause`marketDataTabName`joinTimeOffset! flip (

    // Examples of adding analytics to the pre-packaged analytic types.
    (`reversionAskPrice_30          ; `reversion        ; `.eqea.analytics.reversion                ; `askPrice                               ; `Quote   ; 00:00:30);    
    (`reversionBidPrice_30          ; `reversion        ; `.eqea.analytics.reversion                ; `bidPrice                               ; `Quote   ; 00:00:30);    

    // Examples of adding simple custom analytic types (simple aggregation on already calculated values in our OrderAnalytics table)
    (`strikeToCompletionBidMidPrice ;`simpleCustomAgg   ;`.example.orderExecution.simpleCustomAgg   ; (%;(+;`arrivalAskPrice;`endAskPrice);2) ; `       ; 0Nt );
    (`strikeToCompletionAskMidPrice ;`simpleCustomAgg   ;`.example.orderExecution.simpleCustomAgg   ; (%;(+;`arrivalAskPrice;`endAskPrice);2) ; `       ; 0Nt );

    // Examples of adding more complex custom analytics (aggregations on tick data with multiple where clauses)
    (`countPriceUnderLimitPrice              ;`tickDataAgg  ;`.example.orderExecution.tickDataAgg  ; (count;`i)                              ; `Trade   ; 0Nt );
    (`sumVolumeUnderLimitPrice               ;`tickDataAgg  ;`.example.orderExecution.tickDataAgg  ; (sum;`volume)                           ; `Trade   ; 0Nt );

    // Examples which use as of joins
    (`myArrivalTradePrice           ;`ajExample   ;`.example.orderExecution.ajExample                ; `price                                ; `Trade     ; 00:00:00 );
    (`myArrivalTradePrice_5         ;`ajExample   ;`.example.orderExecution.ajExample                ; `price                                ; `Trade     ; 00:00:05 );
    (`myArrivalTradePrice_10        ;`ajExample   ;`.example.orderExecution.ajExample                ; `price                                ; `Trade     ; 00:00:10 )

    );

From the above, see various analytics grouped - for the sake of this example - into four analytic types:

1. reversion

   • This is an extension of an existing analytic type.

   • Here, you are adding an extra interval for our reversion analytics.

   • You do not need to add any custom functions to generate this analytic.

2. simpleCustomAgg

   • This is an example of a simple calculations that generate an analytics based on to columns that already exist in the OrderAnalytics table.

   • You have specified that a custom function .example.orderExecution.simpleCustomAgg is used to calculate this analytic.

   • For this to work, you must define .example.orderExecution.simpleCustomAgg in our example.eqeaCustomAnalytics.q file using the below q code:

     q

     Copy

     // Function to be used to generate strikeToCompletionBidMidPrice & strikeToCompletionAskMidPrice
     .example.orderExecution.simpleCustomAgg:{[OrderAnalyticsRes]
     // We can use a util function .eqea.util.runSimpleAnalytic to run analytics that are dependent on columns that already exist in the OrderAnalytics table
     cfg:select from .eqea.analytics.cfg where analyticType=`simpleCustomAgg;
     .eqea.util.simpleAnalytics[OrderAnalyticsRes;cfg]
     };
                         

3. tickDataAgg

   • This is an example of using market data in analytic calculations.

   • You have specified that a custom function .example.orderExecution.tickDataAgg should be used to calculate this analytic.

   • To be successful, you must define .example.orderExecution.tickDataAgg in our example.eqeaCustomAnalytics.q file using the below q code:

     q

     Copy

     .example.orderExecution.tickDataAgg:{[OrderAnalyticsRes]
     
      // We can use `.eqea.util.runTickDataAnalytics` to run analytics on tick data tables
     cfg:select from .eqea.analytics.cfg where analyticType=`tickDataAgg;
     // Our where clause changes depending on whether the order is a buy or a sell.
     // By generating wcList from our input OrderAnalytics table we also ensure the limitPrice value relates to the Order we are analyzing.
     wcList:exec whereClause from select whereClause:(((((`BUY`SELL)!(<=;>=)) orderSideCode),\:(`price)),'limitPrice) from OrderAnalyticsRes;
     OrderAnalyticsRes:.eqea.util.tickData.getDataAndAggFromCfg[OrderAnalyticsRes;cfg;wcList;`strikeTime;`orderCompletedTime;0b];
     OrderAnalyticsRes
     
     };
     
                         

4. aj Example

   • This is an example of analytics using the 'as of join' functionality.

   • You have specified that a custom function .example.orderExecution.ajExample should be used to calculate this analytic.

   • To be successful, you must define .example.orderExecution.ajExample in our example.eqeaCustomAnalytics.q file using the below q code:

     q

     Copy

     .example.orderExecution.ajExample:{[OrderAnalyticsRes]
     
         // We can use `.eqea.util.runTickDataAnalytics` to run analytics which retrieve values from tick data using an as of joins
     
         cfg:select from .eqea.analytics.cfg where analyticType=`ajExample;
     
         OrderAnalyticsRes:.eqea.util.asof.ajFromCfg[OrderAnalyticsRes;cfg;`strikeTime;()];
     
         OrderAnalyticsRes
     
         }

Step 3 - Deploy package with custom analytics

When you are ready to test your custom analytics, do the following:

• Add your *eqeaCustomAnalytics*.q file to your package.

# Copy our q file containing our analytics to the `src` directory in our Accelerator package
cp example.eqeaCustomAnalytics.q ${PKG_NAME}/src/

• First ensure original package has been torn down.

kxi pm teardown ${PKG_NAME}

• Push and deploy your updated package directory using the kxi CLI.

kxi pm push ./${PKG_NAME} --force --deploy

Troubleshooting

Analytic defined in .eqea.analytics.cfg but not in the OrderAnalytics table

If you see the following error message:

"Error running .fsi.eqea.generateOrderAnalytics Analytic Defined in .eqea.analytics.cfg but not in OrderAnalytics table: someAnalytic, someOtherAnalytic "

• This error occurs in the Analytic API if an analytic has been defined in the .eqea.analytics.cfg but is not present in the OrderAnalytics table.

• The missing analytic(s) are called out by the error message. In this example, they are someAnalytic and someOtherAnalytic.

• Resolve by ensuring all analytics in .eqea.config.custom.analytics are also defined as columns in the OrderAnalytics in your patch file, as outlined in step 1.

• Reapply the patch, re-push and redeploy when you are confident you have aligned the analytics defined in .eqea.config.custom.analytics and your patch file.

Custom function failing

If you see the following error message:

"Error Encountered Running Order Execution Analytic Function: [ .my.custom.function ] - Error Message: [ rank ] "

• This error occurs when a function encounters an error.

• In this case the function .my.custom.function has encountered a rank error.

• To fix this issue debug your code, repackage and redeploy using the steps in the above example.

• For more information, refer to debugging custom analytics.

Next steps

Now that your custom analytics have been deployed, test them to ensure they work as intended.

Further reading

• Equity Analytics Framework - Util Functions

• generateOrderAnalytics API

• Test Custom Analytics

• Equity Analytics Results Generation