Troubleshooting

This page offers guidance on errors or issues you may encounter when attempting to import data into kdb Insights Enterprise.

Validation failures

Most validation failures are the result of schema mismatches. Read the errors below and fins how to resolve them.

No sym file found at

Make sure you have moved the `sym` file into the PVC when setting up the initial import. Make sure your database satisfies the [prerequisites](#prerequisites) for enumerated columns.

Stray files in DB root: Tables must be listed in schema and symbol file must be called 'sym'

If the listed file names are tables on disk in the database that you want to import, put them in the schema. Otherwise, delete them.

The table has not been configured in the assembly: partitions=

If the table is supposed to be in the schema, add it. Otherwise, remove it from every partition directory.

The table is missing from the database: partitions=

If the table is supposed to be absent from the database being imported, remove it from the schema. Otherwise, ensure that your database has been copied over correctly and that it is valid. A non-corrupt kdb+ database contains data for every partitioned table in every partition directory, even if there are zero rows for that day.

The table has the following extra columns: columns= partitions=

Add the extra columns to the schema.

The table has the following missing columns: columns= partitions=

Remove the missing columns from the schema.

The table has the following missing columns from the .d file: columns= partitions=

Make sure all the files have been copied into the PVC. If this is an issue with the source database, then it is corrupt.

The table has the following incorrect column order: correctOrder= currentIncorrectOrder=

partitions= Reorder the columns in the schema to match that in `currentIncorrectOrder`.

The table has the following incorrect column types: columns= expectedTypes= actualTypes= partitions=

partitions= Update the column types in the schema to match those in `actualTypes`.

Note that these solutions assume the underlying database is not corrupt. Salvaging a corrupt kdb+ database is beyond the scope of this document, although running .Q.chk may resolve some common problems.

Error recovery

Storage Manager has a recovery mechanism: if it is interrupted during a long conversion, on restart it continues where it left off. If an error occurs during conversion, SM rolls back the database to its original state.

Considerations

Encryption

Initial import does not perform encryption or decryption, it solely copies the data into HDB. This has implications when considering the encryption status of the data as well as the target database.

When importing unencrypted data, the database must have encryption disabled before deploying it. Refer to the database settings. If you intend to encrypt the data, you can enable encryption after the initial import has finished.

To import already encrypted data, the encryption key must be loaded into the kdb+ Insights installation. This can be done by creating an encryption secret setting KXI_ENCRYPTION_KEY_FILE and KXI_ENCRYPTION_PASSWORD accordingly.

The encryption key is shared between databases. It is not possible to import data encrypted with a different key if there is already data in the kdb+ Insights system. This restriction can be worked around by decrypting all the databases, replacing the encryption key, then re-encrypting the databases and performing the initial import.

Use the correct configuration

If database encryption is enabled when importing unencrypted data, the import succeeds, but the data is not encrypted in the database until an operation is performed that rewrites the data (e.g. migration between tiers with different compression settings, or disabling and then re-enabling encryption).

If the imported data is encrypted and the database has encryption disabled or has it enabled with the wrong key, the import fails as it's not able to load the data to perform preliminary checks such as checking the data types of columns.