Data Import Troubleshooting
This page offers guidance on errors or issues you may encounter when attempting to import data into kdb Insights Enterprise.
Validation failures
Most validation failures are the result of schema mismatches. Read the errors below and fins how to resolve them.
No sym file found at
Make sure you have moved the `sym` file into the PVC when setting up the initial import. Make sure your database satisfies the prerequisites for enumerated columns.
Stray files in DB root
If the listed file names are tables on disk in the database that you want to import, put them in the schema. Otherwise, delete them.
The table has not been configured in the assembly: partitions=
If the table is supposed to be in the schema, add it. Otherwise, remove it from every partition directory.
The table is missing from the database: partitions=
If the table is supposed to be absent from the database being imported, remove it from the schema. Otherwise, ensure that your database has been copied over correctly and that it is valid. A non-corrupt kdb+ database contains data for every partitioned table in every partition directory, even if there are zero rows for that day.
The table has the following extra columns: columns= partitions=
Add the extra columns to the schema.
The table has the following missing columns: columns= partitions=
Remove the missing columns from the schema.
The table has the following missing columns from the .d file: columns= partitions=
Make sure all the files have been copied into the PVC. If this is an issue with the source database, then it is corrupt.
The table has the following incorrect column order: correctOrder= currentIncorrectOrder=
partitions= Reorder the columns in the schema to match that in `currentIncorrectOrder`.
The table has the following incorrect column types: columns= expectedTypes= actualTypes= partitions=
partitions= Update the column types in the schema to match those in `actualTypes`.
Note
These solutions assume the underlying database is not corrupt. Salvaging a corrupt kdb+ database is beyond the scope of this document, although running .Q.chk may resolve some common problems.
Error recovery
Storage Manager has a recovery mechanism: if it is interrupted during a long conversion, on restart it continues where it left off. If an error occurs during conversion, SM rolls back the database to its original state.
Considerations
Encryption
Initial import does not perform encryption or decryption, it solely copies the data into HDB. This has implications when considering the encryption status of the data as well as the target database.
When importing unencrypted data, the database must have encryption disabled before deploying it. See decrypt and encrypted database. If the intention is to have the data encrypted, you can enable encryption after the initial import has finished to encrypt the data.
To import already encrypted data, the encryption key must be loaded into the kdb+ Insights installation. This can be done by creating an encryption secret. Note that an encryption secret is created by default on installing kdb+ Insights, so in order to install a different key, the default one must be deleted. See changing the encryption key for details.
The encryption key is shared between databases. It is not possible to import data encrypted with a different key if there is already data in the kdb+ Insights system. This restriction can be worked around by decrypting all the databases, replacing the encryption key, then re-encrypting the databases and performing the initial import.
Warning
Use the correct configuration
If database encryption is enabled when importing unencrypted data, the import succeeds, but the data is not encrypted in the database until an operation is performed that rewrites the data. For example, migration between tiers with different compression settings, or disabling and then re-enabling encryption).
If the imported data is encrypted and the database has encryption disabled or has it enabled with the wrong key, the import fails as it's not able to load the data to perform preliminary checks such as checking the data types of columns.