Offline Imports
When it is not possible to establish a real-time data stream from a source, such as an offline system, historic orders or certain CRM providers, Velocidi supports data onboarding through secure file uploads or specific third-party API integrations.
#
Quick-Start#
Upload FilesYou need to make your files available to the CDP prior to importing them. We recommend using the S3 bucket dedicated to your CDP. You can learn more about this process on the Upload Files documentation page.
#
How to ImportOffline imports can be configured using the Dashboard under Collect > Data Imports, by clicking the New Import button.
From here, you can configure these types of imports:
- Order imports - Import order data into the CDP.
- Customer imports - Import CRM data into the CDP.
- Product imports - Import production information into the CDP.
caution
Before starting an import, make sure that:
- Your files are properly encoded as UTF-8 or that the appropriate encoding is selected when importing;
- The IDs (user IDs, order IDs, product IDs...) in the imported files match the ones sent by events;
- Required fields are present in all lines;
- The fields have the correct keys (case sensitive).
#
OrdersExport your CRM orders and import them into the CDP.
#
Use CasesImporting orders into the CDP has 3 main use cases, as described below.
#
Bootstrap Your System with Historic Order DataImporting orders plays an important role when bootstrapping a system. While the Velocidi CDP builds user profiles with attributes produced from the users' interaction with the website, there's usually a large amount of information businesses have on orders that were made before events started being collected. These orders can add valuable information to produce user attributes. The attributes that benefit the most from a bootstrap from past orders are the ones related to next purchase predictions, RFM and lifetime value. The attributes related to the likelihood to buy are produced using different types of events from the users' interaction with the website (besides the actual purchases), so they don't benefit much from an import of orders previously unseen in the system.
#
Ensure the Orders Used by the CDP Are in Sync with Your CRM OrdersOrders can change their state (for example, becoming canceled or refunded) without an explicit event from the user on the website. In those scenarios, having a periodic import of orders from your CRM backend can help the Velocidi CDP have a more accurate view of existing orders.
#
Complement Order Information Collected by Tracking Events with Details Private to the CRM BackendSometimes, the data that is available when producing events to be collected by the Velocidi CDP lacks some details that are private to the CRM backend, and are therefore not directly accessible by the website. Typical scenarios are information missing from specific line items (like product-related data) or incomplete payment information due to actions that happen after the event is triggered. To make sure that the Velocidi CDP has access to relevant order details, even those that are private to the CRM backend, it's helpful to setup a periodic import of orders.
#
Order Events from ImportsOrder imports generate order events. If the order is unknown, an orderPlace
is generated. Otherwise, if the import provides new information, for example, the status of a line item is updated, an orderUpdate
event is generated.
Events generated from order imports are enriched if a product feed is available, considering the provided product ID. Only omitted attributes are added from the product feed. Attributes available on the import are kept on the event and are not replaced by data from the product feed.
#
FormatsWe currently have support for the following 1st and 3rd party formats:
Supported schemas
#
Velocidi CSVThis format is intended for CSV imports where each line is a Velocidi's Order
/LineItem
objects serialized to CSV (Example file).
Relevant notes:
- The export file should be encoded using UTF-8.
- All line items for the same order must be listed sequentially. Otherwise, only the last sequential line items of the order will be persisted;
- Given an order with an arbitrary number of rows, only the first row is used to extract Order details and all rows are used to extract Line item details. This means that Orders with more than 1 Line item do not need to populate the order columns for all columns and can be left empty.
- If the provided
userId
is not hashed (E.g.idType=email
), you can configure the import to hash it (to aidType=email_sha256
) beforehand. - The userId field is validated accordingly to the type specified in userIdType field. You can find more information about user ID types in the User Identifiers documentation.
Column | Required | Type | Description | Details | Example |
---|---|---|---|---|---|
orderId | Required | String | Order identifier | Must be unique by Order | or123 |
userIdType | Required | String | The type of user identifier (e.g. "email") | ||
userId | Required | String | The user identifier | john@email.com | |
currency | Required | String | The currency used in monetary values | Check the supported currencies | EUR |
total | Required | Double | Total value of the order with all discounts already considered, including subtotal, tax and shipping | total = subtotal + tax + shipping | 102.5 |
subtotal | Required | Double | The value of the order, when tax and shipping are excluded | subtotal = total - tax - shipping | 85 |
tax | Required | Double | The total applied tax value for the order | 11.5 | |
shipping | Required | Double | The total shipping costs for the order | 6 | |
discount | Required | Double | The total discounted value for the order | 5 | |
refund | Required | Double | The total refunded value for the order | 0 | |
paymentMethod | String | The payment method | Visa | ||
shippingMethod | String | The shipping method | UPS | ||
shippingCountry | String | The destination country | France | ||
source | String | Source from where the order was placed. Examples are web , phone call , POS . | |||
promotions | List of Strings | Vouchers applied in the purchase. | Strings separated by ; | WINTERSALE | |
createdAt | Required | DateTime | When the order was created | Date format: ISO-8601 | 2020-09-16T10:38:06+0100 |
lastUpdated | DateTime | When the order was last updated | Date format: ISO-8601 | 2020-09-16T20:13:31+0100 | |
lineItem_id | String | A unique identifier for the line item within the order | li125zc | ||
lineItem_productId | Required | String | Product identifier | p125zc | |
lineItem_itemGroupId | String | Identifier for a group of products that come in different versions (variants) | p125 | ||
lineItem_name | String | Product name | My Product | ||
lineItem_sku | Required | String | Product SKU | p125zc-5 | |
lineItem_brand | String | Product brand | Nike | ||
lineItem_category | String | Product category | Men > Shirts > Short Sleeve Shirts | ||
lineItem_categoryList | List of Strings | A hierarchically sorted list of the sub-categories that this product belongs to. The first item on this list should be the most general sub-category, while the last should be the most specific. Any item on this list should correspond to a sub-category of the item immediately preceding it. | Strings separated by ; | Men;Shirts;Short Sleeve Shirts | |
lineItem_currency | Required | String | The currency used in monetary values | Check the supported currencies | EUR |
lineItem_total | Required | Double | Total value of the line item with all discounts already considered, including subtotal, tax and shipping | total = subtotal + tax + shipping | 102.5 |
lineItem_subtotal | Required | Double | The value of the line item, when tax and shipping are excluded | subtotal = total - tax - shipping | 85 |
lineItem_tax | Required | Double | The total applied tax value for the line item | 11.5 | |
lineItem_shipping | Required | Double | The total shipping costs for the line item | 6 | |
lineItem_discount | Required | Double | The total discounted value for the line item | 5 | |
lineItem_refund | Required | Double | The total refunded value for the line item | 0 | |
lineItem_quantity | Required | Int | Number of purchased items | 1 | |
lineItem_promotions | List of Strings | Vouchers applied in the purchase or displayed together with the product | Strings separated by ; | WINTERSALE | |
lineItem_adult | Boolean | Whether the product includes sexually suggestive content. | Must be true or false . Defaults to false | false | |
lineItem_subscriptionDuration | Long | If the purchase relates to a subscription, captures the subscription duration, in milliseconds. | 15552000000 | ||
lineItem_size | String | Product size | S | ||
lineItem_sizeSystem | String | Country of the size system used by your product. | EU | ||
lineItem_status | Required | String | Line item status. placed the order has been placed but the item is yet to be paidpaid the item has been paid but not yet deliveredfulfilled the item has been paid and deliveredrefunded the item has been refundedcancelled the order has been modified to exclude this line item | Supported status: placed , paid , fulfilled , refunded , cancelled | fulfilled |
There's an example file available, showing how each field can be used.
#
Shopify CSVVelocidi supports CSV order exports from Shopify. The documentation for generating a Shopify export can be found here. Please ensure you select Export orders
to export all order information, instead of Export transaction histories
, which only exports the transactions.
Relevant notes:
- Shopify uses the
Name
column to distinguish between different orders inside the CSV file, so it must be unique per Order; - All Line items for the same Order must be listed sequentially;
- Given an order with an arbitrary number of rows, only the first row is used to extract Order details and all rows are used to extract Line item details. This means that Orders with more than 1 Line item do not need to populate the order columns for all columns and can be left empty.
- If the provided
userId
is not hashed (e.g.idType=email
), you can configure the import to hash it (to aidType=email_sha256
) beforehand. - The email field is validated as an email.
- It's important that the extracted order identifier from the shopify import matches the
orderId
provided via tracking events. When importing, you'll be able to select which field to use as the order identifier between theId
andName
columns.
Velocidi will consider the following attributes while importing a Shopify file:
Column | Required | Type | Details |
---|---|---|---|
Name | Required | String | Must be unique by Order and can be the same as Id (Can be used as an order identifier) |
Required | String | ||
Id | Required | String | (Can be used as an order identifier) |
Currency | Required | String | Check the supported currencies |
Total | Required | Double | |
Subtotal | Required | Double | |
Taxes | Required | Double | |
Payment Method | String | ||
Financial Status | Required | String | Supported status: authorized , paid , partially_paid , pending , refunded , partially_refunded , voided |
Refunded Amount | Required | Double | |
Discount Amount | Required | Double | |
Discount Code | String | ||
Shipping Method | String | ||
Shipping Country | String | ||
Shipping | Required | Double | |
Source | String | ||
Created at | Required | DateTime | Date format: yyyy-MM-dd HH:mm:ss Z |
Paid at | DateTime | Date format: yyyy-MM-dd HH:mm:ss Z | |
Fulfilled at | DateTime | Date format: yyyy-MM-dd HH:mm:ss Z | |
Lineitem quantity | Required | Int | |
Lineitem name | Required | String | |
Lineitem price | Required | Double | |
Lineitem discount | Required | Double | |
Lineitem fulfillment status | Required | String | Supported status: pending , partial , restocked , fulfilled |
Lineitem sku | Required | String | |
Vendor | String |
#
CustomersExport your CRM data and import them into the CDP.
Use cases
- Import CRM data to use in activations;
- Complement the users' profile with their CRM data.
Info
We automatically lowercase the following User IDs from the default ID Types list:
Additionally, if a transformation is required for any ID type before being imported (for example hashing plaintext emails), we also lowercase and trim the original ID before transforming it.
We support importing CRM data from different Sources.
#
Files SourcesThe following formats are supported:
Supported schemas
The import will import every field as a custom User Attribute of the User Profile. Some notes:
- The file must have at least one field which identifies the user (See User Identifiers). You can select which field contains the User ID as well as the associated User ID type.
- By default, the attribute merge strategy is merge, which means the imported attributes will be merged with the existing attributes so that non-imported attributes will remain unaffected in the User Profile.
- If importing a CSV file, all attributes will be imported as text. This restricts the ability to segment the
user using numeric rules (e.g.
numberOfOrders > 20
won't work). As a workaround, you can apply castTo transformations to known numeric fields. If importing a JSON-lines file, the attributes will use the original json types.
#
E-goi Contacts SourceThis is a specific third-party integration which loads customer data directly from an E-goi Contact List and imports it as attributes and user ids. If the contact information contains an email, it will be transformed into sha256 representation and merged into any existing profiles. The E-goi contact id is also imported and allows you to make use of the E-goi Destination which will make use of that ID to attach tags to activated users.
To configure this import you should first select E-goi Contacts from the Source Type dropdown and then provide an E-goi API key with permissions to List All Contacts and Get My Account Info, as well as the List ID from which you want to download and import contact data. If you want to import only a subset of Contacts from that List that belong to a given Segment, you may fill the Segment Id input. Otherwise, you may leave that field empty.
An API key can be created in you Integrations dashboard by choosing to Create a New Key and making sure the necessary permissions are selected:
The set of required permissions are the following:
- Get all contacts
- Get all contacts by segment
- Get my account info
The list id can be taken from your E-goi dashboard and should be the numeric ID before its title:
The segment id can be taken from you List view on the E-goi dashboard, and should correspond to the numeric value that prefixes the segment name:
Notice that the customer import from this type of Source does not allow you to specify any transformations.
#
E-goi Attribute MappingThis source imports a subset of the available E-goi Contact fields, and maps them either to CDP user IDs or to user
attributes. User attributes are mapped from E-goi original fields using the egoi.account<ACCOUNT-ID>.list<LIST-ID>.<E-GOI-ATTRIBUTE>
format. <ACCOUNT-ID>
is the E-goi account ID to which the provided API key refers to and is automatically determined,
so you don't have to provide it explicitly. <LIST-ID>
corresponds to the List ID provided in the form and
<E-GOI-ATTRIBUTE>
corresponds to the original Contact field this source will import into the CDP.
For example, importing the status
field from an E-goi Contact might result in
egoi.account123.list321.status
, if the API key used belongs to the E-goi account 123
and the user submitted 321
as
the List to import.
The following describes the fields that are consumed from the E-goi Contacts and how they are mapped into User Attributes:
E-goi Contact Field | CDP User ID or Attribute | Description |
---|---|---|
contact_id | User ID: Internal E-goi User ID Type | The contact_id field is mapped into an internal user ID type whose main purpose is to be used by the E-goi Destination |
email | User ID: Email SHA256 | The E-goi plain-text email field, if available, hashed and mapped to a SHA256 User ID |
status | Attribute: egoi.account<ACCOUNT-ID>.list<LIST-ID>.status | The Contact status |
consent | Attribute: egoi.account<ACCOUNT-ID>.list<LIST-ID>.consent | The Contact consent |
email_status | Attribute: egoi.account<ACCOUNT-ID>.list<LIST-ID>.emailStatus | The Contact email status, if an email is available |
cellphone_status | Attribute: egoi.account<ACCOUNT-ID>.list<LIST-ID>.cellphoneStatus | The Contact cellphone status, if a cellphone is available |
phone_status | Attribute: egoi.account<ACCOUNT-ID>.list<LIST-ID>.phoneStatus | The Contact phone status, if a phone is available |
Please refer to E-goi Contact API documentation for more information about the existing Contact fields and their possible values.
#
ProductsExport your products' information and import them into the CDP.
Use cases
- Complement tracking events.
We have support for importing product information in the following formats:
Supported schemas
Importing product data allows for product-related tracking events to be complemented with the products' information even if the original event did not include them.
For more information, see the Product Feed documentation.
#
Advanced Configurations#
File SourcesWe have support for importing files with the following protocols: http
, https
, s3
and sftp
.

info
For convenience, each Velocidi CDP system has a pre-created AWS S3 bucket that the client can use to upload files, to later be imported, securely. The credentials for these bucket will be supplied to the client during onboarding.
caution
In order to use a s3
bucket other than then provided one, or an authenticated sftp
you should contact
Velocidi support.
#
File EncodingsYou can specify the file encoding of the files to be imported. Most files should use, the very common, "UTF-8" file encoding. However, when exporting files from certain CRM platforms or third-party providers, the files can be encoded with another format which results in garbled text or even an error when importing into the CDP. To address this, file encoding auto-detection can be selected when importing data into the CDP. This process relies on heuristics and is not 100% accurate for all file encodings. If you know the encoding for your file, or auto-detection is failing, it is also possible to explicitly specify the file's encoding.

#
Import SchedulesYou can configure an offline import to execute and repeat on a given schedule.
The schedule can be specified according to years, months, days and hours. You can build the schedule by choosing appropriate values on the dropdown boxes and see the resulting description below. All dates and times are in UTC timezone.


As seen in the examples above, the component lets you choose different repetition periods and shows the textual descriptions of the chosen schedule.
Please note that, if an import gets triggered while another instance pertaining the same schedule is still running, the new one will be skipped.
You can also verify the status of your periodically configured imports.

#
Data TransformationData transformation allows you to rename fields and even map values without having to edit the original file.
The most basic transformations are:
- Rename field (e.g. renaming the field "number_of_orders" to "completedOrders". The "number_of_orders" field will disappear).
- Copy field (e.g. copying the field "number_of_orders" to "completedOrders". The "number_of_orders" field will be unaffected).
- New field (e.g. adding the field "completedOrders" with the value 0 set for all lines)
- Delete field (e.g. Deletes the field "completedOrders")

For the previous (Rename/Copy) transformations, you can also specify some value transformations to apply (i.e. transformations that will affect the values of the fields):
#
Mapping Field ValuesAccepts an exhaustive list of from -> to values to transform. This transformation will iterate over the provided mappings and if the field's values matches the from value, it will be replaced with the to value. If no value matches, the field value will remain unaltered.
Example:
Given the mappings "1" -> "one"
and "2" -> "two"
:
Original value | Resulting value | Description |
---|---|---|
"1" | "one" | Value is transformed |
"2" | "dois" | Value is transformed |
"other" | "other" | Value is unaltered because it doesn't match any of the from values |
1 | 1 | Value 1 is different from value "1" (number instead text) so it doesn't match the "1" -> "one" mapping |
note
When dealing with CSV files, when this transformation takes place, all field values are interpreted as text.
#
Extracting a Custom TimestampAllows parsing a non-standard DateTime format into a ISO 8601 format. Requires the user to specify the custom DateTime pattern which the file to import is using.
Example:
Given the pattern yyyy/MM/dd hh:mm
, the value 2021/01/21 14:25
will be converted to the
ISO 8601 format 2021-01-21T14:25:00Z
.
#
Casting Field Value TypesAllows casting values to a specific type. This is useful mostly for Customer Imports where the values are directly set as user attributes and the type (most commonly, whether we're dealing with text, or a number) is relevant (e.g. to create segments with numeric rules with that attribute). When dealing with Order Imports or Product Imports, casting the fields to the correct types isn't as import since there imports have well-defined schemas. These means that the schemas will perform the transformation implicitly if needed.
The supported cast types are: string
, integer
, float
and bool
.
Example:
Original value | Cast to String | Cast to Integer | Cast to Float | Cast to bool |
---|---|---|---|---|
"2.4" | "2.4" | 2 | 2.4 | true |
"0" | "0" | 0 | 0.0 | false |
"car" | "car" | Error | Error | Error |
"false" | "false" | Error | Error | true |
3.8 | "3.8" | 3 | 3.8 | true |
#
API ReferenceFor advanced, programmatic control, see the API reference for details.