Work with Parquet files in data feeds
Customer Journey Analytics data feeds deliver files in Parquet format. Most modern data warehouses can read Parquet files directly, so you usually don’t need to convert the files before you use them.
Use the following information to understand why data feeds use Parquet, how Parquet compares to other file formats, and how to query or load Parquet files in your data warehouse.
Advantages of using Parquet for data feeds
Parquet is better suited to data feed output than CSV, TSV, or JSON for the following reasons:
-
Wide feeds: Data feeds can have hundreds of columns. Parquet lets a query read only the columns that it needs.
-
Nested fields: Parquet represents nested fields, such as a product list, natively. CSV and TSV can represent them only by encoding them into strings. For more information, see Sub-event components in data feeds.
-
Built-in schema: Each Parquet file stores its own schema, including column names, data types, and nested structures. Because the schema is in the file, your data warehouse can create tables directly from the files. You don’t need to provide a separate header file or schema definition, and the data warehouse doesn’t need to infer data types.
-
Large exports: Parquet usually reduces storage and transfer costs. It also avoids text parsing when you load the data.
How Parquet compares to CSV, TSV, and JSON
CSV, TSV, and JSON are common text-based file formats:
-
CSV and TSV: Each row of data is a line of plain text, with values separated by commas (CSV) or tabs (TSV). Adobe Analytics data feeds deliver TSV files. For other differences between data feeds in Customer Journey Analytics and Adobe Analytics, see Compare data feeds in Customer Journey Analytics and Adobe Analytics.
-
JSON: Each record is stored as text with named fields, and can include nested arrays and objects.
The following table details the differences between Parquet, CSV, TSV, and JSON:
Query or load Parquet files
Data feed files are delivered in Parquet format to the cloud storage location that you selected when you created the data feed.
Rather than download the files, you typically connect your data warehouse to the storage location. You can then use the Parquet files in either of the following ways:
Query the files in place
Define an external table that points to the files in your cloud storage. The data warehouse reads the files when you run a query.
- Snowflake: External tables, or queries against an external stage
- Databricks: External tables over Amazon S3, Azure Data Lake Storage (ADLS), or Google Cloud Storage
- BigQuery: External tables over Google Cloud Storage
- Amazon Redshift: Redshift Spectrum over Amazon S3
Load the files into managed tables
Loading files into managed tables is a common choice for recurring workloads. For example:
- Snowflake:
COPY INTO - BigQuery: Load jobs
- Databricks:
COPY INTOor Auto Loader
Convert Parquet data to CSV or JSON
If your downstream processes require CSV or JSON files, most data warehouses and tools can export Parquet data to either format after you load it. For example, in Snowflake, you can unload query results as CSV or JSON with COPY INTO.