Avro (.avro) Integration with Edilitics
Avro is a widely adopted open-source data serialization framework, renowned for its efficient binary storage format and compatibility across platforms. Its JSON-based schema simplifies data exchange and ensures adaptability to evolving data structures. This makes Avro an essential tool in big data ecosystems, enabling seamless integration with diverse workflows.
This step-by-step guide outlines how to integrate Avro files into Edilitics, manage workflows efficiently, and adhere to platform constraints. By following this guide, you can securely utilize Avro files as data sources for analytical operations.
Before You Begin
Ensure the following prerequisites are met before starting:
-
File Size Limit: Avro files must be 100 MB or less.
-
Column Headers: Include well-defined column headers for accurate processing.
-
Usage Constraints:
-
Avro files can only function as data sources, not destinations.
-
Workflows using Avro files:
-
Support full loads with "Schedule as Once" in Replicate.
-
Allow "Schedule as Once" in Transform.
-
Do not support auto updates or data refreshes in Visualize.
-
-
-
AI Column Insights: Avro files are not eligible for AI Column Insights.
File Security and Management
Edilitics implements robust security and management protocols for Avro files:
-
Security Scans: All uploaded files undergo rigorous validation for risks and data accuracy.
-
Encryption: Files are encrypted during storage and decrypted only when accessed or used in workflows (Replicate, Transform, Visualize).
-
Permanent Deletion: Deleting an integration removes the file from Edilitics systems immediately, ensuring compliance with data governance policies.
Supported Data Structures
Edilitics supports various Avro data types, enabling flexible and scalable workflows:
| Avro Type | Description | Example |
|---|---|---|
| Records | Structured collections of named fields, akin to table rows. | {"name": "John", "age": 30, "email": "john@example.com"} |
| Enums | Enumerations representing fixed sets of values. | "day": {"type": "enum", "symbols": ["Monday", "Tuesday", "Wednesday"]} |
| Arrays | Collections of homogeneous items. | [1, 2, 3, 4] |
| Maps | Key-value pairs with string keys and dynamic values. | {"height": 175, "weight": 70} |
| Unions | Fields capable of holding multiple data types. | "favorite": ["string", "null"] |
| Fixed | Binary data with a specified size. | A 16-byte hash: {"type": "fixed", "size": 16, "name": "md5"} |
These structures provide flexibility for diverse use cases, from transactional data to hierarchical data models.
Steps to Integrate Avro Files
Step 1: Add the Avro Connector
-
Navigate to the Integrations module in Edilitics.
-
Click on New Integration.
- Choose the Avro connector from the list or use the search bar to locate it.
Step 2: Configure the Integration
Enter the following details on the setup screen:
| Field Name | Details |
|---|---|
| Integration Title | A unique identifier for your integration. |
| Integration Description | A concise description of the Avro data being integrated. |
| File Upload | Upload the Avro file directly from your local storage (must be ≤ 100 MB). |
Step 3: Validate and Save
-
Click Test & Save Connection to validate the uploaded file.
-
Edilitics scans the file for format and schema compliance.
-
Upon successful validation, the file is encrypted and securely stored.
Need Assistance? Edilitics Support is Here for You!