Introduction
Data migration is often one of the most challenging aspects of modern software development. Whether you're moving from one CRM to another, consolidating data sources, or building integrations, the process typically involves writing hundreds of lines of boilerplate code to handle API authentication, pagination, rate limiting, and data transformation.
Enter OpenETL - an open-source SDK that promises to simplify data migration across platforms. In this post, we'll explore how OpenETL can help you extract data from platforms like HubSpot in just a few lines of Python code.
What is OpenETL?
OpenETL is an open-source Python library developed by DataOmni Solutions that abstracts away the complexity of working with various platform APIs. Instead of wrestling with API documentation, authentication flows, and pagination logic, OpenETL provides a unified interface for data extraction and migration.
The library handles:
- Authentication - Support for various auth types (Bearer tokens, OAuth, API keys)
- Pagination - Automatic handling of paginated responses
- Rate Limiting - Built-in rate limit management
- Data Transformation - Easy conversion to pandas DataFrames
- Error Handling - Robust error management and retries
A Real-World Example: Extracting HubSpot Contacts
Let's walk through a practical example of using OpenETL to extract all contacts from HubSpot - a task that would normally require dozens of lines of code.
Step 1: Setup and Authentication
from openetl_csdk.connectors.api import hubspot
from openetl_utils.enums import AuthType
import pandas as pd
# Initialize the HubSpot connector
connector = hubspot.Connector()
# Authenticate with your HubSpot access token
ACCESS_TOKEN = "pat-abc-xyz-123-123-123-123"
auth_credentials = {"token": ACCESS_TOKEN}
api_session = connector.connect_to_api(
auth_type=AuthType.BEARER.value,
**auth_credentials
)
The setup is straightforward. OpenETL uses an enum-based approach for authentication types, making it easy to switch between different auth methods without changing your code structure.
Step 2: Fetch Data with Automatic Pagination
Here's where OpenETL really shines:
# Fetch all contacts - pagination handled automatically!
contacts_data = connector.fetch_data(api_session, "get_all_contacts")
That's it. One line of code. OpenETL automatically:
- Makes multiple API requests to handle pagination
- Respects HubSpot's rate limits
- Retries failed requests
- Aggregates all results
In the screenshot example, you can see OpenETL successfully fetched 1,052 contacts across multiple paginated requests (10 batches of 100 contacts, plus a final batch of 52).
Step 3: Transform to DataFrame
# Combine all paginated results into a single DataFrame
all_contacts_df = pd.DataFrame()
for page_response in contacts_data:
page_df = connector.return_final_df(page_response)
all_contacts_df = pd.concat([all_contacts_df, page_df], ignore_index=True)
print(f"✓ Fetched {len(page_df)} contacts")
OpenETL provides a return_final_df() method that converts API responses directly to pandas DataFrames, making it trivial to work with the data using familiar data science tools.
Step 4: Export Results
# Export to CSV
OUTPUT_FILE = "hubspot_contacts.csv"
all_contacts_df.to_csv(OUTPUT_FILE, index=False)
print(f"\n✓ Successfully exported {len(all_contacts_df)} contacts to {OUTPUT_FILE}")
Why OpenETL Stands Out
1. Minimal Boilerplate
The entire HubSpot contact extraction - including authentication, pagination, data transformation, and export - takes less than 35 lines of code. Compare this to the typical 100+ lines you'd need writing raw API calls.
2. Built for Scale
OpenETL automatically handles pagination and batching, making it suitable for extracting thousands or even millions of records without memory issues.
3. Platform Agnostic
While this example uses HubSpot, OpenETL supports multiple platforms through a unified interface. Switching from HubSpot to Salesforce or another platform requires minimal code changes.
4. Production Ready
With built-in error handling, rate limiting, and retry logic, OpenETL is designed for production use cases, not just quick scripts.
Installation
Getting started with OpenETL is simple:
pip install openetl-sdk
Use Cases
OpenETL is perfect for:
- Data Migration Projects - Moving data between CRMs, marketing platforms, or databases
- Data Warehousing - ETL pipelines for business intelligence
- Backup and Archival - Regular exports of critical business data
- Integration Development - Building custom integrations between platforms
- Data Analysis - Extracting data for analysis in pandas, Jupyter notebooks, or BI tools
Performance Considerations
In our example, OpenETL fetched 1,052 contacts efficiently by:
- Making parallel-safe API calls
- Respecting API rate limits (preventing throttling)
- Using efficient data structures
- Processing results in batches to minimize memory usage
What's Next?
OpenETL is actively developed and open-source. You can:
- Explore the code: github.com/RusabKhan/OpenETL
- Explore the PyPI: pypi.org/project/openetl-sdk
- Learn more: dataomnisolutions.com
- Contribute: The project welcomes community contributions
- Request features: Open issues for platforms or features you'd like to see
Conclusion
OpenETL represents a significant step forward in making data migration accessible to Python developers. By abstracting away API complexity while maintaining flexibility and power, it enables developers to focus on what matters - the data itself, not the plumbing.
Whether you're a data engineer building ETL pipelines, a developer creating integrations, or an analyst needing to extract data for analysis, OpenETL provides a clean, pythonic way to get the job done.
The next time you face a data migration challenge, consider reaching for OpenETL. As our HubSpot example shows, what used to take hours of API documentation reading and debugging can now be accomplished in minutes.
Have you used OpenETL in your projects? What platforms would you like to see supported? Share your thoughts in the comments below!
