Data integration in Fivetran has three key facets: prebuilt, fully managed connectors; normalisation of the data you are moving; and provision of analysis-ready schemas for target connectors. Between them this means the process of data integration (and data movement) is almost completely automatic. Moreover, Fivetran operations are idempotent, meaning that however many times you process a particular workflow you will always get the same result. In other words, once a process flow is correct it will always be correct, and data integrity will be maintained. If it fails, automatic recovery is available. Idempotency is achieved, in part, because Fivetran will automatically add or remove columns whenever there is a schema change.
Figure 2 – A range of the connectors available in Fivetran
Fivetran currently offers more than 300 connectors to SaaS applications, on-premises and cloud-hosted databases, file systems, data lakes, cloud data warehouses, and event services (see Figure 2), that have purpose-built to support relevant use cases. In particular, various cloud platforms, including the ‘big 3’ of AWS, Azure, and Google Cloud, as well as Snowflake, Redshift, and Databricks, are supported, as is multi-cloud. For otherwise unsupported data sources, Fivetran will create “lite” connectors on request. The company’s robust partner network provides access to extensive data management capabilities, including data governance, data cataloguing, data masking, and so on. In addition, Fivetran’s data movement functionality can automatically propagate associated metadata alongside the data itself. This is particularly useful if you are, for instance, using Fivetran to feed a data catalogue.
Fivetran provides a robust library of pre-built, SQL-based, dbt data models that can transform, join, and calculate connector-loaded data to fill common reporting requirements. Some of these models can be downloaded and orchestrated within Fivetran directly using Quickstart transformations. You can also integrate your own dbt project into the platform to orchestrate and manage any custom data models you might have. With both methods, you can synchronise model-run orchestration with connector loads, reducing data latency and computational costs. This is visualised in a data lineage graph, providing observability. Integration with dbt also offers version control, logging, alerting, and various other features. Perhaps most notably, this includes data quality functionality that can be built into your data movement pipelines.
Additional capabilities are available, such as support for stream processing (including integration with Apache Kafka), automatic data updates, and automated schema migrations, management, and drift handling. On the latter point, Fivetran will also standardise your schemas for easy querying and API access (for example, by applying deduplication processes). These revamped schemas are fully documented by the product. It also features integrated scheduling for your data movement jobs, and can also set transformations to run automatically whenever data is loaded into your system.
What’s more, security and governance are clearly priorities for Fivetran. Accordingly, the product is certified against industry best practices and other regulations, including GDPR, SOC2, ISO27001, PCI and HIPAA; it encrypts all data, both in transit and at rest, and data moved into the Fivetran environment is deleted as soon as its data movement workflow is verified (or after 24 hours, whichever comes sooner); and it provides role-based access and authorisation, including automated user provisioning, integration with Azure AD (Active Directory), and various other features.
