Before you configure a sync task, you must configure the required data sources on the Data Source Management page. Data Integration supports dozens of data sources, such as MySQL, MaxCompute, Hologres, OSS, and Kafka. The supported synchronization solutions and read/write plugins vary by data source. When you configure a sync task, select the appropriate source and destination data sources based on your requirements.
Supported data sources and synchronization solutions
Data Integration provides three feature modules: batch synchronization, real-time synchronization, and synchronization solutions. You can select a feature module to configure a sync task based on the data sources that the module supports.
DataWorks batch synchronization provides Reader and Writer plugins to read data from and write data to data sources.
DataWorks real-time synchronization lets you use various combinations of source and destination data sources for real-time incremental synchronization of a single table or an entire database.
DataWorks also provides solutions for various synchronization scenarios between different data sources, such as full database batch synchronization and real-time full and incremental synchronization.
In real-time data synchronization scenarios, different data sources may handle DDL and DML operations differently, which can cause compatibility issues. For example, if a table is created in the source, you must manually create the table schema in the destination before the synchronization can continue. For more information about DDL and DML policies, see Supported DML and DDL operations.
To configure and run tasks, go to the Batch Synchronization, Real-time Synchronization, or Synchronization Solutions module.
Data source | Batch synchronization | Real-time synchronization | Full and incremental sync task | |||||
Read single table | Write single table | Read single table (incremental) | Write single table (incremental) | Read full database (batch) | Write full database (batch) | Read single table/full database (full and incremental, real-time) | Write single table/full database (full and incremental, real-time) | |
| - | - | - | - | - | - | - | |
|
| - | - | - | - | - | - | |
|
| - | - | - | - | - | - | |
|
| - |
|
| - | - |
| |
|
| - | - |
| - | - | - | |
|
| - | - | - | - |
| - | |
| - | - | - | - | - | - | - | |
| - | - | - | - | - | - | - | |
|
| - | - | - | - | - | - | |
|
|
|
| - | - | - |
| |
- |
| - |
| - | - | - |
| |
|
| - | - |
| - | - | - | |
|
| - | - | - | - | - | - | |
|
| - | - |
| - | - | - | |
|
| - | - |
| - |
| - | |
Elasticsearch |
|
| - |
| - |
| - |
|
|
| - | - | - | - | - | - | |
GBase8a |
|
| - | - | - | - | - | - |
HBase |
|
| - | - | - | - | - | - |
|
| - | - | - | - | - | - | |
Hive |
|
| - | - | - |
| - | - |
|
|
|
|
|
| - |
| |
| - | - | - | - | - | - | - | |
|
|
|
| - | - | - |
| |
|
| - | - | - | - | - | - | |
|
| - | - | - | - | - | - | |
|
|
| - | - | - | - | - | |
|
| - |
| - |
| - |
| |
|
| - | - | - | - | - | - | |
- |
| - | - | - | - | - | - | |
- |
| - | - | - | - | - | - | |
| - | - | - | - | - | - | - | |
- |
| - | - | - | - | - | - | |
|
| - | - | - | - |
| - | |
|
|
| - |
| - |
| - | |
- |
| - | - | - | - | - | - | |
|
|
| - |
| - |
| - | |
|
| - |
| - |
| - | - | |
|
| - |
| - | - | - | - | |
|
|
| - |
| - |
| - | |
|
| - | - |
| - |
| - | |
|
| - | - |
| - |
| - | |
- |
| - | - | - | - | - | - | |
|
| - | - | - | - | - | - | |
| - | - | - | - | - | - | - | |
|
| - | - | - | - | - | - | |
- |
| - | - | - | - | - | - | |
|
| - | - | - |
| - | - | |
|
| - | - |
| - | - | - | |
|
| - |
| - | - | - | - | |
| - | - | - | - | - | - | - | |
| - | - | - | - | - | - | - | |
- |
| - | - | - | - | - | - | |
Vertica |
|
| - | - | - | - | - | - |
| - | - | - | - | - | - | - | |
Common business scenarios
References
For more information about how to configure a real-time sync task for a single table from a MySQL, Kafka, MaxCompute, or Hologres data source, see Configure a real-time sync task for a single table.