AnalyticDB:Manually migrate data from an Amazon Redshift cluster to an AnalyticDB for PostgreSQL instance

Log on to the Data Transport console.

Create the source and destination data addresses for data synchronization.

1. In the left-side navigation pane, choose Data Online Migration > Data Address.

2. Click Create Data Address to create a source data address.

3. In the Create Data Address panel, configure the parameters that are described in the following table and click OK.

   Parameter	Required	Description
   Name	Yes	The name of the source data address. The name must meet the following requirements: The name is 3 to 63 characters in length. The name must be case-sensitive and can contain lowercase letters, digits, hyphens (-), and underscores (_). The name is encoded in the UTF-8 format and cannot start with a hyphen (-) or an underscore (_).
   Type	Yes	The type of the source data address. Select AWS S3.
   Domain Name	Yes	The endpoint of the source data address. For more information, see the "Amazon S3 endpoints" section of the Amazon Simple Storage Service endpoints and quotas topic.
   AccessKeyId	Yes	The access key pair of the Identity and Access Management (IAM) user that is used to read data from the source data address. Amazon S3 uses the access key pair to check whether the IAM user has the permissions to read data from the source data address.
   SecretAccessKey	Yes
   Bucket	Yes	The name of the Amazon S3 bucket in which the data to be migrated is stored. Note The bucket name cannot contain invalid characters, such as spaces, line breaks, or tab characters.
   Prefix	No	The prefix of the source data address. You can specify a prefix to migrate specific data. If you specify a prefix, only the data in the directory that is specified by the prefix is migrated. A prefix cannot start with a forward slash (/) and must end with a forward slash (/). If you do not specify a prefix, all data in the Amazon S3 bucket is migrated.
   Tunnel	No	The name of the tunnel that you want to use. Important This parameter is required only when you migrate data to the cloud by using Express Connect circuits or VPN gateways or migrate data from self-managed databases to the cloud. If data at the destination data address is stored in a local file system or you need to migrate data over an Express Connect circuit in an environment such as Alibaba Finance Cloud or Apsara Stack, you must create and deploy an agent.
   Agent	No	The name of the agent that you want to use. Important This parameter is required only when you migrate data to the cloud by using Express Connect circuits or VPN gateways or migrate data from self-managed databases to the cloud. You can select up to 30 agents at a time for a specific tunnel.

4. Click Create Data Address again to create a destination data address.

5. In the Create Data Address panel, configure the parameters that are described in the following table and click OK.

   Parameter	Required	Description
   Data Type	Yes	The type of the destination data address. Select OSS.
   Data Name	Yes	The name of the source data address. The name must be 3 to 63 characters in length. The name cannot contain special characters except for hyphens (-) and underscores (_).
   Data Region	Yes	The region in which the destination data address resides.
   Enable Transfer Acceleration	No	If you want to use Data Online Migration together with the transfer acceleration feature of Object Storage Service (OSS), enable the transfer acceleration feature for buckets. The transfer acceleration feature takes effect within 30 minutes after it is enabled. After you enable the feature, wait for 30 minutes before you create a migration job. Important If you use the transfer acceleration feature to access your OSS buckets, you are charged additional transfer acceleration fees. For more information, see Transfer acceleration.
   OSS Endpoint	Yes	The endpoint of the destination data address. Select an endpoint based on the region in which the destination data address resides. For more information, see Regions and endpoints. Note When you migrate data from third-party cloud services to OSS, you must access OSS from a public endpoint over the Internet.
   Access Key Id and Access Key Secret	Yes	The AccessKey pair of the RAM user that is used to write data to the destination data address. The AccessKey pair is used by OSS to check whether the RAM user has the permissions to write data to the destination data address.
   OSS Bucket	Yes	The OSS bucket to which the data is migrated.
   OSS Prefix	No	The prefix of the destination data address. If you specify a prefix, the source data is migrated to the specified directory. The value cannot start with a forward slash (/) and must end with a forward slash (/). Example: data/to/oss/. If you do not specify a prefix, the source data is migrated to the root directory of the destination bucket. Important If the name of a source file starts with a forward slash (/), you must specify an OSS prefix when you specify the destination data address. If no OSS prefix is specified, the migration job fails. For example, if the name of a file to be migrated is /test/test.png, you must specify an OSS prefix, such as oss/. After the migration is complete, the name of the file changes from /test/test.png to oss//test/test.png.

Create an online migration job.

1. In the left-side navigation pane, choose Data Online Migration > Migration Jobs.

2. Click Create Job.

3. In the Create Job panel, read the terms of the migration service and select I agree to the above terms and activate the Data Transport service. Then, click Next.

4. In the Fee Reminder message, click Yes, Go Ahead.

5. In the Create Job panel, configure the parameters that are described in the following table and click Next.

   Parameter	Required	Description
   Job Name	Yes	The name of the source data address. The name must be 3 to 63 characters in length. The name cannot contain special characters except for hyphens (-) and underscores (_).
   Source Data Address	Yes	The source data address that you created.
   Destination Data Address	Yes	The destination data address that you created.
   Specified Directory	No	The directories in which the files and subdirectories are migrated or not migrated. Valid values: Do not filter: migrates all data from the source data address. Exclude: does not migrate the files or subdirectories in the specified directories. For example, you want to migrate all the directories in the root_dir/ directory except root_dir/bad_sub1/ and root_dir/bad_sub2/. In this case, you can select this method and specify bad_sub1/ and bad_sub2/. Contain: migrates only the files and subdirectories in the specified directories. For example, you want to migrate only root_dir/good_sub1/ and root_dir/good_sub2/ in the root_dir/ directory. In this case, you can select this method and specify good_sub1/ and good_sub2/. Note A directory name can contain only digits and letters. If you use special characters, the migration job may fail. A directory name cannot start with a forward slash (/) or a backslash (\), and cannot contain double forward slashes (//), double periods (..), or double quotation marks ("). The character string that consists of all the specified directory names cannot exceed 10 KB in length. A directory name must end with a forward slash (/). For example, you can specify docs/ as the directory name. You can specify up to 20 directories of the Exclude or Contain type.
   Migration Type	Yes	The data migration mode. Valid values: Full: The system migrates all data from the source data address to the destination data address. You must configure the Start Time Point of File parameter. The system migrates all data whose last modification time is later than the specified time. After all data is migrated, the migration job ends. If you change the data at the source data address after the migration job is complete, you can submit another job to migrate all data. In this case, the system migrates only the changed data. Incremental: The system runs a migration job based on the values that you specified for the Migration Interval and Migration Times parameters. You must configure the Start Time Point of File parameter. During the first migration, the system migrates all data whose last modification time is later than the specified time. After the first migration is complete, incremental migration is performed based on the value that you specified for the Migration Interval parameter. For each incremental migration, the system migrates only the data that is created or modified after the previous migration starts and before the current migration starts. If you set the Migration Times parameter to N, full migration is performed once and then incremental migration is performed (N - 1) times. For example, the current time is 08:00, March 10, 2019 and you set the Migration Interval parameter to 1, the Migration Times parameter to 5, and the Start Time Point of File parameter to 08:00, March 5, 2019. When the first migration starts, the system migrates all files whose last modification time is between 08:00, March 5, 2019 and 08:00, March 10, 2019. In this example, the first migration requires one hour to complete and the second migration starts at 10:00, March 10, 2019. The duration from 08:00 to 10:00 consists of one hour for the first migration process and one hour for the migration interval. During the second migration, the system migrates the files whose last modification time is between 08:00, March 10, 2019 and 10:00, March 10, 2019. The migration job consists of one full migration and four incremental migrations. Important Before full or incremental migration starts, the system compares the files at the source data address with the files at the destination data address. If a source file has the same name as a destination file, the destination file is overwritten when one of the following conditions is met: The Content-Type values of the files at the source data address and the destination data address are different. The last modification time of the file at the source data address is later than that of the file at the destination data address. The size of the file at the source data address is different from that of the file at the destination data address.
   Start Time Point of File	Yes	The time to filter data to be migrated. The system migrates the data that is created or modified after the specified time. Valid values: All: migrates all files. Assign: migrates only the files that are created or modified after the specified time. For example, if you set the time to 08:00:00, November 1, 2018, only the files that are created or modified after 08:00:00, November 1, 2018 are migrated.
   Migration Interval	This parameter is required if you set the Migration Type parameter to Incremental.	The default value is 1, and the maximum value is 24. Unit: hours.
   Migration Times	This parameter is required if you set the Migration Type parameter to Incremental.	The default value is 1, and the maximum value is 30.
   File Overwrite Method	Yes	The method that is used to process the file at the source data address whose name is the same as that of a file at the destination data address. Valid values: LastModified: If a source file has the same name as a destination file, the system compares the last modified time indicated by the LastModified property of the files. If the last modification time of the file at the source data address is later than that of the file at the destination data address, the file at the source data address is migrated and the file at the destination data address is overwritten. If the last modification time of the file at the source data address is earlier than that of the file at the destination data address, the file at the source data address is not migrated and the file at the destination data address is retained. If the two files have the same last modification time, the system checks the values of the Size and Content-Type properties of the files. If the two files have the same Size and Content-Type values, the file at the source data address is not migrated. If the two files have different values of at least one property, the file at the source data address is migrated and the file at the destination data address is overwritten. Condition: If a source file has the same name as a destination file, the system compares the values of the LastModified, Size, and Content-Type properties of the files. If all property values of the two files are the same, the file at the source data address is not migrated. If the two files have different values of at least one property, the file at the source data address is migrated and the file at the destination data address is overwritten. All: If a source file has the same name as a destination file, the system performs no comparison and directly overwrites the destination file with the source file. No: If a source file has the same name as a destination file, the system performs no comparison and directly skips the source file in the migration. Warning If you select Condition or LastModified, earlier files may overwrite later files. If you select Condition or LastModified, make sure that the source files have LastModified, Size, and Content-Type values. Otherwise, the overwrite policy may become ineffective and data is not migrated as expected.

6. In the Performance step, go to the Data Prediction section and configure the Data Size and File Count parameters.

   Note

   To ensure that the migration job is successful, we recommend that you estimate the amount of data that you want to migrate as accurately as possible. For more information, see Prepare for a migration job.

7. (Optional) In the Performance step, go to the Flow Control section, specify the Time Range and Max Flow(MB/s) parameters, and then click Add.

   Note

   To ensure business continuity, we recommend that you configure the Time Range and Max Flow parameters based on traffic fluctuations.

8. Click Create and wait until the migration job is complete.

Download and install ossimport in standalone mode. For more information, see Overview.

The ossimport tool in standalone mode has the following file structure:

ossimport
├── bin
│ └── ossimport2.jar  # The JAR package that contains the Master, Worker, TaskTracker, and Console modules.
├── conf
│ ├── local_job.cfg   # The job configuration file.
│   └── sys.properties  # The configuration file that contains system parameters.
├── console.bat         # The Windows command line utility that is used to run tasks in a distributed manner.
├── console.sh          # The Linux command line utility that is used to run tasks in a distributed manner.
├── import.bat          # The script that automatically imports files based on the conf/local_job.cfg configuration file in Windows. The configuration file contains parameters that specify data migration operations such as start, migration, verification, and retry.
├── import.sh           # The script that automatically imports files based on the conf/local_job.cfg configuration file in Linux. The configuration file contains parameters that specify data migration operations, such as start, migration, verification, and retry.
├── logs                # The directory that contains logs.
└ ── README.md # The file that provides a description of ossimport. We recommend that you read the file before you use ossimport.

Configure ossimport in standalone mode.

You need to modify only the following parameters in the conf/local_job.cfg configuration file:

srcType=s3
srcAccessKey=<Your AWS access key>
srcSecretKey=<Your AWS secret key>
srcDomain=<The domain name that corresponds to the region in which the Amazon S3 bucket is deployed>
srcBucket=<The name of the Amazon S3 bucket>
destAccessKey=<Your Alibaba Cloud AccessKey ID>
destSecretKey=<Your Alibaba Cloud AccessKey secret>
destDomain=<The endpoint that corresponds to the region in which the OSS bucket is deployed>
destBucket=<The name of the OSS bucket>
destPrefix=
isSkipExistFile=true

For more information, see Overview.

Run ossimport to synchronize data to OSS. For information about how to use ossimport in standalone mode, see Standalone deployment.