What is ETL?
46 likes22,338 views
ETL (Extract, Transform, Load) is a process that allows companies to consolidate data from multiple sources into a single target data store, such as a data warehouse. It involves extracting data from heterogeneous sources, transforming it to fit operational needs, and loading it into the target data store. ETL tools automate this process, allowing companies to access and analyze consolidated data for critical business decisions. Popular ETL tools include IBM Infosphere Datastage, Informatica, and Oracle Warehouse Builder.
What is ETL?
- 1. WHAT IS ETL? “EXTRACT … TRANSFORM … LOAD” Eng. Ismail El Gayar Software Engineer
- 2. WHY ETL? Companies need a way to analyze their data for critical business decisions. Transactional Database can’t answer complex business questions. A data warehouse provide a common data repository. ETL provide a method of moving the data from various source into a data warehouse.
- 3. ETL CONCEPT A Company data may be scattered in different locations and in different formats. ETL Allows you to: Migrate the data into a data warehouse. Convert the various formats and types to adhere to one consistent system. ETL is a predefined process for access and manipulate source data and loading it into a target database.
- 4. ETL REQUIREMENTS Any ETL Architecture must meet the following requirements: Business Requirement Compliance Requirement Data Profiling Data Security Data Integration Right Data at Right Time Archiving & Uneage Final End User Delivery Interface Available Skills Legacy License Alignment with overall Enterprise Architecture
- 5. THE ETL PROCESS Load The process of Transform writing data into the target The process of database converting data from one form to another Extract The process of reading data from a database
- 6. EXTRACT Gathering the data Raw data that was written directly into the disk Data written to flat files or relational tables from structured source systems Data can be read multiple times, if needed. Cleansing the data Eliminateduplicates or fragmented data Exclude unwanted / unneeded information
- 7. TRANSFORM Preparing the data to be housed in the data warehouse. Converting the extracted data Using rules and lookup tables Combining data Verification/Validity checks Standardization
- 8. LOAD Storing the transformed data in the data warehouse. Batch/Real-time processing Can follow star schema and snowflake schema
- 9. ETL FLOW
- 10. ADVANTAGE OF ETL TOOL Simple, faster and cheaper development Most ETL tools provide a metadata repository, synchronizing metadata from various sources. Most ETL tools deliver good performance, even for very large dataset. Most ETL tools provide impact analysis tools for any proposed schema changes. Most ETL tools have built-in connectors for all the major RDBMS systems
- 11. ADVANTAGE OF ETL TOOL Most ETL tools allow reuse of the existing complex programs. Several ETL tools offers visual Development Environment. Most ETL tools offers built-in scheduler sequencers and documentation. Several ETL tools offer various performance optimization options such as (parallel processing, complex load balancing etc)
- 12. POPULAR ETL TOOLS Tools Company Infosphere Datastage IBM Informatica Informatica Corp DT/Studio Embarcadero Technologies Ab Inito Ab Inito Software Corp Oracle Warehouse Builder ORACLE Microsoft SQL Server Integration Microsoft Transformation Manager ETL Solutions
- 13. THANK YOU