Week2: Programming for Data Analysis
0 likes223 views
This document discusses merging and concatenating data in pandas. It describes how to use the concat() function to combine data from multiple sources like CSV, JSON, and XML files into a single dataframe. The syntax of concat() is explained and examples are provided of merging different arrays and dataframes, setting axes, performing inner and outer joins, and appending rows. Methods like append() are also covered as alternatives to concat() for combining two dataframes.
Week2: Programming for Data Analysis
- 1. Programming for Data Analysis Week 2 Dr. Ferdin Joe John Joseph Faculty of Information Technology Thai – Nichi Institute of Technology, Bangkok
- 2. Today’s lesson Faculty of Information Technology, Thai - Nichi Institute of Technology, Bangkok 2 • Merging • Concatenating • Reshaping • Laboratory
- 3. Merging • Used in pandas to combine data from two sources • Sources can be from same format or different • csv and csv, csv and json, json and xml and a concoction of all these • Similar to numpy array manipulation but effective with pandas Faculty of Information Technology, Thai - Nichi Institute of Technology, Bangkok 3
- 4. Function Used concat() Faculty of Information Technology, Thai - Nichi Institute of Technology, Bangkok 4
- 5. Syntax of concat() Faculty of Information Technology, Thai - Nichi Institute of Technology, Bangkok 5
- 6. Syntax of concat() Faculty of Information Technology, Thai - Nichi Institute of Technology, Bangkok 6 •objs : a sequence or mapping of Series or DataFrame objects. If a dict is passed, the sorted keys will be used as the keys argument, unless it is passed, in which case the values will be selected (see below). Any None objects will be dropped silently unless they are all None in which case a ValueError will be raised. •axis : {0, 1, …}, default 0. The axis to concatenate along.
- 7. Syntax of concat() Faculty of Information Technology, Thai - Nichi Institute of Technology, Bangkok 7 •join : {‘inner’, ‘outer’}, default ‘outer’. How to handle indexes on other axis(es). Outer for union and inner for intersection. •ignore_index : boolean, default False. If True, do not use the index values on the concatenation axis. The resulting axis will be labeled 0, …, n - 1. This is useful if you are concatenating objects where the concatenation axis does not have meaningful indexing information. Note the index values on the other axes are still respected in the join.
- 8. Syntax of concat() Faculty of Information Technology, Thai - Nichi Institute of Technology, Bangkok 8 •keys : sequence, default None. Construct hierarchical index using the passed keys as the outermost level. If multiple levels passed, should contain tuples. •levels : list of sequences, default None. Specific levels (unique values) to use for constructing a MultiIndex. Otherwise they will be inferred from the keys. •names : list, default None. Names for the levels in the resulting hierarchical index.
- 9. Syntax of concat() Faculty of Information Technology, Thai - Nichi Institute of Technology, Bangkok 9 •verify_integrity : boolean, default False. Check whether the new concatenated axis contains duplicates. This can be very expensive relative to the actual data concatenation. •copy : boolean, default True. If False, do not copy data unnecessarily.
- 10. Example Faculty of Information Technology, Thai - Nichi Institute of Technology, Bangkok 10
- 11. Example • Available Data frames: df1, df2 and df3 Faculty of Information Technology, Thai - Nichi Institute of Technology, Bangkok 11
- 12. Creation of arrays Faculty of Information Technology, Thai - Nichi Institute of Technology, Bangkok 12
- 13. Creation of arrays Faculty of Information Technology, Thai - Nichi Institute of Technology, Bangkok 13
- 14. Creation of arrays Faculty of Information Technology, Thai - Nichi Institute of Technology, Bangkok 14
- 15. Creation of Arrays Faculty of Information Technology, Thai - Nichi Institute of Technology, Bangkok 15
- 16. Frames Faculty of Information Technology, Thai - Nichi Institute of Technology, Bangkok 16
- 17. Concatenation Faculty of Information Technology, Thai - Nichi Institute of Technology, Bangkok 17
- 18. Concatenation views Faculty of Information Technology, Thai - Nichi Institute of Technology, Bangkok 18
- 19. Setting other axes Faculty of Information Technology, Thai - Nichi Institute of Technology, Bangkok 19
- 20. Setting other axes Faculty of Information Technology, Thai - Nichi Institute of Technology, Bangkok 20
- 21. Setting other axes Faculty of Information Technology, Thai - Nichi Institute of Technology, Bangkok 21
- 22. Inner Join Faculty of Information Technology, Thai - Nichi Institute of Technology, Bangkok 22
- 23. Outer Join Faculty of Information Technology, Thai - Nichi Institute of Technology, Bangkok 23
- 24. Append() • Alternative to concat() • Combines two dataframes in first index only Faculty of Information Technology, Thai - Nichi Institute of Technology, Bangkok 24
- 25. Append Faculty of Information Technology, Thai - Nichi Institute of Technology, Bangkok 25
- 26. Sort Faculty of Information Technology, Thai - Nichi Institute of Technology, Bangkok 26
- 27. Append multiple dataframes Faculty of Information Technology, Thai - Nichi Institute of Technology, Bangkok 27
- 28. Varying dimension concatenation Faculty of Information Technology, Thai - Nichi Institute of Technology, Bangkok 28
- 29. Appending rows to a dataframe Faculty of Information Technology, Thai - Nichi Institute of Technology, Bangkok 29
- 30. How it works with csv, json and xml • Convert these files to pandas dataframe object • Play with concat or append Faculty of Information Technology, Thai - Nichi Institute of Technology, Bangkok 30
- 31. DSA 207 - Merging • Create two arrays A1 and A2 and convert them into pandas data frame. Merge the data frames and store in A2. Display A2 before and after merging • Merge the given csv files together using pandas and display the first 10 data and last 15 data. Faculty of Information Technology, Thai - Nichi Institute of Technology, Bangkok 31