PySpark - Extracting single value from DataFrame

PySpark - Extracting single value from DataFrame

Last Updated : 17 Jun, 2021

In this article, we are going to extract a single value from the pyspark dataframe columns. To do this we will use the first() and head() functions.

Single value means only one value, we can extract this value based on the column name

Syntax:

dataframe.first()['column name']
Dataframe.head()['Index']

Where,

dataframe is the input dataframe and column name is the specific column
Index is the row and columns.

So we are going to create the dataframe using the nested list.

Python3

# importing module
import pyspark

# importing sparksession from pyspark.sql module
from pyspark.sql import SparkSession

# creating sparksession and giving an app name
spark = SparkSession.builder.appName('sparkdf').getOrCreate()

# list  of students  data 
data =[["1","sravan","vignan"],
       ["2","ojaswi","vvit"],
       ["3","rohith","vvit"],
       ["4","sridevi","vignan"],
       ["1","sravan","vignan"], 
       ["5","gnanesh","iit"]]

# specify column names
columns=['student ID','student NAME','college']

# creating a dataframe from the lists of data
dataframe = spark.createDataFrame(data,columns)

print("Actual data in dataframe")
# show dataframe
dataframe.show()

Output:

Actual data in dataframe
+----------+------------+-------+
|student ID|student NAME|college|
+----------+------------+-------+
|         1|      sravan| vignan|
|         2|      ojaswi|   vvit|
|         3|      rohith|   vvit|
|         4|     sridevi| vignan|
|         1|      sravan| vignan|
|         5|     gnanesh|    iit|
+----------+------------+-------+

Example 1: Python program to extract a single value from a particular column using first().

Python3

# extract single value based on
# column in the dataframe
dataframe.first()['student ID']

Output:

'1'

Example 2: Extract a single value using head().

Python3

# extract single value based
# on column in the dataframe
dataframe.head()[0]

Output:

'1'

Example 3: Extract a single value using head().

Python3

# extract single value based
# on column in the dataframe
dataframe.head()[2]

Output:

'vignan'

PySpark - Extracting single value from DataFrame

gottumukkalabobby

Improve

Article Tags :

Practice Tags :

python

Similar Reads

Extract First and last N rows from PySpark DataFrame

In data analysis, extracting the start and end of a dataset helps understand its structure and content. PySpark, widely used for big data processing, allows us to extract the first and last N rows from a DataFrame. In this article, we'll demonstrate simple methods to do this using built-in functions

Show distinct column values in PySpark dataframe

In this article, we are going to display the distinct column values from dataframe using pyspark in Python. For this, we are using distinct() and dropDuplicates() functions along with select() function. Let's create a sample dataframe. Python3 # importing module import pyspark # importing sparksessi

PySpark - Select Columns From DataFrame

In this article, we will discuss how to select columns from the pyspark dataframe. To do this we will use the select() function. Syntax: dataframe.select(parameter).show() where,Â dataframe is the dataframe nameparameter is the column(s) to be selectedshow() function is used to display the selected

Get value of a particular cell in PySpark Dataframe

In this article, we are going to get the value of a particular cell in the pyspark dataframe. For this, we will use the collect() function to get the all rows in the dataframe. We can specify the index (cell positions) to the collect function Creating dataframe for demonstration: Python3 # importing

Remove duplicates from a dataframe in PySpark

In this article, we are going to drop the duplicate data from dataframe using pyspark in Python Before starting we are going to create Dataframe for demonstration: Python3 # importing module import pyspark # importing sparksession from pyspark.sql module from pyspark.sql import SparkSession # creati

Update Pyspark Dataframe Metadata

In this article, we are going to learn how to update the PySpark data frame in Python. In this article, we will discuss how to update the metadata of a PySpark data frame. Specifically, we will cover the following topics: Understanding the importance of metadata in PySpark DataFramesHow to access an

Read Text file into PySpark Dataframe

In this article, we are going to see how to read text files in PySpark Dataframe. There are three ways to read text files into PySpark DataFrame. Using spark.read.text()Using spark.read.csv()Using spark.read.format().load() Using these we can read a single text file, multiple files, and all files fr

PySpark Count Distinct from DataFrame

In this article, we will discuss how to count distinct values present in the Pyspark DataFrame.Â In Pyspark, there are two ways to get the count of distinct values. We can use distinct() and count() functions of DataFrame to get the count distinct of PySpark DataFrame. Another way is to use SQL coun

Create PySpark dataframe from dictionary

In this article, we are going to discuss the creation of Pyspark dataframe from the dictionary. To do this spark.createDataFrame() method method is used. This method takes two argument data and columns. The data attribute will contain the dataframe and the columns attribute will contain the list of

Select columns in PySpark dataframe

In this article, we will learn how to select columns in PySpark dataframe. Function used: In PySpark we can select columns using the select() function. The select() function allows us to select single or multiple columns in different formats.Â Syntax: dataframe_name.select( columns_names ) Note: We