Hive function-cheat-sheet
1 like588 views
Hive functions cheat sheet provides a listing and descriptions of built-in mathematical, date, and time functions in Hive. It also describes how to create user-defined functions (UDFs), user-defined aggregate functions (UDAFs), and user-defined table generating functions (UDTFs) and how to deploy and drop custom functions.
![Hive Functions Cheat-sheet, by Qubole
How to create and use Hive Functions, Listing of Built-In Functions that are supported in Hive
www.qubole.com QUESTIONS? CALL US 855-HADOOP-HELP
Description
Returns the rounded BIGINT value of the double
Returns the double rounded to d decimal places
Returns the maximum BIGINT value that is equal or less than the double
Returns the minimum BIGINT value that is equal or greater than the double
Returns a random number (that changes from row to row) that is distributed uniformly from 0 to 1.
Specifiying the seed will make sure the generated random number sequence is deterministic.
Returns ea where e is the base of the natural logarithm
Returns the natural logarithm of the argument
Returns the base-10 logarithm of the argument
Returns the base-2 logarithm of the argument
Return the base "base" logarithm of the argument
Return ap
Returns the square root of a
Returns the number in binary format
If the argument is an int, hex returns the number as a string in hex format. Otherwise if the number is a string,
it converts each character into its hex representation and returns the resulting string.
Inverse of hex. Interprets each pair of characters as a hexidecimal number and converts to the character
represented by the number.
Converts a number from a given base to another
Returns the absolute value
Returns the positive value of a mod b
Returns the sine of a (a is in radians)
Returns the arc sin of x if -1<=a<=1 or null otherwise
Returns the cosine of a (a is in radians)
Returns the arc cosine of x if -1<=a<=1 or null otherwise
Returns the tangent of a (a is in radians)
Returns the arctangent of a
Converts value of a from radians to degrees
Converts value of a from degrees to radians
Returns a
Returns -a
Returns the sign of a as '1.0' or '-1.0'
Returns the value of e
Returns the value of pi
Mathematical Functions
Return Type
BIGINT
DOUBLE
BIGINT
BIGINT
double
double
double
double
double
double
double
double
string
string
string
string
double
int double
double
double
double
double
double
double
double
double
int double
int double
float
double
double
Name (Signature)
round(double a)
round(double a, int d)
floor(double a)
ceil(double a), ceiling(double a)
rand(), rand(int seed)
exp(double a)
ln(double a)
log10(double a)
log2(double a)
log(double base, double a)
pow(double a, double p), power(double a, double p)
sqrt(double a)
bin(BIGINT a)
hex(BIGINT a) hex(string a)
unhex(string a)
conv(BIGINT num, int from_base, int to_base),
conv(STRING num, int from_base, int to_base)
abs(double a)
pmod(int a, int b) pmod(double a, double b)
sin(double a)
asin(double a)
cos(double a)
acos(double a)
tan(double a)
atan(double a)
degrees(double a)
radians(double a)
positive(int a), positive(double a)
negative(int a), negative(double a)
sign(double a)
e()
pi()
Description
Converts the number of seconds from unix epoch (1970-01-01 00:00:00 UTC) to a string representing the
timestamp of that moment in the current system time zone in the format of "1970-01-01 00:00:00"
Gets current time stamp using the default time zone.
Converts time string in format yyyy-MM-dd HH:mm:ss to Unix time stamp, return 0 if fail: unix_timestamp
('2009-03-20 11:30:01') = 1237573801
Convert time string with given pattern to Unix time stamp, return 0 if fail: unix_timestamp('2009-03-20',
'yyyy-MM-dd') = 1237532400
Returns the date part of a timestamp string: to_date("1970-01-01 00:00:00") = "1970-01-01"
Returns the year part of a date or a timestamp string: year("1970-01-01 00:00:00") = 1970, year("1970-01-01")
= 1970
Returns the month part of a date or a timestamp string: month("1970-11-01 00:00:00") = 11, month
("1970-11-01") = 11
Return the day part of a date or a timestamp string: day("1970-11-01 00:00:00") = 1, day("1970-11-01") = 1
Returns the hour of the timestamp: hour('2009-07-30 12:58:59') = 12, hour('12:58:59') = 12
Returns the minute of the timestamp
Returns the second of the timestamp
Return the week number of a timestamp string: weekofyear("1970-11-01 00:00:00") = 44, weekofyear
("1970-11-01") = 44
Return the number of days from startdate to enddate: datediff('2009-03-01', '2009-02-27') = 2
Add a number of days to startdate: date_add('2008-12-31', 1) = '2009-01-01'
Subtract a number of days to startdate: date_sub('2008-12-31', 1) = '2008-12-30'
Assumes given timestamp ist UTC and converts to given timezone (as of Hive 0.8.0)
Assumes given timestamp is in given timezone and converts to UTC (as of Hive 0.8.0)
Date Functions
Return Type
string
bigint
bigint
bigint
string
int
int
int
int
int
int
int
int
string
string
timestamp
timestamp
Name (Signature)
from_unixtime(bigint unixtime[, string format])
unix_timestamp()
unix_timestamp(string date)
unix_timestamp(string date, string pattern)
to_date(string timestamp)
year(string date)
month(string date)
day(string date) dayofmonth(date)
hour(string date)
minute(string date)
second(string date)
weekofyear(string date)
datediff(string enddate, string startdate)
date_add(string startdate, int days)
date_sub(string startdate, int days)
from_utc_timestamp(timestamp, string timezone)
to_utc_timestamp(timestamp, string timezone)
Hive Function Meta commands
SHOW FUNCTIONS– lists Hive functions and operators
DESCRIBE FUNCTION [function name]– displays short description of the function
DESCRIBE FUNCTION EXTENDED [function name]– access extended description of the function
Types of Hive Functions
UDF– is a function that takes one or more columns from a row as argument and returns a single value or object. Eg: concat(col1, col2)
UDAF- aggregates column values in multiple rows and returns a single value. Eg: sum(c1)
UDTF— takes zero or more inputs and and produces multiple columns or rows of output. Eg: explode()
Macros— a function that users other Hive functions.
How To Develop UDFs
package org.apache.hadoop.hive.contrib.udf.example;
import java.util.Date;
import java.text.SimpleDateFormat;
import org.apache.hadoop.hive.ql.exec.UDF;
@Description(name = "YourUDFName",
value = "_FUNC_(InputDataType) - using the input datatype X argument, "+
"returns YYY.",
extended = "Example:n"
+ " > SELECT _FUNC_(InputDataType) FROM tablename;")
public class YourUDFName extends UDF{
..
public YourUDFName( InputDataType InputValue ){
..;
}
public String evaluate( InputDataType InputValue ){
..;
}
}
How To Develop UDFs, GenericUDFs, UDAFs, and UDTFs
public class YourUDFName extends UDF{
public class YourGenericUDFName extends GenericUDF {..}
public class YourGenericUDAFName extends AbstractGenericUDAFResolver {..}
public class YourGenericUDTFName extends GenericUDTF {..}
How To Deploy / Drop UDFs
At start of each session:
ADD JAR /full_path_to_jar/YourUDFName.jar;
CREATE TEMPORARY FUNCTION YourUDFName AS 'org.apache.hadoop.hive.contrib.udf.example.YourUDFName';
At the end of each session:
DROP TEMPORARY FUNCTION IF EXISTS YourUDFName;](https://image.slidesharecdn.com/hive-function-cheat-sheet-160325215033/85/Hive-function-cheat-sheet-1-320.jpg)
Hive function-cheat-sheet
- 1. Hive Functions Cheat-sheet, by Qubole How to create and use Hive Functions, Listing of Built-In Functions that are supported in Hive www.qubole.com QUESTIONS? CALL US 855-HADOOP-HELP Description Returns the rounded BIGINT value of the double Returns the double rounded to d decimal places Returns the maximum BIGINT value that is equal or less than the double Returns the minimum BIGINT value that is equal or greater than the double Returns a random number (that changes from row to row) that is distributed uniformly from 0 to 1. Specifiying the seed will make sure the generated random number sequence is deterministic. Returns ea where e is the base of the natural logarithm Returns the natural logarithm of the argument Returns the base-10 logarithm of the argument Returns the base-2 logarithm of the argument Return the base "base" logarithm of the argument Return ap Returns the square root of a Returns the number in binary format If the argument is an int, hex returns the number as a string in hex format. Otherwise if the number is a string, it converts each character into its hex representation and returns the resulting string. Inverse of hex. Interprets each pair of characters as a hexidecimal number and converts to the character represented by the number. Converts a number from a given base to another Returns the absolute value Returns the positive value of a mod b Returns the sine of a (a is in radians) Returns the arc sin of x if -1<=a<=1 or null otherwise Returns the cosine of a (a is in radians) Returns the arc cosine of x if -1<=a<=1 or null otherwise Returns the tangent of a (a is in radians) Returns the arctangent of a Converts value of a from radians to degrees Converts value of a from degrees to radians Returns a Returns -a Returns the sign of a as '1.0' or '-1.0' Returns the value of e Returns the value of pi Mathematical Functions Return Type BIGINT DOUBLE BIGINT BIGINT double double double double double double double double string string string string double int double double double double double double double double double int double int double float double double Name (Signature) round(double a) round(double a, int d) floor(double a) ceil(double a), ceiling(double a) rand(), rand(int seed) exp(double a) ln(double a) log10(double a) log2(double a) log(double base, double a) pow(double a, double p), power(double a, double p) sqrt(double a) bin(BIGINT a) hex(BIGINT a) hex(string a) unhex(string a) conv(BIGINT num, int from_base, int to_base), conv(STRING num, int from_base, int to_base) abs(double a) pmod(int a, int b) pmod(double a, double b) sin(double a) asin(double a) cos(double a) acos(double a) tan(double a) atan(double a) degrees(double a) radians(double a) positive(int a), positive(double a) negative(int a), negative(double a) sign(double a) e() pi() Description Converts the number of seconds from unix epoch (1970-01-01 00:00:00 UTC) to a string representing the timestamp of that moment in the current system time zone in the format of "1970-01-01 00:00:00" Gets current time stamp using the default time zone. Converts time string in format yyyy-MM-dd HH:mm:ss to Unix time stamp, return 0 if fail: unix_timestamp ('2009-03-20 11:30:01') = 1237573801 Convert time string with given pattern to Unix time stamp, return 0 if fail: unix_timestamp('2009-03-20', 'yyyy-MM-dd') = 1237532400 Returns the date part of a timestamp string: to_date("1970-01-01 00:00:00") = "1970-01-01" Returns the year part of a date or a timestamp string: year("1970-01-01 00:00:00") = 1970, year("1970-01-01") = 1970 Returns the month part of a date or a timestamp string: month("1970-11-01 00:00:00") = 11, month ("1970-11-01") = 11 Return the day part of a date or a timestamp string: day("1970-11-01 00:00:00") = 1, day("1970-11-01") = 1 Returns the hour of the timestamp: hour('2009-07-30 12:58:59') = 12, hour('12:58:59') = 12 Returns the minute of the timestamp Returns the second of the timestamp Return the week number of a timestamp string: weekofyear("1970-11-01 00:00:00") = 44, weekofyear ("1970-11-01") = 44 Return the number of days from startdate to enddate: datediff('2009-03-01', '2009-02-27') = 2 Add a number of days to startdate: date_add('2008-12-31', 1) = '2009-01-01' Subtract a number of days to startdate: date_sub('2008-12-31', 1) = '2008-12-30' Assumes given timestamp ist UTC and converts to given timezone (as of Hive 0.8.0) Assumes given timestamp is in given timezone and converts to UTC (as of Hive 0.8.0) Date Functions Return Type string bigint bigint bigint string int int int int int int int int string string timestamp timestamp Name (Signature) from_unixtime(bigint unixtime[, string format]) unix_timestamp() unix_timestamp(string date) unix_timestamp(string date, string pattern) to_date(string timestamp) year(string date) month(string date) day(string date) dayofmonth(date) hour(string date) minute(string date) second(string date) weekofyear(string date) datediff(string enddate, string startdate) date_add(string startdate, int days) date_sub(string startdate, int days) from_utc_timestamp(timestamp, string timezone) to_utc_timestamp(timestamp, string timezone) Hive Function Meta commands SHOW FUNCTIONS– lists Hive functions and operators DESCRIBE FUNCTION [function name]– displays short description of the function DESCRIBE FUNCTION EXTENDED [function name]– access extended description of the function Types of Hive Functions UDF– is a function that takes one or more columns from a row as argument and returns a single value or object. Eg: concat(col1, col2) UDAF- aggregates column values in multiple rows and returns a single value. Eg: sum(c1) UDTF— takes zero or more inputs and and produces multiple columns or rows of output. Eg: explode() Macros— a function that users other Hive functions. How To Develop UDFs package org.apache.hadoop.hive.contrib.udf.example; import java.util.Date; import java.text.SimpleDateFormat; import org.apache.hadoop.hive.ql.exec.UDF; @Description(name = "YourUDFName", value = "_FUNC_(InputDataType) - using the input datatype X argument, "+ "returns YYY.", extended = "Example:n" + " > SELECT _FUNC_(InputDataType) FROM tablename;") public class YourUDFName extends UDF{ .. public YourUDFName( InputDataType InputValue ){ ..; } public String evaluate( InputDataType InputValue ){ ..; } } How To Develop UDFs, GenericUDFs, UDAFs, and UDTFs public class YourUDFName extends UDF{ public class YourGenericUDFName extends GenericUDF {..} public class YourGenericUDAFName extends AbstractGenericUDAFResolver {..} public class YourGenericUDTFName extends GenericUDTF {..} How To Deploy / Drop UDFs At start of each session: ADD JAR /full_path_to_jar/YourUDFName.jar; CREATE TEMPORARY FUNCTION YourUDFName AS 'org.apache.hadoop.hive.contrib.udf.example.YourUDFName'; At the end of each session: DROP TEMPORARY FUNCTION IF EXISTS YourUDFName;