Introduction to python programming 2

Management Analytics
Python
Giovanni Della Lunga
giovanni.dellalunga@gmail.com
MASTER BIG DATA, ANALYTICS AND TECHNOLOGIES FOR MANAGEMENT

Python ABC
A Concise Introduction

»Ok, so we got some basics out of the way. Now, we can try to create a
real program.
»I pulled a problem off of Project Euler. Let’s have some fun.
»Each new term in the Fibonacci sequence is generated by adding the
previous two terms. By starting with 1 and 2, the first 10 terms will
be:
»1, 2, 3, 5, 8, 13, 21, 34, 55, 89, ...
»By considering the terms in the Fibonacci sequence whose values do
not exceed four million, find the sum of the even-valued terms.
Our first real Python program

A Solution Using basic python
from __future__ import print_function
total = 0
f1, f2 = 1, 2
while f1 < 4000000:
if f1 % 2 == 0:
total = total + f1
f1, f2 = f2, f1 + f2
print(total)
Notice we’re using the Python 3.x
version of print here.
Python supports multiple
assignment at once.
Right hand side is fully evaluated
before setting the variables.
Output: 4613732

A solution with functions
def even_fib():
total = 0
f1, f2 = 1, 2
while f1 < 4000000:
if f1 % 2 == 0:
total = total + f1
f1, f2 = f2, f1 + f2
return total
if __name__ == "__main__":
print(even_fib())
The Python interpreter will set some special
environmental variables when it starts executing.
If the Python interpreter is running the module (the
source file) as the main program, it sets the special
__name__ variable to have a value "__main__". This
allows for flexibility is writing your modules.
Note: __name__, as with other built-ins, has two underscores on either
side!

A solution with input
def even_fib(n):
total = 0
f1, f2 = 1, 2
while f1 < n:
if f1 % 2 == 0:
total = total + f1
f1, f2 = f2, f1 + f2
return total
if __name__ == "__main__":
limit = raw_input(“Enter the max Fibonacci number: ")
print(even_fib(int(limit)))
Enter the max Fibonacci number: 4000000
4613732

Modules
» So, we just put together our
first real Python program.
Let’s say we store this
program in a file called fib.py.
» We have just created a
module.
» Modules are simply text files
containing Python definitions
and statements which can be
executed directly or imported
by other modules.
''' Module fib.py '''
def even_fib(n):
total = 0
f1, f2 = 1, 2
while f1 < n:
if f1 % 2 == 0:
total = total + f1
f1, f2 = f2, f1 + f2
return total
if __name__ == "__main__":
limit = raw_input(“Max Fibonacci number: ")

modules
• A module is a file containing Python definitions and statements.
• The file name is the module name with the suffix .py appended.
• Within a module, the module’s name (as a string) is available as the
value of the global variable __name__.
• If a module is executed directly however, the value of the global
variable __name__ will be “__main__”.
• Modules can contain executable statements aside from definitions.
These are executed only the first time the module name is
encountered in an import statement as well as if the file is executed
as a script.

modules
I can run our module directly
at the command line. In this
case, the module’s __name__
variable has the value
“__main__”.
$ python fib.py
Max Fibonacci number:
4000000
4613732
def even_fib(n):
total = 0
f1, f2 = 1, 2
while f1 < n:
if f1 % 2 == 0:
total = total + f1
f1, f2 = f2, f1 + f2
return total
if __name__ == "__main__":

modules
I can import the module
into the interpreter. In this
case, the value of
__name__ is simply the
name of the module itself.
$ python
>>> import fib
>>> fib.even_fib(4000000)
4613732
def even_fib(n):
total = 0
f1, f2 = 1, 2
while f1 < n:
if f1 % 2 == 0:
total = total + f1
f1, f2 = f2, f1 + f2
return total
if __name__ == "__main__":

modules
I can import the module into
the interpreter. In this case,
the value of __name__ is
simply the name of the
module itself.
$ python
>>> import fib
>>> fib.even_fib(4000000)
4613732
Note that we can only access the
definitions of fib as members of the fib
object.
def even_fib(n):
total = 0
f1, f2 = 1, 2
while f1 < n:
if f1 % 2 == 0:
total = total + f1
f1, f2 = f2, f1 + f2
return total
if __name__ == "__main__":

modules
I can import the definitions
of the module directly into
the interpreter.
$ python
>>> from fib import
even_fib
>>> even_fib(4000000)
4613732
To import everything from a module:
>>> from fib import *
def even_fib(n):
total = 0
f1, f2 = 1, 2
while f1 < n:
if f1 % 2 == 0:
total = total + f1
f1, f2 = f2, f1 + f2
return total
if __name__ == "__main__":

Surprising behavior
»Let’s say I have the following Python module. It defines the add_item
function whose arguments are item and item_list, which defaults to
an empty list.
''' Module adder.py '''
def add_item(item, item_list = []):
item_list.append(item) # Add item to end of list
print item_list

Surprising behavior
»Let’s say I have the following Python module. It defines the add_item
function whose arguments are item and item_list, which defaults to
an empty list.
item_list.append(item)
print item_list
$ python
>>> from adder import *
>>> add_item(3, [])
[3]
>>> add_item(4)
[4]
>>> add_item(5)
[4, 5]

Surprising behavior
»This bizarre behavior actually gives us some insight into how Python
works.
print item_list
$ python
>>> add_item(3, [])
[3]
>>> add_item(4)
[4]
>>> add_item(5)
[4, 5]
Python’s default arguments are evaluated once when the
function is defined, not every time the function is called.
This means that if you make changes to a mutable
default argument, these changes will be reflected in
future calls to the function.

Surprising behavior
»This bizarre behavior actually gives us some insight into how Python
works.
print item_list
$ python
>>> add_item(3, [])
[3]
>>> add_item(4)
[4]
>>> add_item(5)
[4, 5]
Python’s default arguments are evaluated once when the
function is defined, not every time the function is called.
This means that if you make changes to a mutable
default argument, these changes will be reflected in
future calls to the function.
Arguments are evaluated at this point!

Surprising behavior
»An easy fix is to use a sentinel default value that tells you when to
create a new
mutable argument.
$ python
>>> add_item(3, [])
[3]
>>> add_item(4)
[4]
>>> add_item(5)
[5]
def add_item(item, item_list = None):
if item_list == None:
item_list = []
print item_list

SQL Databases
Accessing Structured Data

Databases
Commonly, Python applications will need to access a database of some
sort.
As you can imagine, not only is this easy to do in Python but there is a
ton of support for various relational and non-relational databases.
• Databases for which there is module support include:
• MySQL
• PostgreSQL
• Oracle
• SQLite
• Cassandra
• MongoDB
• etc…

Databases
» Even for a certain database, there are a number of module options. For
example, MySQL alone has the following interface modules:
• MySQL for Python (import MySQLdb)
• PyMySQL (import pymysql)
• pyODBC (import pyodbc)
• MySQL Connector/Python (import mysql.connector)
• mypysql (import mypysql)
• etc …
Yes, for every combination of my, py, and sql, there is someone out there
with a “better” implementation of a MySQL module.

Database API Specification
»So which module do you choose? Well, as far as code-writing goes, it
probably won’t make that much of a difference…
Python Enhancement Proposal 249 provides the API specification for
modules that interface with databases. You can access the
specification here.
»The majority of database modules conform to the specification so no
matter which kind of database and/or module you choose, the code
will likely look very similar.

Database api specification
»The module interface is required to have the following:
• connect(args) – a constructor for Connection objects,
through which access is made available. Arguments are database-
dependent.
• Globals apilevel (DB API level 1.0 or 2.0), threadsafety
(integer constant indicating thread safety status), paramstyle
(string constant indicating query parameter style).
• A number of exceptions, including IntegrityError,
OperationalError, DataError, etc.

»So assuming conn = connect(args) yields a Connection
object, we should be able to manipulate our connection via the
following methods:
• conn.close() – close connection.
• conn.commit() – commit pending transaction.
• conn.rollback() – if supported by db, roll back to start of
pending transaction.
• conn.cursor() – return a Cursor object for the connection.

»So c = conn.cursor() should yield a Cursor object. We can
have multiple cursors per connection, but they are not isolated from
one another. The following attributes should be available:
• c.description – a description of the cursor with up to seven
fields.
• c.rowcount – number of rows produced by last execute method.

»So c = conn.cursor() should yield a Cursor object. We can
have multiple cursors per connection, but they are not isolated from
one another. The following methods should be available:
• c.execute[many](op, [params]) – prepare and execute an
operation with parameters where the second argument may be a list
of parameter sequences.
• c.fetch[one|many|all]([s]) – fetch next row, next s rows,
or all remaining rows of result set.
• c.close() – close cursor.
• and others.

»There are a number of optional extensions such as the rownumber
attribute for cursors, which specifies the current row of the result set.
»There are also additional implementation requirements that are not
necessary to be familiar with as a user of the module.
So now we basically understand how most of Python’s database
modules work.

MySQLdb
import MySQLdb
db = MySQLdb.connect("localhost","username", "password", "EmployeeData")
cursor = db.cursor()
sql = "INSERT INTO EMPLOYEE(FIRST_NAME, LAST_NAME, AGE) VALUES ('%s', '%s', '%d')" %
('Caitlin', 'Carnahan', 24)
try:
cursor.execute(sql)
db.commit()
except:
db.rollback()
db.close()

psycopg2
import psycopg2
db = psycopg2.connect(database="mydatabase", user="uname", password="pword")
c = db.cursor()
c.execute ("SELECT * FROM versions")
rows = c.fetchall()
for i, row in enumerate(rows):
print "Row", i, "value = ", row
c.execute("DELETE FROM versions")
c.execute ("DROP TABLE versions")
c.close()
db.close()

SQLite3
»To get a feel for database usage in Python, we’ll play around with the
sqlite3 module, which is a part of the standard library.
»SQLite is a lightweight C-based relational database management
system which uses a variant of the SQL language. The data is
essentially stored in a file which is manipulated by the functions of
the C library that implements SQLite.

A Simple Python Crawler
Infor TV

Factory Pattern
»In class-based programming, the factory method pattern is
a creational pattern that uses factory methods to deal with the
problem of creating objects without having to specify the
exact class of the object that will be created.
»This is done by creating objects by calling a factory method—either
specified in an interface and implemented by child classes, or
implemented in a base class and optionally overridden by derived
classes—rather than by calling a constructor.

urllib
»urllib.request is a Python module for fetching URLs (Uniform
Resource Locators).
»It offers a very simple interface, in the form of the urlopen function.
This is capable of fetching URLs using a variety of different protocols.
It also offers a slightly more complex interface for handling common
situations - like basic authentication, cookies, proxies and so on.
These are provided by objects called handlers and openers.
»https://docs.python.org/3/howto/urllib2.html

NLTK
» The Natural Language Toolkit, or more commonly NLTK,
is a suite of libraries and programs for symbolic and
statistical natural language processing (NLP) for English
written in the Python programming language.
» It was developed by Steven Bird and Edward Loper in
the Department of Computer and Information Science
at the University of Pennsylvania.
» NLTK includes graphical demonstrations and sample
data.
» It is accompanied by a book that explains the underlying
concepts behind the language processing tasks
supported by the toolkit.

Python for Data Analysis
Getting started with pandas

Getting Started
 Excel is easy to use, but scientists need
more powerful tools
 Today we'll learn how to
 Quickly get stats on all of your samples
 Merge data from multiple rows
 Filter data by various criteria
 Merge data from multiple sheets
 All of this comes from a module called
“pandas”, which is included in Anaconda, so
it should already be installed on your
machine
 import pandas as pd
 import numpy as np
This presentation is based
on the ultimate book about
Pandas, written by
Pandas’ Creator

Getting Started
• Reading a file
• The first file we’ll work with is a download of financial data from yahoo finance
related to the Dow Jones index over a span of four years;
• The second one is a compilation of all the car accidents in England from 1979-2004,
to extract all accidents that happened in London in the year 2000. Don’t even try to
open this with Excel it’s too big for it!
• https://realpython.com/blog/python/working-with-large-excel-files-in-pandas/

Input and Output
 How do you get data into and out of Pandas as spreadsheets?
 Pandas can now work with XLS or XLSX files (they didn't use to)
 A tab looks like this: 't', but on your file it looks like a big space
 Can also be comma-delimited, but bioinformatics people always like to
use tabs because there are sometimes commas in our data
 Check which delimiter your file is using before import!
 Import to Pandas:
 df = pd.read_csv('data.csv', sep='t', header=0) # or
header=None if there is no header
 For Excel files, it's the same thing but with read_excel
 Export to text file:
 df.to_csv('data.csv', sep='t', header=True, index=False) #
the values of header and index depend on if you want to print
the column and/or row names

Getting Started
» import pandas as pd # Read the file
» data = pd.read_csv(“^DJI.csv")
» # Output the number of rows
» print("Total rows: {0}".format(len(data)))
» # See which headers are available
» print(list(data))

Pandas Objects
 Like lists, dictionaries, etc., Pandas has two objects:
 Series: like a column in a spreadsheet
 DataFrame: like a spreadsheet – a dictionary of Series objects
 Now type data.head() into your terminal and see what it outputs

Pandas Objects - Series
• A series is a one-dimensional array-like object containing an array of
data and an associated array of data labels called its index;
• The simplest Series is formed from only an array of data:

Pandas Objects – Series
• You can get the array representation and index object of the Series
via its values and index attributes, respectively:
• Often it will be desirable to create a Series with an index identifying
each data point:

Pandas Objects - Series
• There is a connection between pandas series and python;
• Should you have data contained in a Python dict, you can create a Series from it
by passing the dict:
• In this case, 3 values found in
sdata were placed in the
appropriate locations, but since
no value for 'California' was
found, it appears as NaN (not a
number) which is considered in
pandas to mark missing or NA
values

Pandas Objects - DataFrame
• A DataFrame represents a tabular, spreadsheet-like data structure containing an ordered
collection of columns, each of which can be a different value type (numeric, string,
boolean, etc.).
• The DataFrame has both a row and column index; it can be thought of as a dict of Series
(one for all sharing the same index).
• Compared with other such DataFrame-like structures you may have used before (like R’s
data.frame), row oriented and column-oriented operations in DataFrame are treated
roughly symmetrically.

Pandas Objects - DataFrame
• There are numerous ways to construct a DataFrame, though one of the most common is from a dict of
equal-length lists or NumPy arrays
• The resulting DataFrame will have its index assigned automatically as with Series, and the columns are
placed in sorted order:

Viewing Data
 Try the following:
 df.head()
 df.tail()
 df.tail(2)
 df[Close']
 df.columns
 df.index
 df.values
 You should see, in order: the first 5 lines, the last 5 lines, the last 2 lines, only the column Close',
the columns, the indices, and the data
 Unlike other Python data objects, if
you print a Pandas object to the
terminal, it won't flood your screen
because it was designed to be
readable
 What you'll find in the following
sections is that Pandas objects have
a logic that is quite different from
regular Python
 For example, operations happen on
entire columns and rows
 The new Pandas rules exist to make
your life easier, but it means you
have to hold two sets of rules in your
head

Basic Operations
 We'll go back to our data in a moment, but first, create this spreadsheet:
 nums = [[1, 2], [4, 5], [7, 8], [10, 11]]
 numdf = pd.DataFrame(nums, columns=['c1', 'c2'])
 Add a column:
 numdf['c3'] = [3, 6, 9, 12]
 Multiply all elements of a column (give just the name of the column):
 numdf['c1'] = numdf['c1']*2
 Divide all elements of multiple columns (give the DF a list of columns):
 numdf[['c2', 'c3']] = numdf[['c2', 'c3']]/2

Basic Metrics
 Your DataFrame should look like this:
 Now try the following:
 numdf.describe()
 save_stats = numdf.describe()
 What if you want to calculate those yourself?
 numdf.max(axis=0) # across all rows: the default
 numdf.max(axis=1) # across all columns
 Now try the above for numdf.min(), numdf.mean(), numdf.std(), numdf.median(), and
numdf.sum()
 Use what we learned to normalize all columns:
 normdf = (numdf - numdf.mean())/numdf.std()

Indexing and Iterating
 Remember indexing? How does it work with DFs?
 numdf.ix[1, 'c2']
 numdf.ix[1, ['c1', 'c2']]
 numdf.ix[1]
 numdf.ix['c2'] # error
 Exercise: get me 14 from numdf
 Exercise: get me the column c2 for real. Hint: on another slide
 numdf.ix[1, 'c2'] = 5.0
 numdf['c2'][1] = 5.0 # How are they different?

Filtering Data
 Let's go back to our original DF, data
 We only want to see the p-values that passed
 data[Close'] > 20000 # this is a boolean Series
 data[data[Close'] > 20000] # this is called boolean indexing
 Boolean indexing can also do assignments
 With the Accidents7904.file, let’s find all the accidents that happened on a
Sunday. Looking at the headers, there is a Day_of_Weeks field, which
we will use…

Filtering Data
• Let’s make our query more complicated: Find out all accidents that happened on a
Sunday and involved more than twenty cars:
• Let’s add another condition – weather (the code 2 means, “Raining with no heavy
winds”). Add that to our query:

Merge and Join Operation
 Two DFs can be joined by column values:

Sort and Group By
 Let's sort by ‘prezzototale’:
 ordini_sort = ordini_df.sort_values('prezzototale', ascending=False)
 print ordini_sort.head()
 In our data example, we have many orders for each state, how can we count them (pivot table) ?
 ordini_pivot = ordini_df.groupby('stato').size()

JSON
»JavaScript Object Notation (JSON) is an open, human and machine-
readable standard that facilitates data interchange, and along with
XML is the main format for data interchange used on the modern
web.
»JSON supports all the basic data types you’d expect: numbers, strings,
and boolean values, as well as arrays and hashes.

JSON
» Document databases such as MongoDB use JSON documents in order to store
records, just as tables and rows store records in a relational database. Here is an
example of a JSON document:

JSON: Syntax and Structure
» A JSON object is a key-value data format that is typically rendered in curly braces. When you’re
working with JSON, you’ll likely see JSON objects in a .json file, but they can also exist as a JSON
object or string within the context of a program.
» A JSON object looks something like this:
» Although this is a very short example, and JSON could be many lines long, this shows that the
format is generally set up with two curly braces (or curly brackets) that look like this { } on either
end of it, and with key-value pairs populating the space between.

JSON: Syntax and Structure
» By enclosing the variable's value in curly braces, we're
indicating that the value is an object. Inside the object, we
can declare any number of properties using a “key":
"value" pairing, separated by commas.
» There is a strong connection with python dictionaries;
» A slightly more complicated example involves storing two
people in one variable. To do this, we enclose multiple
objects in square brackets, which signifies an array.

Working with Complex Types in JSON
»Nested Objects
 JSON can store nested
objects in JSON format in
addition to nested arrays.
These objects and arrays
will be passed as values
assigned to keys, and
typically will be
comprised of key-value
pairs as well.

Working with Complex Types in JSON
»Nested Arrays
 We may use an array when we
are dealing with a lot of data
that can be easily grouped
together, like when there are
various websites and social
media profiles associated with
a single user.

Introduction to python programming 2

import json
{
"connection1": {
"DSN": "con1",
"UID": "abc",
"PWD": "1234",
"connection_string_python":"test1"
} ,
"connection2": {
"DSN": "con2",
"UID": "def",
"PWD": "1234"
}
}
Connection.json

Introduction to python programming 2

Recommended

More Related Content

What's hot (20)

Similar to Introduction to python programming 2 (20)

More from Giovanni Della Lunga (20)

Recently uploaded (20)

Introduction to python programming 2