Applied Computer Science Concepts in Android

Applied Computer Science
Concepts in Android

Jim Huang ( 黃敬群 )
Developer & Co-Founder, 0xlab
jserv@0xlab.org

Dec 31, 2010 / CSIE, NTU

Applied Computer Science
Concepts in Android

Android = A complete operating system
for mobile computing, open source'd

Rights to copy
© Copyright 2010 0xlab
http://0xlab.org/
contact@0xlab.org
Attribution – ShareAlike 3.0
You are free Corrections, suggestions, contributions and
translations are welcome!
to copy, distribute, display, and perform the work
to make derivative works Latest update:Dec 31, 2010
to make commercial use of the work
Under the following conditions
Attribution. You must give the original author credit.
Share Alike. If you alter, transform, or build upon this work, you may distribute the
resulting work only under a license identical to this one.
For any reuse or distribution, you must make clear to others the license terms of this
work.
Any of these conditions can be waived if you get permission from the copyright holder.
Your fair use and other rights are in no way affected by the above.
License text: http://creativecommons.org/licenses/by-sa/3.0/legalcode

Agenda (1) Android Internals
(2) Compiler: 2D/3D Graphics
(3) OS: Telephony
(4) Virtual Machine: Database

In this presentation, the unaware or indirect applications of
In this presentation, the unaware or indirect applications of
essential computer science concepts are dicussed as showcase.
essential computer science concepts are dicussed as showcase.

Android Internals

… or, the low-level parts ...

Android Internals

• Android Low-Level system
– Architecture View
– Everything from “Zygote”
– Key design concepts in Android
– System components

• Hardware Abstraction Layer (HAL)
– GPS, RIL, Graphics (2D/3D), ...

Android Low-level system

• Architecture
• Everything from “Zygote”
• Key design concepts in Android

Android Low-Level System
[ Architecture ]

GNU/Linux vs. Android

Linux Android

Android = (patched) Linux Kernel + special user-space

Functional View
Applications
Gallery Phone Web Browser Google Maps ・・・・・

Android Framework
Activity Window Content Notification
View System
Manager Manager Manager Manager

Package Telephony Resource Location

System Library Android Runtime
SurfaceFlinger OpenCORE SQLite Class Library

Dalvik Virtual Machine
OpenGL|ES AudioFlinger WebKit

Skia OpenSSL bionic libc Freetype

Linux Kernel

[ Everything from Zygote ]

Android Boot Sequence
Low level initialization High level perspective (Android specific)

ARM based Initialization process
Target Board /etc/init.rc

Android bootloader Start Android Service
USB loader

console
adbd
service manager
Load Linux mountd
debuggerd
zygote
mediaserver
installd
Start kernel
flash_recovery
- /init/main.c

Process View
User-space:: process

Process

PID : 1 PPID : 2
1
PPID : 0

spawn
spawn (fork+exec)
(fork+exec) Process

Process Process spawn
(fork+exec) PPID : 3
PID : 2 PID : 3
PPID : 1 PPID : 1

2
Process Process
PID : 4 PID : 5
PPID : 2 PPID : 3
3

4

fork() 5
Copy on Write

[ Key Design Concepts ]

System Interaction

IPC: Inter-Process Communication

Activities

window

widget

Activities

view

view group

Activity state transition
Task 1

Home Activity1 Activity2 Activity3

1 2 3
Home
Back Key Key
Home Activity4 Activity5 Activity6

4 5 6
Task 2

Activity life cycle
• Entire lifetime
• Visible lifetime
• Foreground lifetime

Services

Activity Activity

Service

Invoking specific service
(provided by Media server)

Home

or
Activity1 Back

Activity2

Broadcast Receivers

Components
Components

Receiver
Broadcast

SMS_RECEIVED
Intent

SMS_RECEIVED

Content Providers

Components

Content Providers Interface
Content Provider Class Components 　 A

Preferences
Content Resolver
Files
Components 　 B

SQLite(DB)
Content Resolver
Network

Content Providers, Intents, Resolver

Components
Intent Components

Activity
Activity
Context.startActivity()
(Intents)
or
Activity.startActivityForResult()

Service
Context.startService()
Context.bindService() or

Broadcast Broadcast

Context.sendBroadcast()
Context.sendOrderedBroadcast() System
Context.sendStickyBroadcast() 3
manifest Components
2 ActivityManager
Class A
Components

Service
Intent Intent
5
to class A 1 To Class A
<class list>
…
class A
4 …

Android Services in Action
Content Telephony Bluetooth Connectivity Location
Manager Service Service Service Manager

Window Activity Package Power
…
Add Service
Service Audio Flinger
Add Service
Manager
daemons Surface
Service Flinger
usbd Manager

adbd System
runtime Server
debuggerd

rild Init Zygote Dalvik VM

Kernel Android Service
Linux Kernel
Dalvik Framework

IPC: Binder(1)
 Binder in Action

 A pool of threads is associated to each service application to process
incoming IPC (Inter-Process Communication).
 Binder performs mapping of object between two processes.
 Binder uses an object reference as an address in a process’s memory
space.
 Synchronous call, reference couting

Understanding

• Essential components
– Compiler: Java
– Virtual Machine: Dalvik
– OS: Linux Kernel SSA form

• Anything else?

Applied Computer Science Concepts in Android

Compilers

Applied in 2D/3D Graphics

OpenGL Anywhere

ARM® PowerVR
Cortex™- A8 SGX
with NEON VFP Graphics

C64x+ Other Display
Peripherals Subsystem
DSP

OpenGL Graphics Pipeline
State Variables Front Buffer

Geometry Fragment
Pipeline Pipeline

Camera Position
Light Sources F
Etc. C G
E
D
Back Buffer
A

C
GL_TRIANGLES A
B B

GL_TRIANLE_FAN
A C
E
F

B
D
Texture Maps
GL_TRIANGLE_STRIP

Texturing Example

Before Texturing After Texturing

Graphics Pipeline
LOD selection
Frustum Culling
Application Portal Culling
…

Clipping
VERTEX SHADER
Programmable

Geometry Division by w
Processing Assembly
Primitive Viewport transform
Backface culling

Scan Conversion
Rasterization FRAGMENT SHADER

Output Output to Device

Vertex and Fragment Shaders
( x, y, z, w ) ( x’, y’, z’, w’ )
( nx, ny, nz ) ( nx’, ny’, nz’ )
( s, t, r, q ) ( s’, t’, r’, q’ )
( r, g, b, a ) ( r’, g’, b’, a’ )

VERTEX
SHADER

( x, y ) ( x, y )
( r’, g’, b’, a’ ) ( r, g, b, a )
( depth’ ) ( depth )

FRAGMENT
SHADER

Embedded code with no lighting

OpenGL|ES
• Added
– Fixed-point and byte data
• Retained
– Vertex Transforms and Lighting (mostly) Embedded code with lighting
– Multi-texturing (2D only)
– Full Scene Antialiasing via Multisampling (Smooth Shaded)
– Alpha blending Surface Normal Vectors

OpenGL|ES 1.1 Example
// Enable fixed-function shading (smooth or flat)
glShadeModel(GL_SMOOTH);

// Define the appearance of triangle surfaces
glMaterialfv(GL_FRONT_AND_BACK, GL_AMBIENT, fMaterialAmbient);
glMaterialf(GL_FRONT_AND_BACK, GL_SHININESS, Shininess);

// Define the appearance and position of a light source
glLightfv(GL_LIGHT0, GL_AMBIENT, fLightAmbient);
glLightModelfv(GL_LIGHT_MODEL_AMBIENT, fAmbient);
glLightfv(GL_LIGHT0, GL_POSITION, fLightPosition);

// Set pointers to geometry and other attributes and draw it
glVertexPointer(3, GL_FLOAT, 0, Vertices);
glTexCoordPointer(2, GL_FLOAT, 0, TexCoords);
glNormalPointer(GL_FLOAT, 0, NormalsPerVertex);
glDrawArrays(GL_TRIANGLES, 0, Count);

38

OpenGL|ES 2.0 Example
// Create a vertex shader object, load source code and compile it
hVertexShader = glCreateShader(GL_VERTEX_SHADER);
glShaderSource(hVertexShader, 1, pVertexShaderSourceCode, NULL);
glCompileShader(hVertexShader);

// Create a shader program and attach the fragment and vertex shaders to it
hProgram = glCreateProgram();
glAttachShader(hProgram, hFragmentShader);
glAttachShader(hProgram, hVertexShader);

// Link and load the new shader programs into the PowerVR SGX
glLinkProgram(hProgram);
glUseProgram(hProgram);

// Set pointers to geometry and other attributes and send to the vertex shader
glVertexAttribPointer(Index, 3, GL_FLOAT, GL_TRUE, Stride, pAttributes);
glDrawArrays(GL_TRIANGLES, 0, Count);

39

GPU = Graphics Processing Unit
 Pipelining
1 2 3
 Number of stages

 Parallelism 1

2
 Number of parallel processes 3

 Parallelism + pipelining
1 2 3
 Number of parallel pipelines 1 2 3
 Operates on 4 tuples
1 2 3
– Position ( x, y, z, w )
– Color( red, green, blue, alpha )
– Texture Coordinates ( s, t, r, q )
4 tuple ops, 1 clock cycle
 SIMD [ Single Instruction Multiple Data ]

Reyes(scanline,polygon) Raytracing

Optimized by SIMD

Hard to be optimized by SIMD
Computing dependency
Dynamic fast-paths required

Specialize color space convertion takes lots of time.
BGRA 444R → RGBA 8888
Technique

Speedup depends on src/dest format:
– 5.4x speedup on average, 19.3x max
speedup: (13.3MB/s to 257.7MB/s)

LLVM = Low Level Virtual Machine

Mandelbort is optimized through LLVM JIT up to 11x.

Android SurfaceFlinger
 Properties
 Can combine 2D/3D surfaces and surfaces from multiple applications
 Surfaces passed as buffers via Binder IPC calls
 Can use OpenGL ES and 2D hardware accelerator for its compositions
 Double-buffering using page-flip

UI Element GLSurfaceView javax .microedition .khronos .opengles

View and Canvas → Surface
View and Canvas → Surface

android. view. View com.google.android.gles_jni

android. view. Surface android graphics. Canvas
.
Java Graphics JAVA
is implemented Framework
via JNI Native
Framework Surface JNI Graphic JNI OpenGL JNI

SurfaceFlinger skia OpenGL|ES
Libui +
Libui +
SurfaceFlinger +
SurfaceFlinger +
libpixelflinger
libpixelflinger libui
Overlay Camera Surface Key / Event format EglWindows

libpixelflinger
FrameBuffer Event Input
Driver Driver

PixelFlinger : software renderer
• Render functions: pointx, linex, recti, trianglex
• Texture and color buffer: activeTexture, bindTexture,
colorBuffer, readBuffer, depthBuffer, BindTextureLod
• …
• Device framebuffer functions: copyPixels,
rasterPos2x, rasterPos2i
• Optimizer: codeflinger (JIT assembler)

I/SurfaceFlinger( 1931): OpenGL informations:

I/SurfaceFlinger( 1931): vendor : Android

I/SurfaceFlinger( 1931): renderer : Android PixelFlinger 1.2

I/SurfaceFlinger( 1931): version : OpenGL ESCM 1.0

GGLAssembler –
GGLAssembler –
codeflinger's JIT Assembler
codeflinger's JIT Assembler

commit 77cadd2ffada95bb3279552e1a29f4bcf4012228
commit 77cadd2ffada95bb3279552e1a29f4bcf4012228
Author: Jim Huang <jserv@0xlab.org>
Author: Jim Huang <jserv@0xlab.org>
Date:   Wed Jan 13 01:01:18 2010 +0800
Date:   Wed Jan 13 01:01:18 2010 +0800

    [libpixelflinger] Adds UXTB16 support to Pixelflinger
    [libpixelflinger] Adds UXTB16 support to Pixelflinger
    ...
    ...
    Uses UXTB16 to extract channels for SIMD operations, rather than creating and
    Uses UXTB16 to extract channels for SIMD operations, rather than creating and
    ANDing with masks. Saves a register and is faster on A8, as UXTB16 result can
    ANDing with masks. Saves a register and is faster on A8, as UXTB16 result can
    feed into first stage of multiply, unlike AND.
    feed into first stage of multiply, unlike AND.


    Also, used SMULWB rather than SMULBB, which allows removal of MOVs used to
    Also, used SMULWB rather than SMULBB, which allows removal of MOVs used to
    rescale results.
    rescale results.


    Code has been scheduled for A8 pipeline, specifically aiming to allow
    Code has been scheduled for A8 pipeline, specifically aiming to allow
    multiplies to issue in pipeline 0, for efficient dual issue operation.
    multiplies to issue in pipeline 0, for efficient dual issue operation.


    Testing on SpriteMethodTest (http://code.google.com/p/appsforandroid/) gives
    Testing on SpriteMethodTest (http://code.google.com/p/appsforandroid/) gives
    8% improvement (12.7 vs. 13.7 fps.)
    8% improvement (12.7 vs. 13.7 fps.)

libpixelflinger/codeflinger/ARMAssembler.cpp
libpixelflinger/codeflinger/ARMAssembler.cpp
@@ 424,5 +424,15 @@ void ARMAssembler::SMLAW(int cc, int y,
@@ 424,5 +424,15 @@ void ARMAssembler::SMLAW(int cc, int y,
     *mPC++ = (cc<<28) | 0x1200080 | (Rd<<16) | (Rn<<12) | (Rs<<8) | (y<<4) | Rm;
     *mPC++ = (cc<<28) | 0x1200080 | (Rd<<16) | (Rn<<12) | (Rs<<8) | (y<<4) | Rm;
}
}

+#if 0
+#if 0
+#pragma mark
+#pragma mark
+#pragma mark Byte/half word extract and extend (ARMv6+ only)...
+#pragma mark Byte/half word extract and extend (ARMv6+ only)...
+#endif
+#endif
+
+
+void ARMAssembler::UXTB16(int cc, int Rd, int Rm, int rotate)
+void ARMAssembler::UXTB16(int cc, int Rd, int Rm, int rotate)
+{
+{
+    *mPC++ = (cc<<28) | 0x6CF0070 | (Rd<<12) | ((rotate >> 3) << 10) | Rm;
+    *mPC++ = (cc<<28) | 0x6CF0070 | (Rd<<12) | ((rotate >> 3) << 10) | Rm; opcode
+}
+}
+
+
}; // namespace android

libpixelflinger/codeflinger/ARMAssemblerProxy.cpp
libpixelflinger/codeflinger/ARMAssemblerProxy.cpp
@@ 195,6 +195,10 @@ void ARMAssemblerProxy::SMLAW(int cc, int y, int Rd, int Rm, int Rs, int Rn) {
@@ 195,6 +195,10 @@ void ARMAssemblerProxy::SMLAW(int cc, int y, int Rd, int Rm, int Rs, int Rn) {
     mTarget>SMLAW(cc, y, Rd, Rm, Rs, Rn);
     mTarget>SMLAW(cc, y, Rd, Rm, Rs, Rn);
}
}

+void ARMAssemblerProxy::UXTB16(int cc, int Rd, int Rm, int rotate) {
+void ARMAssemblerProxy::UXTB16(int cc, int Rd, int Rm, int rotate) {
+    mTarget>UXTB16(cc, Rd, Rm, rotate);
+    mTarget>UXTB16(cc, Rd, Rm, rotate);
+}
+}
+
+


libpixelflinger/codeflinger/texturing.cpp
libpixelflinger/codeflinger/texturing.cpp

@@ 868,6 +869,106 @@ void GGLAssembler::filter24(
@@ 868,6 +869,106 @@ void GGLAssembler::filter24(
     load(txPtr, texel, 0);
     load(txPtr, texel, 0);
}
}

+#if __ARM_ARCH__ >= 6
+#if __ARM_ARCH__ >= 6
+// ARMv6 version, using UXTB16, and scheduled for CortexA8 pipeline
+// ARMv6 version, using UXTB16, and scheduled for CortexA8 pipeline
+void GGLAssembler::filter32(
+void GGLAssembler::filter32(
+        const fragment_parts_t& parts,
+        const fragment_parts_t& parts,
+        pixel_t& texel, const texture_unit_t& tmu,
+        pixel_t& texel, const texture_unit_t& tmu,
+        int U, int V, pointer_t& txPtr,
+        int U, int V, pointer_t& txPtr,
+        int FRAC_BITS)
+        int FRAC_BITS)
+{
+{
...
...

+    UXTB16(AL, temp, pixel, 0);
Reimplement filter32()
+    if (round) {
+    if (round) {
+        ADD(AL, 0, u, u, imm(1<<(adjust1)));
+        ADD(AL, 0, u, u, imm(1<<(adjust1)));
+        MOV(AL, 0, u, reg_imm(u, LSR, adjust));
+        MOV(AL, 0, u, reg_imm(u, LSR, adjust));
+    }
In optimized ASM
+    }
+    LDR(AL, pixellb, txPtr.reg, reg_scale_pre(offsetlb));
+    LDR(AL, pixellb, txPtr.reg, reg_scale_pre(offsetlb));
+    MUL(AL, 0, dh, temp, u);
+    MUL(AL, 0, dh, temp, u);
+    MUL(AL, 0, dl, temp, u);
+    MUL(AL, 0, dl, temp, u);
+    RSB(AL, 0, k, u, imm(0x100));
+    RSB(AL, 0, k, u, imm(0x100));
...
...
+#else
+#else
void GGLAssembler::filter32(
void GGLAssembler::filter32(
         const fragment_parts_t& parts,
         const fragment_parts_t& parts,
         pixel_t& texel, const texture_unit_t& tmu,
         pixel_t& texel, const texture_unit_t& tmu,

Operating System

Applied in Telephony

RIL → Telephony
 Radio Interface Layer(RIL)
 HAL  Android TelephonyManager  baseband modem
 Voice, Data, SMS, SIM, SIMToolkit
 Android RIL  AT command
 RIL API
 Android follows GSM TS 27.007 standard
 RIL implementation
 Qualcomm supports CDMA &
CDMA-GSM Multi-mode

OS Kernels

Monolithic Kernel Micro Kernel
Traditional UNIX, Linux, … Mach, L4, Symbian OS

Exokernel

L4 Microkernel

• Developed by Jochen Liedtke in 1995.
– German National Research Center for IT
• Developed from scratch
• 2nd generation microkernel
• Small and Fast Kernel
– 7 system calls
– 12KB
• In November 2005, NICTA announced that
Qualcomm was deploying NICTA's L4 version
on their Mobile Station Modem chipsets.

Communications
Processor
L4 Interrupt Handler
-> Send Message

L4 microkernel

RPC
System-
Server call
Handler

RPC Interrupt
RPC
RT task
Requests from
Shared Memory Application Processor
to Modem

Virtual Machine

Applied in Database

sqlite3_prepare()
/*
** Compile the UTF8 encoded SQL statement zSql into a statement handle.
*/
int sqlite3_prepare(
  sqlite3 *db,            /* Database handle. */
  const char *zSql,       /* UTF8 encoded SQL statement. */
  int nBytes,             /* Length of zSql in bytes. */
  sqlite3_stmt **ppStmt,  /* OUT: A pointer to the prepared
     statement */
  const char** pzTail     /* OUT: End of parsed string */
)
{
...
      sqlite3VdbeSetNumCols(sParse.pVdbe, 5);
      sqlite3VdbeSetColName(sParse.pVdbe, 0, COLNAME_NAME, "addr", P3_STATIC);
      sqlite3VdbeSetColName(sParse.pVdbe, 1, COLNAME_NAME, "opcode", P3_STATIC);
      sqlite3VdbeSetColName(sParse.pVdbe, 2, COLNAME_NAME, "p1", P3_STATIC);
…
Opcode!

SQLite Virtual Machine (VDBE) 61

…And what about other platforms?

The concept can be applied to all VM platforms

Application virtual machines (short list)
.NET (CLR)
Java Virtual Machine (JVM)
Dalvik virtual machine (Google Android)
PHP (Zend Engine)
Flash Player / AIR - ActionScript Virtual Machine (AVM)
SQLite virtual machine (VDBE; Virtual DataBase Engine)
Perl virtual machine

Inspiration

• Computer science concepts are applied
everywhere in Android software stack.
• Essential system software brings the further
optimizations and feasibility of new technologies.
• Advanced compiler techniques are already
applied in every domain
• Best Practice in Android

http://0xlab.org

Applied Computer Science Concepts in Android

Recommended

More Related Content

What's hot (20)

Viewers also liked (15)

Similar to Applied Computer Science Concepts in Android (20)

More from National Cheng Kung University (20)

Recently uploaded (20)

Applied Computer Science Concepts in Android