Fast and cost effective geospatial analysis pipeline with AWS lambda
0 likes112 views
This document discusses using AWS Lambda for geospatial analysis. It provides examples of serverless configuration files that define Lambda functions with Python 2.7 and 3.6 runtimes. It also describes a Dockerfile and build process for packaging Python dependencies within Lambda deployment packages to reduce their size and optimize startup performance.
![FROM amazonlinux:latest
# Install apt dependencies
RUN yum install -y
gcc gcc-c++ freetype-devel yum-utils findutils openssl-devel
RUN yum -y groupinstall development
RUN curl https://www.python.org/ftp/python/3.6.1/Python-3.6.1.tar.xz | tar -xJ
&& cd Python-3.6.1
&& ./configure --prefix=/usr/local --enable-shared
&& make
&& make install
&& cd ..
&& rm -rf Python-3.6.1
ENV LD_LIBRARY_PATH=/usr/local/lib:$LD_LIBRARY_PATH
# Install Python dependencies
RUN pip3 install xxx--no-binary -t /tmp/vendored -U
# Reduce Lambda package size to fit the 250Mb limit
# Mostly based on https://github.com/jamesandersen/aws-machine-learning-demo
RUN du -sh /tmp/vendored
# This is the list of available modules on AWS lambda Python 3
# ['boto3', 'botocore', 'docutils', 'jmespath', 'pip', 'python-dateutil', 's3transfer', 'setuptools', 'six']
RUN find /tmp/vendored -name "*-info" -type d -exec rm -rdf {} +
RUN rm -rdf /tmp/vendored/boto3/
RUN rm -rdf /tmp/vendored/botocore/
RUN rm -rdf /tmp/vendored/docutils/
RUN rm -rdf /tmp/vendored/dateutil/
RUN rm -rdf /tmp/vendored/jmespath/
RUN rm -rdf /tmp/vendored/s3transfer/
RUN rm -rdf /tmp/vendored/numpy/doc/
# Leave module precompiles for faster Lambda startup
RUN find /tmp/vendored -type f -name '*.pyc' | while read f; do n=$(echo $f | sed 's/__pycache__///' | sed 's/.cpython-36//'); cp $f $n; done;
RUN find /tmp/vendored -type d -a -name '__pycache__' -print0 | xargs -0 rm -rf
RUN find /tmp/vendored -type f -a -name '*.py' -print0 | xargs -0 rm -f
RUN du -sh /tmp/vendored
COPY app /tmp/vendored/app
# Create archive
RUN cd /tmp/vendored && zip -r9q /tmp/package.zip *
# Cleanup
RUN rm -rf /tmp/vendored/
© Civic Vision UG](https://image.slidesharecdn.com/aws-181121235201/85/Fast-and-cost-effective-geospatial-analysis-pipeline-with-AWS-lambda-27-320.jpg)
Fast and cost effective geospatial analysis pipeline with AWS lambda
- 1. Fast and cost effective geospatial analysis pipeline with AWS lambda © Civic Vision UG
- 2. Mila Frerichs © Civic Vision UG
- 3. I help civic advocacy groups © Civic Vision UG
- 4. © Civic Vision UG
- 5. © Civic Vision UG
- 6. © Civic Vision UG
- 7. © Civic Vision UG
- 8. © Civic Vision UG
- 9. © Civic Vision UG
- 10. © Civic Vision UG
- 11. © Civic Vision UG
- 12. © Civic Vision UG
- 13. © Civic Vision UG
- 14. © Civic Vision UG
- 15. © Civic Vision UG
- 16. © Civic Vision UG
- 17. © Civic Vision UG
- 18. Available packages boto3 botocore docutils jmespath pip python-dateutil s3transfer setuptools six © Civic Vision UG
- 19. © Civic Vision UG
- 20. service: foss4g_oceania provider: name: aws runtime: nodejs6.10 functions: hello: handler: handler.hello © Civic Vision UG
- 21. © Civic Vision UG
- 22. service: foss4g_oceania provider: name: aws runtime: python2.7 package: include: - venv/lib/python2.7/site-packages - analysis.py exclude: - venv functions: hello: handler: handler.hello © Civic Vision UG
- 23. import sys, os, json here = os.path.dirname(os.path.realpath(__file__)) env_path = os.path.join(here, "./venv/lib/python2.7/site-packages/") sys.path.append(env_path) © Civic Vision UG
- 24. © Civic Vision UG
- 25. plugins: - serverless-python-requirements © Civic Vision UG
- 26. service: foss4g_oceania provider: name: aws runtime: python3.6 package: artifact: package.zip functions: hello: handler: handler.hello © Civic Vision UG
- 27. FROM amazonlinux:latest # Install apt dependencies RUN yum install -y gcc gcc-c++ freetype-devel yum-utils findutils openssl-devel RUN yum -y groupinstall development RUN curl https://www.python.org/ftp/python/3.6.1/Python-3.6.1.tar.xz | tar -xJ && cd Python-3.6.1 && ./configure --prefix=/usr/local --enable-shared && make && make install && cd .. && rm -rf Python-3.6.1 ENV LD_LIBRARY_PATH=/usr/local/lib:$LD_LIBRARY_PATH # Install Python dependencies RUN pip3 install xxx--no-binary -t /tmp/vendored -U # Reduce Lambda package size to fit the 250Mb limit # Mostly based on https://github.com/jamesandersen/aws-machine-learning-demo RUN du -sh /tmp/vendored # This is the list of available modules on AWS lambda Python 3 # ['boto3', 'botocore', 'docutils', 'jmespath', 'pip', 'python-dateutil', 's3transfer', 'setuptools', 'six'] RUN find /tmp/vendored -name "*-info" -type d -exec rm -rdf {} + RUN rm -rdf /tmp/vendored/boto3/ RUN rm -rdf /tmp/vendored/botocore/ RUN rm -rdf /tmp/vendored/docutils/ RUN rm -rdf /tmp/vendored/dateutil/ RUN rm -rdf /tmp/vendored/jmespath/ RUN rm -rdf /tmp/vendored/s3transfer/ RUN rm -rdf /tmp/vendored/numpy/doc/ # Leave module precompiles for faster Lambda startup RUN find /tmp/vendored -type f -name '*.pyc' | while read f; do n=$(echo $f | sed 's/__pycache__///' | sed 's/.cpython-36//'); cp $f $n; done; RUN find /tmp/vendored -type d -a -name '__pycache__' -print0 | xargs -0 rm -rf RUN find /tmp/vendored -type f -a -name '*.py' -print0 | xargs -0 rm -f RUN du -sh /tmp/vendored COPY app /tmp/vendored/app # Create archive RUN cd /tmp/vendored && zip -r9q /tmp/package.zip * # Cleanup RUN rm -rf /tmp/vendored/ © Civic Vision UG
- 28. # Install Python dependencies RUN pip3 install xxx--no-binary -t /tmp/vendored -U # Reduce Lambda package size to fit the 250Mb limit # Mostly based on https://github.com/jamesandersen/aws-machine-learning-demo RUN du -sh /tmp/vendored © Civic Vision UG
- 29. Adapted from mapbox/landsat-tiler © Civic Vision UG
- 30. all: build package build: docker build --tag fis-server:latest . package: docker run -w /var/task/ --name fis -itd fis-server:latest docker cp fis:/tmp/package.zip package.zip docker stop fis docker rm fis deploy: sls deploy © Civic Vision UG
- 31. © Civic Vision UG
- 32. © Civic Vision UG
- 33. Divide & Conquer © Civic Vision UG
- 34. © Civic Vision UG
- 35. © Civic Vision UG
- 36. © Civic Vision UG
- 37. © Civic Vision UG
- 39. © Civic Vision UG
- 40. © Civic Vision UG
- 41. © Civic Vision UG
- 42. © Civic Vision UG
- 43. © Civic Vision UG
- 44. questions Give me your two questions! © Civic Vision UG
- 45. Resources available at civicvision.de/f4goaws © Civic Vision UG