Going Serverless with AWS Lambda at ReportGarden
- 1. Going Serverless with AWS Lambda Gandhi Jay
- 2. Why talk about Serverless? ● Web Scraping - Actual use case from Tribelocal ○ Use case is to find Business is listed correctly, incorrectly or not listed on directory. ○ Process ■Scrap the directory’s web page with parameter ■Parse businesses and it’s related information ■Find matching record from parsed data ■If any information is incorrect or business is not listed, Notify the user. ○ Each directory might have different way to data.
- 3. Traditional approach ● In Monolith, ○ Create Directory DataLoader ○ Write XPaths/Css Selectors ○ Write code to scrap directory’s webpage ○ Write code to parse Businesses ○ Write code to match Business and find out it’s correct, incorrect, duplicate or not listed. Repeat the process for another directory.
- 4. Traditional approach: is it good? â—Ź As number of locations increases requests are increasing. â—Ź Simple Web scraping problem suddenly becomes distributed computing nightmare.
- 5. ● A service take independent functions ● runs it in parallel "containers" ○ can be monitored separately ○ can be scaled separately and automatically ● Pay per execution. (in ms) ● Lets you focus on application, not infrastructure. What is Serverless? There are actual servers, located somewhere but you don’t have to worry about. Someone said use Serverless, Backend as a Service Function as a Service Serverless+ =
- 6. Serverless Approach â—Ź Choose your runtime. â—Ź Write code for AWS Lambda. â—Ź Put it into S3 â—Ź Configure AWS Lambda â—Ź Configure Services like API Gateway, SNS etc. â—Ź Test it.
- 7. Benefits of Serverless â—Ź Increases flexibility to scale â—‹ Provisioning based on usage, not instances â—‹ Self auto-scaling and auto-provisioning â—Ź Reduced time to write code â—Ź Minimum risk â—‹ No living host or instances â—Ź Decreased Time to Market â—Ź Encourages a modular, well encapsulated, loosely coupled architecture â—Ź Reduced resource cost â—‹ Costs based on precise usage (no usage = no cost)
- 8. Service Providers - Serverless â—Ź AWS Lambda â—Ź Google Cloud Functions â—Ź Azure Functions â—Ź Auth0 Webtask â—Ź IBM OpenWhisk
- 9. At ReportGarden - AWS Lambda â—Ź Function as a Service platform â—‹ Billed per 100ms â—‹ Node.js, Python, C# and JRE(OpenJDK) â—Ź Event-driven â—Ź Asynchronous invocation â—Ź Can be integrated with other AWS service â—Ź AWS Serverless Ecosystem â—‹ Lambda â—‹ API Gateway â—‹ SNS â—‹ SQS â—‹ DynamoDB â—‹ S3 â—‹ Kinesis â—‹ CloudWatch â—‹ Step Functions
- 10. AWS Lambda Runtime Environment â—Ź Memory Available from 128MB to 1.5GB â—‹ Minimum CPU speed and I/O scale will be based on Memory â—Ź 2 Virtual CPUs â—Ź 512MB /tmp storage â—Ź 50MB compressed(jar/zip) for function, 250MB deployment package â—Ź STDOUT and STDERR, goes to CloudWatch Logs AWS Lambda - http://docs.aws.amazon.com/lambda/latest/dg/welcome.html
- 11. Functions runs when events are triggered â—Ź A POST request is sent to API Gateway â—Ź A scheduled cron triggers â—Ź A new item is added to S3 bucket â—Ź An infrastructure event matching rule is found. â—Ź A SAAS webhook is fired. â—Ź A link in email is clicked. â—Ź A new item is added to DB.
- 12. Async, Latency Tolerant WebScraping with AWS Lambda Framework - Scrapie Request with JSON spec AWS API Gateway Trigger Lambda AWS SNS scrape -job AWS Lambda ScraperNotifier Client System AWS Lambda ScraperJob Hit Callback URL with messageid and content. AWS CloudWatch & Rollbar (Monitoring & Logging) AWS CloudFormation (Orchestration) Gradle & Serverless (Build & Deployment) Helium Scraping Service Proxy
- 13. Configure Lambda, SNS, API Gateway…. Still lot of work & We are lazy.
- 14. DEMO â—Ź Prerequisite â—‹ Install NVM - Node Version Manager â—‹ Install serverless â—‹ Install AWS CLI â—‹ AWS configure with your AccessKey and SecretKey â—‹ Good to GO!
- 15. Challenges - Serverless â—Ź State â—Ź Latency â—Ź Loss of control â—Ź Testing/Tooling â—‹ Serverless-offline â—‹ localstack/localstack â—‹ But not lot of the options. â—Ź Very low latency â—‹ High Frequency Trading â—Ź Large Scale, in memory, stateful â—‹ MFTs of TBs of data. â—Ź Long running, stateful â—‹ Synchronous external transactions Terrible use cases - Serverless
- 16. Awesome Serverless use cases. â—Ź Async, Latency tolerant â—‹ Data Pipelines, Automatic Thumbnail generation, New User welcome emails, PDF Generation â—Ź Sync, Latency tolerant â—‹ Web apps, APIs â—Ź Glue â—‹ Infrastructure automation, orchestration â—Ź Analytics Stack â—‹ AWS Athena, AWS Glue
Editor's Notes
- #5: Need to ask why? Bar