Annotate types in large codebase with automated refactoring
0 likes163 views
The document discusses an automated refactoring process for enhancing type annotations in a large Python codebase at Carta, which consists of 1.8 million lines and 120,000 functions. It highlights tools like libcst for modifying Python code and mentions strategies such as using static and runtime analysis to improve type coverage. Additionally, it encourages continuous improvement through automated weekly updates and the engagement of developers.
1 of 20
Download to read offline
Ad
Recommended
Enforcing API Design Rules for High Quality Code Generation
Enforcing API Design Rules for High Quality Code GenerationTim Burks The document discusses the importance of enforcing API design rules and utilizing openAPI tooling for high-quality code generation. It highlights various design guidelines from companies like Google, Microsoft, and IBM, and suggests the implementation of configurable and extensible linters for API compliance. Additionally, it explores the potential of using Protocol Buffers for API data serialization and outlines future directions for API tools and organization-specific linters.
Type Annotations in Python: Whats, Whys and Wows!
Type Annotations in Python: Whats, Whys and Wows!Andreas Dewes The document discusses type annotations in Python. It explains why type annotations are useful for finding bugs, provides a history of type hints in Python from PEP 3107 to PEP 544, and demonstrates three approaches to adding type hints: inline annotations, type comments, and stub files. It also analyzes real-world usage of type hints by checking over 100 popular Python projects on GitHub and discusses additional uses for annotations beyond type checking.
MongoDB World 2018: A Swift Introduction to Swift
MongoDB World 2018: A Swift Introduction to SwiftMongoDB Swift is a new programming language released by Apple in 2014 that is designed to be safer, more modern, and work seamlessly with Objective-C. The document discusses building a MongoDB driver in Swift, highlighting some of Swift's key features like protocols, extensions, generics, and optionals that make it well-suited for this task. Interfacing Swift with existing C-based MongoDB drivers also presents challenges around representing BSON data types and pointer types that the driver had to address. Future work on the driver includes implementing more of the MongoDB API and taking advantage of newer Swift features.
Python_Interview_Questions.pdf
Python_Interview_Questions.pdfSamir P. This document contains frequently asked Python interview questions organized into different sections like Python fundamentals, OOPs concepts, Pandas, Numpy etc. It begins with introductory questions about Python's history, benefits, dynamic and interpreted nature. Then it covers core concepts like data types, modules, scopes, lists vs tuples. The document also includes questions on OOPs concepts, Pandas for data analysis, Numpy for numerical computing and commonly used Python libraries. It ends with examples of Python programs to solve different problems.
C Types - Extending Python
C Types - Extending PythonPriyank Kapadia Ctypes provides an easy way to extend Python with functions from C libraries without needing to write C code or dependencies like SWIG. It works by dynamically loading shared libraries and allowing Python functions to call C functions. Ctypes abstracts away the complexities of type conversions and memory management to make the process simple. Developers can write Python callback functions, define C structures and unions in Python, and interface with C libraries using just a few lines of Python code. This makes ctypes a popular modern alternative for creating Python bindings for C code.
Introduction to Python.pdf
Introduction to Python.pdfRahul Mogal - Python is a programming language created by Guido van Rossum in 1991. It has two major versions: Python 2.0 released in 2000 and Python 3.0 released in 2008. Python 3.0 is the latest version and is not completely backward compatible with Python 2.0.
- Python is used for machine learning, data science, web development, game development, and more. Popular frameworks include TensorFlow, Pandas, Django, and PyGame.
- The document discusses installing Python, different integrated development environments (IDEs) like PyCharm and Visual Studio Code, and covers basic Python concepts like data types, variables, operators, and more.
Nordic API days 2016 - APIs.guru Wikipedia for Web APIs
Nordic API days 2016 - APIs.guru Wikipedia for Web APIsIvan Goncharov The document discusses the adoption and standardization of APIs, highlighting the need for developer-driven approaches and features like human-readable documentation. It emphasizes collaboration with API owners and the importance of community involvement in improving API descriptions through platforms like apis.guru. The roadmap includes enhancing integrations, validating contributions, and the future challenges of data compatibility.
Git risky using git metadata to predict code bug risk
Git risky using git metadata to predict code bug riskPyData The document presents a framework for predicting code bug risk using Git metadata by building a model based on the history of commits. It outlines steps for identifying bug-inducing commits, constructing features, training a binary classifier, and implementing Git hooks to score new commits. Emphasis is placed on the importance of good commit hygiene for effective outcomes in bug tracking and classification.
python programming.pptx
python programming.pptxKaviya452563 The document presents an overview of Python programming, detailing its history, features, advantages, disadvantages, and applications. It further explains how to collect information from Twitter using Python, specifically by utilizing the Tweepy library to access Twitter's API for streaming and tracking tweets. Practical coding examples are provided to demonstrate how to implement these features effectively.
Static code analysis for verification of the 64-bit applications
Static code analysis for verification of the 64-bit applicationsPVS-Studio The document discusses the challenges of porting 32-bit applications to 64-bit systems, focusing on the need for static code analysis to ensure correctness after migration. It outlines common pitfalls faced during this transition and presents various analysis methods, especially emphasizing the viva64 static code analyzer designed for detecting potential errors in 64-bit applications. The article asserts the importance of thorough code checks and introduces ten specific rules for identifying dangerous coding practices related to 64-bit architecture.
Pa1 json requests
Pa1 json requestsaiclub_slides This document discusses using the json and requests modules in Python to interact with APIs and AI models. It explains that json is used to serialize Python dictionaries to strings for transmission over HTTP. The requests module allows Python to send HTTP requests, which can be used to interact with AI models and APIs. The document provides an example of calling a mood analysis AI by sending a json payload with a sentence to the API URL using requests. It instructs the reader to log into an AI platform, find the integration code, and paste it into a repl environment to call the AI model from Python.
Python Linters at Scale.pdf
Python Linters at Scale.pdfJimmy Lai The document discusses the challenges and solutions for managing Python linters in large codebases, focusing on improving developer experience and scaling issues. It highlights various popular linters, their configurations, and practices for optimizing linting processes, including using caching, parallel execution, and custom auto-fixing tools. Additionally, it emphasizes the importance of tracking metrics and establishing consistent linting practices across multiple projects to enhance efficiency and reduce errors.
phases of compiler PPT includes phases of compiler
phases of compiler PPT includes phases of compilerpremajain3 The document provides an overview of compilers, detailing the process of translating source code into a target language through compilation and interpretation methods. It discusses the qualities of a good compiler, principles of compilation, the importance of studying compilation technology, and diverse applications of compiler technology, including program translation and performance optimization. Additionally, it highlights challenges in compiler construction and the integration of various computer science fields.
FDP-faculty deveopmemt program on python
FDP-faculty deveopmemt program on pythonkannikadg This document provides an overview of dictionaries and structuring data in Python. It discusses key-value pairs, creating and initializing dictionaries, accessing and modifying dictionary elements, dictionary methods, use cases for dictionaries, other Python data structures like lists and tuples, combining data structures, best practices, and concludes by emphasizing the importance of dictionaries and structured data in Python code.
The Onward Journey: Porting Twisted to Python 3
The Onward Journey: Porting Twisted to Python 3Craig Rodrigues This document discusses porting the Twisted Python library to Python 3. It describes Twisted as a networking library that uses asynchronous programming. The author ported over 325 pull requests to help move Twisted to Python 3. Some challenges included changes to Python 3 like print becoming a function and str handling, as well as porting old C extensions. Running extensive unit tests was important. While difficult, porting benefits code quality and keeps Twisted compatible with Python's direction.
API Athens Meetup - API standards 22.03.2016
API Athens Meetup - API standards 22.03.2016Ivan Goncharov APIs.guru is a platform that aims to serve as a Wikipedia for web APIs by providing documentation for APIs in one centralized location. It filters out private and unreliable APIs, standardizes documentation into the Swagger 2.0 format, and allows contributors to add and improve API descriptions. APIs.guru also seeks to address challenges of data and schema incompatibility between APIs by developing a system of common scalar and complex data types that can be referenced and converted between APIs.
JavaScript API Deprecation in the Wild: A First Assessment (SANER 2020)
JavaScript API Deprecation in the Wild: A First Assessment (SANER 2020)Andre Hora The document assesses JavaScript API deprecation practices by analyzing 50 popular projects, revealing that only 3% of files included deprecation mechanisms. It identifies various methods developers use to handle deprecation, with replacement messages being common but overall adoption of these practices being low, particularly for JSDoc annotations. Future research is suggested to improve understanding of developer motivations and to create tools for identifying deprecation contexts.
1-Phases of compiler-26-04-2023.pptx
1-Phases of compiler-26-04-2023.pptxvenkatapranaykumarGa The document discusses the different phases and passes of a compiler. It describes the analysis phase which includes lexical analysis, syntax analysis, and semantic analysis. It then discusses the synthesis phase which includes intermediate code generation, code optimization, and code generation. It explains how compilers use multiple passes to handle forward references and reduce memory usage. Finally, it briefly defines different types of compilers such as one pass, two pass, incremental, native code, and cross compilers.
Wondershare Filmora 14.3.2 Crack + License Key Free Download
Wondershare Filmora 14.3.2 Crack + License Key Free Downloadanglekaan18 ➡️ 🌍📱👉COPY & PASTE LINK👉👉👉 https://forum-up.org/download-now/
Wondershare Filmora 14.3.2 Crack is an excellent software for video editing. It is beneficial in perfect video editing with industry standards.
2025-03-20 - How to use AI to your advantage - AI-Driven Development.pdf
2025-03-20 - How to use AI to your advantage - AI-Driven Development.pdfShereef The presentation "AI Driven Development" discusses the evolution and current state of AI in software development, highlighting its strengths and weaknesses. The speaker, along with Shafiq Amin, covers core AI concepts, the history of AI, and introduces tools like Copilot for code generation and debugging. The session includes a live demo showcasing AI's capabilities in creating, testing, and documenting a calculator API using Visual Studio Code. Key points include AI's proficiency in code completion, debugging, and generating boilerplate code, while cautioning about its limitations such as hallucinations, handling complex decisions, and ethical considerations. The presentation emphasizes the importance of using AI tools to enhance productivity while maintaining vigilance through code reviews and testing.
https://drive.google.com/drive/folders/1Lb8VIE82f7WP9FwpPN5_QqYPNdwwmi88
AOMEI Backupper Crack 2025 FREE Download
AOMEI Backupper Crack 2025 FREE Downloadmuhammadwaqaryounus6 🌍📱👉COPY LINK & PASTE ON GOOGLE http://drfiles.net/ 👈🌍
AOMEI Backupper offers free and affordable data backup & disaster recovery software to protect your computer.
AOMEI Backupper is a backup and recovery software for Windows. It allows users to back up and restore their system, disks, partitions, and files.
AOMEI partition assistant crack makes produce, resize, shift, combine, Crack rupture without having any reduction of your valuable information.
AOMEI Backupper Standard is the best free backup software and provides you easy backup & disaster recovery solutions for Windows 11, 10, 8.1, 8, 7,5.
Direct License file Link Below https://dr-up-community.info/ AOMEI Backupper offers free and affordable data backup & disaster recovery .
Wondershare PDFelement Pro Crack FREE Download
Wondershare PDFelement Pro Crack FREE Downloadwaqarcracker5 👉📱 COPY & PASTE LINK 👉 https://drfiles.net/
Wondershare PDFelement Pro is a PDF editing program that allows users to read, edit, convert, and share PDFs.
With pdfelement pro, you can convert your original format to DOC, XLS, PPT, BMP, GIF, JPG, PNG, TIFF, ePub, HTML and RTF formats. Read more. 0 TRACKS(0:00).
Wondershare PDFelement Crack is an impressive application for editing and customizing PDF files with easily. With it, you can easily view .
Wondershare PDFelement Crack is an impressive application for editing and customizing PDF files with easily. With it, you can easily view documents.
Python Course In Chandigarh
Python Course In ChandigarhExcellence Academy Excellence Technology offers one of the best Python courses in Chandigarh, focusing on both desktop and web application development. The curriculum includes fundamental and advanced topics such as data structures, flow control, and object-oriented programming. Additionally, the institute provides various industrial training durations to accommodate different learning needs.
Pa2 session 4
Pa2 session 4aiclub_slides The document discusses connecting an AI model to Python using HTTP requests. It covers converting pandas dataframes to dictionaries, using the requests module to send HTTP requests to an AI, and interacting with a mood classification AI by copying code from its integration page. The JSON module is also discussed for converting response dictionaries to strings. An exercise is provided to calculate accuracy and confusion matrix by sending dictionary formatted mood data through the AI service.
C# 4.0 and .NET 4.0
C# 4.0 and .NET 4.0Buu Nguyen This document discusses new features in C# 4.0 and .NET 4.0 including optional and named parameters, covariance and contravariance, dynamic binding, threading using tasks, the task parallel library (TPL), and PLINQ. It includes sample code and questions to test understanding.
Pa1 json requests
Pa1 json requestsaiclub_slides The document discusses using Python modules like JSON and Requests to interact with APIs and AI models. It explains that JSON is commonly used to transmit data over the web and can be dumped and loaded from Python dictionaries and strings. The Requests module allows sending HTTP requests in Python to APIs and AI services. It provides an example of sending a request to a mood analysis AI with a payload containing sentence data and converting the response. The document also outlines steps for connecting an AI model in Navigator to a Python application using the integration code snippet.
Overview of python 2019
Overview of python 2019Samir Mohanty This document provides an overview of the Python programming language. It covers Python basics like syntax, datatypes, modules, and control structures. It also discusses topics like functions, classes, files, and popular Python modules. The document contains an agenda that outlines these topics and provides code samples to illustrate Python concepts hands-on. It aims to equip readers with foundational Python programming knowledge.
[PyCon US 2025] Scaling the Mountain_ A Framework for Tackling Large-Scale Te...
[PyCon US 2025] Scaling the Mountain_ A Framework for Tackling Large-Scale Te...Jimmy Lai Managing tech debt in large legacy codebases isn’t just a challenge—it’s an ongoing battle that can drain developer productivity and morale. In this talk, I’ll introduce a Python-powered Tech Debt Framework bar-raiser designed to help teams tackle even the most daunting tech debt problems with 100,000+ violations. This open-source framework empowers developers and engineering leaders by: - Tracking Progress: Measure and visualize the state of tech debt and trends over time. - Recognizing Contributions: Celebrate developer efforts and foster accountability with contribution leaderboards and automated shoutouts. - Automating Fixes: Save countless hours with codemods that address repetitive debt patterns, allowing developers to focus on higher-priority work.
Through real-world case studies, I’ll showcase how we: - Reduced 70,000+ pyright-ignore annotations to boost type-checking coverage from 60% to 99.5%. - Converted a monolithic sync codebase to async, addressing blocking IO issues and adopting asyncio effectively.
Attendees will gain actionable strategies for scaling Python automation, fostering team buy-in, and systematically reducing tech debt across massive codebases. Whether you’re dealing with type errors, legacy dependencies, or async transitions, this talk provides a roadmap for creating cleaner, more maintainable code at scale.
PyCon JP 2024 Streamlining Testing in a Large Python Codebase .pdf
PyCon JP 2024 Streamlining Testing in a Large Python Codebase .pdfJimmy Lai The document discusses strategies for streamlining testing in a growing Python codebase at Zip, which has doubled to 2.5 million lines of code with 100 developers. Key strategies include parallel execution of tests, caching dependencies to save time, and skipping unnecessary tests to improve efficiency. The insights aim to enhance quality assurance and developer experience through optimized continuous integration practices.
More Related Content
Similar to Annotate types in large codebase with automated refactoring (20)
python programming.pptx
python programming.pptxKaviya452563 The document presents an overview of Python programming, detailing its history, features, advantages, disadvantages, and applications. It further explains how to collect information from Twitter using Python, specifically by utilizing the Tweepy library to access Twitter's API for streaming and tracking tweets. Practical coding examples are provided to demonstrate how to implement these features effectively.
Static code analysis for verification of the 64-bit applications
Static code analysis for verification of the 64-bit applicationsPVS-Studio The document discusses the challenges of porting 32-bit applications to 64-bit systems, focusing on the need for static code analysis to ensure correctness after migration. It outlines common pitfalls faced during this transition and presents various analysis methods, especially emphasizing the viva64 static code analyzer designed for detecting potential errors in 64-bit applications. The article asserts the importance of thorough code checks and introduces ten specific rules for identifying dangerous coding practices related to 64-bit architecture.
Pa1 json requests
Pa1 json requestsaiclub_slides This document discusses using the json and requests modules in Python to interact with APIs and AI models. It explains that json is used to serialize Python dictionaries to strings for transmission over HTTP. The requests module allows Python to send HTTP requests, which can be used to interact with AI models and APIs. The document provides an example of calling a mood analysis AI by sending a json payload with a sentence to the API URL using requests. It instructs the reader to log into an AI platform, find the integration code, and paste it into a repl environment to call the AI model from Python.
Python Linters at Scale.pdf
Python Linters at Scale.pdfJimmy Lai The document discusses the challenges and solutions for managing Python linters in large codebases, focusing on improving developer experience and scaling issues. It highlights various popular linters, their configurations, and practices for optimizing linting processes, including using caching, parallel execution, and custom auto-fixing tools. Additionally, it emphasizes the importance of tracking metrics and establishing consistent linting practices across multiple projects to enhance efficiency and reduce errors.
phases of compiler PPT includes phases of compiler
phases of compiler PPT includes phases of compilerpremajain3 The document provides an overview of compilers, detailing the process of translating source code into a target language through compilation and interpretation methods. It discusses the qualities of a good compiler, principles of compilation, the importance of studying compilation technology, and diverse applications of compiler technology, including program translation and performance optimization. Additionally, it highlights challenges in compiler construction and the integration of various computer science fields.
FDP-faculty deveopmemt program on python
FDP-faculty deveopmemt program on pythonkannikadg This document provides an overview of dictionaries and structuring data in Python. It discusses key-value pairs, creating and initializing dictionaries, accessing and modifying dictionary elements, dictionary methods, use cases for dictionaries, other Python data structures like lists and tuples, combining data structures, best practices, and concludes by emphasizing the importance of dictionaries and structured data in Python code.
The Onward Journey: Porting Twisted to Python 3
The Onward Journey: Porting Twisted to Python 3Craig Rodrigues This document discusses porting the Twisted Python library to Python 3. It describes Twisted as a networking library that uses asynchronous programming. The author ported over 325 pull requests to help move Twisted to Python 3. Some challenges included changes to Python 3 like print becoming a function and str handling, as well as porting old C extensions. Running extensive unit tests was important. While difficult, porting benefits code quality and keeps Twisted compatible with Python's direction.
API Athens Meetup - API standards 22.03.2016
API Athens Meetup - API standards 22.03.2016Ivan Goncharov APIs.guru is a platform that aims to serve as a Wikipedia for web APIs by providing documentation for APIs in one centralized location. It filters out private and unreliable APIs, standardizes documentation into the Swagger 2.0 format, and allows contributors to add and improve API descriptions. APIs.guru also seeks to address challenges of data and schema incompatibility between APIs by developing a system of common scalar and complex data types that can be referenced and converted between APIs.
JavaScript API Deprecation in the Wild: A First Assessment (SANER 2020)
JavaScript API Deprecation in the Wild: A First Assessment (SANER 2020)Andre Hora The document assesses JavaScript API deprecation practices by analyzing 50 popular projects, revealing that only 3% of files included deprecation mechanisms. It identifies various methods developers use to handle deprecation, with replacement messages being common but overall adoption of these practices being low, particularly for JSDoc annotations. Future research is suggested to improve understanding of developer motivations and to create tools for identifying deprecation contexts.
1-Phases of compiler-26-04-2023.pptx
1-Phases of compiler-26-04-2023.pptxvenkatapranaykumarGa The document discusses the different phases and passes of a compiler. It describes the analysis phase which includes lexical analysis, syntax analysis, and semantic analysis. It then discusses the synthesis phase which includes intermediate code generation, code optimization, and code generation. It explains how compilers use multiple passes to handle forward references and reduce memory usage. Finally, it briefly defines different types of compilers such as one pass, two pass, incremental, native code, and cross compilers.
Wondershare Filmora 14.3.2 Crack + License Key Free Download
Wondershare Filmora 14.3.2 Crack + License Key Free Downloadanglekaan18 ➡️ 🌍📱👉COPY & PASTE LINK👉👉👉 https://forum-up.org/download-now/
Wondershare Filmora 14.3.2 Crack is an excellent software for video editing. It is beneficial in perfect video editing with industry standards.
2025-03-20 - How to use AI to your advantage - AI-Driven Development.pdf
2025-03-20 - How to use AI to your advantage - AI-Driven Development.pdfShereef The presentation "AI Driven Development" discusses the evolution and current state of AI in software development, highlighting its strengths and weaknesses. The speaker, along with Shafiq Amin, covers core AI concepts, the history of AI, and introduces tools like Copilot for code generation and debugging. The session includes a live demo showcasing AI's capabilities in creating, testing, and documenting a calculator API using Visual Studio Code. Key points include AI's proficiency in code completion, debugging, and generating boilerplate code, while cautioning about its limitations such as hallucinations, handling complex decisions, and ethical considerations. The presentation emphasizes the importance of using AI tools to enhance productivity while maintaining vigilance through code reviews and testing.
https://drive.google.com/drive/folders/1Lb8VIE82f7WP9FwpPN5_QqYPNdwwmi88
AOMEI Backupper Crack 2025 FREE Download
AOMEI Backupper Crack 2025 FREE Downloadmuhammadwaqaryounus6 🌍📱👉COPY LINK & PASTE ON GOOGLE http://drfiles.net/ 👈🌍
AOMEI Backupper offers free and affordable data backup & disaster recovery software to protect your computer.
AOMEI Backupper is a backup and recovery software for Windows. It allows users to back up and restore their system, disks, partitions, and files.
AOMEI partition assistant crack makes produce, resize, shift, combine, Crack rupture without having any reduction of your valuable information.
AOMEI Backupper Standard is the best free backup software and provides you easy backup & disaster recovery solutions for Windows 11, 10, 8.1, 8, 7,5.
Direct License file Link Below https://dr-up-community.info/ AOMEI Backupper offers free and affordable data backup & disaster recovery .
Wondershare PDFelement Pro Crack FREE Download
Wondershare PDFelement Pro Crack FREE Downloadwaqarcracker5 👉📱 COPY & PASTE LINK 👉 https://drfiles.net/
Wondershare PDFelement Pro is a PDF editing program that allows users to read, edit, convert, and share PDFs.
With pdfelement pro, you can convert your original format to DOC, XLS, PPT, BMP, GIF, JPG, PNG, TIFF, ePub, HTML and RTF formats. Read more. 0 TRACKS(0:00).
Wondershare PDFelement Crack is an impressive application for editing and customizing PDF files with easily. With it, you can easily view .
Wondershare PDFelement Crack is an impressive application for editing and customizing PDF files with easily. With it, you can easily view documents.
Python Course In Chandigarh
Python Course In ChandigarhExcellence Academy Excellence Technology offers one of the best Python courses in Chandigarh, focusing on both desktop and web application development. The curriculum includes fundamental and advanced topics such as data structures, flow control, and object-oriented programming. Additionally, the institute provides various industrial training durations to accommodate different learning needs.
Pa2 session 4
Pa2 session 4aiclub_slides The document discusses connecting an AI model to Python using HTTP requests. It covers converting pandas dataframes to dictionaries, using the requests module to send HTTP requests to an AI, and interacting with a mood classification AI by copying code from its integration page. The JSON module is also discussed for converting response dictionaries to strings. An exercise is provided to calculate accuracy and confusion matrix by sending dictionary formatted mood data through the AI service.
C# 4.0 and .NET 4.0
C# 4.0 and .NET 4.0Buu Nguyen This document discusses new features in C# 4.0 and .NET 4.0 including optional and named parameters, covariance and contravariance, dynamic binding, threading using tasks, the task parallel library (TPL), and PLINQ. It includes sample code and questions to test understanding.
Pa1 json requests
Pa1 json requestsaiclub_slides The document discusses using Python modules like JSON and Requests to interact with APIs and AI models. It explains that JSON is commonly used to transmit data over the web and can be dumped and loaded from Python dictionaries and strings. The Requests module allows sending HTTP requests in Python to APIs and AI services. It provides an example of sending a request to a mood analysis AI with a payload containing sentence data and converting the response. The document also outlines steps for connecting an AI model in Navigator to a Python application using the integration code snippet.
Overview of python 2019
Overview of python 2019Samir Mohanty This document provides an overview of the Python programming language. It covers Python basics like syntax, datatypes, modules, and control structures. It also discusses topics like functions, classes, files, and popular Python modules. The document contains an agenda that outlines these topics and provides code samples to illustrate Python concepts hands-on. It aims to equip readers with foundational Python programming knowledge.
More from Jimmy Lai (20)
[PyCon US 2025] Scaling the Mountain_ A Framework for Tackling Large-Scale Te...
[PyCon US 2025] Scaling the Mountain_ A Framework for Tackling Large-Scale Te...Jimmy Lai Managing tech debt in large legacy codebases isn’t just a challenge—it’s an ongoing battle that can drain developer productivity and morale. In this talk, I’ll introduce a Python-powered Tech Debt Framework bar-raiser designed to help teams tackle even the most daunting tech debt problems with 100,000+ violations. This open-source framework empowers developers and engineering leaders by: - Tracking Progress: Measure and visualize the state of tech debt and trends over time. - Recognizing Contributions: Celebrate developer efforts and foster accountability with contribution leaderboards and automated shoutouts. - Automating Fixes: Save countless hours with codemods that address repetitive debt patterns, allowing developers to focus on higher-priority work.
Through real-world case studies, I’ll showcase how we: - Reduced 70,000+ pyright-ignore annotations to boost type-checking coverage from 60% to 99.5%. - Converted a monolithic sync codebase to async, addressing blocking IO issues and adopting asyncio effectively.
Attendees will gain actionable strategies for scaling Python automation, fostering team buy-in, and systematically reducing tech debt across massive codebases. Whether you’re dealing with type errors, legacy dependencies, or async transitions, this talk provides a roadmap for creating cleaner, more maintainable code at scale.
PyCon JP 2024 Streamlining Testing in a Large Python Codebase .pdf
PyCon JP 2024 Streamlining Testing in a Large Python Codebase .pdfJimmy Lai The document discusses strategies for streamlining testing in a growing Python codebase at Zip, which has doubled to 2.5 million lines of code with 100 developers. Key strategies include parallel execution of tests, caching dependencies to save time, and skipping unnecessary tests to improve efficiency. The insights aim to enhance quality assurance and developer experience through optimized continuous integration practices.
EuroPython 2024 - Streamlining Testing in a Large Python Codebase
EuroPython 2024 - Streamlining Testing in a Large Python CodebaseJimmy Lai The document outlines strategies for optimizing testing in a large Python codebase at Zip, which consists of 2.5 million lines of code and over 100 developers. It emphasizes the need for quality assurance and efficient testing practices using tools like pytest and continuous integration workflows to handle challenges such as increasing test execution time and coverage. Key strategies discussed include parallel execution, caching dependencies, skipping unnecessary tests, and modernizing runner infrastructure for faster and cost-effective testing.
EuroPython 2022 - Automated Refactoring Large Python Codebases
EuroPython 2022 - Automated Refactoring Large Python CodebasesJimmy Lai Like many companies with multi-million-line Python codebases, Carta has struggled to adopt best practices like Black formatting and type annotation. The extra work needed to do the right thing competes with the almost overwhelming need for new development, and unclear code ownership and lack of insight into the size and scope of type problems add to the burden. We’ve greatly mitigated these problems by building an automated refactoring pipeline that applies Black formatting and backfills missing types via incremental Github pull requests. Our refactor applications use LibCST and MonkeyType to modify the Python syntax tree and use GitPython/PyGithub to create and manage pull requests. It divides changes into small, easily reviewed pull requests and assigns appropriate code owners to review them. After creating and merging more than 3,000 pull requests, we have fully converted our large codebase to Black format and have added type annotations to more than 50,000 functions. In this talk, you’ll learn to use LibCST to build automated refactoring tools that fix general Python code quality issues at scale and how to use GitPython/PyGithub to automate the code review process.
The journey of asyncio adoption in instagram
The journey of asyncio adoption in instagramJimmy Lai The document discusses the adoption of asyncio at Instagram, detailing its challenges and the subsequent benefits like improved API performance and reduced CPU idle time. It outlines key features of asyncio, common myths, and migration strategies from synchronous to asynchronous coding practices. The implementation insights share methods to optimize performance and achieve better concurrency, ultimately leading to enhanced user experience.
Data Analyst Nanodegree
Data Analyst NanodegreeJimmy Lai Hung-Che Lai successfully completed the Data Analyst Nanodegree program from Udacity in 2016. The certificate verifies that Hung-Che Lai learned data analysis skills and discovered insights from data. Sebastian Thrun, CEO of Udacity, certified that Hung-Che Lai completed the program on October 19, 2016.
Distributed system coordination by zookeeper and introduction to kazoo python...
Distributed system coordination by zookeeper and introduction to kazoo python...Jimmy Lai The document provides an overview of Apache ZooKeeper, a coordination service for distributed systems, and introduces the Kazoo Python library for interacting with it. It details common tasks that ZooKeeper simplifies, such as leader election and configuration management, and explains concepts like znodes and session states. Additionally, practical examples demonstrate how to implement distributed coordination patterns like locks and master-worker relationships using ZooKeeper.
Continuous Delivery: automated testing, continuous integration and continuous...
Continuous Delivery: automated testing, continuous integration and continuous...Jimmy Lai 该文档介绍了持续发布的概念，强调了自动化测试、持续集成和版本控制在确保软件交付质量中的重要性。它详细描述了用Python构建部署流水线的步骤，包括单元测试、功能测试和自动部署的实践。文中还提到了一些主流工具，如Git和Jenkins，以支持这些过程的实施。
Build a Searchable Knowledge Base
Build a Searchable Knowledge BaseJimmy Lai The document outlines the construction and search capabilities of a searchable knowledge base, emphasizing the importance of efficient interfaces for data navigation. It discusses specific technologies such as Python, Solr, and DBpedia for data aggregation and search functionalities including string matching, synonym search, and geo search. It concludes with potential applications of such a knowledge base in personal assistants and question answering systems.
[LDSP] Solr Usage
[LDSP] Solr UsageJimmy Lai The document outlines a backend API solution for fast prototyping using Solr, detailing steps for adding a new core, defining the schema, feeding and updating data, and performing queries. Key instructions include creating a core, editing schema.xml, and using JSON format for data updates and queries. The next steps involve understanding Solr's mechanisms and integrating with Django Rest Framework for further development.
[LDSP] Search Engine Back End API Solution for Fast Prototyping
[LDSP] Search Engine Back End API Solution for Fast PrototypingJimmy Lai The document outlines a backend API solution for rapid prototyping of a search engine using Linux, Django, Solr, and Python. It includes installation instructions, data preprocessing steps, and details for creating a geo search API for a Taiwan movie dataset. Further steps for understanding Solr and Django REST framework functionalities are also suggested.
Text classification in scikit-learn
Text classification in scikit-learnJimmy Lai This document provides an overview of text classification in Scikit-learn. It discusses setting up necessary packages in Ubuntu, loading and preprocessing text data from the 20 newsgroups dataset, extracting features from text using CountVectorizer and TfidfVectorizer, performing feature selection, training classification models, evaluating performance through cross-validation, and visualizing results. The goal is to classify newsgroup posts by topic using machine learning techniques in Scikit-learn.
Big data analysis in python @ PyCon.tw 2013
Big data analysis in python @ PyCon.tw 2013Jimmy Lai The document outlines the use of Python for big data analysis, covering key tools and frameworks such as Scrapy for web scraping, MongoDB for database management, and Scikit-learn for machine learning. It discusses the advantages of Python including its readability and productivity, as well as practical examples for data collection, analysis, and visualization. The presentation also provides references and resources for further learning in big data applications with Python.
Text Classification in Python – using Pandas, scikit-learn, IPython Notebook ...
Text Classification in Python – using Pandas, scikit-learn, IPython Notebook ...Jimmy Lai The document discusses text classification in Python utilizing libraries such as pandas, scikit-learn, and matplotlib. It covers the entire process from data collection and feature extraction to classification model training and performance evaluation using iPython notebooks for fast prototyping. Additionally, it includes practical examples, demo code, and references for further learning.
Software development practices in python
Software development practices in pythonJimmy Lai The document outlines software development practices in Python, focusing on runtime environments, source code management, unit testing, coding conventions, documentation, and automation. Key tools discussed include virtualenv, Mercurial for version control, and Sphinx for documentation. The document emphasizes the importance of test coverage, coding standards (PEP 8), and automation through Fabric to enhance development efficiency.
Fast data mining flow prototyping using IPython Notebook
Fast data mining flow prototyping using IPython NotebookJimmy Lai The document discusses a workflow for fast data mining prototyping using IPython Notebook, highlighting its features such as interactive web IDE, code execution flexibility, and data visualization capabilities. It exemplifies text classification using a newsgroup dataset and details various stages involved in data mining along with a demo code for practical application. The goal is to classify articles according to their newsgroup, showcasing results and observations from experiments with feature extraction.
Documentation with sphinx @ PyHug
Documentation with sphinx @ PyHugJimmy Lai The document provides a comprehensive guide on using Sphinx for Python documentation, covering setup, document types, reStructuredText syntax, and autodoc features. It includes instructions for installing Sphinx, configuring Apache, and generating PDF documentation. Key topics include documenting system architecture, usage, classes, modules, and functions.
Apache thrift-RPC service cross languages
Apache thrift-RPC service cross languagesJimmy Lai The document describes the Apache Thrift framework, which enables cross-language RPC services, supporting multiple programming languages such as C++, Java, and Python. It outlines the steps for defining data structures and service interfaces using IDL, generating language bindings, and writing server and client scripts for a machine learning prediction service. An example is provided, demonstrating the process of predicting outcomes using a pre-trained model in a server-client architecture.
NetworkX - python graph analysis and visualization @ PyHug
NetworkX - python graph analysis and visualization @ PyHugJimmy Lai NetworkX is a Python package for analyzing and visualizing graphs and networks. It allows users to construct graphs from data, model network topology and examine properties like centrality and connectivity. The document provides instructions on installing NetworkX and links to tutorials, demonstrates analyzing a social network from a PTT bulletin board, and lists the top users by PageRank centrality.
When big data meet python @ COSCUP 2012
When big data meet python @ COSCUP 2012Jimmy Lai The document presents a presentation by Jimmy Lai on the intersection of big data and Python, outlining the concepts, benefits, and technical issues associated with big data. It discusses various tools, techniques, and libraries available in Python for data collection, storage, processing, and visualization. Additionally, the document highlights initiatives from the White House to support big data education and technology development.
Ad
Recently uploaded (20)
Introduction to Python Programming Language
Introduction to Python Programming Languagemerlinjohnsy This PPT covers features, applications, variable, data types and statements in Python
Call For Papers - 17th International Conference on Wireless & Mobile Networks...
Call For Papers - 17th International Conference on Wireless & Mobile Networks...hosseinihamid192023 17th International Conference on Wireless & Mobile Networks (WiMoNe 2025) will provide
an excellent international forum for sharing knowledge and results in theory, methodology and
applications of Wireless & Mobile computing Environment. Current information age is witnessing
a dramatic use of digital and electronic devices in the workplace and beyond. Wireless, Mobile
Networks & its applications had received a significant and sustained research interest in terms of
designing and deploying large scale and high performance computational applications in real life.
The aim of the conference is to provide a platform to the researchers and practitioners from both
academia as well as industry to meet and share cutting-edge development in the field.
Rapid Prototyping for XR: Lecture 3 - Video and Paper Prototyping
Rapid Prototyping for XR: Lecture 3 - Video and Paper PrototypingMark Billinghurst This is lecture 3 in the course on Rapid Prototyping for XR, taught by Mark Billinghurst on June 10th 2025. This lecture is about Video and Paper prototyping.
Fatality due to Falls at Working at Height
Fatality due to Falls at Working at Heightssuserb8994f It is related to fatality due to falls at working at height
Rapid Prototyping for XR: Lecture 2 - Low Fidelity Prototyping.
Rapid Prototyping for XR: Lecture 2 - Low Fidelity Prototyping.Mark Billinghurst This is lecture 2 on the Rapid Prototyping for XR course taught by Mark Billingurst on June 10th 2025. This lecture is about Low Fidelity Prototyping.
Tesla-Stock-Analysis-and-Forecast.pptx (1).pptx
Tesla-Stock-Analysis-and-Forecast.pptx (1).pptxmoonsony54 this is data science ppt for tesla stock (linear regression)
Deep Learning for Natural Language Processing_FDP on 16 June 2025 MITS.pptx
Deep Learning for Natural Language Processing_FDP on 16 June 2025 MITS.pptxresming1 This gives an introduction to how NLP has evolved from the time of World War II till this date through the advances in approaches, architectures and word representations. From rule based approaches, it advanced to statistical approaches. from traditional machine learning algorithms it advanced to deep neural network architectures. Deep neural architectures include recurrent neural networks, long short term memory, gated recurrent units, seq2seq models, encoder decoder models, transformer architecture, upto large language models and vision language models which are multimodal in nature.
Proposal for folders structure division in projects.pdf
Proposal for folders structure division in projects.pdfMohamed Ahmed Proposal for folders structure division in projects
MATERIAL SCIENCE LECTURE NOTES FOR DIPLOMA STUDENTS
MATERIAL SCIENCE LECTURE NOTES FOR DIPLOMA STUDENTSSAMEER VISHWAKARMA This PPT contains notes for taking lecture at diploma level.
Machine Learning - Classification Algorithms
Machine Learning - Classification Algorithmsresming1 This covers traditional machine learning algorithms for classification. It includes Support vector machines, decision trees, Naive Bayes classifier , neural networks, etc.
It also discusses about model evaluation and selection. It discusses ID3 and C4.5 algorithms. It also describes k-nearest neighbor classifer.
Abraham Silberschatz-Operating System Concepts (9th,2012.12).pdf
Abraham Silberschatz-Operating System Concepts (9th,2012.12).pdfShabista Imam Complete book of operating system edition 9
Introduction to Natural Language Processing - Stages in NLP Pipeline, Challen...
Introduction to Natural Language Processing - Stages in NLP Pipeline, Challen...resming1 Lecture delivered in 2021. This gives an introduction to Natural Language Processing. It describes the use cases of NLP in daily life. It discusses the stages in NLP Pipeline. It highlights the challenges involved covering the different levels of ambiguity that could arise. It also gives a brief note on the present scenario with the latest language models, tools and frameworks/libraries for NLP.
Rapid Prototyping for XR: Lecture 5 - Cross Platform Development
Rapid Prototyping for XR: Lecture 5 - Cross Platform DevelopmentMark Billinghurst This is lecture 5 in the course about Rapid Prototyping for XR, taught by Mark Billinghurst. This lecture is about Cross Platform Development.
Ad
Annotate types in large codebase with automated refactoring
- 1. Annotate Types in Large Codebase with Automated Refactoring Jimmy Lai, Software Engineer at Carta Feb. 9, 2022
- 4. Tech Stack …
- 5. A Large Python Codebase Python code 1.8 million lines 27,000 files 120,000 functions ~200 active developers Lots of TypeError, AttributeError, ValueError
- 6. Type Annotation and Mypy Mypy: Argument 1 to "add" has incompatible type "str"; expected "int"
- 7. Automated Refactoring Automated code changes for fixing large scale tech debt (Code Formatting, Type Annotation, Dead Code Cleanup) LibCST Features: ● Concrete Syntax Tree ● Transformer and Matcher API ● Metadata with static analysis Recommended tool: LibCST A library for modifying Python code easily.
- 8. Code Review with Pull Requests Pull Request Pull Request Pull Request Pull Request
- 13. Add missing types based on static analysis
- 16. MonkeyType: add missing types based on runtime data 1. Collect types by running Python program. 2. Aggregate collected types and apply to the code using LibCST. Run test cases and apply types:
- 17. Make it more fun! Automated weekly updates and leaderboards!
- 18. Fully Typed Function Coverage 2018 2021 automated refactoring
- 19. Production Type Error Improvement
- 20. 20  Carta We are hiring! https://tinyurl.com/carta-jobs Carta Engineering Blog https://medium.com/building-carta Contact: [email protected]