Author Posts

Around this time in 2015, W3C introduced WebAssembly, a small binary format that promises to bring near-native performance to the web. Since then it has been well received by web developers, with some going as far as to say that the "death of JavaScript is near." It is also supported in all the major browsers including Mozilla, Chrome, Safari, and Edge. While WebAssembly was initially designed with the web in mind, it would be a waste not to take its performance and security benefits to go “beyond the web” environments as well. This year we are seeing many initiatives pushing WebAssembly beyond the web. One of them is by Fastly, an edge cloud platform provider. Beginning this year, Fastly open sourced its WebAssembly compiler and runtime, named Lucet. With Lucet, Fastly’s edge cloud can execute tens of thousands of WebAssembly programs simultaneously. We had a great opportunity to interview Fastly’s CTO Tyler McMullen, who gave us insight into why and how they came up with Lucet, what sets it apart from other WebAssembly compilers, the inner workings and design decisions behind Lucet, and more.   Here are some of the highlights from the interview: Benefits of WebAssembly beyond the Web It is exciting to think that we will be able to experience near-native experience on the web. But WebAssembly also aims to solve another major concern of today’s times: security. “WebAssembly was designed for performance, and also for security. WebAssembly programs carry much stronger security guarantees than native code, with comparable performance. That makes it a great candidate for the edge cloud, where we can use the Lucet compiler and runtime to execute WebAssembly programs in isolation from each other, at a much lower resource and performance cost than competing approaches to multi-tenant isolation of native code, like processes, containers, or virtual machines.” Along with these security and performance benefits, the growing support for WebAssembly by compilers like LLVM (since its version 8 release) also makes it suitable for non-web environments. McMullen adds, “Besides security, the other aspect that makes WebAssembly attractive beyond the browser is maturing support by compilers, most notably the LLVM toolchain, used by the Clang C compiler and Rust language compiler, among others. Rather than having to build a new language, or a new compiler, to emit code with the security guarantees we need, we can use the WebAssembly output of any compiler. And it means that tons of existing programs can be compiled to WebAssembly with minimal modification.” How Lucet ensures security With security being one of the major focus areas of Lucet, we asked McMullen how security in Lucet works. “WebAssembly provides a set of guarantees about the security and safety of the code that can be verified during compilation. But those guarantees only hold if verification and compilation are done correctly. Those guarantees also require the runtime to cooperate. So there are a lot of moving pieces here that need to work in concert with each other. Lucet takes a security-by-contract approach to this problem. The compilation phase builds up a set of constraints for the runtime. Those constraints get embedded into the compiled artifact. The runtime then picks up those constraints and enforces them while loading and running the module. This lets us enforce things like which functions a module will be allowed to import for the embedding program, how much memory it will attempt to use, as well as the layout of that memory. So, the security guarantees that Lucet provides end up being enforced with a combination of the compiler, runtime, and the embedding program.” Compilation in Lucet Lucet is designed to compile a code written in C/Rust to WebAssembly and then compile this to native. So, why can’t we directly compile code written in C/Rust to native code? McMullen says that this will give you control over the behavior of the generated code. “If you used a typical C or Rust compiler you’d have relatively little in the way of guarantees about the behavior of the generated code. With Rust you’d have a bit more in that you could guarantee memory safety, but that’s not sufficient by itself. On the other hand, we could certainly create a new C or Rust compiler that guaranteed all the safety guarantees we’ve already discussed, but that would be a tremendous amount of work and would require still more work for each language you wanted to safely compile. We chose WebAssembly because it provides many of the safety and performance guarantees we’re looking for and -- just as importantly -- also has community support. Rather than reinventing the wheel over and over again, we as a community can work together toward a common goal.” Lucet is still in its early stages of development. McMullen shares what the Lucet team is up to now: “Prior to open sourcing Lucet, we focused on WebAssembly programs emitted by a couple of compilers - LLVM via Clang and Rustc, and AssemblyScript. Supporting that subset of WebAssembly was sufficient to launch Terrarium late last year, where users can create complex web services that are compiled and deployed on demand. Since the Lucet announcement, we’ve seen interest and contributions from other languages, including Swift, Golang, Zig, and Wam. We’ve fixed a bunch of the spec compliance issues that blocked these users, and are actively working on fixing the remaining ones now.” To support, or not to support JavaScript, that is the question While building WebAssembly runtimes today, developers have two paths to choose from: either supporting JavaScript or not. Lucet follows the latter one, which helps it be simple yet performant. "Security and resource consumption also drove our design here. Modern, fast JavaScript engines are quite complex, require lots of RAM, startup time, and -- in order to make them fast -- highly advanced JIT compilers. These requirements run counter to what Fastly does. By dropping JavaScript, we can dramatically reduce the complexity and increase the performance of our system. To be clear, reducing complexity isn’t just about making life easier on ourselves. By cutting out the massive complexity of JavaScript we can also reduce the attack surface and increase confidence in our safety guarantees." In the myriad of WebAssembly runtimes, what sets Lucet apart There are currently quite a few WebAssembly runtimes, for instance, Nebulet, Wasmjit, Life, including the ones very similar to Lucet like Wasmer and Wasmtime. We were curious to know what differences Lucet brings to the table. “Lucet was designed from the ground up for multi-tenant, highly concurrent use cases, which matches the runtime requirements of Fastly’s edge cloud. The major design decisions that differentiate it are all focused on performance and resource consumption in our use case, where we need to launch WebAssembly instances for each request our edge cloud handles. Adam Foltzer, a senior software engineer at Fastly, wrote a detailed post on our design and benchmarked its performance here. Lucet shares a major component with the Wasmtime runtime, the Cranelift code generation engine. Wasmtime is currently designed for a single-tenant use case, and supports in-process compilation of WebAssembly, often called JIT. We are collaborating with the maintainers of Wasmtime on Cranelift, and on runtime implementations of the WebAssembly System Interface (WASI).” Why Fastly chose Rust for implementing Lucet Looking at Rust’s memory and thread safety guarantees, a supportive community, and a quickly evolving toolchain, many major projects are being written or rewritten in Rust. One of them is Servo, an HTML rendering engine that will eventually replace Firefox’s rendering engine. Mozilla is also using Rust to rewrite many key parts of Firefox under Project Quantum. More recently, Facebook chose Rust to implement its controversial Libra blockchain. And Fastly’s decision to choose Rust as Lucet’s implementation language was focused on security: “As for why we chose to write Lucet in Rust, the biggest reason was again safety. Writing compilers is complex work. Rust lets us take much of that complexity, describe it with types, and let the Rust compiler check our work in much deeper ways than other languages allow. It lets us focus on the problem we’re trying to solve, rather than the incidental issues of complex software.” Fastly on the future of Rust and WebAssembly In the past few years, Fastly seems to be focusing on Rust and WebAssembly. McMullen believes these languages will be central to the future and will impact key domains in tech. While Rust enables developers to write both highly efficient and safe code, WebAssembly gives you the flexibility of writing code in your choice of language and platform. “With our role in the internet, efficiency is of utmost importance. That’s why, traditionally, the type of software we build has been done with lower level languages like C and C++. We still, today, write and maintain quite a bit of software in C. There are some problems where C is still the correct option. That domain of C -- and to a lesser extent, processor-specific assembly code -- has been largely unassailable for decades as we’ve developed languages that make writing software faster and easier, but at the cost of efficiency. That’s been a great detriment to the entire industry because of how easy it is to write unsafe C code. We believe that Rust has finally been the language to change that. It allows us to write highly efficient code while also providing incredible safety. Now, WebAssembly. WebAssembly has the potential to provide something that we’ve never, in the history of computing, managed to accomplish: a common platform. It was designed to run in a browser, but manages to provide the other components that are needed: efficiency, safety, and platform-independence. We imagine a future in which a WebAssembly module can be run in a browser, on your watch, on your phone, on your TV, in the games you play, and inside server software. We’re still a ways off from that and many pieces are still needed. Lucet is our attempt at providing a WebAssembly compiler and runtime that is made to be used across many different use cases. The first one is Fastly’s edge, but we want to see many more.” Fastly on its other products and projects Limitations in the legacy CDNs that Fastly’s edge cloud platform addresses A CDN or Content Delivery Network consists of a geographically distributed group of servers that work together to ensure that content requested by a user reaches to them as fast as possible. However, it has many limitations like bulky XML based configuration files and specifications. McMullen adds, “Legacy CDNs suffer from a number of technical limitations that make them particularly ill-equipped to address changing consumer expectations, not to mention, developer and enterprise requirements. We’ve all had those online experiences when a site crashes or is non-responsive when we need it most, and our mission is to fuel the next modern digital experience, an experience that’s fast, secure, and reliable. By and large, traditional CDNs are black box solutions that are limited in their ability to provide real-time visibility and control, largely as a result of their outdated architecture, which adds cost and limits developers’ flexibility to expand on functionality.” Fastly’s edge cloud platform is not that -- rather, it aims to address these limitations by bringing data closer to the user. “As a result, developers have not been truly empowered to pursue digital transformations, despite many attempts for improvement within the industry,” he adds. What other projects by Fastly we should look forward to Fastly is continuously contributing towards making the internet better and safer by getting involved in projects like QUIC, Encrypted SNI, and standardizing WASI. Last year Fastly made three of its projects available on Fastly Labs: Terrarium, Fiddle, and Insights. When asked what else it is working on, McMullen shared, “Fastly Labs is heavily dependent on experimentation. If the experiment goes well and we think it’ll be useful for others, then we release it. We have quite a few experiments currently underway, and many of them are around the items listed in the question: ESNI, QUIC, WASI, as well as others like DNS-over-HTTPS. More iteration on what we have now is also in the cards. Lucet has come a long way, but it still has so much room to grow. Expect to see some pretty compelling developments in performance, safety, and features there.” Follow Tyler McMullen on Twitter: @tbmcmullen Learn more about Fastly and its edge-cloud platform at Fastly’s official website. Fastly open sources Lucet, a native WebAssembly compiler and runtime Fastly, edge cloud platform, files for IPO Rust’s original creator, Graydon Hoare on the current state of system programming and safety

The definition of DevOps is a hotly disputed topic among amateur practitioners and experienced engineers alike. Ironically, DevOps was actually supposed to bring some order into the messy and chaotic environment of IT software development. In DevOps Paradox, DevOps expert Viktor Farcic talks to fellow industry figures who reveal their perspectives on the trend and what it means to them. In this article, we’ll see what some prominent people in the DevOps community have to say about DevOps. The quotes in this article are taken directly from the book. So, how are we supposed to incorporate DevOps into our organizations if we don't even know what it is? Let’s hear Viktor’s thoughts about what DevOps is and what are it’s trends and future aspects: What is DevOps and why do we need it? What is DevOps and why do we need it? What is the most important thing DevOps helps us achieve? What are the factors that drive the development of DevOps? Viktor Farcic: Almost everyone gives a different answer to the question “What is DevOps?”—there is a huge discrepancy between the idea and the implementation. I believe that the main objective of DevOps is to enable self-sufficient product-oriented teams capable of having a full control of their products. That is in stark contrast with the way many companies operate today. Normally, a lifecycle of an application is split between many teams. Business analysts define requirements, architects work on guidelines that must be followed and frameworks that must be used, developers write code, testers are in charge of validations, operators deploy new releases, and so on and so forth. The problem is that each of those groups belong to different departments and have different and often opposing objectives. Instead of fostering collaboration towards a common goal, different teams (departments) are looking for their own shortsighted interests. DevOps tries to remove organization based on the type of tasks performed and unite all the expertise required for the whole lifecycle of an application into a single team reporting to a single person. It forces us to work together and it builds empathy. Is DevOps a process? Or a set of technologies? What's your perspective on this area of debate? Viktor: DevOps is neither of the two. Unlike some agile frameworks (e.g. Scrum), there is no prescribed process to follow. Similarly, there is no technology we can adapt that will convert us into “DevOps” teams. It is only an idea that developers, operators, and everyone else needs to work together instead of being isolated in different silos. That does not mean that technology is not important; it is, but often for other than obvious reasons. Every new technology is created by a group of people that worked together to create it. As such, it always reflects processes of those involved in creating it. Those processes, on the other hand, are a result of the culture of the people following it. In other words, every tech is a result of certain processes created as a result of the culture of the team that worked on it. So, even though we use an end result, it is a product of a process created in a specific culture. If we adopt a technology that does not match our own processes and culture, it will produce suboptimal results at best. So, we must either adopt technology that matches our processes and culture or use it to change them. One cannot work without the other. All in all, DevOps is an idea, not much more than that. It’s up to us to figure out which processes and technology will help us make it reality. What does a DevOps engineer do? Is it even a real job role? What are the core roles of DevOps Engineers in terms of development and Infrastructure? Viktor: I don't think there is such a thing as a “DevOps Engineer.” The term was invented by people who were not ready to apply the changes DevOps leads to. Most of the time, a “DevOps Engineer” is just a different name for someone working in shared services, operations, infrastructure, or whichever department was first to be renamed into DevOps. Do you think DevOps is experiencing an identity crisis? Viktor: DevOps was never defined as a process. Agile, for example, got quite a few implementations that tell people what to do. Among others, we got Scrum that clearly defines what to do. We could even argue that Scrum, as being a set of practices that must be followed, is against the spirit of Agile, but that’s a conversation for some other time. What matters is that no one defined the process behind DevOps. There is no such thing as a set of steps that must be followed daily or weekly. It is just an idea that we should work together and not throw things over department walls. As such, the way to accomplish that is open to interpretation. So, DevOps never had a clear identity, so it cannot have an identity crisis either. It’s just an idea, and it’s up to each one of us to try to figure out how to make it reality. The biggest challenges in DevOps today What are the biggest challenges in DevOps at the moment? Viktor: Currently, DevOps is mostly misunderstood. More often than not, companies just rename a department. In some companies, shared services become DevOps teams; in others it is infrastructure, operations, or any other department. It’s as if it was a race and the first department to change their name into “DevOps” was a winner. Logically, changing the name means nothing and does not result in any tangible improvement. The key challenges are related to people and culture. DevOps is not easy because it challenges current organizational structure, it restructures power within an organization, and it questions the need for the existence of many departments. As such, middle management is often against it because it is perceived as a risk to their position. At the same time, people who spent many years doing the same thing over and over again feel that their credibility is at risk if the structure that allowed them to climb company ladder is removed. Congratulations on the release of DevOps Paradox. Could you talk a little bit about the idea behind it and what you hope it achieves? Viktor: I go to a lot of conferences and I realized that scheduled talks are not the main takeaway from them. True, I learned things by listening to them, but the primary reason I continue attending are “corridor talks.” Conferences are a great opportunity for me to find interesting people and have amazing discussions. Unlike scheduled talks, those conversations are not structured. I do not prepare a list of questions for the next person I’ll meet in between talks or at a party. Instead, we’d just start talking about a random thing that happens to be interesting. I wanted to bring those types of conversations to people who cannot travel the world and be every moth in a different conference in a different country. So, I did not have any real goals for this book, other than speaking with people about any topic, as long as it is related to DevOps. Since DevOps can be anything related to software development, you can say that the scope of the book is as broad as it can be. My true goal was to enjoy having conversations with people. I did not prepare questions in advance. Instead, I just gathered people I’d like to speak with if I’d meet them in a conference and say, “Let’s have a coffee and see what you were up to since the last time we met”. Some of those I interviewed are my friends, while others I met for the first time. Some work for huge enterprises, while others work in startups. Some worked in software industry for many years, while others are young up-and-coming experts. I wanted to make sure that the book gives as many different opinions as possible. Find Viktor Farcic's DevOps Paradox on the Packt store. Read the first chapter for free on the Packt subscription platform.

IntroductionNavigating the world of data and AI systems can be overwhelming.Their complexity often makes it difficult to visualize how data engineering, research (data science and machine learning), and production roles (AI engineering, ML engineering, MLOps) work together to form an end-to-end system.As a data engineer, your work finishes when standardized data is ingested into a data warehouse or lake.As a researcher, your work ends after training the optimal model on a static dataset and registering it.As an AI or ML engineer, deploying the model into production often signals the end of your responsibilities.As an MLOps engineer, your work finishes when operations are fully automated and adequately monitored for long-term stability.But is there a more intuitive and accessible way to comprehend the entire end-to-end data and AI ecosystem?Absolutely—through the FTI architecture.Let’s quickly dig into the FTI architecture and apply it to a production LLM & RAG use case. Figure 1: The mess of bringing structure between the common elements of an ML system.Introducing the FTI architectureThe FTI architecture proposes a clear and straightforward mind map that any team or person can follow to compute the features, train the model, and deploy an inference pipeline to make predictions.The pattern suggests that any ML system can be boiled down to these 3 pipelines: feature, training, and inference.This is powerful, as we can clearly define the scope and interface of each pipeline. Ultimately, we have just 3 instead of 20 moving pieces, as suggested in Figure 1, which is much easier to work with and define.Figure 2 shows the feature, training, and inference pipelines. We will zoom in on each one to understand its scope and interface.Figure 2: FTI architectureBefore going into the details, it is essential to understand that each pipeline is a separate component that can run on different processes or hardware. Thus, each pipeline can be written using a different technology, by a different team, or scaled differently.The feature pipelineThe feature pipeline takes raw data as input, processes it, and outputs the features and labels required by the model for training or inference.Instead of directly passing them to the model, the features and labels are stored inside a feature store. Its responsibility is to store, version, track, and share the features.By saving the features into a feature store, we always have a state of our features. Thus, we can easily send the features to the training and inference pipelines.The training pipelineThe training pipeline takes the features and labels from the features stored as input and outputs a trained model(s).The models are stored in a model registry. Its role is similar to that of feature stores, but the model is the first-class citizen this time. Thus, the model registry will store, version, track, and share the model with the inference pipeline.The inference pipelineThe inference pipeline takes as input the features and labels from the feature store and the trained model from the model registry. With these two, predictions can be easily made in either batch or real-time mode.As this is a versatile pattern, it is up to you to decide what you do with your predictions. If it’s a batch system, they will probably be stored in a DB. If it’s a real-time system, the predictions will be served to the client who requested them.The most important thing you must remember about the FTI pipelines is their interface. It doesn’t matter how complex your ML system gets — these interfaces will remain the same.The final thing you must understand about the FTI pattern is that the system doesn’t have to contain only 3 pipelines. In most cases, it will include more.For example, the feature pipeline can be composed of a service that computes the features and one that validates the data. Also, the training pipeline can comprise the training and evaluation components.Applying the FTI architecture to a use caseThe FTI architecture is tool-agnostic, but to better understand how it works, let’s present a concrete use case and tech stack.Use case: Fine-tune an LLM on your social media data (LinkedIn, Medium, GitHub) and expose it as a real-time RAG application. Let’s call it your LLM Twin.As we build an end-to-end system, we split it into 4 pipelines:The data collection pipeline (owned by the DE team)The FTI pipelines (owned by the AI teams)As the FTI architecture defines a straightforward interface, we can easily connect the data collection pipeline to the ML components through a data warehouse, which, in our case, is a MongoDB NoSQL DB.The feature pipeline (the second ML-oriented data pipeline) can easily extract standardized data from the data warehouse and preprocess it for fine-tuning and RAG.The communication between the two is done solely through the data warehouse. Thus, the feature pipeline isn’t aware of the data collection pipeline and how it collected the raw data. Figure 3: LLM Twin high-level architectureThe feature pipeline does two things:chunks, embeds and loads the data to a Qdrant vector DB for RAG;generates an instruct dataset and loads it into a versioned ZenML artifact.The training pipeline ingests a specific version of the instruct dataset, fine-tunes an open-source LLM from HuggingFace, such as Llama 3.1, and pushes it to a HuggingFace model registry.During the research phase, we use a Comet ML experiment tracker to compare multiple fine-tuning experiments and push only the best one to the model registry.During production, we can automate the training job and use our LLM evaluation strategy or canary tests to check if the new LLM is fit for production.As the input dataset and output model registry are decoupled, we can quickly launch our training jobs using ML platforms like AWS SageMaker.ZenML orchestrates the data collection, feature, and training pipelines. Thus, we can easily schedule them or run them on demand orThe end-to-end RAG application is implemented in the inference pipeline side, which accesses fresh documents from the Qdrant vector DB and the latest model from the HuggingFace model registry.Here, we can implement advanced RAG techniques such as query expansion, self-query and rerank to improve the accuracy of our retrieval step for better context during the generation step.The fine-tuned LLM will be deployed to AWS SageMaker as an inference endpoint. Meanwhile, the rest of the RAG application is hosted as a FastAPI server, exposing the end-to-end logic as REST API endpoints.The last step is to collect the input prompts and generated answers with a prompt monitoring tool such as Opik to evaluate the production LLM for things such as hallucinations, moderation or domain-specific problems such as writing tone and style.SummaryThe FTI architecture is a powerful mindmap that helps you connect the dots in the complex data and AI world, as illustrated in the LLM Twin use case.Unlock the full potential of Large Language Models with the "LLM Engineer's Handbook" by Paul Iusztin and Maxime Labonne. Dive deeper into real-world applications, like the FTI architecture, and learn how to seamlessly connect data engineering, ML pipelines, and AI production. With practical insights and step-by-step guidance, this handbook is an essential resource for anyone looking to master end-to-end AI systems. Don’t just read about AI—start building it. Get your copy today and transform how you approach LLM engineering!Author BioPaul Iusztin is a senior ML and MLOps engineer at Metaphysic, a leading GenAI platform, serving as one of their core engineers in taking their deep learning products to production. Along with Metaphysic, with over seven years of experience, he built GenAI, Computer Vision and MLOps solutions for CoreAI, Everseen, and Continental. Paul's determined passion and mission are to build data-intensive AI/ML products that serve the world and educate others about the process. As the Founder of Decoding ML, a channel for battle-tested content on learning how to design, code, and deploy production-grade ML, Paul has significantly enriched the engineering and MLOps community. His weekly content on ML engineering and his open-source courses focusing on end-to-end ML life cycles, such as Hands-on LLMs and LLM Twin, testify to his valuable contributions.

Key takeaways The transformer architecture has proved to be revolutionary in outperforming the classical RNN and CNN models in use today. Artificial intelligence is simply a recent form of automation, just like all other automation. AI consultants will always be necessary to implement AI. Understand transformers from a cognitive science perspective with the book Transformers for Natural Language Processing. The transformer architecture is both revolutionary and disruptive making it the hottest Algorithm in AI. It is a game-changer for Natural Language Understanding (NLU), a subset of Natural Language Processing (NLP), which has become one of the pillars of artificial intelligence in a global digital economy.​ Transformers can outperform the classical RNN and CNN models in use today. We interviewed artificial intelligence expert Denis Rothman about transformers, it's advancement in artificial intelligence & NLP, and his recent book Transformers for Natural Language Processing. What's the significance of AI language understanding in the tech world today and what role do transformers play in it? Artificial intelligence-driven language understanding is expanding exponentially. It has become the pillar of language modeling, chatbots, personal assistants, question answering, text summarizing, speech-to-text, sentiment analysis, machine translation, and more. The Transformer, introduced by Google, provides novel approaches to language understanding through a novel self-attention architecture. OpenAI offers transformer technology, and Facebook's AI Research department provides high-quality datasets. Overall, the Internet giants have made transformers available to all, as you will discover in my book. The transformer architecture is both revolutionary and disruptive. The Transformer and subsequent transformer architectures and models are revolutionary because they changed the way we think of NLP and artificial intelligence itself. The architecture of the Transformer is not an evolution. It breaks with the past, leaving RNNs and CNNs behind. It takes us closer to seamless machine intelligence that will match human intelligence in the years to come. What should deep learning & NLP practitioners keep in mind while starting their career with transformers? The world of artificial intelligence is undergoing an exponential evolution in NLP due to the amount of data available. As this evolution expands to all domains, new abilities are required. NLP will not just be about downloading a model and getting to work in terms of software. You will have to analyze the quality of what a transformer model produces to fine-tune it. In turn, to analyze NLP properly, a minimum knowledge in linguistics will become mandatory. Linguistics will enable you to understand the building blocks and structure of a language. Grammar will increase your ability to analyze the output of a transformer.  Otherwise, your team will have to hire a linguist, which will increase the project's cost and threaten the Return On Investment(ROI) of the team. What are some future advancements that you anticipate in transformers and NLP? Transformers have wiped RNNs off the map at this point. They represent the industrialization of artificial intelligence. As artificial intelligence, transformers are taking AI from the hype to an industrial level. Unlike traditional deep learning models, transformers contain optimized layers for GPUs and CPUs. In the future, creating NLP models will require machine architecture awareness. Machine performance will be the key to more efficient models. Not everybody can purchase or rent a supercomputer to train a model. Learning how to design tailored transformer models based on optimized datasets will become mandatory to face competition. What are some of the popular myths around transformers prevalent in the tech market? Many people believe that transformers can perform all NLP tasks with a model such as GPT-3. Nothing can be further from the truth. Google, Microsoft, Facebook, and Amazon, for example, need data for their everyday business and powerful NLP transformer models to analyze the billions of words coming in every day. However, the tasks are limited to their marketing usage. If you need to implement a transformer in a specific area, you will have to build datasets. You will also have to build pipelines with classical algorithms and queries to process the data, the inputs, and manage the outputs. In real-life, that means that artificial intelligence is only a component in a long chain of classical algorithms and processes. How was your experience building one of the very first word2matrix embedding solutions? In the early 1980s, I managed a company with many students who wanted to learn a language. I had a choice. Increase the number of teachers or automate vast portions of the process. I decided to go for automation. Any intelligent system requires calculations. I found that converting words and word pieces into numbers was far more efficient than directly analyzing the words. I thus create a word2vector system, patented it in 1982, wrote a textbook, and implemented it in our company. Students began to take specific courses independently in our lab without a teacher. I then went further in the next few years, writing one of the first Cognitive NLP Chatbots with was successfully implemented for an industrial amount of students. Being the author of three cutting-edge AI solutions, what is your take on the shrinkage of job opportunities due to AI? Automation began centuries ago with water mills, windmills, textile machines, locomotives, and more recently, motorized personal vehicles in the early 20th century. Tractors replaced millions of jobs in the fields. Services are no exception. In the 1950s, hundreds of thousands of tellers, actual humans, worked in banks around the world. Today everybody goes to an ATM. ATM stands for Automated Teller Machine(ATM). “Automated teller,” says it all. A person performing a service was automated. Software is the automation of human tasks from the beginning, from accounting to stock market management and thousands of tasks. Artificial intelligence is simply a recent form of automation, just like all other automation. AI cannot replace traditional mathematics in physics. The calculation of differential equations driving rockets and satellites requires classical software precision, not artificial intelligence. AI is only a component of automation, like when cars replaced horses and all of the jobs that went with horse-driven transportation. AI will not replace everything because AI is useless in many fields. AI consultants will always be necessary to implement AI. Why has Python become the most suitable language for natural language processing? It’s important not to confuse the concepts of “most used” and “most suitable.” Python is a great intuitive language to learn AI and NLP. But it’s not a prerequisite. Python is easy to use and run, making it the shortest path, at this point, to take to learn AI. But do not be mistaken. C++ skills will also be required in large real-life projects, for example. My advice. Learn AI with Python at full speed. Do some implementations with Python. But learn other languages such as C++, Java, and more. Real-life pipelines require classical processes and algorithms, not only AI. In some projects, C++ will boost performances, for example. Tell us about your book, Transformers for Natural Language Processing. What trajectory does your book follow to help its readers master transformers? Reading my book on transformers will help you save weeks and maybe months of effort trying to understand how they work by watching videos and reading blogs. The reader will begin by learning the original Transformer in depth. Once the transformer's building blocks are mastered, the reader will learn how to train and fine-tune a transformer. The reader will then build and run the main transformer models such as BERT, RoBERTa, GPT-2, T5, and more. The models will be applied to NLP tasks such as document summarization, Q&As, semantic analysis, and a wide range of NLP tasks. The book contains a method to analyze fake news with transformers. The book also goes beyond the architecture of transformers and into the world of usage. You will learn how to build, train, fine-tune, and implement transformers.

Over the past few years, we have seen some advanced technologies in artificial intelligence shaping human life. Deep learning (DL) has become the main driving force in bringing new innovations in almost every industry. We are sure to continue to see DL everywhere. Most of the companies including startups are already integrating deep learning into their own day-to-day process. Deep learning techniques and algorithms have made building advanced neural networks practically feasible, thanks to high-level open source libraries such as TensorFlow, Keras, PyTorch and more. We recently interviewed Valentino Zocca, a deep learning expert and the author of the book, Python Deep Learning. Valentino explains why deep learning is getting so much hype, and what's the roadmap ahead in terms of new technologies and libraries. He will also talks about how major vendors and tech-savvy startups adopt deep learning within their organization. Being a consultant and an active developer he is expecting a better approach than back-propagation for carrying out various deep learning tasks. Author’s Bio Valentino Zocca graduated with a Ph.D. in mathematics from the University of Maryland, USA, with a dissertation in symplectic geometry, after having graduated with a laurel in mathematics from the University of Rome. He spent a semester at the University of Warwick. After a post-doc in Paris, Valentino started working on high-tech projects in the Washington, D.C. area and played a central role in the design, development, and realization of an advanced stereo 3D Earth visualization software with head tracking at Autometric, a company later bought by Boeing. At Boeing, he developed many mathematical algorithms and predictive models, and using Hadoop, he has also automated several satellite-imagery visualization programs. He has since become an expert on machine learning and deep learning and has worked at the U.S. Census Bureau and as an independent consultant both in the US and in Italy. He has also held seminars on the subject of machine learning and deep learning in Milan and New York. Currently, Valentino lives in New York and works as an independent consultant to a large financial company, where he develops econometric models and uses machine learning and deep learning to create predictive models. But he often travels back to Rome and Milan to visit his family and friends. Key Takeaways Deep learning is one of the most adopted techniques used in image and speech recognition and anomaly detection research and development areas. Deep learning is not the optimum solution for every problem faced. Based on the complexity of the challenge, the neural network building can be tricky. Open-source tools will continue to be in the race when compared to enterprise software. More and more features are expected to improve on providing efficient and powerful deep learning solutions. Deep learning is used as a tool rather than a solution across organizations. The tool usage can differ based on the problem faced. Emerging specialized chips expected to bring more developments in deep learning to mobile, IoT and security domain. Valentino Zocca states We have a quantity vs. quality problem. We will be requiring better paradigms and approaches in the future which can be improved through research driven innovative solutions instead of relying on hardware solutions. We can make faster machines, but our goal is really to make more intelligent machines for performing accelerated deep learning and distributed training. Full Interview Deep learning is as much infamous as it is famous in the machine learning community with camps supporting and opposing the use of DL passionately. Where do you fall on this spectrum? If you were given a chance to convince the rival camp with 5-10 points on your stand about DL, what would your pitch be like? The reality is that Deep Learning techniques have their own advantages and disadvantages. The areas where Deep Learning clearly outperforms most other machine learning techniques are in image and speech recognition and anomaly detection. One of the reasons why Deep Learning does so much better is that these problems can be decomposed into a hierarchical set of increasingly complex structures, and, in multi-layer neural nets, each layer learns these structures at different levels of complexity. For example, an image recognition, the first layers will learn about the lines and edges in the image. The subsequent layers will learn how these lines and edges get together to form more complex shapes, like the eyes of an animal, and finally the last layers will learn how these more complex shapes form the final image. However, not every problem can suitably be decomposed using this hierarchical approach. Another issue with Deep Learning is that it is not yet completely understood how it works, and some areas, for example, banking, that are heavily regulated, may not be able to easily justify their predictions. Finally, many neural nets may require a heavier computational load than other classical machine learning techniques. Therefore, the reality is that one still needs a proficient machine learning expert who deeply understands the functioning of each approach and can make the best decision depending on each problem. Deep Learning is not, at the moment, a complete solution to any problem, and, in general, there can be no definite side to pick, and it really depends on the problem at hand. Deep learning can conquer tough challenges, no doubt. However, there are many common myths and realities around deep learning. Would you like to give your supporting reasoning on whether the following statements are myth or fact? You need to be a machine learning expert or a math geek to build deep learning models We need powerful hardware resources to use deep learning Deep learning models are always learning, they improve with new data automagically Deep learning is a black box, so we should avoid using it in production environments or in real-world applications. Deep learning is doomed to fail. It will be replaced eventually by data sparse, resource economic learning methods like meta-learning or reinforcement learning. Deep learning is going to be central to the progress of AGI (artificial general intelligence) research Deep Learning has become almost a buzzword, therefore a lot of people are talking about it, sometimes misunderstanding how it works. People hear the word DL together with "it beats the best player at go", "it can recognize things better than humans" etc., and people think that deep learning is a mature technology that can solve any problem. In actuality, deep learning is a mature technology only for some specific problems, you do not solve everything with deep learning and yet at times, whatever the problem, I hear people asking me "can't you use deep learning for it?" The truth is that we have lots of libraries ready to use for deep learning. For example, you don’t need to be a machine learning expert or a math geek to build simple deep learning models for run-of-the-mill problems, but in order to solve for some of the challenges that less common issues may present, a good understanding of how a neural network works may indeed be very helpful. Like everything, you can find a grain of truth in each of those statements, but they should not be taken at face value. With MLaaS being provided by many vendors from Google to AWS to Microsoft, deep learning is gaining widespread adoption not just within large organizations but also by data-savvy startups. How do you view this trend? More specifically, is deep learning being used differently by these two types of organizations? If so, what could be some key reasons? Deep Learning is not a monolithic approach. We have different types of networks, ANNs, CNNs, LSTMs, RNNs, etc. Honestly, it makes little sense to ask if DL is being used differently by different organizations. Deep Learning is a tool, not a solution, and like all tools it should be used differently depending on the problem at hand, not depending on who is using it. There are many open source tools and enterprise software (especially the ones which claim you don't need to code much) in the race. Do you think this can be the future where more and more people will opt for ready-to-use (MLaaS) enterprise backed cognitive tools like IBM Watson rather than open-source tools? This holds true for everything. At the beginning of the internet, people would write their own HTML code for their web pages, now we use tools who do most of the work for us. But if we want something to stand-out we need a professional designer. The more a technology matures, the more ready-to-use tools will be available, but that does not mean that we will never need professional experts to improve on those tools and provide specialized solutions. Deep learning is now making inroads to mobile, IoT and security domain as well. What makes DL great for these areas? What are some challenges you see while applying DL in these new domains? I do not have much experience with DL in mobiles, but that is clearly a direction that is becoming increasingly important. I believe we can address these new domains by building specialized chips. Deep learning is a deeply researched topic within machine learning and AI communities. Every year brings us new techniques from neural nets to GANs, to capsule networks that then get widely adopted both in research and in real-world applications. What are some cutting-edge techniques you foresee getting public attention in deep learning in 2018 and in the near future? And why? I am not sure we will see anything new in 2018, but I am a big supporter of the idea that we need a better paradigm that can excel more at inductive reasoning rather than just deductive reasoning. At the end of last year, even DL pioneer Geoff Hinton admitted that we need a better approach than back-propagation, however, I doubt we will see anything new coming out this year, it will take some time. We keep hearing noteworthy developments in AI and deep learning by DeepMind and OpenAI. Do you think they have the required armory to revolutionize how deep learning is performed? What are some key challenges for such deep learning innovators? As I mentioned before, we need a better paradigm, but what this paradigm is, nobody knows. Gary Marcus is a strong proponent of introducing more structure in our networks, and I do concur with him, however, it is not easy to define what that should be. Many people want to use the brain as a model, but computers are not biological structures, and if we had tried to build airplanes by mimicking how a bird flies we would not have gone very far. I think we need a clean break and a new approach, I do not think we can go very far by simply refining and improving what we have. Improvement in processing capabilities and the availability of custom hardware have propelled deep learning into production-ready environments in recent years. Can we expect more chips and other hardware improvements in the coming years for GPU accelerated deep learning and distributed training? What other supporting factors will facilitate the growth of deep learning? Once again, foreseeing the future is not easy, however, as these questions are related, I think only so much can be gained by improving chips and GPUs. We have a quantity vs. quality problem. We can improve quantity (of speed, memory, etc.) through hardware improvements, but the real problem is that we need a real quality improvement, better paradigms, and approaches, that needs to be achieved through research and not with hardware solutions. We can make faster machines, but our goal is really to make more intelligent machines. A child can learn by seeing just a few examples, we should be able to create an approach that allows a machine to also learn from few examples, not by cramming millions of examples in a short time. Would you like to add anything more to our readers? Deep Learning is a fascinating discipline, and I would encourage anyone who wanted to learn more about it to approach it as a research project, without underestimating his or her own creativity and intuition. We need new ideas. If you found this interview to be interesting, make sure you check out other insightful interviews on a range of topics: Blockchain can solve tech’s trust issues – Imran Bashir “Tableau is the most powerful and secure end-to-end analytics platform”: An interview with Joshua Milligan “Pandas is an effective tool to explore and analyze data”: An interview with Theodore Petrou

Microsoft Windows is one of the two most common OSes, and managing its security has spawned the discipline of Windows security. Kali Linux is the premier platform for testing and maintaining Windows security. Kali is built on the Debian distribution of Linux and shares the legendary stability of that OS. This lets you focus on network penetration, password cracking, and using forensics tools, and not the OS. In this interview, we talk to two experts, Wolf Halton and Bo Weaver, on using Kali Linux for pentesting. We also discuss their book Kali Linux 2018: Windows Penetration Testing - Second Edition. Read also: Kali Linux 2018 for testing and maintaining Windows security - Interview with Wolf Halton and Bo Weaver - Part 2 Kali Linux is the premier platform for testing and maintaining Windows security. According to you, what makes it ideal to use? Bo Weaver: First, it runs on Linux and is built on Debian Linux.  Second, the people at Offensive Security do a fantastic job of keeping it updated and stable with the latest tools to support not just pentesting but also forensics work or network analysis and diagnostics.  You can tell that this platform is built and maintained by real security experts and isn’t some distro thrown together by some marketing folks to make a buck. Wolf Halton: Kali is a very stable and extensible open source platform.  Offensive Security’s first security platform, BackTrack, was customised in a non-Posix way, breaking from UNIX or other Linux distros by putting the security tools in unexpected places in the filesystem.  Since Kali was first released, they used Debian Testing as a base, and adhered to the usual file locations. This made Kali Linux far easier to use. The normalization of the OS behind the Kali Linux distro makes it more productivity-friendly than most of the other “Security Distros,” which are usually too self-consciously different. Here, the developers are building their space in the mass of distros by how quirky the interface or how customizable the installation process has to be. Why do you love working with Kali Linux? Bo Weaver: I appreciate it’s stability.  In all the years I have used Kali on a daily basis, I have had only one failure to update properly.  Even with this one failure, I didn’t have any data loss. I run Kali as my “daily driver” on both my personal and company laptop, so one failure in all that time is nothing.  I even do my writing from my Kali machines. Yes I do all my normal computing from a normal user account and NOT root! I don’t have to go looking for a tool. Any tool that I need is either installed or is in the repo.  Since everything comes from the same repo, updates to all my tools and the system is just a simple command to keep everything updated. Wolf Halton: Kali is a stable platform, based upon a major distribution with which I am very familiar.  There are over 400 security tools in the Kali repos, and it can also draw directly from the Debian Testing repos for even more tools.  I always add a few applications on top of the installation default set of packages, but the menus work predictably, allowing me to install what I need without having to create a whole new menu system to get to them. Can you tell the readers about some advantages and disadvantages of using Kali Linux for pentesting? Bo Weaver: I really can’t think of a disadvantage. The biggest advantage is that all these tools are in one toolbox (Kali). I remember a time when building a pentesting machine would take a week, having to go out, and find and build the tools separately.  Most tools had to be manually compiled for the machine. Remember “make”, “make install”? Then to have it bork over a missing library file. In less than an hour, you can have a working pentesting machine running. As mentioned earlier, Kali has the tools to do any security job, not just pentesting, such as pulling evidence from a laptop for legal reasons,  analyzing a network, finding what is breaking your network, breaking into a machine because the passwords are lost. Also, it runs on anything from a high-end workstation to a Raspberry Pi or a USB drive with no problem. Wolf Halton: The biggest disadvantage is for Windows-Centric users who have never used any other operating system.  In our book, we try to ease these users into the exciting world of Linux. The biggest advantage is that the Kali Linux distro is in constant development.  I can be sure that there will be a Kali distro available even if I wander off for a year.  This is a great benefit for people who only use Linux when they want to run an ad hoc penetration test. Can you give us a specific example (real or fictional) of why Kali Linux is the ideal solution to go for? Bo Weaver: There are other distros out there for this use.  Most don’t have the completeness of toolsets. Most security distros are set up to be run from a DVD and only contain a few tools to do a couple of tasks and not all security tasks.  BlackArch Linux is the closest to Kali in comparison. BlackArch is built on Arch Linux which is a bleeding-edge distro which doesn’t have the stability of Debian.  Sometimes Arch will bork on an update due to bleeding-edge buggy code in an update. This is fine in a testing environment but when working in production, you need your system to run at the time of testing.  It’s embarrassing to call the customer and say you lost three hours on a test fixing your machine. I’m not knocking BlackArch. They did a fine job on the build and the toolsets included. I just don’t trust Arch to be stable enough for me.  This is not saying anything bad about Arch Linux. It does have its place in the distro world and does a fine job of filling its place in this world. Some people like bleeding edge, it’s just a personal choice. The great thing about Linux overall is that you have choices.  You’re not locked into one way a system looks or works. Kali comes with five different desktop environments, so you can choose which one is the best for you.  I personally like KDE. Wolf Halton: I have had to find tools for various purposes: Tools to recover data from failed hard-drives, Tools to stress-test hundreds of systems at a time, Tools to check whole networks at a time for vulnerabilities, Tools to check for weak passwords, Tools to perform Phishing tests on email users, Tools to break into Windows machines, security appliances and network devices. Kali Linux is the one platform where I could find multiple tools to perform all of these tasks and many more. Congratulations on your recent book, Kali Linux 2018: Windows Penetration Testing - Second Edition. Can you elaborate on the key takeaways for readers? Bo Weaver: I hope the readers come out with a greater understanding of system and network security and how easy it is to breach a system if simple and proper security rules are not followed.  By following simple no-cost rules like properly updating your systems and proper network segmentation, you can defeat most of the exploits in the book. Over the years, Wolf and I have been asked by a lot of Windows Administrators “How do you do a pentest?”  This person doesn’t want a simple glossed over answer. They are an engineer and understand their systems and how they work; they want a blow by blow description on actually how you broke it, so they can understand the problem and properly fix it.  The book is the perfect solution for them. It contains methods we use in our work on a daily basis, from scanning to post exploitation work. Also, I hope the readers find how easy Linux is to use as a desktop workstation and the advantages in security when using Linux as your workstation OS and do the switch from Windows to the Linux Desktop. I want to thank the readers of our book and hope they walk away with a greater understanding of system security. Wolf Halton: The main thing we tried to do with both the first and second edition of this book is to give a useful engineer-to-engineer overview of the possibilities of using Kali to test one’s own network, and including very specific approaches and methods to prove their network’s security.  We never write fictionalized, unworkable testing scenarios, as we believe our readers want to actually know how to improve their craft and make their networks safer, even though there is no budget for fancy-schmancy proprietary Windows-based security tools that make their non-techie managers feel safer. The world of pentesting is still edgy and interesting, and we try to infuse the book with our own keen interest in testing and developing attack models before the Red-Team hackers get there. Thanks Bo and Wolf for a very insightful perspective into the world of pentesting and on Kali Linux! Readers, if you are looking for help to quickly pentest your system and network using easy-to-follow instructions and support images, Kali Linux 2018: Windows Penetration Testing - Second Edition might just be the book for you. Author Bio Wolf Halton is an Authority on Computer and Internet Security, a best selling author on Computer Security, and the CEO of Atlanta Cloud Technology. He specializes in—business continuity, security engineering, open source consulting, marketing automation, virtualization and data center restructuring, network architecture, and Linux administration. Bo Weaver is an old school ponytailed geek. His first involvement with networks was in 1972 while in the US Navy working on a R&D project called ARPA NET. Bo has been working with and using Linux daily since the 1990's and a promoter of Open Source. (Yes, Bo runs on Linux.) He now works as the senior penetration tester and security researcher for CompliancePoint a Atlanta based security consulting company. Pentest tool in focus: Metasploit Kali Linux 2018.2 released How artificial intelligence can improve pentesting

This article is an excerpt from the book, "RAG-Driven Generative AI", by Denis Rothman. Explore the transformative potential of RAG-driven LLMs, computer vision, and generative AI with this comprehensive guide, from basics to building a complex RAG pipeline.IntroductionOn a bright Monday morning, Dakota sits down to get to work and is called by the CEO of their software company, who looks quite worried. An important fire department needs a conversational AI agent to train hundreds of rookie firefighters nationwide on drone technology. The CEO looks dismayed because the data provided is spread over many websites around the country. Worse, the management of the fire department is coming over at 2 PM to see a demonstration to decide whether to work with Dakata’s company or not. Dakota is smiling. The CEO is puzzled.  Dakota explains that the AI team can put a prototype together in a few hours and be more than ready by 2 PM and get to work. The strategy is to divide the AI team into three sub-teams that will work in parallel on three pipelines based on the reference Deep Lake, LlamaIndex and OpenAI RAG program* they had tested and approved a few weeks back.  Pipeline 1: Collecting and preparing the documents provided by the fire department for this Proof of Concept(POC). Pipeline 2: Creating and populating a Deep Lake vector store with the first batch of documents while the Pipeline 1 team continues to retrieve and prepare the documents. Pipeline 3: Indexed-based RAG with LlamaIndex’s integrated OpenAI LLM performed on the first batch of vectorized documents. The team gets to work at around 9:30 AM after devising their strategy. The Pipeline 1 team begins by fetching and cleaning a batch of documents. They run Python functions to remove punctuation except for periods and noisy references within the content. Leveraging the automated functions they already have through the educational program, the result is satisfactory.  By 10 AM, the Pipeline 2 team sees the first batch of documents appear on their file server. They run the code they got from the RAG program* to create a Deep Lake vector store and seamlessly populate it with an OpenAI embedding model, as shown in the following excerpt: from llama_index.core import StorageContext vector_store_path = "hub://denis76/drone_v2" dataset_path = "hub://denis76/drone_v2" # overwrite=True will overwrite dataset, False will append it vector_store = DeepLakeVectorStore(dataset_path=dataset_path, overwrite=True)  Note that the path of the dataset points to the online Deep Lake vector store. The fact that the vector store is serverless is a huge advantage because there is no need to manage its size, storage process and just begin to populate it in a few seconds! Also, to process the first batch of documents, overwrite=True, will force the system to write the initial data. Then, starting the second batch,  the Pipeline 2 team can run overwrite=False, to append the following documents. Finally,  LlamaIndex automatically creates a vector store index: storage_context = StorageContext.from_defaults(vector_store=vector_store) # Create an index over the documents index = VectorStoreIndex.from_documents(documents, storage_context=storage_context) By 10:30 AM, the Pipeline 3 team can visualize the vectorized(embedded) dataset in their Deep Lake vector store. They create a LlamaIndex query engine on the dataset: from llama_index.core import VectorStoreIndex vector_store_index = VectorStoreIndex.from_documents(documents) … vector_query_engine = vector_store_index.as_query_engine(similarity_top_k=k, temperature=temp, num_output=mt) Note that the OpenAI Large Language Model is seamlessly integrated into LlamaIndex with the following parameters: k, in this case, k=3, specifies the number of documents to retrieve from the vector store. The retrieval is based on the similarity of embedded user inputs and embedded vectors within the dataset. temp, in this case temp=0.1, determines the randomness of the output. A low value such as 0.1 forces the similarity search to be precise. A higher value would allow for more diverse responses, which we do not want for this technological conversational agent. mt, in this case, mt=1024, determines the maximum number of tokens in the output. A cosine similarity function was added to make sure that the outputs matched the sample user inputs: from sentence_transformers import SentenceTransformer model = SentenceTransformer('all-MiniLM-L6-v2') def calculate_cosine_similarity_with_embeddings(text1, text2):     embeddings1 = model.encode(text1)     embeddings2 = model.encode(text2)     similarity = cosine_similarity([embeddings1], [embeddings2])     return similarity[0][0] By 11:00 AM, all three pipeline teams are warmed up and ready to go full throttle! While the Pipeline 2 team was creating the vector store and populating it with the first batch of documents, the Pipeline 1 team prepared the next several batches. At 11:00 AM, Dakota gave the green light to run all three pipelines simultaneously. Within a few minutes, the whole RAG-driven generative AI system was humming like a beehive! By 1:00 PM, Dakota and the three pipeline teams were working on a PowerPoint slideshow with a copilot. Within a few minutes, it was automatically generated based on their scenario. At 1:30 PM, they had time to grab a quick lunch. At 2:00 pm, the fire department management, Dakota’s team, and the CEO of their software company were in the meeting room.  Dakota’s team ran the PowerPoint slide show and began the demonstration with a simple input:  user_input="Explain how drones employ real-time image processing and machine learning algorithms to accurately detect events in various environmental conditions." The response displayed was satisfactory: Drones utilize real-time image processing and machine learning algorithms to accurately detect events in various environmental conditions by analyzing data captured by their sensors and cameras. This technology allows drones to process visual information quickly and efficiently, enabling them to identify specific objects, patterns, or changes in the environment in real-time. By employing these advanced algorithms, drones can effectively monitor and respond to different situations, such as wildfires, wildlife surveys, disaster relief efforts, and agricultural monitoring with precision and accuracy. Dakota’s team then showed that the program could track and display the original documents the response was based on. At one point, the fire department’s top manager, Taylor, exclaimed, “Wow, this is impressive! It’s exactly what we were looking for! " Of course, Dakato’s CEO began discussing the number of users, cost, and timelines with Taylor. In the meantime, Dakota and the rest of the fire department’s team went out to drink some coffee and get to know each other. Fire departments intervene at short notice efficiently for emergencies. So can expert-level AI teams! https://github.com/Denis2054/RAG-Driven-Generative-AI/blob/main/Chapter03/Deep_Lake_LlamaIndex_OpenAI_RAG.ipynb ConclusionIn facing a high-stakes, time-sensitive challenge, Dakota and their AI team demonstrated the power and efficiency of RAG-driven generative AI. By leveraging a structured, multi-pipeline approach with tools like Deep Lake, LlamaIndex, and OpenAI’s advanced models, the team was able to integrate scattered data sources quickly and effectively, delivering a sophisticated, real-time conversational AI prototype tailored for firefighter training on drone technology. Their success showcases how expert planning, resourceful use of AI tools, and teamwork can transform a complex project into a streamlined solution that meets client needs. This case underscores the potential of generative AI to create responsive, practical solutions for critical industries, setting a new standard for rapid, high-quality AI deployment in real-world applications.Author Bio Denis Rothman graduated from Sorbonne University and Paris-Diderot University, and as a student, he wrote and registered a patent for one of the earliest word2vector embeddings and word piece tokenization solutions. He started a company focused on deploying AI and went on to author one of the first AI cognitive NLP chatbots, applied as a language teaching tool for Mo�t et Chandon (part of LVMH) and more. Denis rapidly became an expert in explainable AI, incorporating interpretable, acceptance-based explanation data and interfaces into solutions implemented for major corporate projects in the aerospace, apparel, and supply chain sectors. His core belief is that you only really know something once you have taught somebody how to do it.

As per a recent IDC study, the forecast for new jobs demanding Salesforce skills shows a huge surge from last year. The numbers reveal that the demand is set to create 3.3 million jobs in the Salesforce ecosystem by 2022.  Additionally, among Indeed’s top 10 best jobs include Salesforce-specific, Salesforce Administrator ranking 4th and Salesforce Developer ranking at 6th place. Though Salesforce admins are not developers, but they create easy-to-use dashboards, intelligent workflows and applications for any project. They keep the Salesforce users happy and business processes smart, hence they are high in demand. Companies, especially in the US, know the potential and value Salesforce admins bring and are making serious human capital investments. We recently interviewed, Enrico Murru, a Solution and Technical Architect, a platinum Salesforce partner and Salesforce MVP to discuss the Salesforce ecosystem, his Salesforce expert journey, various certifications for Salesforce admins, and how they enhance their careers. Enrico is the author of the latest edition of our book, Salesforce Advanced Administrator Guide. This guide extends beyond being an administrator certification and covers advanced platform features and functions such as configuration, automation, security, and customization. It is packed with exam-oriented questions and mock tests to help you earn advanced administrator credentials. On the Salesforce ecosystem and Enrico’s journey to becoming a Salesforce MVP As per a recent 10K Advisors research, the Salesforce ecosystem is innovating faster than the talent can keep pace. This has resulted in great career opportunities but also introduced challenges for Salesforce end-users. How is Salesforce dealing with the challenges? How can administrators and developers leverage growth opportunities in Salesforce? When I started working with Salesforce about 10 years ago, I had never heard about the Salesforce ecosystem in my life: honestly Italy was not a hot market at that time, that’s why my (small at the time) company had a chance to work with big customers...we were among the few Salesforce system integrators in our whole country, after all. About 4 to 5 years ago things changed dramatically and Italy finally aligned with the rest of the world: Salesforce was in high demand among all kinds of companies (small or huge, no difference). The Italian market is one of the fastest growing; we started growing more and more due to increasing number of customers joining us but we started suffering from lack of professionals. We built an internal academy but it wasn’t enough, we still needed (and currently need) more developers, administrators and business analysts, the demand has exceeded the supply! The amount of “free to access documentation” is huge, the Salesforce Ohana has produced tons of content with blogs, webinars and tens of books. When Salesforce delivered Trailhead to the world we all had a boost in training: learning Salesforce became ever easier! No surprise the number of people getting certified has increased drastically, and it’s not uncommon now to see people with 5, 10 or 20 certifications on their career backpack: you don’t need to stay hours and hours with your head in a book, now you can learn 15 minutes a time when you are free between your working tasks. This is a HUGE revolution: learn a bit often and you keep yourself always on the trail, for free! From now on, anyone can become a Salesforce trailblazer and start building their trail: a lot of people have decided to change jobs and dipped into the Salesforce world with few to no experience in computer science. However when it’s time to get a certification, especially when it is your first certification, Trailhead is not enough: you need some real-world experience (no Trailhead can prepare you enough, experience is an amazing fuel for increasing your overall knowledge). A book can be a good compromise to boost your knowledge while giving you the right amount of experience that the author melt on each topic, and that’s why I chose to start this amazing trail with Packt: I wanted to do something I’ve never done before (writing a book) while delivering then Ohana more chances to pass a certification...I guess this is a win-win situation! How did you start your journey of becoming a Salesforce expert? Did being a Java developer, help you in some way? What motivated you to make the choice? Good question and the answer is that I have to thank the randomness that we can encounter daily on our lives (we can call it destiny, if you prefer). I started working as a Java developer (I came from an Electronic Engineering MSc) for a small company in my local town (Cagliari, Italy). After a while I got bored of what I was doing (boredom is a fuel for me) then I decided to move to Ireland. I got immediately the day after I landed in Cork a new job with a great income (compared to what I was earning in Italy)...but I was not 100% sure if I wanted to move abroad and that’s why I rejected that position and got back to Italy (some say it was an act of cowardice, I partially agree but I was not ready to change my life so much at that time). After just 2 months from my return home, my boss told me about a new opportunity: moving to northern Italy to join WebResults, a small company (we were just 15 people, including the CEO and CTO) that worked with something called “Salesforce”. I accepted the challenge and moved for 6 months with my spouse-to-be to WebResults headquarters: I discovered the world of Salesforce and I immediately fell in love with it. In a few weeks I learnt all that I needed to start my journey as a Salesforce developer. Years to come, I’m still working with WebResults (that in the meanwhile has been acquired by Engineering Spa, the greatest Italian consultant company) as a Salesforce Solution and Technical Architect (the amount of time I spend on coding at work has dramatically dropped unfortunately) and with the honorable Salesforce MVP title I try to evangelize my company and all the Salesforce Ohana buddies anyway I can! So if you ask me if my Java dev position helped me to arrive where I am, the answer is “definitely yes” but there is a lot more in the story! On various Salesforce certifications and why he wrote a book There are many certifications available for beginners as well as for experienced CRM developers. How should one go about choosing them? How do different Salesforce certification programs enhance a developer’s career? If you want to start your journey with Salesforce you have to choose primarily among the following paths (more details at https://trailhead.salesforce.com/credentials/administratoroverview, but you can build your own trail!): Administrator Developer Marketer Consultant Architect In my experience any aspiring Salesforce consultant should start from bases, even though she is a skilled business analysts with 20 years of experience: you need to know how the Salesforce Lightning Platform works and the best way is to get your hands dirty. Whether you wanna start as an administrator or a developer, I always recommend you face administrator skills at the beginning: a good developer should be a good administrator as well! As far as Marketer and Consultant paths are concerned, they are more related to your knowledge of specific products of the platform such as Marketing Cloud, Pardot, Field Service, Community Cloud, Einstein Analytics and many others. The Architect path brings you to the Mount Olympus of all certifications - the Technical Architect certification, which any Salesforce trailblazer aspire to get one day (and I’m one of them). Some think that owning a Salesforce certification doesn’t necessarily indicate your proficiency in the technologies involved but I do not agree with them. When I tried to get the Salesforce Advanced Administrator exam I really thought I had the required skills to pass but I failed...why? Because I didn’t study some of the topics and I wasn’t that skilled on such topics either (you’ll read this story in the book as well). That’s why I needed hours of study to pass the exam, and thanks to that deep study I learnt new Salesforce stuff and increased my proficiency in features I hadn’t actually ever used, making me the “most skilled” guy in my company regarding Omni-Channel or Salesforce Knowledge. This is an absolute win for both you and your company: certifications are meant to make you a trailblazer. Needless to say headhunters really love Salesforce certifications (my owning 20 certifications  attracts tens of contact requests on my social channels). Your book, Salesforce Advanced Administrator Certification Guide promises to give administrators a deeper knowledge of advanced Salesforce features for administrators. Why should one read this book? How is it different from other available Salesforce certification guides in the market? At first I want to say that the Salesforce Advanced Administrator Certification is a bit mistreated by administrators (as far as I’ve seen in my career): it is usually considered too hard or too complex for the skills you earn…”after all I’m already an administrator why should I become an advanced administrator”? You should my friend, the amount of things you learn is really huge, you’ll keep playing with features such as Lightning Knowledge, Omni-Channel, Live Chat, Lightning Content, features that maybe you’ve never used before, or exploring in depth the world of Salesforce automation with Process Builder, Lightning Flows, Entitlements and Approvals or knowing everything related to security and sharing of records (and many many more). Why should you choose this book? It covers extensively all required topics for the Salesforce Advanced Administrator certification keeping in mind the requirements for the exam as well. While the number of topics is too large for us to cover anything and everything for each topic, you’ll get a good amount of knowledge, suggestions and external references to ensure you reach the Salesforce Advanced Administrator certification goal. On the challenges faced by Salesforce administrators What are some of the challenges faced by Salesforce administrators today? How is Salesforce as a platform helping overcome these challenges? Can Salesforce administrators become developers too and vice versa? What is next for Salesforce? The biggest challenge that Salesforce admins face day after day is keeping pace with the extraordinarily growing Salesforce ecosystems: new companies join the Lightning Platform and new features are delivered release after release. It is more than mandatory that consultant companies and, in general, IT divisions reserve a percentage of their employees time for continuous learning, to allow Salesforce admins and devs to stay on track with the changing environment. Learning is a cost for sure, when you study you are not productive, but the benefits of a skilled and always on top employee overtakes its cost. And I see no obstacles for administrators to start their developer path as well: all they need is passion, curiosity and patience, Rome wasn’t built in a day and your developer skills won’t for sure. Trailhead is the starting point for any career path and I guess in the coming years we’ll see artificial intelligence absolutely stealing the show in Salesforce world and so admins should be prepared for the revolution that is taking place year after year. On making an impact in the Salesforce community You have created highly popular Salesforce browser extensions like ORGanizer. Tell us about how this came about? What does it take to build such successful products? Are you working on or planning to work on similar projects now? I said that boredom is my fuel: when I get bored I usually start a new project or a new hobby, and ORGanizer for Salesforce Chrome & Firefox extension (available at https://organizer.enree.co) is no different. It started as a personal project to ease my daily work with Salesforce projects, by adding little features that could speed up my administrative and coding tasks, while increasing my overall productivity. Then I thought, why not deliver this cool thing to my Salesforce Ohana? That’s where I believe the community took notice of me and it has remained one of the main reasons for my Salesforce MVP nomination. After the cool experience of writing a book, which is something that has been on my check list since I was a child, I have a few side projects related to Salesforce with some trailblazer friends, that I believe will have a great impact on the Ohana. And, why not, perhaps another book in 2020? Author Bio Enrico Murru is a Solution and Technical Architect at WebResults (an engineering company), an Italian platinum Salesforce partner, and an Independent Software Vendor (ISV). He has completed his MSc in Electronic Engineering at the University of Cagliari in 2007. In 2013, he launched a blog named Nerd @ Work. In 2016, he was nominated as the first Italian Salesforce MVP for his commitment to the Salesforce community. Then over the course of 3 years, Murru gained 20 Salesforce certifications, including the Salesforce Technical Architect certification. In 2016, he started one of the most popular projects, the ORGanizer for Salesforce Chrome and Firefox extension. You can follow him on Twitter @Enreeco, LinkedIn, GitHub, Trailblazer Community as well as on his personal blog page. Are you planning to embark on the journey of being a Salesforce Advanced Administrator? Confused about the various Salesforce certification programs and don’t know what to choose? Grab this book right now! The Salesforce Advanced Administrator Certification Guide will help you master data access security, monitoring and auditing, and understanding best practices for handling change management and data across organizations. What makes Salesforce Lightning Platform a powerful, fast and intuitive user interface What are the challenges of adopting AI-powered tools in Sales? How Salesforce can help Salesforce is buying Tableau in a $15.7 billion all-stock deal Salesforce’s open sourcing Centrifuge: A library for accelerating JVM restarts Build a custom Admin Home page in Salesforce CRM Lightning Experience

It’s well known that there’s a toxic element to tech culture. And although it isn’t new, it has nevertheless surfaced and become more visible thanks to the increasing maturity of the platforms that are today shaping public discourse. As those platforms empower new voices to speak and allow new communities to organize, the very fabric of the culture on which many of them were built - hyper-masculine, competitive, and with scant disregard for the wider implications of their decisions on users - becomes the target of critique. But while everything from sexual harassment cover-ups to content moderation crises signal deep rooted issues inside the tech industry, substantially transforming tech’s cultural problems is a problem that’s more difficult to solve. It’s also one that many leading organizations and individuals seem to be unwilling to properly engage with. This is where April Wensel comes in. She’s made it her mission to help tackle issues of toxicity and ultimately transform tech culture with her organization, Compassionate Coding. What is Compassionate Coding? Compassionate Coding was launched in 2016 as a “response to a lot of the problems I saw in the tech industry with culture,” Wensel tells me when we spoke recently over Skype. “The common thread,” she explains, “was a lack of concern for human beings that are involved in technology or affected by technology.” This is particularly significant for Wensel. While it might be tempting to see the Google Walkout, the Cambridge Analytica scandal, and the controversy around Rekognition as nothing more than a collection of troubling but ultimately unrelated issues, it’s vital that we understand them together. [caption id="attachment_28750" align="alignright" width="300"] via compassionatecoding.com[/caption] “For things to really change - we can’t approach each issue as one problem,” Wensel says. “They really have the same root problem, which is this lack of compassion.” Compassion is an important and very deliberate word. It wasn’t chosen purely for its alliterative impact. “I chose compassion because I see compassion as a really rational thing; not just an intangible thing.” Compassion is, Wensel continues, “a more active form of empathy. Empathy allows you to feel what others are feeling, compassion allows you to see suffering, and - the important piece - to want to alleviate suffering.” Compassion as an antidote to toxic tech culture To talk about compassion in the tech industry is provocative. She tells me she recalls someone on Reddit describing the idea of compassionate coding as ‘girly '. But she tries to “tune out” online resistance, adopting a measured attitude: “whenever you have new, challenging ideas people get defensive.” Even if people aren’t aggressively opposed to her ideas, initially there was a distinct unwillingness to really engage with the ideas she was putting forward. “I… saw that it wasn’t cool to talk about these things. If you started talking about humans or whatnot, people are like oh, you must be a designer or you must be in product… No, I’m a developer. I just care about the people we’re impacting.” Crucial to this attitude is Wensel’s point that compassionate coding is something that can have real effects at every level. She describes it as “a new way of weighing decisions on a daily basis… it goes from high level things like what are we building? to low level things - what should I name this variable to make it easier for somebody in the future to understand?” Distributing power through diversity The context into which Compassionate Coding has entered the world is complex. High profile scandals need attention and action, but they are only the tip of the iceberg. They are symptomatic of low-lying problems that often pass unnoticed. Diversity is a good example of this. Although it’s often framed in the somewhat prosaic context of equal opportunities, it’s actually a powerful way of breaking apart privilege and the concentration of power that allows harmful products to be released and discrimination to find its way into organizational practices. By bringing people from a diverse range of backgrounds with different experiences into positions of authority and influence, the decisions that are made at all levels are supported by a greater awareness of context. In effect, decision making becomes more rigorous. Similarly, organizations themselves become safer and more welcoming places for employees from minority backgrounds because networks of support can form, making challenging malpractice or even abuse less of a risk professionally. This is something Wensel is well aware of. She takes umbrage with the concept of ‘diversity of thought’ which she sees as a way to mask a lack of genuine diversity. “A lot of companies claim they have diversity of thought…” she says, “that are all white men.” “You can’t really have true diversity of thought if everybody has come from the same background and hasn’t had any of the challenges that people from minority backgrounds might face.” The barriers to diversity are largely structural problems that can be felt far beyond tech. But according to Wensel, there are nevertheless cultural issues unique to the industry that are compounding the problem: “If you say you value diversity but really one of your values is the efficiency or perceived efficiency that comes when everyone thinks the same way then you have to realise that you’re gonna have to make some concessions in terms of creating a bit of discomfort when people are debating issues… because there is going to be some conflict when you create these diverse spaces.” Put another way, in an industry where you’re expected to move quickly and adapt, where you’re constantly looking for efficiency, diversity is always going to be an issue. It brings friction. For Wensel, the role Compassionate Coding can play in supporting diversity and inclusion is one where it helps to shift the industry mindset away from one that is scared of friction, to one where friction is vital if we’re to build better, safer, and more secure software. She points out that diversity isn’t just an initiative, it must be something that is constantly practiced: “Inclusion has to be a daily practice and so you need somebody who is in a position of power who can help establish inclusive practices,” she says. But it also needs to be something organizations need to invest in: “companies need to be paying people to do this because a lot of times the burden falls on underrepresented groups in the company and that’s not right.” Read next: Github Sponsors: Could corporate strategy eat FOSS culture for dinner? The problem with meritocracy If diversity can help unlock a better way of working in the tech industry, there are still other industry shibboleths that need to be slayed. According to Wensel, one of these is meritocracy. It is, she argues, often used as cover by those that are resistant to genuine diversity. “A lot of time in tech people want to talk about a meritocracy… [Recode co-founder] Kara Swisher says tech is more like a mirrortocracy because the people who succeed look like the ones who are already in the industry.” https://youtu.be/ng4sbQHCGLQ But what makes this problem worse is the fact that tech’s meritocracy is haunted by stereotypes and assumptions about what it means to be a developer. She points to a study done by IBM in the sixties that aimed to find out “what makes a good, strong programmer.” “They found among other things that programmers like puzzles, and they don’t like people… So it created a stereotype of what it means to be a good developer, and part of that was not liking people. And the reason that was so important - even though it was back in the sixties - is that IBM was a very influential company in terms of establishing tech culture,” Wensel says. Stack Overflow’s negative impact on tech culture What has further exacerbated this issue is how influential figures have helped to reinforce these stereotypes, effectively buying into the image of a programmer put forward in IBM’s research. In particular, Wensel calls out Stack Overflow and its founders Joel Spolsky and Jeff Atwood. “If you read through some of their old blogs from the early 2000s,” she says, “you can see a lot of the elements of the toxic culture that I talk about in so much of my work. Things like... hyper-competition… an over focus on aggressive competition… things like zero sum thinking. There’s an elitism - there’s not enough for everybody and some people are better than others.” Wensel suggests the attitudes of Atwood and Spolsky have been instrumental in forming the worst elements of the website “where the focus is not on helping people, but on accumulating points in the game of stack overflow.” Wensel detailed her experiences of Stack Overflow and offered an incisive critique of the website in a post on Medium in 2018. She reveals that although she has used Stack Overflow since its launch in 2008 (the year she graduated from her Computer Science class) “the condescending and blatantly rude responses on the site” have dissuaded her from ever actually creating an account. Although the Compassionate Code founder can see that the site is trying to change things, she believes it can still do a lot more (in her post she adds this response from Stack overflow employee Joe Friend). The problem, however, is that this would be a risk for the company. “They really have to be willing to alienate their audience - the ones who are contributing to the toxic culture.” Ultimately this highlights the problem facing many companies and communities in the tech industry - inclusivity and diversity aren’t things that can simply be integrated into established patterns and beliefs. Those beliefs and values need to change too. Which can, of course be painful. Dismantling the hierarchy of tech skills Again, it’s important to note that Wensel’s criticisms aren’t just on the grounds of civility or accessibility. It’s ultimately bad for the industry as a whole and bad for users. It helps to cultivate an engineering culture where certain skills are overvalued while others are excluded. This has consequences for how we view ourselves in the industry (we're never good enough, and we constantly have to compete), but it also means the sort of work and thought that should go into building and delivering software is viewed as less important. “None of this is productive and none of this is creating value. We need people doing all of these roles, and so which one of these has more prestige shouldn’t be an issue” Wensel argues. “That’s why one of very clear indications that there’s a problem in the culture is the fact that we are obsessed with the need to rank skills... software projects are failing for people reasons. And yet people who are good with people and technology are seen as too soft… they’re put in a box of not being technical.” Wensel argues that we need to stop worrying about who is and who isn’t a developer. “There’s no such thing as a real developer. If you write code you’re a developer... that’s enough… Developers are no better than designers, or product managers, or salespeople… that hierarchy is even more entrenched because it’s often reflected in salaries - so developers get paid disproportionately more than all these other roles.” The myth of scarcity and the tech skills gap What’s more, Wensel believes this hierarchy of programming skills is actually helping to perpetuate the notion of a tech skills gap. She believes the idea that there is a scarcity of “tech talent” is a “myth.” “I think there’s tons of talent in tech that’s being overlooked for reasons of unconscious bias, stereotypes…” she explains. “Once we start to bring in these people to the table who are out there already - very talented, very skilled - it will start to melt away this whole putting developers on a pedestal… developers don’t belong on a pedestal, they’re just doing a job like anybody else.” Wensel believes we will - and need to - move towards a world where programming skills lose their “prestige”. Having Python or React on your CV, for example, should really be no different to saying you know how to use Excel. “As these skills become seen for what they are, which is just something that anybody can learn if they put in the time, then I think that the prestige around them will be reduced.” How Agile is changing what it means to be a developer We’re moving towards a world where the solipsism of the valorization of technical skill becomes outdated thanks to broader industry trends. With DevOps forcing developers to become accountable for the full lifecycle of their code, and distributed systems engineering requiring a holistic awareness of a complex network of dependencies, it’s clear that more sensitivity about how your code is interacting with and impacting users in the real world is more important in software engineering than it ever has. “Over and over again I see both in formal studies and anecdotally… what’s causing software projects to fail or to be delayed... are people problems. Coordination problems, planning problems resourcing, all of that - not purely technical problems,” says Wensel. That said, Wensel nevertheless views Agile as a trend that’s positive for the industry. “A lot of the ideas behind agile software development are really positive in a lot of ways I see it as the first step in bringing emotional intelligence to the software team because you’re asked to consider the end user…” Read next: DevOps Engineering and Full-Stack Development – 2 Sides of the Same Agile Coin However, she also says that software engineering practices and philosophies like Agile only go so far. “The problem is that they [proponents of Agile] didn’t bring in the ethics there. So you can still create a lot of value very efficiently with agile development without considering the long term impact.” Agile is a good context for Wensel to drive her mission forward - but it can’t improve things on it own. Read next: Honeycomb CEO Charity Majors discusses observability and dealing with “the coming armageddon of complexity” [Interview] Putting Compassionate Coding into practice It’s clear that Compassionate Coding is needed in today’s software industry. Yes, tech culture’s toxicity is damaging and dangerous for everyone, but it’s also not fit for purpose. It’s stopping us from evolving and building the software people actually need. Think of it this way: it’s stopping us from putting users first at a time when the very idea of the individual feels vulnerable, thanks to a whirlwind of reactionary politics and rampant, unsustainable capitalism. However, it’s important that we actually see Compassionate Coding as something that can be practiced, both by individuals and organizations. The 4 levels of compassionate coding Wensel explained compassionate coding as involving 4 key ‘levels’. These levels turn the concept into something practical, that every individual and team can actually go and do themselves. “It’s how you treat yourself with compassion, how you treat your coworkers, your collaborators with compassion, how you treat your direct users of the software you’re creating… and how you treat the community at large who may or may not be people who use your product,” she says. Wensel is not only continuing to deliver training sessions and keynotes for her clients, but is also writing a book which will make her ideas more accessible. I asked her what advice she would offer individuals and businesses that want to follow her lead now. “The biggest thing people can do,” she says, “is to analyze their own thinking… Do a bit of meta-cognition to understand how do I think? Where do I have biases? At an organizational level, businesses should be “prioritizing talking about these issues, making it safe to talk about these issues, hiring people who understand these issues and can improve your company in these ways” she says. The importance of the individual in tackling tech's toxicity But Wensel still believes in the importance of individuals in enacting change. “It’s humans all the way down and all the way up… Leadership in a company and [the issue of] who makes decisions is just... another set of humans, and so I think changing individuals is really powerful.” Her approach is ultimately one that espouses the values of Compassionate Coding. “You can’t control the outcome but you can control the actions you take. So I have a lot of faith in the change that motivated individuals can make.” If everyone in the industry could adopt that attitude we’d surely be some way towards not better professional lives and better experiences and products for users. Follow April on Twitter: @aprilwensel  Other projects that are making the tech industry better April cited a number of organizations that she believes are doing great and important work across the tech industry: Project Include, an organization that wants to accelerate diversity in the industry. Black Girls Code, which aims to improve the number of women of color in the digital sector. Elephant in the Valley, which is tackling gender disparity in Silicon Valley. Kapor Center, removing barriers for underrepresented groups in tech. Learn more about the issues they're helping to solve, and support them if you can.

Odoo is one of the fastest growing open source, business application development software products available. It comes with: Powerful GUI, Performance optimization, Integrated in-app purchase features Fast-growing community to transform and modernize businesses We recently interviewed Yenthe Van Ginneken, an Odoo developer, highly active in the Odoo community and recipient of Odoo best contributor of the year 2016 and Odoo community hero 2017. He spoke to us about his journey with Odoo, his thoughts on Odoo’s past, present and future, and the Odoo community. Expert's Bio Yenthe Van Ginneken, currently the technical team leader at Odoo Experts, has been an Odoo developer for over four years. He has won two awards, “Best contributor of the year 2016” and the “Odoo community hero” award in 2017. He loves improving software and teaching other people the best practices for Odoo development on his blog. You can read his Odoo blog, follow him on Twitter or reach out to him on LinkedIn. Key Takeaways Odoo is scalable and flexible to the extent that everyone, from a small startup to a giant tech company can benefit from it. It is ahead of quite a lot of ERP systems with its clean UI, advanced modules integration and the flexibility of its technical framework. Python is the preferred language of choice among most developers that want to use the Odoo framework, especially for automating and scaling tasks. The Odoo community is diverse and vast. By contributing and regularly interacting with other members, you will gain deeper insights into many different aspects of Odoo development. A great way to learn to develop in Odoo and quickly grow is actually by helping in the community. Odoo 12 will reportedly improve data processing, better report insights, and support for OCR (Optical Character Recognition) for handling documents among other exciting updates. Full Interview On who should use Odoo Odoo is more than an ERP tool. According to you, What is Odoo? Who will benefit from adopting Odoo? What made you choose Odoo?   For me, Odoo is more than an ERP. Odoo literally allows me to make any module or functionality that I can think of. Since Odoo is so flexible and scalable I believe that almost everybody can benefit from adopting Odoo. Whether you’re a small start-up or a giant tech company. The most important part to be able to benefit from adopting Odoo is adjusting the processes and mindset to use Odoo, not adjusting Odoo for the company. The projects that work the best and have the best benefit are those that don’t over-engineer and try to focus on the main company processes. I personally chose Odoo after I got an opportunity to become an Odoo developer at a company in Belgium. After the job offer I visited Odoo.com and saw the massive amount of functionalities in Odoo (while being free!) and I was genuinely amazed. After looking at the technical framework and all the default options provided by the framework I was sure that I would love to develop and implement in Odoo. Since that day I never stopped working with Odoo. On journey from OpenERP to Odoo Odoo started off as OpenERP and then in 2014, it moved beyond just ERP and was renamed Odoo. How has Odoo’s journey been so far since then? What do you think are the key milestones achieved by Odoo till date? Since the renaming from OpenERP to Odoo the company has seen a rapid growth. A bit after changing the name Odoo also introduced the enterprise version which was, in my opinion, the turning point for Odoo S.A. It allowed Odoo to keep its open source strength and market share while also gathering funds to fund the ongoing growth of the product. The big investments that are being made in the Research and Development team allow them to keep improving year after year. The main strengths and key milestones from Odoo are absolutely its flexibility, a great framework and the fact that most of the possibilities are already in Odoo by default. On the drive behind contributing to the Odoo community You are highly active in the Odoo community. How did you get into contributing for Odoo? How has this experience improved you as a developer? According to you, what are the key challenges the Odoo community is facing currently? My very first contribution started in the second half of 2014 and weren’t very significant at first. I noticed that Odoo 8, at that point the newest version, was not very well translated and had a lot of inconsistency so I started translating it in Dutch. From there on I noticed that it could have had quite a big impact and in fact could improve the ERP. It didn’t take long before I started contributing in other ways. Reporting issues, fixing bugs, maintaining bug reports and helping other people on the official help forums. By contributing to all these different subjects I got introduced to more domains and gained more insights. Thanks to my involvement with the community, I’ve learned that there is more than one side to developing and implementing projects. I believe it made me a better programmer and made me think a lot more about ways to code custom development for projects. Without being active in a community and contributing you’ll be blindsided by your own perspective. It is a great way to get challenged and you’ll see more cases by being active in the community than you could ever see on your own. The Odoo community faces a few challenges at this point. It is difficult to maintain the right balance between the enterprise version and community (free) version. There are not a lot of very active contributors to the official Odoo code and Odoo is behind on handling fixes/bug reports made by community members. This results in some community members not feeling appreciated or heard. Hiring a second community manager might be a good way to resolve these issues though. The most difficult challenge for both Odoo and the Odoo community is to make everybody feel heard and give every person the ability to contribute in the way he or she can. When there is enough help from Odoo and the community feels supported there is a possibility for a great and thriving community. On how to learn Odoo effectively As a person who has a strong hold over Odoo development, what is the typical learning curve for someone getting into Odoo, as a consultant? What is the best way to start developing in Odoo? What should one watch out for while learning? The learning curve can be quite long and can have its challenges. Usually, if you don’t have any experience with Odoo and only know basic Python it’ll take about six months before you really get to know the ins and outs of Odoo. The best way to learn to develop Odoo is probably the same as with most things in technology: dive in! Make sure you get the basics right and understand how the main functionalities work before going deeper. A great way to learn to develop in Odoo and to quickly grow is actually by helping in the community. You can get insight and help from experienced developers while also contributing to the community, it’s a win-win. Start small and build your way up to the details. It is important to find good documentation and tutorials though. At the moment there are still quite some blog posts and tutorials that are from quite a low quality. Because of this I actually started writing my own tutorials, which explain concepts step by step with samples. You can find it at https://odoo.yenthevg.com Editor’s note: Check out our collection of Odoo Books and Videos to master Odoo development. On the upcoming Odoo 12 release Odoo 12 is expected to be released later this year. What’s got you excited about this new release? Quite a lot! Every release has loads of new features that are announced and it’s an exciting time, every time. The introduction of a report designer for functional people is one of the best (known) new features. The improved reporting tools for data insight will become a great improvement too. The biggest announcements are made at Odoo Experience in October and are not publicly available yet so we’ll have to wait for that. On the future of ERP There is a lot happening in the area of ERP and BI: self-service analytics, real-time analytics, agile BI development etc. Where do you foresee the ERP market headed? We've seen ERP/CRM systems getting powerful inbuilt analytics systems, what do you think is next for the industry? What is Odoo’s role here? As with any sector in IT, a lot is becoming very data-driven. In the future integration and usage of data will only grow. I expect the combination of BI and AI to become a powerful way to process and handle data on unseen scales. Odoo itself has already hinted at improved data processing, better report insights and support for OCR (Optical Character Recognition) for handling documents. Odoo has been ahead of quite a lot of ERP systems with its clean UI, advanced modules integration and the flexibility of its technical framework for years. I expect Odoo will also be leading the way for handling all this data and getting important statistics out of it. I’m quite sure it is only a matter of time before Odoo starts working on even better BI reporting and tools. On Python and automation Automation is everywhere today and becoming an integral part of organizations and processes. Python and automation have gone hand in hand since Python’s early days. Today Python is one of the top programming languages. How do you see Python’s evolution over the years in the area of automation? What are the top ways you use Python for automation, today? It is for a reason that Python is so popular. It is flexible, quite quick to program with and the options are virtually endless. In the next years, Python will only become more popular and this will also be the case for automation projects made with Python. I personally use the Odoo framework with Python as a backbone for nearly everything that I automate (and in fact also for non-automated tasks). The projects vary from automatically handling stock moves to automatically updating remote instances to automatically getting full diagnostic reports. The combination of the programming language and the framework from Odoo allows me to automate tasks and deploy them on a big scale. ERP tool in focus: Odoo 11 How to Scaffold a New module in Odoo 11 A step by step guide to creating Odoo Addon Modules

IntroductionIn today’s data-driven world, businesses rely heavily on robust tools to transform raw data into actionable insights. Among these tools, Microsoft Power BI stands out as a leader, renowned for its versatility and user-friendliness. From its humble beginnings as an Excel add-in, Power BI has evolved into a comprehensive enterprise business intelligence platform, competing with industry giants like Tableau and Qlik. This journey of transformation reflects not only Microsoft’s innovation but also the growing need for accessible, scalable analytics solutions.As a data specialist who has transitioned from traditional data warehousing to modern analytics platforms, I’ve witnessed firsthand how Power BI empowers both technical and non-technical users. It has become an indispensable tool, offering capabilities that bridge the gap between data modeling and visualization, catering to everyone from citizen developers to seasoned data analysts. This article explores the evolution of Power BI, its role in democratizing data analytics, and its integration into broader solutions like Microsoft Fabric, highlighting why mastering Power BI is critical for anyone pursuing a career in analytics.The Changing Tide for Data Analysts When you think of business intelligence in the modern era, Power BI is often the first tool that comes to mind. However, this wasn't always the case. Originally launched as an add-in for Microsoft Excel, Power BI quickly evolved into a comprehensive enterprise business intelligence platform in a few years competing with the likes of Qlik and Tableau—a true testament to its capabilities. As a data specialist, what really impresses me about Power BI's evolution is how Microsoft has continuously improved its user-friendliness, making both data modeling and visualizing more accessible, catering to both technical professionals and business users.  As a data specialist, initially working with traditional data warehousing, and now with modern data lakehouse-based analytics platforms, I’ve come to appreciate the capabilities that Power BI brings to the table. It empowers me to go beyond the basics, allowing me to develop detailed semantic layers and create impactful visualizations that turn raw data into actionable insights. This capability is crucial in delivering truly comprehensive, end-to-end analytics solutions. For technical folk like me, by building on our experiences working with these architectures and the deep understanding of the technologies and concepts that drive them, integrating Power BI into the workflow is a smooth and intuitive process. The transition to including Power BI in my solutions feels almost like a natural progression, as it seamlessly complements and enhances the existing frameworks I work with. It's become an indispensable tool in my data toolkit, helping me to push the boundaries of what's possible in analytics. In recent years, there has been a noticeable increase in the number of citizen developers and citizen data scientists. These are non-technical professionals who are well-versed in their business domains and dabble with technology to create their own solutions. This trend has driven the development of a range of low-code/no-code, visual tools such as Coda, Appian, OutSystems, Shopify, and Microsoft’s Power Platform. At the same time, the role of the data analyst has significantly expanded. More organizations are now entrusting data analysts with responsibilities that were traditionally handled by technology or IT departments. These include tasks like reporting, generating insights, data governance, and even managing the organization’s entire analytics function. This shift reflects the growing importance of data analytics in driving business decisions and operations. As a data specialist, I’ve been particularly impressed by how Power BI has evolved in terms of user-friendliness, catering not just to tech-savvy professionals but also to business users. Microsoft has continuously refined Power BI, simplifying complex tasks and making it easy for users of all skill levels to connect, model, and visualize data. This focus on usability is what makes Power BI such a powerful tool, accessible to a wide range of users. For non-technical users, Power BI offers a short learning curve, enabling them to connect to and model data for reporting without needing to rely on Excel, which they might be more familiar with. Once the data is modeled, they can explore a variety of visualization options to derive insights. Moreover, Power BI’s capabilities extend beyond simple reporting, allowing users to scale their work into a full-fledged enterprise business intelligence system. Many data analysts are now looking to deepen their understanding of the broader solutions and technologies that support their work. This is where Microsoft Fabric becomes essential. Fabric extends Power BI by transforming it into a comprehensive, end-to-end analytics platform, incorporating data lakes, data warehouses, data marts, data engineering, data science, and more. With these advanced capabilities, technical work becomes significantly easier, enabling data analysts to take their skills to the next level and realize their full potential in driving analytics solutions.  If you're considering a career in analytics and business intelligence, it's crucial to master the fundamentals and gain a comprehensive understanding of the necessary skills. With the field rapidly evolving, staying ahead means equipping yourself with the right knowledge to confidently join this dynamic industry. The Complete Power BI Interview Guide is designed to guide you through this process, providing the essential insights and tools you need to jump on board and thrive in your analytics journey. ConclusionConclusionMicrosoft Power BI has redefined the analytics landscape by making advanced business intelligence capabilities accessible to a wide audience, from technical professionals to business users. Its seamless integration into modern analytics workflows and its ability to support end-to-end solutions make it an invaluable tool in today’s data-centric environment. With the rise of citizen developers and expanded responsibilities for data analysts, tools like Power BI and platforms like Microsoft Fabric are paving the way for more innovative and comprehensive analytics solutions.For aspiring professionals, understanding the fundamentals of Power BI and its ecosystem is key to thriving in the analytics field. If you're looking to master Power BI and gain the confidence to excel in interviews and real-world scenarios, The Complete Power BI Interview Guide is an invaluable resource. From the core PowerBI concepts to interview preparation and onboarding tips and tricks, The Complete Power BI Interview Guide is the ultimate resource for beginners and aspiring Power BI job seekers who want to stand out from the competition.Author BioGogula is an analytics and BI architect born and raised in Sri Lanka. His childhood was spent dreaming, while most of his adulthood was and is spent working with technology. He currently works for a technology and services company based out of Colombo. He has accumulated close to 20 years of experience working with a diverse range of customers across various domains, including insurance, healthcare, logistics, manufacturing, fashion, F&B, K-12, and tertiary education. Throughout his career, he has undertaken multiple roles, including managing delivery, architecting, designing, and developing data & AI solutions. Gogula is a recipient of the Microsoft MVP award more than 15 times, has contributed to the development and standardization of Microsoft certifications, and holds over 15 data & AI certifications. In his leisure time, he enjoys experimenting with and writing about technology, as well as organizing and speaking at technology meetups. 

Data today is the world’s most important resource. However, without properly visualizing your data to discover meaningful insights, it’s useless. Creating visualizations helps in getting a clearer and concise view of the data, making it more tangible for (non-technical) audiences. To further illustrate this, below are questions aimed at giving you an idea why data visualization is so important and why Python should be your choice. In a recent interview, Tim Großmann and Mario Döbler, the authors of the course titled, ‘Data Visualization with Python’, shared with us the importance of Data visualization and why Python is the best fit to carry out Data Visualization. Key Takeaways Data visualization is a great way, and sometimes the only way, to make sense of the constantly growing mountain of data being generated today. With Python, you can create self-explanatory, concise, and engaging data visuals, and present insights that impact your business. Your data visualizations will make information more tangible for the stakeholders while telling them an interesting story. Visualizations are a great tool to transfer your understanding of the data to a less technical co-worker. This builds a faster and better understanding of data. Python is the most requested and used language in the industry. Its ease of use and the speed at which you can manipulate and visualize data, combined with the number of available libraries makes Python the best choice. Full Interview Why is Data Visualization important? What problem is it solving? As the amount of data grows, the need for developers with knowledge of data analytics and especially data visualization spikes. In recent years we have experienced an exponential growth of data. Currently, the amount of data doubles every two years. For example, more than eight thousand tweets are sent per second; and more than eight hundred photos are uploaded to Instagram per second. To cope with the large amounts of data, visualization is essential to make it more accessible and understandable. Everyone has heard of the saying that a picture is worth a thousand words. Humans process visual data better and faster than any other type of data. Another important point is that data is not necessarily the same as information. Often people aren’t interested in the data, but in some information hidden in the data. Data visualization is a great tool to discover the hidden patterns and reveal the relevant information. It bridges the gap between quantitative data and human reasoning, or in other words, visualization turns data into meaningful information. What other similar solutions or tools are out there? Why is Python better? Data visualizations can be created in many ways using many different tools. MATLAB and R are two of the available languages that are heavily used in the field of data science and data visualization. There are also some non-coding tools like Tableau which are used to quickly create some basic visualizations. However, Python is the most requested and used language in the industry. Its ease of use and the speed at which you can manipulate and visualize data, combined with the number of available libraries makes Python the best choice. In addition to all the mentioned perks, Python has an incredibly big ecosystem with thousands of active developers. Python really differs in a way that allows users to also build their own small additions to the tools they use, if necessary. There are examples of pretty much everything online for you to use, modify, and learn from. How can Data Visualization help developers? Give specific examples of how it can solve a problem. Working with, and especially understanding, large amounts of data can be a hard task. Without visualizations, this might even be impossible for some datasets. Especially if you need to transfer your understanding of the data to a less technical co-worker, visualizations are a great tool for a faster and better understanding of data. In general, looking at your data visualized often speaks more than a thousand words. Imagine getting a dataset which only consists of numerical columns. Getting some good insights into this data without visualizations is impossible. However, even with some simple plots, you can often improve your understanding of even the most difficult datasets. Think back to the last time you had to give a presentation about your findings and all you had was a table with numerical values in it. You understood it, but your colleagues sat there and scratched their heads. Instead had you created some simple visualizations, you would have impressed the entire team with your results. What are some best practices for learning/using Data Visualization with Python? Some of the best practices you should keep in mind while visualizing data with Python are: Start looking and experimenting with examples Start from scratch and build on it Make full use of documentation Use every opportunity you have with data to visualize it To know more about the best practices in detail, read our detailed notes on 4 tips for learning Data Visualization with Python. What are some myths/misconceptions surrounding Data Visualization with Python? Data visualizations are just for data scientists Its technologies are difficult to learn Data visualization isn’t needed for data insights Data visualization takes a lot of time Read about these myths in detail in our article, ‘Python Data Visualization myths you should know about’. Data visualization in combination with Python is an essential skill when working with data. When properly utilized, it is a powerful combination that not only enables you to get better insights into your data but also gives you the tool to communicate results better. Data nowadays is everywhere so developers of every discipline should be able to work with it and understand it. About the authors Tim Großmann Tim Großmann is a CS student with interest in diverse topics ranging from AI to IoT. He previously worked at the Bosch Center for Artificial Intelligence in Silicon Valley in the field of big data engineering. He’s highly involved in different Open Source projects and actively speaks at meetups and conferences about his projects and experiences. Mario Döbler Mario Döbler is a graduate student with a focus in deep learning and AI. He previously worked at the Bosch Center for Artificial Intelligence in Silicon Valley in the field of deep learning, using state-of-the-art algorithms to develop cutting-edge products. Currently, he dedicates himself to apply deep learning to medical data to make health care accessible to everyone. Setting up Apache Druid in Hadoop for Data visualizations [Tutorial] 8 ways to improve your data visualizations Getting started with Data Visualization in Tableau  

Change might feel like a natural part of working in the software industry. But in truth it's not natural at all; it takes a hell of a lot of effort to do things differently. That's what software developers Shelby Cohen and Katie Levy found out when they decided that Kotlin could be a better programming language option when it came to their company's engineering team. As two relatively junior developers at financial software provider Intuit, Shelby and Katie didn't only have to take on the challenge of building out training programs and providing resources to thousands of developers across Intuit, they also had to negotiate internal hierarchies and politics that can prove resistant to change. To learn more about what this process was like, as well as why they're so passionate about Kotlin, I spoke to them over email. Visit the Packt store to explore Kotlin eBooks and videos. How did you get started in software engineering? Shelby Cohen: I’ve always been interested in solving problems and engaging with new challenges. In high school I really enjoyed math and one of my math teachers encouraged me to join the High School Robotics Club. It was all men and I didn’t feel like I fit in. With the help and encouragement of my teacher I helped start the Women Robotics Club at my high school. This is where I was exposed to programming for the first time and that inspired me to study computer science in college. During my junior year, I learned about Intuit’s co-op program at a hackathon and thought it was a great opportunity. I then travelled from New York to San Diego to spend a semester working at Intuit. I learned so much and was exposed to a lot of mentorship opportunities which led to me accepting a full time position as a Software Engineer I in 2017. Intuit really values teaching their employees and helping them continue to grow and develop so it felt like a natural fit. Why Kotlin? What’s unique about Kotlin? Katie Levy: Kotlin is an open source, cross-platform programming language, designed to interoperate with Java. What’s nice about it is that it allows for an easy transition to start using a functional programming style while also being safer and more concise than Java code. Kotlin is unique because it is very clean, simple, clear and removes a lot of the redundant, boilerplate code that’s in Java which allows the developer to focus on the business logic. It’s my favorite language to program in as I can write high quality code faster by using its built-in language features. It can be used on any application running on the JVM, including Spring boot backend services, Android apps, and even JavaScript applications. How/where did you learn about it? Was it an immediate thing or did it take time for you to decide to do this? Shelby: I volunteered at KotlinConf back in 2018, and it was such a great opportunity to meet a lot of the engineers and staff from Jetbrains. I got to meet a senior executive at Jetbrains and shared some of the projects I was working on at Intuit. He asked to be a guest on his podcast, TalkingKotlin, and through this conference I got to know some of the most influential people in the Kotlin community. How did people respond? Were they resistant to something new? Katie: My team was definitely hesitant to start learning the language. One of the engineers on my team was especially resistant and tried to identify flaws in the language, using anything he could come up with as a reason why we shouldn’t use Kotlin. In those cases it’s important to identify the real issue the engineers are having with the new language — is it the language itself or is it something else? With that particular engineer, I found out that he was feeling like he didn’t have the bandwidth to take on the work he was being assigned. To combat this, I created a training program for the team so we could learn together, build up the team’s domain knowledge on the language, and so everyone’s workload was more visible. How did you go about driving adoption? Katie: Influencing and driving change is a hard project to scope. It can mean many different things to different people. For us, when we were starting out, we wanted to introduce 500 engineers to Kotlin, and wanted 90% of them to start coding in Kotlin. We ended up exceeding our goal, reaching 370 engineers internal to Intuit and 4,329 external to Intuit. We want to improve the industry by encouraging engineers to develop in a more concise and less error-prone language. More recently, we were able to present on Kotlin to over 500 software engineers at LambdaWorld in Spain. Afterward, we had engineers wanting to take pictures with us and all saying they want to start using the language. We found that speaking at conferences helped us meet our goals, and scale our efforts. What were the challenges? Shelby: In addition to some of the initial resistance, one of the biggest challenges is the internal hierarchy. When I introduced Kotlin to my team, a lot of engineers were more senior than me and at the time, I wasn’t confident about the value that I was bringing to the team. I implemented group code reviews, sent out resources, and walked the team through examples as they were learning Kotlin. Once the team had a good foundation for Kotlin, I implemented a flatter teaching structure and encouraged everyone to learn and teach each other a specific part of the language. This was really effective because everyone learns from different teaching styles and team members felt more empowered in their work. Have you learned anything else about Kotlin throughout the process? And about engineering in general? Shelby: One of the most valuable things I learned from this experience is don’t be afraid to ask for advice on how to influence at scale and connect with others who have driven change at your company. This way you can learn from other’s experiences. Katie and I reached out to a lot of leaders making an impact in their area of expertise to learn from them and get feedback and advice on our journey. This is something we will continue to do as we keep learning and facing other challenging problems. Thanks to Shelby and Katie for talking to us - it's clear they have a lot of passion for Kotlin, but more importantly they also have a great sense of how to engage and support other developers. Follow Shelby on Twitter: @shelbyc0hen Follow Katie on Twitter: @klevy110

Python is currently one of the most popular and desired programming languages. Primarily for its simplicity, agility and ability to be used across a variety of software development projects. Although an Object Oriented language, Python supports an array of programming paradigms, including Functional Programming. Functional Programming is a paradigm that treats computation as the evaluation of math functions. It’s quite advantageous as it allows for efficient parallel programming and error-free code. We recently interviewed Steven Lott, a true Python professional and best-selling author of a number of Python books. Steven talks a bit about modern Python and how the language adapts well to the Functional paradigm, offering developers a range of solutions to build modern, cloud based applications. He also talks about his most recent book, the second edition of his best seller, Functional Python Programming, and how it will benefit developers looking to enter the world of functional programming with Python. About Steven Steven F. Lott has been programming since the 70s, when computers were large, expensive, and rare. As a contract software developer and architect, he has worked on hundreds of projects, from very small to very large. He's been using Python to solve business problems for over 10 years. Steven is a technomad who lives in various places on the east coast of the U.S. Follow his technology blog to stay updated on the latest trends in tech. You may also connect with him on LinkedIn. Key Takeaways Why learn Python? One of the major reasons developers appreciate Python, is because it’s simple, and has the ability to create succinct and expressive programs. For example, data scientists prefer Python because they can build sophisticated analytical tools using simple functions and produce useful results. Why Functional programming? The functional programming paradigm forms the perfect foundation for developers and architects to build and design modern architectures like Serverless. But Python isn’t inherently functional. Although an object oriented language at heart, Python can create higher order functions and other functional features. Python 3 has made this easier. Steven’s Python 4 wish list: In future versions of Python, Steven hopes to see a wider use of Unicode operator characters like × in addition, * for multiplication, and ÷ in addition to / for division. Also, he expects PyPy and RPython projects to be more widely used; future Pythons versions will benefit from optimizations and restructuring the interpreter. One of the most helpful features of Python 3 are type hints, which allow one to write clear and implicitly documented code while preventing the invoking of methods with wrong data types. Steven’s latest edition of Functional programming with Python, explores type hints in depth along with core language features such as lambdas, generator expressions, functions, and callable objects. Full Interview Python is one of the top programming languages. List down top 3 features of Python that make programmers love it.. It seems like programmers love Python primarily because it allows them to create succinct, expressive programs. Before long, they learn the vast library of code - is another reason for Python's immense popularity. Many people adopt Python because of the low barrier to entry: it really is as simple as download and start working. You've been working with Python for over a decade. How has your experience been with Python as a primary development language? Over my 40-year career, I've used a variety of languages. And I've found Python to be extremely productive. A team can build and deploy microservices-based applications at a tremendous pace. Data scientists can build sophisticated analytical tools using simple functions to produce useful results without the overheads of complex compile and build environments. Back when Python 3 just came into existence, we saw certain resistance to the notion of Python being apt for functional programming. How would you say Python has progressed since then? At its core, Python is an object-oriented language. Consequently some functional features aren't central. One of the essential functional design tools -- creating higher-order functions -- has always been part of Python. The wider use of generator functions in Python 3 has made functional Python programming much more common. Many Python applications are hybrids, mixing object-oriented and functional features of the language. Now that type hints are available, it becomes practical to use mypy to confirm that the code is very likely to work properly. For a developer who's picking up the Functional Programming paradigm for the first time, what do you think are the prerequisites? Functional programming is closely aligned to the core mathematical ideas of functions and functional composition. As a consequence, a minimal background in programming could be advantageous to help leverage essential function definitions and avoid needless state change. For programmers already heavily invested in procedural programming, it may be helpful to set the idea of stateful objects aside. In the above context, how does your book, Functional Python Programming, Second Edition, prepare its readers to be industry ready? What are the key takeaways for readers from your title and how does it help with the learning curve? The examples in the book are related to exploratory data analysis, an important skill in the broader area of data science. They also focus on the standard library, allowing someone to apply the functional design approach to other libraries and tools. I think a focus on the core language features (e.g., lambdas, generator expressions, functions, callable objects) provide a foundation that allows a programmer to apply the core ideas more widely to different kinds of problems and other software packages. What new and updated content is available in this edition, for developers who've purchased your previous book? Almost all of the examples have been rewritten to include type hints. This can be an important quality check helping to ensure the Python code works. When used with doctest examples, it becomes relatively easy to provide reliable, correct code. In a few cases, external packages (i.e., the pymonad library) don't have type hints and the examples reflect this gap. Can you throw some light on Functional Reactive Programming and how FRP with Python is boosting the implementation of modern architectures like Cloud Native and Serverless? The central idea of serverless programming -- a collection of isolated functions -- fits the functional programming paradigm very elegantly. The processing is generally stateless, with stand-alone functions waiting for their inputs. Ideas like "choreography" of web services work with this idea of stateless functions that respond to an input by producing an output. This leads to careful separation of persistence and state change from the other transformational processing. This helps create software with easy-to-understand behavior and implementation code that's very expressive of the algorithm. As Python inches towards a 4.0, what do you think should/can be changed/rectified in the language, for the better? At some point, I expect the PyPy and RPython projects to create some optimizations leading to a fundamental restructuring of the interpreter. Perhaps these changes could remove the need for the GIL (Global Interpreter Lock) by exposing a minimal kernel of code that requires exclusive access to internal data structures. Of more general interest, I'd hope to see wider use of Unicode operator characters like × in addition to * for multiplication, and ÷ in addition to / for division. Perhaps ∻ could be adopted for truncated division. This could also lead to use of ℝ instead of float and ℤ instead of int providing a more mathematical look to type hints. It would be nice to expand the use of the Unicode character set to create more readable programs. List down 3 reasons for developers to choose your book as the book of choice for Functional programming in Python. Programmers who want to create succinct and expressive code often find functional design to help them fulfill this. The Functional Python Programming book provides extensive examples that show multiple ways to achieve this goal. In many cases, generator functions and lazy processing can be a large performance improvement. A generator function can use less memory than a large data collection, and this change can be helpful. This book will provide number of examples of lazy processing to avoid creating large, in-memory collections. It can be difficult to get started with type hints. The use case in this book show type hints in somewhat more complex real-world situations. If you enjoyed this interview, head over to check out Steven’s latest edition of Functional Python Programming, He is leveraging Python to implement microservices and ETL pipelines. His other titles with Packt Publishing include Python Essentials, Mastering Object-Oriented Python, Functional Python Programming, and Python for Secret Agents. What is the difference between functional and object oriented programming? Building functional programs with F# Seven wrongs don’t make the one right: Solving a problem with Functional Javascript

Introduction Are you an Excel user looking to push your data analysis capabilities beyond the familiar cells and formulas? If so, you're about to embark on a transformative journey. With the integration of R and Python, you can elevate Excel into a powerhouse of advanced data analysis and visualization. In this blog post, inspired by the book "Extending Excel with Python and R," co-authored by myself and David Kun, we will dive deep into practical implementation, focusing on how to automate data visualization in Excel using these powerful programming languages. Practical Implementation: Creating Advanced Data Visualizations In the world of data analysis, visual representation is key to understanding complex datasets. Excel, while equipped with basic charting tools, often requires enhancement for more sophisticated visuals. By integrating R and Python, you can create dynamic and detailed graphs that bring your data to life. Task: Automating Data Visualization with Python and R Step-by-Step Guide Step 1: Set Up Your Environment Before jumping into visualization, ensure you have the necessary tools installed. You will need: Excel: Ensure you have a version that supports VBA (Visual Basic for Applications). Python: Install Python on your computer. You can download it from the official Python website. R: Similarly, install R from the Comprehensive R Archive Network (CRAN). Libraries: For Python, install `pandas`, `matplotlib`, and `openpyxl` using pip. For R, install `ggplot2` and `readxl`.  Step 2: Importing Data Begin by importing your Excel data into Python or R. Here’s a Python snippet using pandas:  In R, use readxl:  Step 3: Creating Visualizations Python Example Using Matplotlib, you can create a simple line plot: Python Example   R Example With ggplot2, the process is equally straightforward where df is some data frame loaded in:  Step 4: Integrating Visualizations into Excel Once your visualization is created, the next step is to integrate it back into Excel. This can be done manually, or you can automate it using VBA or an API endpoint. Python Integration Using openpyxl, you can embed images:   R Integration For R, you might automate this process using R scripts that interact with Excel via VBA or other packages like `officer`.  Step 5: Automating the Entire Workflow To automate, consider using Python scripts executed from Excel VBA or R scripts called through Excel's RExcel plugin. This way, you can refresh data and update visualizations with minimal effort. Conclusion By integrating R and Python with Excel, you unlock a realm of possibilities for data visualization and analysis, turning Excel from a simple spreadsheet tool into a comprehensive data analytics suite. This guide provides a snapshot of what you can achieve, and with further exploration, the potential is limitless. Author Bio Steven Sanderson is a Manager of Applications with a deep passion for data and its compliments: cleaning, analysis, visualization and communication. He is known primarily for his work in R. After his MPH, Steven continued his work in the healthcare industry as a clinical decision support analyst working his way up to Manager of Applications at Stony Brook Medicine for Patient Financial Services. He currently is focused on expanding functions in his healthyverse suite of packages while also slimming them down and expanding their robustness. He also now enjoys helping mentor junior employees to set them up for success. David Kun is a mathematician and actuary who has always worked in the gray zone between quantitative teams and ICT, aiming to build a bridge. He is a co-founder and director of Functional Analytics, the creator of the ownR infinity platform. As a data scientist, he also uses ownR for his daily work. His projects include time series analysis for demand forecasting, computer vision for design automation, and visualization. Looking to Master Excel with Python and R?If you're excited about extending Excel’s capabilities with powerful tools like Python and R, Extending Excel with Python and R, authored by Steven Sanderson, David Kun, offers an in-depth guide to seamlessly integrating these languages into your Excel workflow. It covers everything from automating data tasks to advanced visualizations, all tailored for Excel enthusiasts.