Search icon CANCEL
Subscription
0
Cart icon
Your Cart (0 item)
Close icon
You have no products in your basket yet
Save more on your purchases! discount-offer-chevron-icon
Savings automatically calculated. No voucher code required.
Arrow left icon
Explore Products
Best Sellers
New Releases
Books
Videos
Audiobooks
Learning Hub
Newsletter Hub
Free Learning
Arrow right icon
timer SALE ENDS IN
0 Days
:
00 Hours
:
00 Minutes
:
00 Seconds

How-To Tutorials

7019 Articles
article-image-how-to-build-a-music-recommendation-system-with-pagerank-algorithm
Vijin Boricha
13 Feb 2018
6 min read
Save for later

How to Build a music recommendation system with PageRank Algorithm

Vijin Boricha
13 Feb 2018
6 min read
[box type="note" align="" class="" width=""]This article is an excerpt from a book Mastering Spark for Data Science written by Andrew Morgan and Antoine Amend. In this book, you will learn about advanced Spark architectures, how to work with geographic data in Spark, and how to tune Spark algorithms to scale them linearly.[/box] In today’s tutorial, we will learn to build a recommender with PageRank algorithm. The PageRank algorithm Instead of recommending a specific song, we will recommend playlists. A playlist would consist of a list of all our songs ranked by relevance, most to least relevant. Let's begin with the assumption that people listen to music in a similar way to how they browse articles on the web, that is, following a logical path from link to link, but occasionally switching direction, or teleporting, and browsing to a totally different website. Continuing with the analogy, while listening to music one can either carry on listening to music of a similar style (and hence follow their most expected journey), or skip to a random song in a totally different genre. It turns out that this is exactly how Google ranks websites by popularity using a PageRank algorithm. For more details on the PageRank algorithm visit: h t t p ://i l p u b s . s t a n f o r d . e d u :8090/422/1/1999- 66. p d f . The popularity of a website is measured by the number of links it points to (and is referred from). In our music use case, the popularity is built as the number hashes a given song shares with all its neighbors. Instead of popularity, we introduce the concept of song commonality. Building a Graph of Frequency Co-occurrence We start by reading our hash values back from Cassandra and re-establishing the list of song IDs for each distinct hash. Once we have this, we can count the number of hashes for each song using a simple reduceByKey function, and because the audio library is relatively small, we collect and broadcast it to our Spark executors: val hashSongsRDD = sc.cassandraTable[HashSongsPair]("gzet", "hashes") val songHashRDD = hashSongsRDD flatMap { hash => hash.songs map { song => ((hash, song), 1) } } val songTfRDD = songHashRDD map { case ((hash,songId),count) => (songId, count) } reduceByKey(_+_) val songTf = sc.broadcast(songTfRDD.collectAsMap()) Next, we build a co-occurrence matrix by getting the cross product of every song sharing a same hash value, and count how many times the same tuple is observed. Finally, we wrap the song IDs and the normalized (using the term frequency we just broadcast) frequency count inside of an Edge class from GraphX: implicit class Crossable[X](xs: Traversable[X]) { def cross[Y](ys: Traversable[Y]) = for { x <- xs; y <- ys } yield (x, y) val crossSongRDD = songHashRDD.keys .groupByKey() .values .flatMap { songIds => songIds cross songIds filter { case (from, to) => from != to }.map(_ -> 1) }.reduceByKey(_+_) .map { case ((from, to), count) => val weight = count.toDouble / songTfB.value.getOrElse(from, 1) Edge(from, to, weight) }.filter { edge => edge.attr > minSimilarityB.value } val graph = Graph.fromEdges(crossSongRDD, 0L) We are only keeping edges with a weight (meaning a hash co-occurrence) greater than a predefined threshold in order to build our hash frequency graph. Running PageRank Contrary to what one would normally expect when running a PageRank, our graph is undirected. It turns out that for our recommender, the lack of direction does not matter, since we are simply trying to find similarities between Led Zeppelin and Spirit. A possible way of introducing direction could be to look at the song publishing date. In order to find musical influences, we could certainly introduce a chronology from the oldest to newest songs giving directionality to our edges. In the following pageRank, we define a probability of 15% to skip, or teleport as it is known, to any random song, but this can be obviously tuned for different needs: val prGraph = graph.pageRank(0.001, 0.15) Finally, we extract the page ranked vertices and save them as a playlist in Cassandra via an RDD of the Song case class: case class Song(id: Long, name: String, commonality: Double) val vertices = prGraph .vertices .mapPartitions { vertices => val songIds = songIdsB .value .vertices .map { case (songId, pr) => val songName = songIds.get(vId).get Song(songId, songName, pr) } } vertices.saveAsCassandraTable("gzet", "playlist") The reader may be pondering the exact purpose of PageRank here, and how it could be used as a recommender? In fact, our use of PageRank means that the highest ranking songs would be the ones that share many frequencies with other songs. This could be due to a common arrangement, key theme, or melody; or maybe because a particular artist was a major influence on a musical trend. However, these songs should be, at least in theory, more popular (by virtue of the fact they occur more often), meaning that they are more likely to have mass appeal. On the other end of the spectrum, low ranking songs are ones where we did not find any similarity with anything we know. Either these songs are so avant-garde that no one has explored these musical ideas before, or alternatively are so bad that no one ever wanted to copy them! Maybe they were even composed by that up-and-coming artist you were listening to in your rebellious teenage years. Either way, the chance of a random user liking these songs is treated as negligible. Surprisingly, whether it is a pure coincidence or whether this assumption really makes sense, the lowest ranked song from this particular audio library is Daft Punk's–Motherboard it is a title that is quite original (a brilliant one though) and a definite unique sound. To summarize, we have learnt how to build a complete recommendation system for a song playlist. You can check out the book Mastering Spark for Data Science to deep dive into Spark and deliver other production grade data science solutions. Read our post on how deep learning is revolutionizing the music industry. And here is how you can analyze big data using the pagerank algorithm.  
Read more
  • 0
  • 0
  • 47410

article-image-ten-tips-to-successfully-migrate-from-on-premise-to-microsoft-azure
Savia Lobo
13 Dec 2019
11 min read
Save for later

Ten tips to successfully migrate from on-premise to Microsoft Azure 

Savia Lobo
13 Dec 2019
11 min read
The decision to start using Azure Cloud Services for your IT infrastructure seems simple. However, to succeed, a cloud migration requires hard work and good planning. At Microsoft Ignite 2018, Eric Berg, an Azure Lead Architect at COMPAREX, a Microsoft MVP Azure + Cloud and Data Center Management, shared ‘Ten tips for a successful migration from on-premises to Azure’, based on their day-to-day learnings. Eric shares known issues, common pitfalls, and best practices to get started. Further Reading To gain a deep understanding of various Azure services related to infrastructure, applications, and environments, you can check out our book Microsoft Azure Administrator – Exam Guide AZ-103 by Sjoukje Zaal. This book is also an effective guide for acquiring the skills needed to pass the Exam AZ-103, with effective mock tests and solutions so that you can confidently crack this exam. Tip #1: Have your Azure Governance Set One needs to have a basic plan of what they are going to do with Azure. Consider Azure Governance as the basis for Cloud Adoption. Berg says, “if you don't have a plan for what you do with Azure, it will hurt you.” To run something on Azure is good, but to keep it secure is the key thing. Here, Governance rule sets help users to audit and figure out if everything is running as expected. One of the key parts of Azure Governance is Networking. Hence one should consider a networking concept that suits both the company and the business. Microsoft is moving really fast; in 2018, to connect to the US and Europe you had to use a VPN then came global v-net peering, and now we have ESRI virtual WAN. Such advancements allow a concept to further grow and always use the top of the edge technologies while adoption of such a rule set enables customers to try a lot of things on their own. Tip #2: Think about different requirements From an IT perspective, every organization wants control, focus on its IT, and also to ensure that everything is compliant. Many organizations also want to write policies in place. On the other hand, the human resource department section wants to be totally agile and innovative and wants to consume services and self-service without feeling the need to communicate with IT. “I've seen so many human resource departments doing their own contracts with external partners building some fancy new hiring platforms and IT didn't know anything about it,” Berg points out. When it comes to Cloud, each and every member of the company should be aware and should be involved. It is simply not just an IT-dependent decision, but is company dependent. Tip #3: Assess your infrastructure Berg says organizations should assess their environment. Migrating your servers as they are to Azure is not the right thing to do. This is because in Azure the decision between 8 and 16 gigabytes of RAM is a decision between 100 and 200 percent of the cost. Hence, right scaling or a good assessment is extremely important and this cannot be achieved by running a script once for 10 minutes and you know what your VMs are doing. Instead, you should at least run an assessment for one month or even three months to see some peaks and some low times. This is like a good assessment where you know what you really need to migrate your systems better. Keep a check on your inventory and also on your contracts to check if you are allowed to migrate your ERP system or CRM system to Azure. As some contracts state that the “deployment of this solution outside of the premises of the company needs some extra contract and some extra cost,” Berg warns. Migrating to Azure is technically easy but difficult from a contract perspective. Also, you should define your needs for migration to a cloud platform. If you don't get value out of your migration don't do it. Berg advises, don't migrate to Azure because everybody does or because it's cool or fancy. Tip #4: Do not rebuild your on-premises structures on Cloud Cloud needs trust. Organizations often try to bring in the old stuff on the on-premises infrastructures such as the external DMZ, the internal DMZ, and also 15 security layers. Berg said they use intune, a cloud-based service in the enterprise mobility management (EMM) space that helps enable your workforce to be productive while keeping your corporate data protected, along with Office 365 on a cloud.  In tune doesn't stick to a DMZ; even if you want to deploy your application or use the latest tech such as BOTS, cognitive services, etc. It may not fit totally into a structured network design on the cloud. On the other hand, there will be disconnected subscriptions, i.e. there will be subscriptions with no connection to your on-premises network. This problem has to be dealt with on a security level. New services need new ways. If you are not agile your IT won't be agile. If you need 16 days or six weeks to deploy a server and you want to stick to those rules and processes, then Azure won't be beneficial for you as there will be no value in it for you. Tip #5: Azure consumption is billed If you spin up a VM that costs $25,000 a month you have to pay for it. The M-series VMs have 128 cores 4 terabytes of RAM and are simply amazing. If you deployed using Windows Server and SQL Server Enterprise, the cost goes up to $58,000 a month for just one VM. When you migrate to Azure and you start integrating new things you probably have to change your own business model. To implement tech such as facial recognition, and others you have to set up a cost management tool for usage tracking. There are many usage APIs and third-party tools available. Proper cost management into the Azure infrastructure helps to divide costs. If you put everything into one subscription, one resource group, where everyone is the owner. Here, the problem won’t be the functioning but you will not be able to figure out who's responsible for what. Instead, a good structure of subscriptions, a good role-based access control, a good tagging policy will help you to figure out cost better. Tip #6: Identity is the new perimeter Azure Ad is the center of everything. To access a user’s data center is not easy these days as it needs access within the premises, then into the data center, then log into the user’s own premises infrastructure. If anyone has a user’s login ID, they are inside the user’s Azure AD, the user’s visa VPN, and also on their on-premises data center. Hence identity is a key part of security. “So, don’t think about using MFA, use MFA. Don't think about using Privileged Identity Management, use it because that's the only way to secure your infrastructure probably and get an insight into who is using what in my infrastructure and how is it going,” Berg warns. In the modern workplace, one can work from anywhere. However, one needs to have proper security levels in place. Secure devices, secure identity, secure access ways to MFA, and so on. Stay cautious. Tip #7: Include your users Users are the most important part of any ecosystem. So, when you migrate servers or the entire on-premise architecture, inform them. What if you have a CRM system fully in the cloud and there's no local cache on the system anymore? This won't fit the needs of your customers or internal customers and this is why organizations should inform them of their plans. They should also ask them what they really need and this will, in turn, help the organizations. Berg illustrated this point with a project in Germany that includes a customer with a very specific project that wanted the product to decrease their response times. The client needs up to two days to answer a customer's email because the project product is very complex and they have a very spread documentation library and it's hard. Their internal goal is to bring down the product response to ten minutes--from two days to 10 minutes. Berg said they considered using a bot, some cognitive services and Azure search, and a plug-in an Outlook. So you get the mail you just search for your product and everything will be figured out. The documentation, the fact sheets, and the standard email template for answering such a thing. The solution proposed was good; both Berg and the IT liked it. However, when the sales team was asked, they said such a solution would steal their jobs. The mistake here was Sales was not included in the process of finding this solution. To rectify this, organizations should include all stakeholders. Focus on benefits, have some key users because they will help you to spread the word over. In the above case, explain and evangelize the sales teams as they are afraid because they don't know and don't understand what happens if you have a bot and some cognitive services to figure out which document is right. This won’t steal their job but instead, help to do better at their job with improved efficiency. Train and educate so they are able to use it, check processes and consider changes. Managed services can help you focus. Back up, monitoring, patching, this is something somebody can do for you. Instead, organizations can now focus on after the migration such as integrating new services, improving right scaling, optimizing cost, optimizing performance, staying up-to-date with all the changes in Azure, etc. Tip #8: Consider Transformation instead of Migration Consider a transformation instead of a migration. Build some logical blocks, don't move an ERP system without your database or the other way around. Berg suggests: To adopt technical and licensing showstoppers define your infrastructure requirements check your compatibility to migrate update helpdesk about SLAs Ask if Azure is really helping me (to figure out or to cover my assets or is it getting better or maybe worse). Tip #9: Keep up to date Continuous learning and continuous knowledge are key to growth. As Azure releases a lot of changes very often, users are notified of these latest updates via emails or via Azure news. Organizations should review their architecture on a regular basis, Berg says. VPN to global v-net peering to Global WAN so that you can change your infrastructure quite fast. Audit your governance not on a yearly basis may be monthly or quarterly. Consider changes fast; don't think two years about a change because then it will not be any more interesting. If there's a new opportunity, grab it, use it and three weeks later probably drop it away. But avoid thinking for two months or more else it will be too late. Tip #10: Plan for the future Do some end to end planning, think about the end-to-end solution; who's using it, what's my back end on this, and so on. Save money and forecast your costs. Keep an eye on resources that probably spread because someone runs the script without knowing what they are doing.  Simply migrating an IIS server with a static website to Azure is not actual cloud migration. Instead, customers should consider moving their servers to a static storage website, to a web app, etc. but not in the Windows VM. Berg concludes by saying that an important migration step is to move from infrastructure. Everybody migrates infrastructure to Azure because that's easy because it's just migrating from one VM to another VM. Customers should not ‘only’ migrate. They should also start an optimization, move forward to platform services, be more agile, think about new ways and most importantly get rid of all on-premise old stuff. Berg adds, “In five years probably nobody will talk about infrastructure as a service anymore because everybody has migrated and optimized it already.” To stay more compliant with corporate standards and SLAs, learn how to configure Azure subscription policies with “Microsoft Azure Administrator – Exam Guide AZ-103” by Packt Publishing. 5 reasons Node.js developers might actually love using Azure [Sponsored by Microsoft] Azure Functions 3.0 released with support for .NET Core 3.1! Microsoft announces Azure Quantum, an open cloud ecosystem to learn and build scalable quantum solutions
Read more
  • 0
  • 0
  • 47349

article-image-design-documentation
Packt
23 Jan 2014
19 min read
Save for later

The Design Documentation

Packt
23 Jan 2014
19 min read
(For more resources related to this topic, see here.) The design documentation provides written documentation of the design factors and the choices the architect has made in the design to satisfy the business and technical requirements. The design documentation also aids in the implementation of the design. In many cases where the design architect is not responsible for the implementation, the design documents ensure the successful implementation of the design by the implementation engineer. Once you have created the documentation for a few designs, you will be able to develop standard processes and templates to aid in the creation of design documentation. Documentation can vary from project to project. Many consulting companies and resellers have standard documentation templates that they use when designing solutions. A properly documented design should include the following information: Architecture design Implementation plan Installation guide Validation test plan Operational procedures This information can be included in a single document or separated into different documents. VMware provides Service Delivery Kits to VMware partners. These kits can be found on the VMware Partner University portal at http://www.vmware.com/go/partneruniversity, which provides documentation templates that can be used as a foundation for creating design documents. If you do not have access to these templates, example outlines are provided in this article to assist you in developing your own design documentation templates. The final steps of the design process include gaining customer approval to begin implementation of the design and the implementation of the design. Creating the architecture design document The architecture design document is a technical document describing the components and specifications required to support the solution and ensure that the specific business and technical requirements of the design are satisfied. An excellent example of an architecture design document is the Cloud Infrastructure Architecture Case Study White Paper article that can be found at http://www.vmware.com/files/pdf/techpaper/cloud-infrastructure-achitecture-case-study.pdf. The architect creates the architecture design document to document the design factors and the specific choices that have been made to satisfy those factors. The document serves as a way for the architect to show his work when making design decisions. The architecture design document includes the conceptual, logical, and physical designs. How to do it... The architecture design document should include the following information: Purpose and overview Executive summary Design methodology Conceptual design Logical management, storage, compute, and network design Physical management, storage, compute, and network design How it works... The Purpose and Overview section of the architecture design includes the Executive Summary section. The Executive Summary section provides a high-level overview of the design and the goals the design will accomplish, and defines the purpose and scope of the architecture design document. The following is an example executive summary in the Cloud Infrastructure Architecture Case Study White Paper : Executive Summary: This architecture design was developed to support a virtualization project to consolidate 100 existing physical servers on to a VMware vSphere 5.x virtual infrastructure. The primary goals this design will accomplish are to increase operational efficiency and to provide high availability of customer-facing applications. This document details the recommended implementation of a VMware virtualization architecture based on specific business requirements and VMware recommended practices. The document provides both logical and physical design considerations for all related infrastructure components including servers, storage, networking, management, and virtual machines. The scope of this document is specific to the design of the virtual infrastructure and the supporting components. The purpose and overview section should also include details of the design methodology the architect has used in creating the architecture design. This should include the processes followed to determine the business and technical requirements along with definitions of the infrastructure qualities that influenced the design decisions. Design factors, requirements, constraints, and assumptions are documented as part of the conceptual design. To document the design factors, use a table to organize them and associate them with an ID that can be easily referenced. The following table illustrates an example of how to document the design requirements: ID Requirement R001 Consolidate the existing 100 physical application servers down to five servers R002 Provide capacity to support growth for 25 additional application servers over the next five years R003 Server hardware maintenance should not affect application uptime R004 Provide N+2 redundancy to support a hardware failure during normal and maintenance operations The conceptual design should also include tables documenting any constraints and assumptions. A high-level diagram of the conceptual design can also be included. Details of the logical design are documented in the architecture design document. The logical design of management, storage, network, and compute resources should be included. When documenting the logical design document, any recommended practices that were followed should be included. Also include references to the requirements, constraints, and assumptions that influenced the design decisions. When documenting the logical design, show your work to support your design decisions. Include any formulas used for resource calculations and provide detailed explanations of why design decisions were made. An example table outlining the logical design of compute resource requirements is as follows: Parameter Specification Current CPU resources required 100 GHz *CPU growth 25 GHz CPU required (75 percent utilization) 157 GHz Current memory resources required 525 GB *Memory growth 131 GB Memory required (75 percent utilization) 821 GB Memory required (25 percent TPS savings) 616 GB *CPU and memory growth of 25 additional application servers (R002) Similar tables will be created to document the logical design for storage, network, and management resources. The physical design documents have the details of the physical hardware chosen along with the configurations of both the physical and virtual hardware. Details of vendors and hardware models chosen and the reasons for decisions made should be included as part of the physical design. The configuration of the physical hardware is documented along with the details of why specific configuration options were chosen. The physical design should also include diagrams that document the configuration of physical resources, such as physical network connectivity and storage layout. A sample outline of the architecture design document is as follows: Cover page: It includes the customer and project names Document version log: It contains the log of authors and changes made to the document Document contacts: It includes the subject matter experts involved in the creation of the design Table of contents: It is the index of the document sections for quick reference List of tables: It is the index of tables included in the document for quick reference List of figures: It is the index of figures included in the document for quick reference Purpose and overview: This section consists of an executive summary to provide an overview of the design and the design methodology followed in creating the design Conceptual design: It is the documentation of the design factors: requirements, constraints, and assumptions Logical design: It has the details of the logical management, storage, network, and compute design Physical design: It contains the details of the selected hardware and the configuration of the physical and virtual hardware Writing an implementation plan The implementation plan documents the requirements necessary to complete the implementation of the design. The implementation plan defines the project roles and defines what is expected of the customer and what they can expect during the implementation of the design. This document is sometimes referred to as the statement of work. It defines the key points of contact, the requirements that must be satisfied to start the implementation, any project documentation deliverables, and how changes to the design and implementation will be handled. How to do it... The implementation plan should include the following information: Purpose statement Project contacts Implementation requirements Overview of implementation steps Definition of project documentation deliverables Implementation of change management How it works... The purpose statement defines the purpose and scope of the document. The purpose statement of the implementation plan should define what is included in the document and provide a brief overview of the goals of the project. The purpose statement is simply an introduction so that someone reading the document can gain a quick understanding of what the document contains. The following is an example purpose statement: This document serves as the implementation plan and defines the scope of the virtualization project. This document identifies points of contact for the project, lists implementation requirements, provides a brief description of each of the document deliverables, deliverables, and provides an overview of the implementation process for the data-center virtualization project. The scope of this document is specific to the implementation of the virtual data-center implementation and the supporting components as defined in the Architecture Design. Key project contacts, their roles, and their contact information should be included as part of the implementation plan document. These contacts include customer stakeholders, project managers, project architects, and implementation engineers. The following is a sample table that can be used to document project contacts for the implementation plan: Role Name Contact information Customer project sponsor     Customer technical resource     Project manager     Design architect     Implementation engineer     QA engineer     Support contacts for hardware and software used in the implementation plan may also be included in the table, for example, contact numbers for VMware support or other vendor support. Implementation requirements contain the implementation dependencies to include the access and facility requirements. Any hardware, software, and licensing that must be available to implement the design is also documented here. Access requirements include the following: Physical access to the site. Credentials necessary for access to resources. These include active directory credentials and VPN credentials (if remote access is required). Facility requirements include the following: Power and cooling to support the equipment that will be deployed as part of the design Rack space requirements Hardware, software, and licensing requirements include the following: vSphere licensing Windows or other operating system licensing Other third-party application licensing Software (ISO, physical media, and so on) Physical hardware (hosts, array, network switches, cables, and so on) A high-level overview of the steps required to complete the implementation is also documented. The details of each step are not a part of this document; only the steps that need to be performed will be included. For example: Procurement of hardware, software, and licensing. Scheduling of engineering resources. Verification of access and facility requirements. Performance of an inventory check for the required hardware, software, and licensing. Installation and configuration of storage array. Rack, cable, and burn-in of physical server hardware. Installation of ESXi on physical servers. Installation of vCenter Server. Configuration of ESXi and vCenter. Testing and verification of implementation plan. Migration of physical workloads to virtual machines. Operational verification of the implementation plan. The implementation overview may also include an implementation timeline documenting the time required to complete each of the steps. Project documentation deliverables are defined as part of the implementation plan. Any documentation that will be delivered to the customer once the implementation has been completed should be detailed here. Details include the name of the document and a brief description of the purpose of the document. The following table provides example descriptions of the project documentation deliverables: Document Description Architecture design This is a technical document describing the vSphere components and specifications required to achieve a solution that addresses the specific business and technical requirements of the design. Implementation plan This identifies implementation roles and requirements. It provides a high-level map of the implementation and deliverables detailed in the design. It documents change management procedures. Installation guide This document provides detailed, step-by-step instructions on how to install and configure the products specified in the architecture design document. Validation test plan This document provides an overview of the procedures to be executed post installation to verify whether or not the infrastructure is installed correctly. It can also be used at any point subsequent to the installation to verify whether or not the infrastructure continues to function correctly. Operational procedures This document provides detailed, step-by-step instructions on how to perform common operational tasks after the design is implemented. How changes are made to the design, specifically changes made to the design factors, must be well documented. Even a simple change to a requirement or an assumption that cannot be verified can have a tremendous effect on the design and implementation. The process for submitting a change, researching the impact of the change, and approving the change should be documented in detail. The following is an example outline for an implementation plan: Cover page: It includes the customer and project names Document version log: It contains the log of authors and changes made to the document Document contacts: It includes the subject matter experts involved in the creation of the design Table of contents: It is the index of document sections for quick reference List of tables: It is the index of tables included in the document for quick reference List of figures: It is the index of figures included in the document for quick reference Purpose statement: It defines the purpose of the document Project contacts: It is the documentation of key project points of contact Implementation requirements: It provides the access, facilities, hardware, software, and licensing required to complete the implementation Implementation overview: It is the overview of the steps required to complete the implementation Project deliverables: It consists of the documents that will be provided as deliverables once implementation has been completed Developing an installation guide The installation guide provides step-by-step instructions for the implementation of the architecture design. This guide should include detailed information about how to implement and configure all the resources associated with the virtual datacenter project. In many projects, the person creating the design is not the person responsible for implementing the design. The installation guide outlines the steps necessary to implement the physical design outlined in the architecture design document. The installation guide should provide details about the installation of all components, including the storage and network configurations required to support the design. In a complex design, multiple installation guides may be created to document the installation of the various components required to support the design. For example, separate installation guides may be created for the storage, network, and vSphere installation and configuration. How to do it... The installation guide should include the following information: Purpose statement Assumption statement Step-by-step instructions to implement the design How it works... The purpose statement simply states the purpose of the document. The assumption statement describes any assumptions the document's author has made. Commonly, an assumption statement simply states that the document has been written, assuming that the reader is familiar with virtualization concepts and the architecture design. The following is an example of a basic purpose and assumption statement that can be used for an installation guide: Purpose: This document provides a guide for installing and configuring the virtual infrastructure design defined in the Architecture Design. Assumptions: This guide is written for an implementation engineer or administrator who is familiar with vSphere concepts and terminologies. The guide is not intended for administrators who have no prior knowledge of vSphere concepts and terminology. The installation guide should include details on implementing all areas of the design. It should include configuration of the storage array, physical servers, physical network components, and vSphere components. The following are just a few examples of installation tasks to include instructions for: Storage array configurations Physical network configurations Physical host configurations ESXi installation vCenter Server installation and configuration Virtual network configuration Datastore configuration High availability, distributed resource scheduler, storage DRS, and other vSphere components installation and configuration The installation guide should provide as much detail as possible. Along with the step-by-step procedures, screenshots can be used to provide installation guidance. The following screenshot is an example taken from an installation guide that details enabling and configuring the Software iSCSI adapter: The following is an example outline for an installation guide: Cover page: It includes the customer and project names Document version log: It contains the log of authors and changes made to the document Document contacts: It includes the subject matter experts involved in the creation of the design Table of contents: It is the index of document sections for quick reference List of tables: It is the index of tables included in the document for quick reference List of figures: It is the index of figures included in the document for quick reference Purpose statement: It defines the purpose of the document Assumption statement: It defines any assumptions made in creating the document Installation guide: It provides the step-by-step installation instructions to be followed when implementing the design Creating a validation test plan The validation test plan documents how the implementation will be verified. It documents the criteria that must be met to determine the success of the implementation and the test procedures that should be followed when validating the environment. The criteria and procedures defined in the validation test plan determine whether or not the design requirements have been successfully met. How to do it... The validation test plan should include the following information: Purpose statement Assumption statement Success criteria Test procedures How it works... The purpose statement defines the purpose of the validation test plan and the assumption statement documents any assumptions the author of the plan has made in developing the test plan. Typically, the assumptions are that the testing and validation will be performed by someone who is familiar with the concepts and the design. The following is an example of a purpose and assumption statement for a validation test plan: Purpose: This document contains testing procedures to verify that the implemented configurations specified in the Architecture Design document successfully addresses the customer requirements. Assumptions: This document assumes that the person performing these tests has a basic understanding of VMware vSphere and is familiar with the accompanying design documentation. This document is not intended for administrators or testers who have no prior knowledge of vSphere concepts and terminology. The success criteria determines whether or not the implemented design is operating as expected. More importantly, these criteria determine whether or not the design requirements have been met. Success is measured based on whether or not the criteria satisfies the design requirements. The following table shows some examples of success criteria defined in the validation test plan: Description Measurement Members of the active directory group vSphere administrators are able to access vCenter as administrators Yes/No Access is denied to users outside the vSphere administrators active directory group Yes/No Access to a host using the vSphere Client is permitted when lockdown mode is disabled Yes/No Access to a host using the vSphere Client is denied when lockdown mode is enabled Yes/No Cluster resource utilization is less than 75 percent. Yes/No If the success criteria are not met, the design does not satisfy the design factors. This can be due to a misconfiguration or error in the design. Troubleshooting will need to be done to identify the issue or modifications to the design may need to be made. Test procedures are performed to determine whether or not the success criteria have been met. Test procedures should include testing of usability, performance, and recoverability. Test procedures should include the test description, the tasks to perform the test, and the expected results of the test. The following table provides some examples of usability testing procedures: Test description Tasks to perform test Expected result vCenter administrator access Use the vSphere Web Client to access the vCenter Server. Log in as a user who is a member of the vSphere administrators AD group. Administrator access to the inventory of the vCenter Server vCenter access: No permissions Use the vSphere Web Client to access the vCenter Server. Log in as a user who is not a member of the vSphere administrators AD group. Access is denied Host access: lockdown mode disabled Disable lockdown mode through the DCUI. Use the vSphere Client to access the host and log in as root. Direct access to the host using the vSphere Client is successful Host access: lockdown mode enabled Re-enable lockdown mode through the DCUI. Use the vSphere Client to access the host and log in as root. Direct access to the host using the vSphere Client is denied The following table provides some examples of reliability testing procedures: Test description Tasks to perform test Expected result Host storage path failure Disconnect a vmnic providing IP storage connectivity from the host The disconnected path fails, but I/O continues to be processed on the surviving paths. A network connectivity alarm should be triggered and an e-mail should be sent to the configured e-mail address. Host storage path restore Reconnect the vmnic providing IP storage connectivity The failed path should become active and begin processing the I/O. Network connectivity alarms should clear. Array storage path failure Disconnect one network connection from the active SP The disconnected paths fail on all hosts, but I/O continues to be processed on the surviving paths. Management network redundancy Disconnect the active management network vmnic The stand-by adapter becomes active. Management access to the host is not interrupted. A loss-of-network redundancy alarm should be triggered and an e-mail should be sent to the configured e-mail address. These are just a few examples of test procedures. The actual test procedures will depend on the requirements defined in the conceptual design. The following is an example outline of a validation test plan: Cover page: It includes the customer and project names Document version log: It contains the log of authors and changes made to the document Document contacts: It includes the subject matter experts involved in the creation of the design Table of contents: It is the index of document sections for quick reference List of tables: It is the index of tables included in the document for quick reference List of figures: It is the index of figures included in the document for quick reference Purpose statement: It defines the purpose of the document Assumption statement: It defines any assumptions made in creating the document Success criteria: It is a list of criteria that must be met to validate the successful implementation of the design Test Procedures: It is a list of test procedures to follow, including the steps to follow and the expected results
Read more
  • 0
  • 2
  • 47263

article-image-working-with-shaders-in-c-to-create-3d-games
Amarabha Banerjee
15 Jun 2018
28 min read
Save for later

Working with shaders in C++ to create 3D games

Amarabha Banerjee
15 Jun 2018
28 min read
A shader is a computer program that is used to do image processing such as special effects, color effects, lighting, and, well, shading. The position, brightness, contrast, hue, and other effects on all pixels, vertices, or textures used to produce the final image on the screen can be altered during runtime, using algorithms constructed in the shader program(s). These days, most shader programs are built to run directly on the Graphical Processing Unit (GPU). In this article, we are going to get acquainted with shaders and implement our own shader infrastructure for the example engine. Shader programs are executed in parallel. This means, for example, that a shader might be executed once per pixel, with each of the executions running simultaneously on different threads on the GPU. The amount of simultaneous threads depends on the graphics card specific GPU, with modern cards sporting processors in the thousands. This all means that shader programs can be very performant and provide developers with lots of creative flexibility. The following article is a part of the book Mastering C++ game Development written by Mickey Macdonald. With this book, you can create advanced games with C++. Shader languages With advances in graphics card technology, more flexibility has been added to the rendering pipeline. Where at one time developers had little control over concepts such as fixed-function pipeline rendering, new advancements have allowed programmers to take deeper control of graphics hardware for rendering their creations. Originally, this deeper control was achieved by writing shaders in assembly language, which was a complex and cumbersome task. It wasn't long before developers yearned for a better solution. Enter the shader programming languages. Let's take a brief look at a few of the more common languages in use. C for graphics (Cg) is a shading language originally developed by the Nvidia graphics company. Cg is based on the C programming language and, although they share the same syntax, some features of C were modified and new data types were added to make Cg more suitable for programming GPUs. Cg compilers can output shader programs supported by both DirectX and OpenGL. While Cg was mostly deprecated, it has seen a resurgence in a new form with its use in the Unity game engine. High-Level Shading Language (HLSL) is a shading language developed by the Microsoft Corporation for use with the DirectX graphics API. HLSL is again modeled after the C programming language and shares many similarities to the Cg shading language. HLSL is still in development and continues to be the shading language of choice for DirectX. Since the release, DirectX 12 the HLSL language supports even lower level hardware control and has seen dramatic performance improvements. OpenGL Shading Language (GLSL) is a shading language that is also based on the C programming language. It was created by the OpenGL Architecture Review Board (OpenGL ARB) to give developers more direct control of the graphics pipeline without having to use ARB assembly language or other hardware-specific languages. The language is still in open development and will be the language we will focus on in our examples. Building a shader program infrastructure Most modern shader programs are composed of up to five different types of shader files: fragment or pixel shaders, vertex shaders, geometry shaders, compute shaders, and tessellation shaders. When building a shader program, each of these shader files must be compiled and linked together for use, much like how a C++ program is compiled and linked. Next, we are going to walk you through how this process works and see how we can build an infrastructure to allow for easier interaction with our shader programs. To get started, let's look at how we compile a GLSL shader. The GLSL compiler is part of the OpenGL library itself, and our shaders can be compiled within an OpenGL program. We are going to build an architecture to support this internal compilation. The whole process of compiling a shader can be broken down into some simple steps. First, we have to create a shader object, then provide the source code to the shader object. We can then ask the shader object to be compiled. These steps can be represented in the following three basic calls to the OpenGL API. First, we create the shader object: GLuint vertexShader = glCreateShader(GL_VERTEX_SHADER); We create the shader object using the glCreateShader() function. The argument we pass in is the type of shader we are trying to create. The types of shaders can be GL_VERTEX_SHADER, GL_FRAGMENT_SHADER, GL_GEOMETRY_SHADER, GL_TESS_EVALUATION_SHADER, GL_TESS_CONTROL_SHADER, or GL_COMPUTE_SHADER. In our example case, we are trying to compile a vertex shader, so we use the GL_VERTEX_SHADER type. Next, we copy the shader source code into the shader object: GLchar* shaderCode = LoadShader("shaders/simple.vert"); glShaderSource(vertexShader, 1, shaderCode, NULL); Here we are using the glShaderSource() function to load our shader source to memory. This function accepts an array of strings, so before we call glShaderSource(), we create a pointer to the start of the shaderCode array object using a still-to-be-created method. The first argument to glShaderSource() is the handle to the shader object. The second is the number of source code strings that are contained in the array. The third argument is a pointer to an array of source code strings. The final argument is an array of GLint values that contains the length of each source code string in the previous argument. Finally, we compile the shader: glCompileShader(vertexShader); The last step is to compile the shader. We do this by calling the OpenGL API method, glCompileShader(), and passing the handle to the shader that we want compiled. Of course, because we are using memory to store the shaders, we should know how to clean up when we are done. To delete a shader object, we can call the glDeleteShader() function. Deleting a Shader ObjectShader objects can be deleted when no longer needed by calling glDeleteShader(). This frees the memory used by the shader object. It should be noted that if a shader object is already attached to a program object, as in linked to a shader program, it will not be immediately deleted, but rather flagged for deletion. If the object is flagged for deletion, it will be deleted when it is detached from the linked shader program object. Once we have compiled our shaders, the next step we need to take before we can use them in our program is to link them together into a complete shader program. One of the core aspects of the linking step involves making the connections between input variables from one shader to the output variables of another and making the connections between the input/output variables of a shader to appropriate locations in the OpenGL program itself. Linking is much like compiling the shader. We create a new shader program and attach each shader object to it. We then tell the shader program object to link everything together. The steps to accomplish this in the OpenGL environment can be broken down into a few calls to the API, as follows: First, we create the shader program object: GLuint shaderProgram = glCreateProgram(); To start, we call the glCreateProgram() method to create an empty program object. This function returns a handle to the shader program object which, in this example, we are storing in a variable named shaderProgram. Next, we attach the shaders to the program object: glAttachShader(shaderProgram, vertexShader); glAttachShader(shaderProgram, fragmentShader); To load each of the shaders into the shader program, we use the glAttachShader() method. This method takes two arguments. The first argument is the handle to the shader program object, and the second is the handle to the shader object to be attached to the shader program. Finally, we link the program: glLinkProgram(programHandle); When we are ready to link the shaders together we call the glLinkProgram() method. This method has only one argument: the handle to the shader program we want to link. It's important that we remember to clean up any shader programs that we are not using anymore. To remove a shader program from the OpenGL memory, we call glDeleteProgram() method. The glDeleteProgram() method takes one argument: the handle to the shader program that is to be deleted. This method call invalidates the handle and frees the memory used by the shader program. It is important to note that if the shader program object is currently in use, it will not be immediately deleted, but rather flagged for deletion. This is similar to the deletion of shader objects. It is also important to note that the deletion of a shader program will detach any shader objects that were attached to the shader program at linking time. This, however, does mean the shader object will be deleted immediately unless those shader objects have already been flagged for deletion by a previous call to the glDeleteShader() method. So those are the simplified OpenGL API calls required to create, compile, and link shader programs. Now we are going to move onto implementing some structure to make the whole process much easier to work with. To do this, we are going to create a new class called ShaderManager. This class will act as the interface for compiling, linking, and managing the cleanup of shader programs. To start with, let's look at the implementation of the CompileShaders() method in the ShaderManager.cpp file. I should note that I will be focusing on the important aspects of the code that pertain to the implementation of the architecture. The full source code for this chapter can be found in the Chapter07 folder in the GitHub repository. void ShaderManager::CompileShaders(const std::string& vertexShaderFilePath, const std::string& fragmentShaderFilepath) { m_programID = glCreateProgram(); m_vertexShaderID = glCreateShader(GL_VERTEX_SHADER); if (m_vertexShaderID == 0){ Exception("Vertex shader failed to be created!"); } m_fragmentShaderID = glCreateShader(GL_FRAGMENT_SHADER); if (m_fragmentShaderID == 0){ Exception("Fragment shader failed to be created!"); } CompileShader(vertexShaderFilePath, m_vertexShaderID); CompileShader(fragmentShaderFilepath, m_fragmentShaderID); } To begin, for this example we are focusing on two of the shader types, so our ShaderManager::CompileShaders() method accepts two arguments. The first argument is the file path location of the vertex shader file, and the second is the file path location to the fragment shader file. Both are strings. Inside the method body, we first create the shader program handle using the glCreateProgram() method and store it in the m_programID variable. Next, we create the handles for the vertex and fragment shaders using the glCreateShader() command. We check for any errors when creating the shader handles, and if we find any we throw an exception with the shader name that failed. Once the handles have been created, we then call the CompileShader() method, which we will look at next. The CompileShader() function takes two arguments: the first is the path to the shader file, and the second is the handle in which the compiled shader will be stored. The following is the full CompileShader() function. It handles the look and loading of the shader file from storage, as well as calling the OpenGL compile command on the shader file. We will break it down chunk by chunk: void ShaderManager::CompileShader(const std::string& filePath, GLuint id) { std::ifstream shaderFile(filePath); if (shaderFile.fail()){ perror(filePath.c_str()); Exception("Failed to open " + filePath); } //File contents stores all the text in the file std::string fileContents = ""; //line is used to grab each line of the file std::string line; //Get all the lines in the file and add it to the contents while (std::getline(shaderFile, line)){ fileContents += line + "n"; } shaderFile.close(); //get a pointer to our file contents c string const char* contentsPtr = fileContents.c_str(); //tell opengl that we want to use fileContents as the contents of the shader file glShaderSource(id, 1, &contentsPtr, nullptr); //compile the shader glCompileShader(id); //check for errors GLint success = 0; glGetShaderiv(id, GL_COMPILE_STATUS, &success); if (success == GL_FALSE){ GLint maxLength = 0; glGetShaderiv(id, GL_INFO_LOG_LENGTH, &maxLength); //The maxLength includes the NULL character std::vector<char> errorLog(maxLength); glGetShaderInfoLog(id, maxLength, &maxLength, &errorLog[0]); //Provide the infolog in whatever manor you deem best. //Exit with failure. glDeleteShader(id); //Don't leak the shader. //Print error log and quit std::printf("%sn", &(errorLog[0])); Exception("Shader " + filePath + " failed to compile"); } } To start the function, we first use an ifstream object to open the file with the shader code in it. We also check to see if there were any issues loading the file and if, there were, we throw an exception notifying us that the file failed to open: std::ifstream shaderFile(filePath); if (shaderFile.fail()) { perror(filePath.c_str()); Exception("Failed to open " + filePath); } Next, we need to parse the shader. To do this, we create a string variable called fileContents that will hold the text in the shader file. We then create another string variable named line; this will be a temporary holder for each line of the shader file we are trying to parse. Next, we use a while loop to step through the shader file, parsing the contents line by line and saving each loop into the fileContents string. Once all the lines have been read into the holder variable, we call the close method on the shaderFile ifstream object to free up the memory used to read the file: std::string fileContents = ""; std::string line; while (std::getline(shaderFile, line)) { fileContents += line + "n"; } shaderFile.close(); You might remember from earlier in the chapter that I mentioned that when we are using the glShaderSource() function, we have to pass the shader file text as a pointer to the start of a character array. In order to meet this requirement, we are going to use a neat trick where we use the C string conversation method built into the string class to allow us to pass back a pointer to the start of our shader character array. This, in case you are unfamiliar, is essentially what a string is: const char* contentsPtr = fileContents.c_str(); Now that we have a pointer to the shader text, we can call the glShaderSource() method to tell OpenGL that we want to use the contents of the file to compile our shader. Then, finally, we call the glCompileShader() method with the handle to the shader as the argument: glShaderSource(id, 1, &contentsPtr, nullptr); glCompileShader(id); That handles the compilation, but it is a good idea to provide ourselves with some debug support. We implement this compilation debug support by closing out the CompileShader() function by first checking to see if there were any errors during the compilation process. We do this by requesting information from the shader compiler through glGetShaderiv() function, which, among its arguments, takes an enumerated value that specifies what information we would like returned. In this call, we are requesting the compile status: GLint success = 0; glGetShaderiv(id, GL_COMPILE_STATUS, &success); Next, we check to see if the returned value is GL_FALSE, and if it is, that means we have had an error and should ask the compiler for more information about the compile issues. We do this by first asking the compiler what the max length of the error log is. We use this max length value to then create a vector of character values called errorLog. Then we can request the shader compile log by using the glGetShaderInfoLog() method, passing in the handle to the shader file the number of characters we are pulling, and where we want to save the log: if (success == GL_FALSE){ GLint maxLength = 0; glGetShaderiv(id, GL_INFO_LOG_LENGTH, &maxLength); std::vector<char> errorLog(maxLength); glGetShaderInfoLog(id, maxLength, &maxLength, &errorLog[0]); Once we have the log file saved, we go ahead and delete the shader using the glDeleteShader() method. This ensures we don't have any memory leaks from our shader: glDeleteShader(id); Finally, we first print the error log to the console window. This is great for runtime debugging. We also throw an exception with the shader name/file path, and the message that it failed to compile: std::printf("%sn", &(errorLog[0])); Exception("Shader " + filePath + " failed to compile"); } ... That really simplifies the process of compiling our shaders by providing a simple interface to the underlying API calls. Now, in our example program, to load and compile our shaders we use a simple line of code similar to the following: shaderManager.CompileShaders("Shaders/SimpleShader.vert", "Shaders/SimpleShader.frag"); Having now compiled the shaders, we are halfway to a useable shader program. We still need to add one more piece, linking. To abstract away some of the processes of linking the shaders and to provide us with some debugging capabilities, we are going to create the LinkShaders() method for our ShaderManager class. Let's take a look and then break it down: void ShaderManager::LinkShaders() { //Attach our shaders to our program glAttachShader(m_programID, m_vertexShaderID); glAttachShader(m_programID, m_fragmentShaderID); //Link our program glLinkProgram(m_programID); //Note the different functions here: glGetProgram* instead of glGetShader*. GLint isLinked = 0; glGetProgramiv(m_programID, GL_LINK_STATUS, (int *)&isLinked); if (isLinked == GL_FALSE){ GLint maxLength = 0; glGetProgramiv(m_programID, GL_INFO_LOG_LENGTH, &maxLength); //The maxLength includes the NULL character std::vector<char> errorLog(maxLength); glGetProgramInfoLog(m_programID, maxLength, &maxLength, &errorLog[0]); //We don't need the program anymore. glDeleteProgram(m_programID); //Don't leak shaders either. glDeleteShader(m_vertexShaderID); glDeleteShader(m_fragmentShaderID); //print the error log and quit std::printf("%sn", &(errorLog[0])); Exception("Shaders failed to link!"); } //Always detach shaders after a successful link. glDetachShader(m_programID, m_vertexShaderID); glDetachShader(m_programID, m_fragmentShaderID); glDeleteShader(m_vertexShaderID); glDeleteShader(m_fragmentShaderID); } To start our LinkShaders() function, we call the glAttachShader() method twice, using the handle to the previously created shader program object, and the handle to each shader we wish to link, respectively: glAttachShader(m_programID, m_vertexShaderID); glAttachShader(m_programID, m_fragmentShaderID); Next, we perform the actual linking of the shaders into a usable shader program by calling the glLinkProgram() method, using the handle to the program object as its argument: glLinkProgram(m_programID); We can then check to see if the linking process has completed without any errors and provide ourselves with any debug information that we might need if there were any errors. I am not going to go through this code chunk line by line since it is nearly identical to what we did with the CompileShader() function. Do note, however, that the function to return the information from the linker is slightly different and uses glGetProgram* instead of the glGetShader* functions from before: GLint isLinked = 0; glGetProgramiv(m_programID, GL_LINK_STATUS, (int *)&isLinked); if (isLinked == GL_FALSE){ GLint maxLength = 0; glGetProgramiv(m_programID, GL_INFO_LOG_LENGTH, &maxLength); //The maxLength includes the NULL character std::vector<char> errorLog(maxLength); glGetProgramInfoLog(m_programID, maxLength, &maxLength, &errorLog[0]); //We don't need the program anymore. glDeleteProgram(m_programID); //Don't leak shaders either. glDeleteShader(m_vertexShaderID); glDeleteShader(m_fragmentShaderID); //print the error log and quit std::printf("%sn", &(errorLog[0])); Exception("Shaders failed to link!"); } Lastly, if we are successful in the linking process, we need to clean it up a bit. First, we detach the shaders from the linker using the glDetachShader() method. Next, since we have a completed shader program, we no longer need to keep the shaders in memory, so we delete each shader with a call to the glDeleteShader() method. Again, this will ensure we do not leak any memory in our shader program creation process: glDetachShader(m_programID, m_vertexShaderID); glDetachShader(m_programID, m_fragmentShaderID); glDeleteShader(m_vertexShaderID); glDeleteShader(m_fragmentShaderID); } We now have a simplified way of linking our shaders into a working shader program. We can call this interface to the underlying API calls by simply using one line of code, similar to the following one: shaderManager.LinkShaders(); So that handles the process of compiling and linking our shaders, but there is another key aspect to working with shaders, which is the passing of data to and from the running program/the game and the shader programs running on the GPU. We will look at this process and how we can abstract it into an easy-to-use interface for our engine next. Working with shader data One of the most important aspects of working with shaders is the ability to pass data to and from the shader programs running on the GPU. This can be a deep topic, and much like other topics in this book has had its own dedicated books. We are going to stay at a higher level when discussing this topic and again will focus on the two needed shader types for basic rendering: the vertex and fragment shaders. To begin with, let's take a look at how we send data to a shader using the vertex attributes and Vertex Buffer Objects (VBO). A vertex shader has the job of processing the data that is connected to the vertex, doing any modifications, and then passing it to the next stage of the rendering pipeline. This occurs once per vertex. In order for the shader to do its thing, we need to be able to pass it data. To do this, we use what are called vertex attributes, and they usually work hand in hand with what is referred to as VBO. For the vertex shader, all per-vertex input attributes are defined using the keyword in. So, for example, if we wanted to define a vector 3 input attribute named VertexColour, we could write something like the following: in vec3 VertexColour; Now, the data for the VertexColour attribute has to be supplied by the program/game. This is where VBO come in. In our main game or program, we make the connection between the input attribute and the vertex buffer object, and we also have to define how to parse or step through the data. That way, when we render, the OpenGL can pull data for the attribute from the buffer for each call of the vertex shader. Let's take a look a very simple vertex shader: #version 410 in vec3 VertexPosition; in vec3 VertexColour; out vec3 Colour; void main(){ Colour = VertexColour; gl_Position = vec4(VertexPosition, 1.0); } In this example, there are just two input variables for this vertex shader, VertexPosition and VertexColor. Our main OpenGL program needs to supply the data for these two attributes for each vertex. We will do so by mapping our polygon/mesh data to these variables. We also have one output variable named Colour, which will be sent to the next stage of the rendering pipeline, the fragment shader. In this example, Colour is just an untouched copy of VertexColour. The VertexPosition attribute is simply expanded and passed along to the OpenGL API output variable gl_Position for more processing. Next, let's take a look at a very simple fragment shader: #version 410 in vec3 Colour; out vec4 FragColour; void main(){ FragColour = vec4(Colour, 1.0); } In this fragment shader example, there is only one input attribute, Colour. This input corresponds to the output of the previous rendering stage, the vertex shader's Colour output. For simplicity's sake, we are just expanding the Colour and outputting it as the variable FragColour for the next rendering stage. That sums up the shader side of the connection, so how do we compose and send the data from inside our engine? We can accomplish this in basically four steps. First, we create a Vertex Array Object (VAO) instance to hold our data: GLunit vao; Next, we create and populate the VBO for each of the shaders' input attributes. We do this by first creating a VBO variable, then, using the glGenBuffers() method, we generate the memory for the buffer objects. We then create handles to the different attributes we need buffers for, assigning them to elements in the VBO array. Finally, we populate the buffers for each attribute by first calling the glBindBuffer() method, specifying the type of object being stored. In this case, it is a GL_ARRAY_BUFFER for both attributes. Then we call the glBufferData() method, passing the type, size, and handle to bind. The last argument for the glBufferData() method is one that gives OpenGL a hint about how the data will be used so that it can determine how best to manage the buffer internally. For full details about this argument, take a look at the OpenGL documentation: GLuint vbo[2]; glGenBuffers(2, vbo); GLuint positionBufferHandle = vbo[0]; GLuint colorBufferHandle = vbo[1]; glBindBuffer(GL_ARRAY_BUFFER,positionBufferHandle); glBufferData(GL_ARRAY_BUFFER, 9 * sizeof(float), positionData, GL_STATIC_DRAW); glBindBuffer(GL_ARRAY_BUFFER, colorBufferHandle); glBufferData(GL_ARRAY_BUFFER, 9 * sizeof(float), colorData, GL_STATIC_DRAW); The third step is to create and define the VAO. This is how we will define the relationship between the input attributes of the shader and the buffers we just created. The VAO contains this information about the connections. To create a VAO, we use the glGenVertexArrays() method. This gives us a handle to our new object, which we store in our previously created VAO variable. Then, we enable the generic vertex attribute indexes 0 and 1 by calling the glEnableVertexAttribArray() method. By making the call to enable the attributes, we are specifying that they will be accessed and used for rendering. The last step makes the connection between the buffer objects we have created and the generic vertex attribute indexes the match too: glGenVertexArrays( 1, &vao ); glBindVertexArray(vao); glEnableVertexAttribArray(0); glEnableVertexAttribArray(1); glBindBuffer(GL_ARRAY_BUFFER, positionBufferHandle); glVertexAttribPointer(0, 3, GL_FLOAT, GL_FALSE, 0, NULL); glBindBuffer(GL_ARRAY_BUFFER, colorBufferHandle); glVertexAttribPointer(1, 3, GL_FLOAT, GL_FALSE, 0, NULL); Finally, in our Draw() function call, we bind to the VAO and call glDrawArrays() to perform the actual render: glBindVertexArray(vaoHandle);glDrawArrays(GL_TRIANGLES, 0, 3 ); Before we move on to another way to pass data to the shader, there is one more piece of this attribute connection structure we need to discuss. As mentioned, the input variables in a shader are linked to the generic vertex attribute we just saw, at the time of linking. When we need to specify the relationship structure, we have a few different choices. We can use what are known as layout qualifiers within the shader code itself. The following is an example: layout (location=0) in vec3 VertexPosition; Another choice is to just let the linker create the mapping when linking, and then query for them afterward. The third and the one I personally prefer is to specify the relationship prior to the linking process by making a call to the glBindAttribLocation() method. We will see how this is implemented shortly when we discuss how to abstract these processes. We have described how we can pass data to a shader using attributes, but there is another option: uniform variables. Uniform variables are specifically used for data that changes infrequently. For example, matrices are great candidates for uniform variables. Within a shader, a uniform variable is read-only. That means the value can only be changed from outside the shader. They can also appear in multiple shaders within the same shader program. They can be declared in one or more shaders within a program, but if a variable with a given name is declared in more than one shader, its type must be the same in all shaders. This gives us insight into the fact that the uniform variables are actually held in a shared namespace for the whole of the shader program. To use a uniform variable in your shader, you first have to declare it in the shader file using the uniform identifier keyword. The following is what this might look like: uniform mat4 ViewMatrix; We then need to provide the data for the uniform variable from inside our game/program. We do this by first finding the location of the variable using the glGetUniformLocation() method. Then we assign a value to the found location using one of the glUniform() methods. The code for this process could look something like the following: GLuint location = glGetUniformLocation(programHandle," ViewMatrix "); if( location >= 0 ) { glUniformMatrix4fv(location, 1, GL_FALSE, &viewMatrix [0][0]) } We then assign a value to the uniform variable's location using the glUniformMatrix4fv() method. The first argument is the uniform variable's location. The second argument is the number of matrices that are being assigned. The third is a GL bool type specifying whether or not the matrix should be transposed. Since we are using the GLM library for our matrices, a transpose is not required. If you were implementing the matrix using data that was in row-major order, instead of column-major order, you might need to use the GL_TRUE type for this argument. The last argument is a pointer to the data for the uniform variable. Uniform variables can be any GLSL type, and this includes complex types such as structures and arrays. The OpenGL API provides a glUniform() function with the different suffixes that match each type. For example, to assign to a variable of type vec3, we would use glUniform3f() or glUniform3fv() methods. (the v denotes multiple values in the array). So, those are the concepts and techniques for passing data to and from our shader programs. However, as we did for the compiling and linking of our shaders, we can abstract these processes into functions housed in our ShaderManager class. We are going to focus on working with attributes and uniform variables. First, we will look at the abstraction of adding attribute bindings using the AddAttribute() function of the ShaderManger class. This function takes one argument, the attribute's name, to be bound as a string. We then call the glBindAttribLocation() function, passing the program's handle and the current index or number of attributes, which we increase on call, and finally the C string conversion of the attributeName string, which provides a pointer to the first character in the string array. This function must be called after compilation, but before the linking of the shader program: void ShaderManager::AddAttribute(const std::string& attributeName) { glBindAttribLocation(m_programID, m_numAttributes++, attributeName.c_str()); } For the uniform variables, we create a function that abstracts looking up the location of the uniform in the shader program, the GetUniformLocation() function. This function again takes only one variable which is a uniform name in the form of a string. We then create a temporary holder for the location and assign it the returned value of the glGetUniformLocation() method call. We check to make sure the location is valid, and if not we throw an exception letting us know about the error. Finally, we return the valid location if found: GLint ShaderManager::GetUniformLocation(const std::string& uniformName) { GLint location = glGetUniformLocation(m_programID, uniformName.c_str()); if (location == GL_INVALID_INDEX) { Exception("Uniform " + uniformName + " not found in shader!"); } return location; } This gives us the abstraction for binding our data, but we still need to assign which shader should be used for a certain draw call, and to activate any attributes we need. To accomplish this, we create a function in the ShaderManager called Use(). This function will first set the current shader program as the active one using the glUseProgram() API method call. We then use a for loop to step through the list of attributes for the shader program, activating each one: void ShaderManager::Use(){ glUseProgram(m_programID); for (int i = 0; i < m_numAttributes; i++) { glEnableVertexAttribArray(i); } } Of course, since we have an abstracted way to enable the shader program, it only makes sense that we should have a function to disable the shader program. This function is very similar to the Use() function, but in this case, we are setting the program in use to 0, effectively making it NULL, and we use the glDisableVertexAtrribArray() method to disable the attributes in the for loop: void ShaderManager::UnUse() { glUseProgram(0); for (int i = 0; i < m_numAttributes; i++) { glDisableVertexAttribArray(i); } } The net effect of this abstraction is we can now set up our entire shader program structure with a few simple calls. Code similar to the following would create and compile the shaders, add the necessary attributes, link the shaders into a program, locate a uniform variable, and create the VAO and VBO for a mesh: shaderManager.CompileShaders("Shaders/SimpleShader.vert", "Shaders/SimpleShader.frag"); shaderManager.AddAttribute("vertexPosition_modelspace"); shaderManager.AddAttribute("vertexColor"); shaderManager.LinkShaders(); MatrixID = shaderManager.GetUniformLocation("ModelViewProjection"); m_model.Init("Meshes/Dwarf_2_Low.obj", "Textures/dwarf_2_1K_color.png"); Then, in our Draw loop, if we want to use this shader program to draw, we can simply use the abstracted functions to activate and deactivate our shader, similar to the following code: shaderManager.Use(); m_model.Draw(); shaderManager.UnUse(); This makes it much easier for us to work with and test out advanced rendering techniques using shaders. Here in this article, we have discussed how advanced rendering techniques, hands-on practical knowledge of game physics and shaders and lighting can help you to create advanced games with C++. If you have liked this above article, check out the complete book Mastering C++ game Development.  How to use arrays, lists, and dictionaries in Unity for 3D game development Unity 2D & 3D game kits simplify Unity game development for beginners How AI is changing game development
Read more
  • 0
  • 0
  • 47216

article-image-create-a-travel-app-with-xamarin
Sugandha Lahoti
20 Jun 2018
14 min read
Save for later

Create a travel app with Xamarin [Tutorial]

Sugandha Lahoti
20 Jun 2018
14 min read
Just like the beginning of many new mobile projects, we will start with an idea. In this tutorial, we will create a new Xamarin.Forms mobile app named TripLog with an initial app structure and user interface. Like the name suggests, it will be an app that will allow its users to log their travel adventures. Although the app might not solve any real-world problems, it will have features that will require us to solve real-world architecture and coding problems. This article is an excerpt from the book Mastering Xamaring.Forms by Ed Snider. Defining features Before we get started, it is important to understand the requirements and features of the TripLog app. We will do this by quickly defining some of the high-level things this app will allow its users to do: View existing log entries (online and offline) Add new log entries with the following data: Title Location using GPS Date Notes Rating Sign into the app Creating the initial app To start off the new TripLog mobile app project, we will need to create the initial solution architecture. We can also create the core shell of our app's user interface by creating the initial screens based on the basic features we have just defined. Setting up the solution We will start things off by creating a brand new, blank Xamarin.Forms solution within Visual Studio by performing the following steps: In Visual Studio, click on File | New Solution. This will bring up a series of dialog screens that will walk you through creating a new Xamarin.Forms solution. On the first dialog, click on App on the left-hand side, under the Multiplatform section, and then select Blank Forms App, as shown in the following screenshot: On the next dialog screen, enter the name of the app, TripLog, ensure that Use Portable Class Library is selected for the Shared Code option, and that Use XAML for user interface files option is checked, as shown in the following screenshot: The Xamarin.Forms project template in Visual Studio for Windows will use a .NET Standard library instead of a Portable Class Library for its core library project. As of the writing of this book, the Visual Studio for Mac templates still use a Portable Class Library. On the final dialog screen, simply click on the Create button, as follows: After creating the new Xamarin.Forms solution, you will have several projects created within it, as shown in the following screenshot: There will be a single portable class library project and two platform-specific projects, as follows: TripLog: This is a portable class library project that will serve as the core layer of the solution architecture. This is the layer that will include all our business logic, data objects, Xamarin.Forms pages, and other non-platform-specific code. The code in this project is common and not specific to a platform, and can therefore, be shared across the platform projects. TripLog.iOS: This is the iOS platform-specific project containing all the code and assets required to build and deploy the iOS app from this solution. By default, it will have a reference to the TripLog core project. TripLog.Droid: This is the Android platform-specific project containing all the code and assets required to build and deploy the Android app from this solution. By default, it will have a reference to the TripLog core project. If you are using Visual Studio for Mac, you will only get an iOS and an Android project when you create a new Xamarin.Forms solution. 
To include a Windows (UWP) app in your Xamarin.Forms solution, you will need to use Visual Studio for Windows. 
Although the screenshots and samples used throughout this book are demonstrated using Visual Studio for Mac, the code and concepts will also work in Visual Studio for Windows. Refer to the Preface of this book for further details on software and hardware requirements that need to be met to follow along with the concepts in this book. You'll notice a file in the core library named App.xaml, which includes a code-behind class in App.xaml.cs named App that inherits from Xamarin.Forms.Application. Initially, the App constructor sets the MainPage property to a new instance of a ContentPage named TripLogPage that simply displays some default text. The first thing we will do in our TripLog app is build the initial views, or screens, required for our UI, and then update that MainPage property of the App class in App.xaml.cs. Updating the Xamarin.Forms packages If you expand the Packages folder within each of the projects in the solution, you will see that Xamarin.Forms is a NuGet package that is automatically included when we select the Xamarin.Forms project template. It is possible that the included NuGet packages need to be updated. Ensure that you update them in each of the projects within the solution so that you are using the latest version of Xamarin.Forms. Creating the main page The main page of the app will serve as the entry point into the app and will display a list of existing trip log entries. Our trip log entries will be represented by a data model named TripLogEntry. Models are a key pillar in the Model-View-ViewModel (MVVM) pattern and data binding, which we will explore more in our tutorial, How to add MVVM pattern and data binding to our Travel app. For now, we will create a simple class that will represent the TripLogEntry model. Let us now start creating the main page by performing the following steps: First, add a new Xamarin.Forms XAML  ContentPage to the core project and name it MainPage. Next, update the MainPage property of the App class in App.xaml.cs to a new instance of Xamarin.Forms.NavigationPage whose root is a new instance of TripLog.MainPage that we just created: public App() { InitializeComponent(); MainPage = new NavigationPage(new MainPage()); } Delete TripLogPage.xaml from the core project as it is no longer needed. Create a new folder in the core project named Models. Create a new empty class file in the Models folder named TripLogEntry. Update the TripLogEntry class with auto-implemented properties representing the attributes of an entry: public class TripLogEntry { public string Title { get; set; } public double Latitude { get; set; } public double Longitude { get; set; } public DateTime Date { get; set; } public int Rating { get; set; } public string Notes { get; set; } } Now that we have a model to represent our trip log entries, we can use it to display some trips on the main page using a ListView control. We will use a DataTemplate to describe how the model data should be displayed in each of the rows in the ListView using the following XAML in the ContentPage.Content tag in MainPage.xaml: <ContentPage xmlns="http://xamarin.com/schemas/2014/forms" xmlns:x="http://schemas.microsoft.com/winfx/2009/xaml" x:Class="TripLog.MainPage" Title="TripLog"> <ContentPage.Content> <ListView x:Name="trips"> <ListView.ItemTemplate> <DataTemplate> <TextCell Text="{Binding Title}" Detail="{Binding Notes}" /> </DataTemplate> </ListView.ItemTemplate> </ListView> </ContentPage.Content> </ContentPage> In the main page's code-behind, MainPage.xaml.cs, we will populate the ListView ItemsSource with a hard-coded collection of TripLogEntry objects. public partial class MainPage : ContentPage { public MainPage() { InitializeComponent(); var items = new List<TripLogEntry> { new TripLogEntry { Title = "Washington Monument", Notes = "Amazing!", Rating = 3, Date = new DateTime(2017, 2, 5), Latitude = 38.8895, Longitude = -77.0352 }, new TripLogEntry { Title = "Statue of Liberty", Notes = "Inspiring!", Rating = 4, Date = new DateTime(2017, 4, 13), Latitude = 40.6892, Longitude = -74.0444 }, new TripLogEntry { Title = "Golden Gate Bridge", Notes = "Foggy, but beautiful.", Rating = 5, Date = new DateTime(2017, 4, 26), Latitude = 37.8268, Longitude = -122.4798 } }; trips.ItemsSource = items; } } At this point, we have a single page that is displayed as the app's main page. If we debug the app and run it in a simulator, emulator, or on a physical device, we should see the main page showing the list of log entries we hard-coded into the view, as shown in the following screenshot. Creating the new entry page The new entry page of the app will give the user a way to add a new log entry by presenting a series of fields to collect the log entry details. There are several ways to build a form to collect data in Xamarin.Forms. You can simply use a StackLayout and present a stack of Label and Entry controls on the screen, or you can also use a TableView with various types of ViewCell elements. In most cases, a TableView will give you a very nice default, platform-specific look and feel. However, if your design calls for a more customized aesthetic, you might be better off leveraging the other layout options available in Xamarin.Forms. For the purpose of this app, we will use a TableView. There are some key data points we need to collect when our users log new entries with the app, such as title, location, date, rating, and notes. For now, we will use a regular EntryCell element for each of these fields. We will update, customize, and add things to these fields later in this book. For example, we will wire the location fields to a geolocation service that will automatically determine the location. We will also update the date field to use an actual platform-specific date picker control. For now, we will just focus on building the basic app shell. In order to create the new entry page that contains a TableView, perform the following steps: First, add a new Xamarin.Forms XAML ContentPage to the core project and name it NewEntryPage. Update the new entry page using the following XAML to build the TableView that will represent the data entry form on the page: <ContentPage xmlns="http://xamarin.com/schemas/2014/forms" xmlns:x="http://schemas.microsoft.com/winfx/2009/xaml" x:Class="TripLog.NewEntryPage" Title="New Entry"> <ContentPage.Content> <TableView Intent="Form"> <TableView.Root> <TableSection> <EntryCell Label="Title" /> <EntryCell Label="Latitude" Keyboard="Numeric" /> <EntryCell Label="Longitude" Keyboard="Numeric" /> <EntryCell Label="Date" /> <EntryCell Label="Rating" Keyboard="Numeric" /> <EntryCell Label="Notes" /> </TableSection> </TableView.Root> </TableView> </ContentPage.Content> </ContentPage> Now that we have created the new entry page, we need to add a way for users to get to this new screen from the main page. We will do this by adding a New button to the main page's toolbar. In Xamarin.Forms, this is accomplished by adding a ToolbarItem to the ContentPage.ToolbarItems collection and wiring up the ToolbarItem.Clicked event to navigate to the new entry page, as shown in the following XAML: <!-- MainPage.xaml --> <ContentPage> <ContentPage.ToolbarItems> <ToolbarItem Text="New" Clicked="New_Clicked" /> </ContentPage.ToolbarItems> </ContentPage> // MainPage.xaml.cs public partial class MainPage : ContentPage { // ... void New_Clicked(object sender, EventArgs e) { Navigation.PushAsync(new NewEntryPage()); } } To handle navigation between pages, we will use the default Xamarin.Forms navigation mechanism. When we run the app, we will see a New button on the toolbar of the main page. Clicking on the New button should bring us to the new entry page, as shown in the following screenshot: We will need to add a save button to the new entry page toolbar so that we can save new items. The save button will be added to the new entry page toolbar in the same way the New button was added to the main page toolbar. Update the XAML in NewEntryPage.xaml to include a new ToolbarItem, as shown in the following code: <ContentPage> <ContentPage.ToolbarItems> <ToolbarItem Text="Save" /> </ContentPage.ToolbarItems> <!-- ... --> </ContentPage> When we run the app again and navigate to the new entry page, we should now see the Save button on the toolbar, as shown in the following screenshot: Creating the entry detail page When a user clicks on one of the log entry items on the main page, we want to take them to a page that displays more details about that particular item, including a map that plots the item's location. Along with additional details and a more in-depth view of the item, a detail page is also a common area where actions on that item might take place, such as, editing the item or sharing the item on social media. The detail page will take an instance of a TripLogEntry model as a constructor parameter, which we will use in the rest of the page to display the entry details to the user. In order to create the entry detail page, perform the following steps: First, add a new Xamarin.Forms XAML ContentPage to the project and name it DetailPage. Update the constructor of the DetailPage class in DetailPage.xaml.cs to take a TripLogEntry parameter named entry, as shown in the following code: public class DetailPage : ContentPage { public DetailPage(TripLogEntry entry) { // ... } } Add the Xamarin.Forms.Maps NuGet package to the core project and to each of the platform-specific projects. This separate NuGet package is required in order to use the Xamarin.Forms Map control in the next step. Update the XAML in DetailPage.xaml to include a Grid layout to display a Map control and some Label controls to display the trip's details, as shown in the following code: <ContentPage xmlns="http://xamarin.com/schemas/2014/forms" xmlns:x="http://schemas.microsoft.com/winfx/2009/xaml" xmlns:maps="clr-namespace:Xamarin.Forms.Maps;assembly=Xamarin.Forms.Maps" x:Class="TripLog.DetailPage"> <ContentPage.Content> <Grid> <Grid.RowDefinitions> <RowDefinition Height="4*" /> <RowDefinition Height="Auto" /> <RowDefinition Height="1*" /> </Grid.RowDefinitions> <maps:Map x:Name="map" Grid.RowSpan="3" /> <BoxView Grid.Row="1" BackgroundColor="White" Opacity=".8" /> <StackLayout Padding="10" Grid.Row="1"> <Label x:Name="title" HorizontalOptions="Center" /> <Label x:Name="date" HorizontalOptions="Center" /> <Label x:Name="rating" HorizontalOptions="Center" /> <Label x:Name="notes" HorizontalOptions="Center" /> </StackLayout> </Grid> </ContentPage.Content> </ContentPage> Update the detail page's code-behind, DetailPage.xaml.cs, to center the map and plot the trip's location. We also need to update the Label controls on the detail page with the properties of the entry constructor parameter: public DetailPage(TripLogEntry entry) { InitializeComponent(); map.MoveToRegion(MapSpan.FromCenterAndRadius( new Position(entry.Latitude, entry.Longitude), Distance.FromMiles(.5))); map.Pins.Add(new Pin { Type = PinType.Place, Label = entry.Title, Position = new Position(entry.Latitude, entry.Longitude) }); title.Text = entry.Title; date.Text = entry.Date.ToString("M"); rating.Text = $"{entry.Rating} star rating"; notes.Text = entry.Notes; } Next, we need to wire up the ItemTapped event of the ListView on the main page to pass the tapped item over to the entry detail page that we have just created, as shown in the following code: <!-- MainPage.xaml --> <ListView x:Name="trips" ItemTapped="Trips_ItemTapped"> <!-- ... --> </ListView> // MainPage.xaml.cs public MainPage() { // ... async void Trips_ItemTapped(object sender, ItemTappedEventArgs e) { var trip = (TripLogEntry)e.Item; await Navigation.PushAsync(new DetailPage(trip)); // Clear selection trips.SelectedItem = null; } } Finally, we will need to initialize the Xamarin.Forms.Maps library in each platform-specific startup class (AppDelegate for iOS and MainActivity for Android) using the following code: // in iOS AppDelegate global::Xamarin.Forms.Forms.Init(); Xamarin.FormsMaps.Init(); LoadApplication(new App()); // in Android MainActivity global::Xamarin.Forms.Forms.Init(this, bundle); Xamarin.FormsMaps.Init(this, bundle); LoadApplication(new App()); Now, when we run the app and tap on one of the log entries on the main page, it will navigate us to the details page to see more detail about that particular log entry, as shown in the following screenshot: We built a simple three-page app with static data, leveraging the most basic concepts of the Xamarin.Forms toolkit. We used the default Xamarin.Forms navigation APIs to move between the three pages. Stay tuned for our next post to learn about adding MVVM pattern and data binding to this Travel app. If you enjoyed reading this excerpt, check out this book Mastering Xamaring.Forms. Xamarin Forms 3, the popular cross-platform UI Toolkit, is here! Five reasons why Xamarin will change mobile development Creating Hello World in Xamarin.Forms_sample
Read more
  • 0
  • 1
  • 47209

article-image-basic-website-using-nodejs-and-mysql-database
Packt
14 Jul 2016
5 min read
Save for later

Basic Website using Node.js and MySQL database

Packt
14 Jul 2016
5 min read
In this article by Fernando Monteiro author of the book Node.JS 6.x Blueprints we will understand some basic concepts of a Node.js application using a relational database (Mysql) and also try to look at some differences between Object Document Mapper (ODM) from MongoDB and Object Relational Mapper (ORM) used by Sequelize and Mysql. For this we will create a simple application and use the resources we have available as sequelize is a powerful middleware for creation of models and mapping database. We will also use another engine template called Swig and demonstrate how we can add the template engine manually. (For more resources related to this topic, see here.) Creating the baseline applications The first step is to create another directory, I'll use the root folder. Create a folder called chapter-02. Open your terminal/shell on this folder and type the express command: express –-git Note that we are using only the –-git flag this time, we will use another template engine but we will install it manually. Installing Swig template Engine The first step to do is change the default express template engine to use Swig, a pretty simple template engine very flexible and stable, also offers us a syntax very similar to Angular which is denoting expressions just by using double curly brackets {{ variableName }}. More information about Swig can be found on the official website at: http://paularmstrong.github.io/swig/docs/ Open the package.json file and replace the jade line for the following: "swig": "^1.4.2" Open your terminal/shell on project folder and type: npm install Before we proceed let's make some adjust to app.js, we need to add the swig module. Open app.js and add the following code, right after the var bodyParser = require('body-parser'); line: var swig = require('swig'); Replace the default jade template engine line for the following code: var swig = new swig.Swig(); app.engine('html', swig.renderFile); app.set('view engine', 'html'); Refactoring the views folder Let's change the views folder to the following new structure: views pages/ partials/ Remove the default jade files form views. Create a file called layout.html inside pages folder and place the following code: <!DOCTYPE html> <html> <head> </head> <body> {% block content %} {% endblock %} </body> </html> Create a index.html inside the views/pages folder and place the following code: {% extends 'layout.html' %} {% block title %}{% endblock %} {% block content %} <h1>{{ title }}</h1> Welcome to {{ title }} {% endblock %} Create a error.html page inside the views/pages folder and place the following code: {% extends 'layout.html' %} {% block title %}{% endblock %} {% block content %} <div class="container"> <h1>{{ message }}</h1> <h2>{{ error.status }}</h2> <pre>{{ error.stack }}</pre> </div> {% endblock %} We need to adjust the views path on app.js, replace the code on line 14 for the following code: // view engine setup app.set('views', path.join(__dirname, 'views/pages')); At this time we completed the first step to start our MVC application. In this example we will use the MVC pattern in its full meaning, Model, View, Controller. Creating controllers folder Create a folder called controllers inside the root project folder. Create a index.js inside the controllers folder and place the following code: // Index controller exports.show = function(req, res) { // Show index content res.render('index', { title: 'Express' }); }; Edit the app.js file and replace the original index route app.use('/', routes); with the following code: app.get('/', index.show); Add the controller path to app.js on line 9, replace the original code, with the following code: // Inject index controller var index = require('./controllers/index'); Now it's time to get if all goes as expected, we run the application and check the result. Type on your terminal/shell the following command: npm start Check with the following URL: http://localhost:3000, you'll see the welcome message of express framework. Removing the default routes folder Remove the routes folder and its content. Remove the user route from the app.js, after the index controller and on line 31. Adding partials files for head and footer Inside views/partials create a new file called head.html and place the following code: <meta charset="utf-8"> <title>{{ title }}</title> <link rel='stylesheet' href='https://cdnjs.cloudflare.com/ajax/libs/twitter-bootstrap/4.0.0-alpha.2/css/bootstrap.min.css'> <link rel="stylesheet" href="/stylesheets/style.css"> Inside views/partials create a file called footer.html and place the following code: <script src='https://cdnjs.cloudflare.com/ajax/libs/jquery/2.2.1/jquery.min.js'></script> <script src='https://cdnjs.cloudflare.com/ajax/libs/twitter-bootstrap/4.0.0-alpha.2/js/bootstrap.min.js'></script> Now is time to add the partials file to layout.html page using the include tag. Open layout.html and add the following highlighted code: <!DOCTYPE html> <html> <head> {% include "../partials/head.html" %} </head> <body> {% block content %} {% endblock %} {% include "../partials/footer.html" %} </body> </html> Finally we are prepared to continue with our project, this time our directories structure looks like the following image: Folder structure Summaray In this article, we are discussing the basic concept of Node.js and Mysql database and we also saw how to refactor express engine template and use another resource like Swig template library to build a basic website. Resources for Article: Further resources on this subject: Exception Handling in MySQL for Python [article] Python Scripting Essentials [article] Splunk's Input Methods and Data Feeds [article]
Read more
  • 0
  • 0
  • 47023
Unlock access to the largest independent learning library in Tech for FREE!
Get unlimited access to 7500+ expert-authored eBooks and video courses covering every tech area you can think of.
Renews at $19.99/month. Cancel anytime
article-image-setting-up-openai-playground
Henry Habib
13 Feb 2024
9 min read
Save for later

Setting Up OpenAI Playground

Henry Habib
13 Feb 2024
9 min read
Dive deeper into the world of AI innovation and stay ahead of the AI curve! Subscribe to our AI_Distilled newsletter for the latest insights. Don't miss out – sign up today!This article is an excerpt from the book, OpenAI API Cookbook, by Henry Habib. Integrate the ChatGPT API into various domains ranging from simple wrappers to knowledge-based assistants, multi-model, and conversational applicationsIntroductionThe OpenAI Playground is an interactive web-based interface designed to allow users to experiment with OpenAI’s language models, including ChatGPT. It’s a place where you can learn about the capabilities of these models by entering prompts and seeing the responses generated in real time. This platform acts as a sandbox where developers, researchers, and curious individuals alike can experiment, learn, and even prototype their ideas.In the Playground, you have the freedom to engage in a wide range of activities. You can test out different versions of the AI models, experimenting with various prompts to see how the model responds, and you can play around with different parameters to influence the responses generated. It provides a real-time glimpse into how these powerful AI models think, react, and create based on your input.Setting up your OpenAI Playground environmentGetting readyBefore you start, you need to create an OpenAI Platform account.Navigate to https://platform.openai.com/and sign in to your OpenAI account. If you do not have an account, you can sign up for free with an email address. Alternatively, you can log in to OpenAI with a valid Google, Microsoft, or Apple account. Follow the instructions to complete the creation of your account. You may need to verify your identity with a valid phone number.How to do it…1. After you have successfully logged in, navigate to Profile in the top right-hand menu, select Personal, and then select Usage from the left-hand side menu. Alternatively, you can navigate to https://platform.openai.com/account/usage after logging in. This page shows the usage of your API, but more importantly, it shows you how many credits you have available.2. Normally, OpenAI provides you a $5 credit with a new account, which you should be able to see under the Free Trial Usage section of the page. If you do have credits, proceed to step 4. If, however, you do not have any credits, you will need to upgrade and set up a paid account.3. You need not set up a paid account if you have received free credits. If you run out of free credits, however, here is how you can set up a paid account: select Billing from the left-hand side menu and then select Overview. Then, select the Set up paid account button. You will be prompted to enter your payment details and set a dollar threshold, which can be set to any level of spend that you are comfortable with. Note that the amount of credits required to collectively execute every single recipe contained in this book is not likely to exceed $5.4. After you have created an OpenAI Platform account, you should be able to access the Playground by selecting Playground from the top menu bar, or by navigating to https://platform. openai.com/playground.How it works…The OpenAI Playground interface is, in my experience, clean, intuitive, and designed to provide users easy access to OpenAI’s powerful language models. The Playground is an excellent place to learn how the models perform under different settings, allowing you to experiment with parameters such as temperature and max tokens, which influence the randomness and length of the outputs respectively. The changes you make are instantly reflected in the model’s responses, offering immediate feedback.As shown in Figure 1.1, the Playground consists of three sections: the System Message, the Chat Log, and the Parameters. You will learn more about these three features in the Running a completion request in the OpenAI Playground recipe.Figure 1.1 – The OpenAI PlaygroundNow, your Playground is set up and ready to be used. You can use it to run completion requests and see how varying your prompts and parameters affect the response from OpenAI.Running a completion request in the OpenAI PlaygroundIn this recipe, we will actually put the Playground in action and execute a completion request from OpenAI. Here, you will see the power of the OpenAI API and how it can be used to provide completions for virtually any prompt.Getting readyEnsure you have an OpenAI Platform account with available usage credits. If you don’t, please follow the Setting up your OpenAI API Playground environment recipe. All the recipes in this chapter will have this same requirement.How to do it…Let’s go ahead and start testing the model with the Playground. Let’s create an assistant that writes marketing slogans:1. Navigate to the OpenAI Playground.2. In the System Message, type in the following: You are an assistant that creates marketing slogans based on descriptions of companies. Here, we are clearly instructing the model of its role and context.3. In the Chat Log, populate the USER message with the following: A company that writes engaging mystery novels.4. Select the Submit button on the bottom of the page.5. You should now see a completion response from OpenAI. In my case (Figure 1.2), the response isUnlock the Thrilling Pages of Suspense with Our Captivating Mystery Novels!Figure 1.2 – The OpenAI Playground with prompt and completionNoteSince OpenAI’s LLMs are probabilistic, you will likely not see the same outputs as me. In fact, if you run this recipe multiple times, you will likely see different answers, and that is expected because it is built into the randomness of the model.How it works…OpenAI’s text generation models utilize a specific neural network architecture termed a transformer. Before delving deeper into this, let’s unpack some of these terms:Neural network architecture: At a high level, this refers to a system inspired by the human brain’s interconnected neuron structure. It’s designed to recognize patterns and can be thought of as the foundational building block for many modern AI systems.Transformer: This is a type of neural network design that has proven particularly effective for understanding sequences, making it ideal for tasks involving human language. It focuses on the relationships between words and their context within a sentence or larger text segment.In machine learning, unsupervised learning typically refers to training a model without any labeled data, letting the model figure out patterns on its own. However, OpenAI’s methodology is more nuanced. The models are initially trained on a vast corpus of text data, supervised with various tasks. This helps them predict the next word in a sentence, for instance. Subsequent refinements are made using Reinforcement Learning through Human Feedback (RLHF), where the model is further improved based on feedback from human evaluators.Through this combination of techniques and an extensive amount of data, the model starts to capture the intricacies of human language, encompassing context, tone, humor, and even sarcasm.In this case, the completion response is provided based on both the System Message and the Chat Log. The System Message serves a critical role in shaping and guiding the responses you receive from Open AI, as it dictates the model’s persona, role, tone, and context, among other attributes. In our case, the System Message contains the persona we want the model to take: You are an assistant that creates marketing slogans based on descriptions of companies.The Chat Log contains the history of messages that the model has access to before providing its response, which contains our prompt, A company that writes engaging mystery novels.Finally, the parameters contain more granular settings that you can change for the model, such as temperature. These significantly change the completion response from OpenAI. We will discuss temperature and other parameters in greater detail in Chapter 3.There’s more…It is worth noting that ChatGPT does not read and understand the meaning behind text – instead, the responses are based on statistical probabilities based on patterns it observed during training.The model does not understand the text in the same way that humans do; instead, the completions are generated based on statistical associations and patterns that have been trained in the model’s neural network from a large body of similar text. Now, you know how to run completion requests with the OpenAI Playground. You can try this feature out for your own prompts and see what completions you get. Try creative prompts such as write me a song about lightbulbs or more professional prompts such as explain Newton's first law.ConclusionIn conclusion, the OpenAI Playground offers a dynamic environment for exploring the capabilities of language models like ChatGPT. By setting up your account and navigating through its features, you can unlock endless possibilities for creativity, learning, and innovation. Experiment with prompts, adjust parameters, and observe real-time responses to gain insights into AI's potential. Whether you're a developer, researcher, or curious individual, the Playground provides a sandbox for unleashing your imagination and understanding AI's intricacies. With each completion request, you delve deeper into the world of artificial intelligence, discovering its nuances and expanding your horizons. Start your journey today and witness the power of AI in action.Author BioHenry Habib is a Manager at one of the world's top management consulting firms, advising F500 companies on analytics and operations, with a particular focus on building intelligent AI-driven solutions and tools to create impact. He is a passionate online instructor and educator, amassing a of more than 150K paid students and facilitating technical programs at large banks and governmental.A proponent in the no-code and LLM revolution, he believes that anyone can now create powerful and intelligent applications without any deep technical skills. Henry resides in Toronto, Canada with his wife, and enjoys reading AI research papers and playing tennis in his free time.
Read more
  • 0
  • 0
  • 47001

article-image-build-a-foodie-bot-with-javascript
Gebin George
03 May 2018
7 min read
Save for later

Build a foodie bot with JavaScript

Gebin George
03 May 2018
7 min read
Today, we are going to build a chatbot that can search for restaurants based on user goals and preferences. Let us begin by building Node.js modules to get data from Zomato based on user preferences. Create a file called zomato.js. Add a request module to the Node.js libraries using the following command in the console: This tutorial has been taken from Hands-On Chatbots and Conversational UI Development. > npm install request --save In zomato.js, add the following code to begin with: var request = require('request'); var baseURL = 'https://developers.zomato.com/api/v2.1/'; var apiKey = 'YOUR_API_KEY'; var catergories = null; var cuisines = null; getCategories(); getCuisines(76); Replace YOUR_API_KEY with your Zomato key. Let's build functions to get the list of categories and cuisines at startup. These queries need not be run when the user asks for a restaurant search because this information is pretty much static: function getCuisines(cityId){ var options = { uri: baseURL + 'cuisines', headers: { 'user-key': apiKey }, qs: {'city_id':cityId}, method: 'GET' } var callback = function(error, response, body) { if (error) { console.log('Error sending messages: ', error) } else if (response.body.error) { console.log('Error: ', response.body.error) } else { console.log(body); cuisines = JSON.parse(body).cuisines; } } request(options,callback); } The preceding code will fetch a list of cuisines available in a particular city (identified by a Zomato city ID). Let us add the code for identifying the list of categories: function getCategories(){ var options = { uri: baseURL + 'categories', headers: { 'user-key': apiKey }, qs: {}, method: 'GET' } var callback = function(error, response, body) { if (error) { console.log('Error sending messages: ', error) } else if (response.body.error) { console.log('Error: ', response.body.error) } else { categories = JSON.parse(body).categories; } } request(options,callback); } Now that we have the basic functions out of our way, let us code in the restaurant search code: function getRestaurant(cuisine, location, category){ var cuisineId = getCuisineId(cuisine); var categoryId = getCategoryId(category); var options = { uri: baseURL + 'locations', headers: { 'user-key': apiKey }, qs: {'query':location}, method: 'GET' } var callback = function(error, response, body) { if (error) { console.log('Error sending messages: ', error) } else if (response.body.error) { console.log('Error: ', response.body.error) } else { console.log(body); locationInfo = JSON.parse(body).location_suggestions; search(locationInfo[0], cuisineId, categoryId); } } request(options,callback); } function search(location, cuisineId, categoryId){ var options = { uri: baseURL + 'search', headers: { 'user-key': apiKey }, qs: {'entity_id': location.entity_id, 'entity_type': location.entity_type, 'cuisines': [cuisineId], 'categories': [categoryId]}, method: 'GET' } var callback = function(error, response, body) { if (error) { console.log('Error sending messages: ', error) } else if (response.body.error) { console.log('Error: ', response.body.error) } else { console.log('Found restaurants:') var results = JSON.parse(body).restaurants; console.log(results); } } request(options,callback); } The preceding code will look for restaurants in a given location, cuisine, and category. For instance, you can search for a list of Indian restaurants in Newington, Edinburgh that do delivery. We now need to integrate this with the chatbot code. Let us create a separate file called index.js. Let us begin with the basics: var restify = require('restify'); var builder = require('botbuilder'); var request = require('request'); var baseURL = 'https://developers.zomato.com/api/v2.1/'; var apiKey = 'YOUR_API_KEY'; var catergories = null; var cuisines = null; Chapter 6 [ 247 ] getCategories(); //setTimeout(function(){getCategoryId('Delivery')}, 10000); getCuisines(76); //setTimeout(function(){getCuisineId('European')}, 10000); // Setup Restify Server var server = restify.createServer(); server.listen(process.env.port || process.env.PORT || 3978, function () { console.log('%s listening to %s', server.name, server.url); }); // Create chat connector for communicating with // the Bot Framework Service var connector = new builder.ChatConnector({ appId: process.env.MICROSOFT_APP_ID, appPassword: process.env.MICROSOFT_APP_PASSWORD }); // Listen for messages from users server.post('/foodiebot', connector.listen()); Add the bot dialog code to carry out the restaurant search. Let us design the bot to ask for cuisine, category, and location before proceeding to the restaurant search: var bot = new builder.UniversalBot(connector, [ function (session) { session.send("Hi there! Hungry? Looking for a restaurant?"); session.send("Say 'search restaurant' to start searching."); session.endDialog(); } ]); // Search for a restaurant bot.dialog('searchRestaurant', [ function (session) { session.send('Ok. Searching for a restaurant!'); builder.Prompts.text(session, 'Where?'); }, function (session, results) { session.conversationData.searchLocation = results.response; builder.Prompts.text(session, 'Cuisine? Indian, Italian, or anything else?'); }, function (session, results) { session.conversationData.searchCuisine = results.response; builder.Prompts.text(session, 'Delivery or Dine-in?'); }, function (session, results) { session.conversationData.searchCategory = results.response; session.send('Ok. Looking for restaurants..'); getRestaurant(session.conversationData.searchCuisine, session.conversationData.searchLocation, session.conversationData.searchCategory, session); } ]) .triggerAction({ matches: /^search restaurant$/i, confirmPrompt: 'Your restaurant search task will be abandoned. Are you sure?' }); Notice that we are calling the getRestaurant() function with four parameters. Three of these are ones that we have already defined: cuisine, location, and category. To these, we have to add another: session. This passes the session pointer that can be used to send messages to the emulator when the data is ready. Notice how this changes the getRestaurant() and search() functions: function getRestaurant(cuisine, location, category, session){ var cuisineId = getCuisineId(cuisine); var categoryId = getCategoryId(category); var options = { uri: baseURL + 'locations', headers: { 'user-key': apiKey }, qs: {'query':location}, method: 'GET' } var callback = function(error, response, body) { if (error) { console.log('Error sending messages: ', error) } else if (response.body.error) { console.log('Error: ', response.body.error) } else { console.log(body); locationInfo = JSON.parse(body).location_suggestions; search(locationInfo[0], cuisineId, categoryId, session); } } request(options,callback); } function search(location, cuisineId, categoryId, session){ var options = { uri: baseURL + 'search', headers: { 'user-key': apiKey }, qs: {'entity_id': location.entity_id, 'entity_type': location.entity_type, 'cuisines': [cuisineId], 'category': categoryId}, method: 'GET' } var callback = function(error, response, body) { if (error) { console.log('Error sending messages: ', error) } else if (response.body.error) { console.log('Error: ', response.body.error) } else { console.log('Found restaurants:') console.log(body); //var results = JSON.parse(body).restaurants; //console.log(results); var resultsCount = JSON.parse(body).results_found; console.log('Found:' + resultsCount); session.send('I have found ' + resultsCount + ' restaurants for you!'); session.endDialog(); } } request(options,callback); } Once the results are obtained, the bot responds using session.send() and ends the dialog: Now that we have the results, let's present them in a more visually appealing way using cards. To do this, we need a function that can take the results of the search and turn them into an array of cards: function presentInCards(session, results){ var msg = new builder.Message(session); msg.attachmentLayout(builder.AttachmentLayout.carousel) var heroCardArray = []; var l = results.length; if (results.length > 10){ l = 10; } for (var i = 0; i < l; i++){ var r = results[i].restaurant; var herocard = new builder.HeroCard(session) .title(r.name) .subtitle(r.location.address) .text(r.user_rating.aggregate_rating) .images([builder.CardImage.create(session, r.thumb)]) .buttons([ builder.CardAction.imBack(session, "book_table:" + r.id, "Book a table") ]); heroCardArray.push(herocard); } msg.attachments(heroCardArray); return msg; } And we call this function from the search() function: function search(location, cuisineId, categoryId, session){ var options = { uri: baseURL + 'search', headers: { 'user-key': apiKey }, qs: {'entity_id': location.entity_id, 'entity_type': location.entity_type, 'cuisines': [cuisineId], 'category': categoryId}, method: 'GET' } var callback = function(error, response, body) { if (error) { console.log('Error sending messages: ', error) } else if (response.body.error) { console.log('Error: ', response.body.error) } else { console.log('Found restaurants:') console.log(body); var results = JSON.parse(body).restaurants; var msg = presentInCards(session, results); session.send(msg); session.endDialog(); } } request(options,callback); } Here is how it looks: We saw how to build a restaurant search bot, that gives you restaurant suggestions as per your preference. If you found our post useful check out Chatbots and Conversational UI Development. Top 4 chatbot development frameworks for developers How to create a conversational assistant using Python My friend, the robot: Artificial Intelligence needs Emotional Intelligence    
Read more
  • 0
  • 0
  • 46967

article-image-introducing-swift-programming-language
Packt
19 Apr 2016
25 min read
Save for later

Introducing the Swift Programming Language

Packt
19 Apr 2016
25 min read
In this article by Steven Daniel, the author of Apple Watch App Development, we will introduce ourselves to Apple's Swift programming language. At WWDC 2014, Apple introduced a brand new programming language called Swift. The Swift programming language brings concise syntax, type safety, and modern programming language features to Mac and iOS developers. (For more resources related to this topic, see here.) Since its release, the Apple developer community has responded with great excitement, and developers are rapidly starting to adopt this new language within their own applications. The Swift language is the future of developing on Apple's platforms. This article includes the following topics: Learning how to register as an Apple developer Learning how to download and install Xcode development tools Introduction to the Swift programming language Learning how to work with Xcode playgrounds Introduction to the newest additions in Swift 2.0 Registering as an Apple developer Before you can begin building iOS applications for your iOS devices, you must first join as a registered user of Apple Developer Program in order to download all of the necessary components to your computer. The registration process is free and provides you with access to the iOS SDK and other developer resources that are really useful to get you started. The following short list outlines some of the things that you will be able to access once you become a registered member of Apple Developer Program: It provides helpful “Getting Started” guides to help you get up and running quickly It gives you helpful tips that show you how to submit your apps to App Store It provides the ability to download the current releases of iOS software It provides the ability to beta test the releases of iOS and the iOS SDK It provides access to Apple Developer Forums Whether you develop applications for the iPhone or iPad, these use the same OS and iOS SDK that allows you to create universal apps that will work with each of these devices. On the other hand, Apple Watch uses an entirely different OS called watchOS. To prepare your computer for iOS development, you need to register as an Apple developer. This free process gives you access to the basic levels of development that allow you to test your app using iOS Simulator without the ability to sell your app on Apple App Store. The steps are as follows: To sign up to Apple Developer Program, you will need to go to https://developer.apple.com/programs/ and then click on the Enroll button to proceed, as shown in the following screenshot: Next, click on the Start Your Enrollment button, as shown in the following screenshot: Once you sign up, you will then be able to download the iOS SDK and proceed with installing it onto your computer. You will then become an official member of Apple Developer Program. You will then be able to download beta software so that you can test them on your actual device hardware as well as having the freedom to distribute your apps to your end users. In the next section, we will look at how to download and install Xcode development tools. Getting and installing Xcode development tools In this section, we will take a look at what Integrated Development Environments (IDEs) and Software Development Kits (SDKs) are needed to develop applications for the iOS platform, which is Apple's operating system for mobile devices. We will explain the importance of each tool's role in the development cycle and the tools required to develop applications for the iOS platform, which are as follows: An Intel-based Mac computer running OS X Yosemite (10.10.2) or later with the latest point release and security patches installed is required. This is so that you can install the latest version of the Xcode development tool. Xcode 6.4 or later is required. Xcode is the main development tool for iOS. You need Xcode 6.4 minimum as this version includes Swift 1.2, and you must be registered as an Apple developer. The iOS SDK consists of the following components: Component Description Xcode This is the main IDE that enables you to develop, edit, and debug your native applications for the iOS and Mac platforms using the Objective-C or Swift programming languages. iOS Simulator This is a Cocoa-based application that enables you to debug your iOS applications on your computer without the need of having an iOS device. There are many iOS features that simply won't work within Simulator, so a device is required if an application uses features such as the Core Location and MapKit frameworks. Instruments These are the analysis tools that help you optimize your applications and monitor memory leaks during the execution of your application in real time. Dashcode This enables you to develop web-based iOS applications and dashboard widgets. Once you are registered, you will need to download and install Xcode developer tools by performing the following steps: Begin by downloading and installing Xcode from Mac App Store at https://itunes.apple.com/au/app/xcode/id497799835?mt=12. Select either the Free or Install button on the App Store page. Once it completes the installation process, you will be able to launch Xcode.app from your Applications folder. You can find additional development tools from the Apple developer website at https://developer.apple.com/. In the next section, we will be look at what exactly Xcode playgrounds are and how you can use them to experiment with designing code algorithms prior to incorporating the code into your project. So, let's get started. Introduction to Xcode playgrounds A playground is basically an interactive Swift coding environment that displays the results of each statement as updates are made without having the need to compile and run a project. You can use playgrounds to learn and explore Swift, prototype parts of your app, and create learning environments for others. The interactive Swift code environment lets you experiment with algorithms, explore system APIs, and even create your very own custom views without the need of having to create a project. Once you perfect your code in the playground, simply move this code into your project. Given that playgrounds are highly interactive, they are a wonderful vehicle for distributing code samples with instructive documentation and can even be used as an alternative medium for presentations. With the new Xcode 7 IDE, you can incorporate rich text comments with bold, italic, and bulleted lists with the addition of having the ability to embed images and links. You can even embed resources and support Swift source code in the playground to make the experience incredibly powerful and engaging, while the visible code remains simple. Playgrounds provide you with the ability to do the following: Share curriculum to teach programming with beautiful text and interactive code Design a new algorithm and watch its results every step of the way Create new tests and verify that they work before promoting them into your test suite Experiment with new APIs to hone your Swift coding skills Turn your experiments into documentation with example code that runs directly within the playground Let's begin by opening the Xcode IDE and explore how to create a new playground file for the first time. Perform the following steps: Open the Xcode.app application either using the finder in your Applications directory or using Apple's Launchpad. If you've never created or opened an Xcode project before, you will be presented with the following screen: In the Welcome to Xcode dialog, select the Get started with a playground option. If this dialog doesn't appear, you can navigate to File | New | Playground… or simply press Shift + Option + Command + N. Next, enter SwiftLanguageBasics as the name of your playground. Then, ensure that you choose iOS as the platform that we will target. Click on the Next button to proceed to the next step in the wizard. Specify the location where you would like to save your project. Then, click on the Create button to save your playground at the specified location. Once your project is created, you will be presented with the default playground template, as shown in the following screenshot: In the next section, you will begin learning about some of the Swift language basics, start adding lines of code within this playground file, and see the results that we get when they are executed. Introduction to the Swift language In this section, we will introduce some of the new and exciting features of the Swift programming language. So, let's get started. Variables, constants, strings, and semicolons Our next step is to familiarize ourselves with the differences between variables, constants, strings, and semicolons in a bit more detail. We will work with and use Xcode playgrounds to put each of these into practice. Variables A variable is a value that can change. Every variable contains a name, called the variable name, and must contain a data type. The data type indicates what sort of value the variable represents, such as whether it is an integer, a floating point number, or a string. Let's take a look at how we can put this into practice and create a variable in Swift. First, let's start by revealing the console output window by navigating to View | Debug Area | Show Debug Area. Next, clear the contents of the playground template and replace them with the following code snippet: /*: # Swift Language Basics – Variables : Created by Steven F. Daniel : Copyright © 2015 GENIESOFT STUDIOS. All Rights Reserved. */ var myGreeting = "Welcome to Learning the basics of Swift Programming" print(myGreeting, terminator: "") As you begin to type in the code, you should immediately see the Welcome to Learning the basics of Swift text magically appear in the right-hand pane in which the assignment takes place and it appears once more for the print statement. The right-hand pane is great to show you smaller output, but for longer debugging output, you would normally take a look at the Xcode console. Constants A constant is basically a value that cannot be changed. Creating these constant variables prevents you from performing accidental assignments and can even improve performance. Let's take a look at how we can put this into practice and create a constant variable in Swift. Clear the contents of the playground template and replace them with the following code snippet: /*: # Swift Language Basics – Constants : Created by Steven F. Daniel : Copyright © 2015 GENIESOFT STUDIOS. All Rights Reserved. */ let myGreeting = "Welcome to Learning the basics" print(myGreeting, terminator: "") myGreeting += " of Swift Programming" Take a look at the screenshot now: As you begin to type in the code, you will immediately receive an error message stating that you cannot assign myGreeting to our let value because the object is not mutable. In Swift, you can control the mutability of the built-in Swift types by using either the let or var keywords during declaration. Strings A string is basically an ordered collection of characters—for example "hello, world". In Swift, strings are represented by the String data type, which represents a collection of values of the char data type. You can use strings to insert constants, variables, literals, and expressions into longer strings in a process known as string interpolation, which we will cover later on in this article. This makes it easy to create custom string values for display, storage, and printing. Let's take a look at how we can put this into practice, create a String variable in Swift, and utilize some of the string methods. Clear the contents of the playground template and replace them with the following code snippet: /*: # Swift Language Basics – Strings : Created by Steven F. Daniel : Copyright © 2015 GENIESOFT STUDIOS. All Rights Reserved. */ import Foundation let myGreeting = "Welcome to Swift Language Basics, working with Strings" // Make our String uppercase print(myGreeting.uppercaseString) // Append exclamation mark at the end of the string var newGreeting = myGreeting.stringByAppendingString("!!!") print(newGreeting) Take a look at the screenshot now: As you can note in the preceding code snippet, we began by importing the Foundation framework class, which contains several APIs to deal with objects such as strings and dates. Next, we declared our myGreeting constant variable and then assigned a default string. We then used the uppercaseString method of the string object to perform a function to make all of the characters within our string uppercase. In our next step, we will declare a new variable called newGreeting and call the stringByAppendingString method to append additional characters at the end of our string. For more information on using the String class, you can consult the Swift programming language documentation at https://developer.apple.com/library/ios/documentation/Swift/Conceptual/Swift_Programming_Language/StringsAndCharacters.html. Semicolons As you would have probably noticed so far, the code you wrote doesn't contain any semicolons. This is because in Swift, these are only required if you want to write multiple statements on a single line. Let's take a look at a code example to see how we can put this into practice. Delete the contents of the playground template and replace them with the following code snippet: /*: # Swift Language Basics – Semicolons : Created by Steven F. Daniel : Copyright © 2015 GENIESOFT STUDIOS. All Rights Reserved. */ import Foundation var myGreeting = "Welcome to Swift Language" let newString = myGreeting + " Basics, ".uppercaseString + "working with semicolons"; print(newString) Take a look the following screenshot now: As you can note in the preceding code snippet, we began by declaring our myGreeting variable and then assigned a default string. In our next step, we declared a new variable called newString, concatenated the details from our myGreeting string, and used the uppercaseString method, which cycles through each character within our string, making our characters uppercase. Next, we appended the additional working with semicolons string to the end of our string and finally used the print statement to output the contents of our newString variable to the console window. As you must have noticed, we included a semicolon to the end of the statement; this is because in Swift, you are required to include semicolons if you want to write multiple statements on a single line. Numeric types and conversion In this section, we will take a look at how we can perform arithmetic operations on our Swift variables. In this example, we will look at how to calculate the area of a triangle, given a base and height value. Let's take a look at a code example to see how we can put this into practice. Clear the contents of the playground template and replace them with the following code snippet: /*: # Swift Language Basics - Numeric Types and Conversion : Created by Steven F. Daniel : Copyright © 2015 GENIESOFT STUDIOS. All Rights Reserved. */ import Foundation // method to calculate the area of a triangle func calcTriangleArea(triBase: Double, triHeight: Double) -> Double { return (triBase * triHeight) / 2 } // Declare our base and height of our triangle let base = 20.0 let height = 120.0 // Calculate and display the area of the triangle and print ("The calculated Area is: " + String(calcTriangleArea(base, triHeight: height))); Take a look at the following screenshot now: As you can note in the preceding code snippet, we started by creating our calcTriangleArea function method, which accepts a base and a height parameter value in order to calculate the area of the triangle. In our next step, we declared two variables, base and height, which contain the assigned values that will be used to calculate the base and the height of our triangle. Next, we made a call to our calcTriangleArea method, passing in the values for our base and height before finally using the print statement to output the calculated area of our triangle to the console window. An important feature of the Swift programming language is that all numeric data type conversions must be explicit, regardless of whether you want to convert to a data type containing more of less precision. Booleans, tuples, and string interpolation In this section, we will look at the various features that come with the Swift programming language. We will look at the improvements that Swift has over Objective-C when it comes to using Booleans and string interpolation before finally discussing how we can use tuples to access elements from a string. Booleans Boolean variables in Swift are basically defined using the Bool data type. This data type can only hold values containing either true or false. Let's take a look at a code example to see how we can put this into practice. Clear the contents of the playground template and replace them with the following code snippet: /*: # Swift Language Basics – Booleans : Created by Steven F. Daniel : Copyright © 2015 GENIESOFT STUDIOS. All Rights Reserved. */ import Foundation let displaySettings : Bool = true print("Display Settings is: " + (displaySettings ? "ON" : "OFF")) Take a look at the following screenshot now: As you can note from the preceding code snippet, we started by declaring our constant variable called displaySettings and assigned it a default Boolean value of true. Next, we performed a check to see whether the value of our displaySettings variable is set to true and called our print statement to output the Display Settings is: ON value to the console window. In Objective-C, you would assign a value of 1 and 0 to denote true and false; this is no longer the case with Swift because Swift doesn't treat 1 as true and 0 as false. You need to explicitly use the actual Boolean values to stay within Swift's data type system. Let's replace the existing playground code with the following code snippet to take a look at what would happen if we changed our value from true to 1: /*: # Swift Language Basics – Booleans : Created by Steven F. Daniel : Copyright © 2015 GENIESOFT STUDIOS. All Rights Reserved. */ import Foundation let displaySettings : Bool = 1 // This will cause an error!!! print("Display Settings is: " + (displaySettings ? "ON" : "OFF")) Take a look at the following screenshot now: As you can note from the previous screenshot, Swift detected that we were assigning an integer value to our Boolean data type and threw an error message. Tuples Tuples provide you with the ability to group multiple values into a single compound value. The values contained within a tuple can be any data type, and therefore are not required to be of the same type. Let's take a look at a code example, to see how we can put this into practice. Clear the contents of the playground template and replace them with the following code snippet: /*: # Swift Language Basics – Tuples : Created by Steven F. Daniel : Copyright © 2015 GENIESOFT STUDIOS. All Rights Reserved. */ import Foundation // Define our Address Details var addressDetails = ("Apple Inc.", "1 Infinite Loop", "Cupertino, California", "United States"); print(addressDetails.0) // Get the Name print(addressDetails.1) // Address print(addressDetails.2) // State print(addressDetails.3) // Country Take a look at the following screenshot now: As you can note from the preceding code snippet, we started by declaring a tuple variable called addressDetails that contains a combination of strings. Next, we accessed each of the tuple elements by referencing their index values and displayed each of these elements in the console window. Let's say that you want to modify the contents of the first element within your tuple. Add the following code snippet after your var addressDetails variable: // Modify the element within our String addressDetails.0 = "Apple Computers, Inc." Take a look at the following screenshot now: As you can note from the preceding screenshot, we modified our first component within our tuple to the Apple Computers, Inc value. If you do not want modifications to be made to your variable, you can just change the var keyword to let, and the assignment would result in a compilation error. You can also express your tuples by referencing them using their named elements. This makes it really useful as you can ensure that your users know exactly what the element refers to. If you express your tuples using their named elements, you will still be able to access your elements using their index notation, as can be seen in the following highlighted code snippet: /*: # Swift Language Basics – Tuples : Created by Steven F. Daniel : Copyright © 2015 GENIESOFT STUDIOS. All Rights Reserved. */ import Foundation // Define our Address Details var addressDetails = (company:"Apple Inc.", Address:"1 Infinite Loop", City:"Cupertino, California", Country:"United States"); // Accessing our Tuple by using their NAMES print("Accessing our Tuple using their NAMESn") print(addressDetails.company) // Get the Name print(addressDetails.Address) // Address print(addressDetails.City) // State print(addressDetails.Country) // Country // Accessing our Tuple by using their index notation print("nAccess our Tuple using their index notation:n") print(addressDetails.0) // Get the Name print(addressDetails.1) // Address print(addressDetails.2) // State print(addressDetails.3) // Country Take a look at the following screenshot now: As you can note from what we covered so far about tuples, these are really cool and are basically just like any other data type in Swift; they can be really powerful to use within your own programs. String interpolation String interpolation means embedding constants, variables, as well as expressions within your string literals. In this section, we will take a look at an example of how you can use this. Clear the contents of the playground template and replace them with the following code snippet: /*: # Swift Language Basics - String Interpolation : Created by Steven F. Daniel : Copyright © 2015 GENIESOFT STUDIOS. All Rights Reserved. */ import Foundation // Define our Address Details var addressDetails = (company:"Apple Inc.", Address:"1 Infinite Loop", City:"Cupertino, California", Country:"United States"); // Use String Interpolation to format output print("Apple Headquarters are located at: nn" + addressDetails.company + ",n" + addressDetails.Address + "n" + addressDetails.City + "n" + addressDetails.Country); Take a look at the following screenshot now: As you can note from the preceding code snippet, we started by declaring a tuple variable called addressDetails that contains a combination of strings. Next, we performed a string concatenation to generate our output in the format that we want by accessing each of the tuple elements using their index values and displaying each of these elements in the console window. Let's take this a step further and use string interpolation to place our address detail information into string variables. The result will still be the same, but I just want to show you the power of using tuples with the Swift programming language. Clear the contents of the playground template and replace them with the following highlighted code snippet: /*: # Swift Language Basics - String Interpolation : Created by Steven F. Daniel : Copyright © 2015 GENIESOFT STUDIOS. All Rights Reserved. */ import Foundation // Use String Interpolation to place elements into string initializers var addressDetails = ("Apple Inc.", "1 Infinite Loop", "Cupertino, California", "United States"); let (Company, Address, City, Country) = addressDetails print("Apple Headquarters are located at: nn" + Company + ",n" + Address + "n" + City + "n" + Country); Take a look at the following screenshot now: As you can note from the preceding code snippet, we removed the named types from our addressDetails string contents, created a new type using the let keyword, and assigned placeholders for each of our tuple elements. This is very handy as it not only makes your code a lot more readable but you can also continue to create additional placeholders for the additional fields that you create. Controlling the flow In this section, we will take a look at how to use the for…in loop to iterate over a set of statements within the body of the loop until a specific condition is met. The for…in loops The for…in loops basically perform a set of statements over a certain number of times until a specific condition is met, which is typically handled by incrementing a counter each time until the loop ends. Let's take a look at a code example to see how we can put this into practice. Clear the contents of the playground template and replace them with the following code snippet: /*: # Swift Language Basics - Control Flow : Created by Steven F. Daniel : Copyright © 2015 GENIESOFT STUDIOS. All Rights Reserved. */ import Foundation // Perform a fibonacci loop using the For-In Loop var fibonacci = 0 var iTemp = 1 var jTemp = 0 for iterator in 0...19 { jTemp = fibonacci fibonacci += iTemp iTemp = jTemp print("Fibonacci: " + String(fibonacci), terminator: "n") } Take a look at the following screenshot now: The preceding code demonstrates the for…in loop and the closed range operator (...). These are often used together, but they are entirely independent. As you can note from the preceding code snippet, we declared the exact same variables: fibonacci, iTemp, and jTemp. Next, we used the for…in loop to iterate over our range, which is from 0 to 19, while displaying the current Fibonacci value in the console window. What's new in Swift 2.0 In this section, we will take a look at some of the new features that come as part of the Swift 2.0 programming language. Error handling Error handling is defined as the process of responding to and recovering from error conditions within your program. The Swift language provides first-class support for throwing, catching, propagating, and manipulating recoverable errors at runtime. In Swift, these are referred to as throwing functions and throwing methods. In Swift 2.0, error handling has vastly improved and adds an additional layer of safety to error checking. You can use the throws keyword to specify which functions and method are most likely to cause an error. You can implement and use the do, try, and catch keywords to handle something that could likely throw an error. Let's take a look at a code example to see how we can put this into practice. Clear the contents of the playground template and replace them with the following code snippet: /*: # Swift Language Basics - What's new in Swift 2.0 : Created by Steven F. Daniel : Copyright © 2015 GENIESOFT STUDIOS. All Rights Reserved. */ import Foundation enum EncryptionError: ErrorType { case Empty case Short } // Method to handle the Encryption func encryptString(str: String, withPassword password: String) throws -> String { if password.characters.count > 0 { // Password is valid } else { throw EncryptionError.Empty } if password.characters.count >= 5 { // Password is valid } else { throw EncryptionError.Short } // Begin constructing our encrypted string let encrypted = password + str + password return String(encrypted.characters.reverse()) } // Call our method to encrypt our string do { let encrypted = try encryptString("Encrypted String Goes Here", withPassword: "123") print(encrypted) } catch EncryptionError.Empty { print("You must provide a password.") } catch EncryptionError.Short { print("Passwords must be at least five characters.") } catch { print("An error occurred!") } Take a look at the following screenshot now: As you can note in the preceding code, we began by creating an enum object that derives from the ErrorType class so that we could create and throw an error. Next, we created a method called encryptString that takes two parameters: str and password. This method performed a check to ensure that we didn't pass an empty password. If our method determines that we did not specify a valid password, we will automatically throw an error using EncryptionError.Empty and exit from this method. Alternatively, if we provide a valid password and string to encrypt, our string will be encrypted. Binding Binding in Swift is something new and provides a means of checking whether a variable contains a valid value prior to continuing and exiting from the method otherwise. Fortunately, Swift 2.0 provides you with exactly this, and it is called the guard keyword. Let's go back to our previous code snippet and take a look at how we can implement the guard statement to our conditional checking within our encryptedString method. Modify the contents of the playground template and replace them with the following highlighted sections: // Method to handle the Encryption func encryptString(str: String, withPassword password: String) throws -> String { guard password.characters.count > 0 else { throw EncryptionError.Empty } guard password.characters.count >= 5 else { throw EncryptionError.Short } // Begin constructing our encrypted string let encrypted = password + str + password return String(encrypted.characters.reverse()) } As you can note in the preceding code snippet, using the guard keyword, you can provide a code block to perform a conditional check within the else statement that will run if the condition fails. This will make your code cleaner as the guard statement lets you trap invalid parameters from being passed to a method. Any conditions you would have checked using if before you can now check using guard. Protocol extensions In Swift 2.0, you have the ability to extend protocols and add additional implementations for properties and methods. For example, you can choose to add additional methods to the String or Array classes, as follows: /* # What's new in Swift 2.0 - Protocol Extensions The first content line displayed in this block of rich text. */ import Foundation let greeting = "Working with Swift Rocks!" // Extend the String class to include additional methods extension CustomStringConvertible { var uCaseString: String { return "(self.description.uppercaseString)!!!" } } print(greeting.uCaseString) Take a look at the following screenshot now: As you can note in the preceding code, we extended the String class using the CustomStringConvertible protocol, which most of the Foundation class objects conform to. Using protocol extensions, they provide you with a wide variety of ways to extend the base classes so that you can add and implement your very own custom functionalities. Summary In this article, we explored how to go about downloading and installing Xcode development tools and then moved on to discussing and using playgrounds to write Swift code to get to grips with some of the Swift programming language features. Next, we looked at some of the newest features that come as part of the Swift 2.0 language. Resources for Article: Further resources on this subject: Exploring Swift[article] Playing with Swift[article] Introduction to WatchKit[article]
Read more
  • 0
  • 0
  • 46898

article-image-building-scalable-microservices
Packt
18 Jan 2017
33 min read
Save for later

Building Scalable Microservices

Packt
18 Jan 2017
33 min read
In this article by Vikram Murugesan, the author of the book Microservices Deployment Cookbook, we will see a brief introduction to concept of the microservices. (For more resources related to this topic, see here.) Writing microservices with Spring Boot Now that our project is ready, let's look at how to write our microservice. There are several Java-based frameworks that let you create microservices. One of the most popular frameworks from the Spring ecosystem is the Spring Boot framework. In this article, we will look at how to create a simple microservice application using Spring Boot. Getting ready Any application requires an entry point to start the application. For Java-based applications, you can write a class that has the main method and run that class as a Java application. Similarly, Spring Boot requires a simple Java class with the main method to run it as a Spring Boot application (microservice). Before you start writing your Spring Boot microservice, you will also require some Maven dependencies in your pom.xml file. How to do it… Create a Java class called com.packt.microservices.geolocation.GeoLocationApplication.java and give it an empty main method: package com.packt.microservices.geolocation; public class GeoLocationApplication { public static void main(String[] args) { // left empty intentionally } } Now that we have our basic template project, let's make our project a child project of Spring Boot's spring-boot-starter-parent pom module. This module has a lot of prerequisite configurations in its pom.xml file, thereby reducing the amount of boilerplate code in our pom.xml file. At the time of writing this, 1.3.6.RELEASE was the most recent version: <parent> <groupId>org.springframework.boot</groupId> <artifactId>spring-boot-starter-parent</artifactId> <version>1.3.6.RELEASE</version> </parent> After this step, you might want to run a Maven update on your project as you have added a new parent module. If you see any warnings about the version of the maven-compiler plugin, you can either ignore it or just remove the <version>3.5.1</version> element. If you remove the version element, please perform a Maven update afterward. Spring Boot has the ability to enable or disable Spring modules such as Spring MVC, Spring Data, and Spring Caching. In our use case, we will be creating some REST APIs to consume the geolocation information of the users. So we will need Spring MVC. Add the following dependencies to your pom.xml file: <dependencies> <dependency> <groupId>org.springframework.boot</groupId> <artifactId>spring-boot-starter-web</artifactId> </dependency> </dependencies> We also need to expose the APIs using web servers such as Tomcat, Jetty, or Undertow. Spring Boot has an in-memory Tomcat server that starts up as soon as you start your Spring Boot application. So we already have an in-memory Tomcat server that we could utilize. Now let's modify the GeoLocationApplication.java class to make it a Spring Boot application: package com.packt.microservices.geolocation; import org.springframework.boot.SpringApplication; import org.springframework.boot.autoconfigure.SpringBootApplication; @SpringBootApplication public class GeoLocationApplication { public static void main(String[] args) { SpringApplication.run(GeoLocationApplication.class, args); } } As you can see, we have added an annotation, @SpringBootApplication, to our class. The @SpringBootApplication annotation reduces the number of lines of code written by adding the following three annotations implicitly: @Configuration @ComponentScan @EnableAutoConfiguration If you are familiar with Spring, you will already know what the first two annotations do. @EnableAutoConfiguration is the only annotation that is part of Spring Boot. The AutoConfiguration package has an intelligent mechanism that guesses the configuration of your application and automatically configures the beans that you will likely need in your code. You can also see that we have added one more line to the main method, which actually tells Spring Boot the class that will be used to start this application. In our case, it is GeoLocationApplication.class. If you would like to add more initialization logic to your application, such as setting up the database or setting up your cache, feel free to add it here. Now that our Spring Boot application is all set to run, let's see how to run our microservice. Right-click on GeoLocationApplication.java from Package Explorer, select Run As, and then select Spring Boot App. You can also choose Java Application instead of Spring Boot App. Both the options ultimately do the same thing. You should see something like this on your STS console: If you look closely at the console logs, you will notice that Tomcat is being started on port number 8080. In order to make sure our Tomcat server is listening, let's run a simple curl command. cURL is a command-line utility available on most Unix and Mac systems. For Windows, use tools such as Cygwin or even Postman. Postman is a Google Chrome extension that gives you the ability to send and receive HTTP requests. For simplicity, we will use cURL. Execute the following command on your terminal: curl http://localhost:8080 This should give us an output like this: {"timestamp":1467420963000,"status":404,"error":"Not Found","message":"No message available","path":"/"} This error message is being produced by Spring. This verifies that our Spring Boot microservice is ready to start building on with more features. There are more configurations that are needed for Spring Boot, which we will perform later in this article along with Spring MVC. Writing microservices with WildFly Swarm WildFly Swarm is a J2EE application packaging framework from RedHat that utilizes the in-memory Undertow server to deploy microservices. In this article, we will create the same GeoLocation API using WildFly Swarm and JAX-RS. To avoid confusion and dependency conflicts in our project, we will create the WildFly Swarm microservice as its own Maven project. This article is just here to help you get started on WildFly Swarm. When you are building your production-level application, it is your choice to either use Spring Boot, WildFly Swarm, Dropwizard, or SparkJava based on your needs. Getting ready Similar to how we created the Spring Boot Maven project, create a Maven WAR module with the groupId com.packt.microservices and name/artifactId geolocation-wildfly. Feel free to use either your IDE or the command line. Be aware that some IDEs complain about a missing web.xml file. We will see how to fix that in the next section. How to do it… Before we set up the WildFly Swarm project, we have to fix the missing web.xml error. The error message says that Maven expects to see a web.xml file in your project as it is a WAR module, but this file is missing in your project. In order to fix this, we have to add and configure maven-war-plugin. Add the following code snippet to your pom.xml file's project section: <build> <plugins> <plugin> <groupId>org.apache.maven.plugins</groupId> <artifactId>maven-war-plugin</artifactId> <version>2.6</version> <configuration> <failOnMissingWebXml>false</failOnMissingWebXml> </configuration> </plugin> </plugins> </build> After adding the snippet, save your pom.xml file and perform a Maven update. Also, if you see that your project is using a Java version other than 1.8. Again, perform a Maven update for the changes to take effect. Now, let's add the dependencies required for this project. As we know that we will be exposing our APIs, we have to add the JAX-RS library. JAX-RS is the standard JSR-compliant API for creating RESTful web services. JBoss has its own version of JAX-RS. So let's  add that dependency to the pom.xml file: <dependencies> <dependency> <groupId>org.jboss.spec.javax.ws.rs</groupId> <artifactId>jboss-jaxrs-api_2.0_spec</artifactId> <version>1.0.0.Final</version> <scope>provided</scope> </dependency> </dependencies> The one thing that you have to note here is the provided scope. The provide scope in general means that this JAR need not be bundled with the final artifact when it is built. Usually, the dependencies with provided scope will be available to your application either via your web server or application server. In this case, when Wildfly Swarm bundles your app and runs it on the in-memory Undertow server, your server will already have this dependency. The next step toward creating the GeoLocation API using Wildfly Swarm is creating the domain object. Use the com.packt.microservices.geolocation.GeoLocation.java file. Now that we have the domain object, there are two classes that you need to create in order to write your first JAX-RS web service. The first of those is the Application class. The Application class in JAX-RS is used to define the various components that you will be using in your application. It can also hold some metadata about your application, such as your basePath (or ApplicationPath) to all resources listed in this Application class. In this case, we are going to use /geolocation as our basePath. Let's see how that looks: package com.packt.microservices.geolocation; import javax.ws.rs.ApplicationPath; import javax.ws.rs.core.Application; @ApplicationPath("/geolocation") public class GeoLocationApplication extends Application { public GeoLocationApplication() {} } There are two things to note in this class; one is the Application class and the other is the @ApplicationPath annotation—both of which we've already talked about. Now let's move on to the resource class, which is responsible for exposing the APIs. If you are familiar with Spring MVC, you can compare Resource classes to Controllers. They are responsible for defining the API for any specific resource. The annotations are slightly different from that of Spring MVC. Let's create a new resource class called com.packt.microservices.geolocation.GeoLocationResource.java that exposes a simple GET API: package com.packt.microservices.geolocation; import java.util.ArrayList; import java.util.List; import javax.ws.rs.GET; import javax.ws.rs.Path; import javax.ws.rs.Produces; @Path("/") public class GeoLocationResource { @GET @Produces("application/json") public List<GeoLocation> findAll() { return new ArrayList<>(); } } All the three annotations, @GET, @Path, and @Produces, are pretty self explanatory. Before we start writing the APIs and the service class, let's test the application from the command line to make sure it works as expected. With the current implementation, any GET request sent to the /geolocation URL should return an empty JSON array. So far, we have created the RESTful APIs using JAX-RS. It's just another JAX-RS project: In order to make it a microservice using Wildfly Swarm, all you have to do is add the wildfly-swarm-plugin to the Maven pom.xml file. This plugin will be tied to the package phase of the build so that whenever the package goal is triggered, the plugin will create an uber JAR with all required dependencies. An uber JAR is just a fat JAR that has all dependencies bundled inside itself. It also deploys our application in an in-memory Undertow server. Add the following snippet to the plugins section of the pom.xml file: <plugin> <groupId>org.wildfly.swarm</groupId> <artifactId>wildfly-swarm-plugin</artifactId> <version>1.0.0.Final</version> <executions> <execution> <id>package</id> <goals> <goal>package</goal> </goals> </execution> </executions> </plugin> Now execute the mvn clean package command from the project's root directory, and wait for the Maven build to be successful. If you look at the logs, you can see that wildfly-swarm-plugin will create the uber JAR, which has all its dependencies. You should see something like this in your console logs: After the build is successful, you will find two artifacts in the target directory of your project. The geolocation-wildfly-0.0.1-SNAPSHOT.war file is the final WAR created by the maven-war-plugin. The geolocation-wildfly-0.0.1-SNAPSHOT-swarm.jar file is the uber JAR created by the wildfly-swarm-plugin. Execute the following command in the same terminal to start your microservice: java –jar target/geolocation-wildfly-0.0.1-SNAPSHOT-swarm.jar After executing this command, you will see that Undertow has started on port number 8080, exposing the geolocation resource we created. You will see something like this: Execute the following cURL command in a separate terminal window to make sure our API is exposed. The response of the command should be [], indicating there are no geolocations: curl http://localhost:8080/geolocation Now let's build the service class and finish the APIs that we started. For simplicity purposes, we are going to store the geolocations in a collection in the service class itself. In a real-time scenario, you will be writing repository classes or DAOs that talk to the database that holds your geolocations. Get the com.packt.microservices.geolocation.GeoLocationService.java interface. We'll use the same interface here. Create a new class called com.packt.microservices.geolocation.GeoLocationServiceImpl.java that extends the GeoLocationService interface: package com.packt.microservices.geolocation; import java.util.ArrayList; import java.util.Collections; import java.util.List; public class GeoLocationServiceImpl implements GeoLocationService { private static List<GeoLocation> geolocations = new ArrayList<>(); @Override public GeoLocation create(GeoLocation geolocation) { geolocations.add(geolocation); return geolocation; } @Override public List<GeoLocation> findAll() { return Collections.unmodifiableList(geolocations); } } Now that our service classes are implemented, let's finish building the APIs. We already have a very basic stubbed-out GET API. Let's just introduce the service class to the resource class and call the findAll method. Similarly, let's use the service's create method for POST API calls. Add the following snippet to GeoLocationResource.java: private GeoLocationService service = new GeoLocationServiceImpl(); @GET @Produces("application/json") public List<GeoLocation> findAll() { return service.findAll(); } @POST @Produces("application/json") @Consumes("application/json") public GeoLocation create(GeoLocation geolocation) { return service.create(geolocation); } We are now ready to test our application. Go ahead and build your application. After the build is successful, run your microservice: let's try to create two geolocations using the POST API and later try to retrieve them using the GET method. Execute the following cURL commands in your terminal one by one: curl -H "Content-Type: application/json" -X POST -d '{"timestamp": 1468203975, "userId": "f1196aac-470e-11e6-beb8-9e71128cae77", "latitude": 41.803488, "longitude": -88.144040}' http://localhost:8080/geolocation This should give you something like the following output (pretty-printed for readability): { "latitude": 41.803488, "longitude": -88.14404, "userId": "f1196aac-470e-11e6-beb8-9e71128cae77", "timestamp": 1468203975 } curl -H "Content-Type: application/json" -X POST -d '{"timestamp": 1468203975, "userId": "f1196aac-470e-11e6-beb8-9e71128cae77", "latitude": 9.568012, "longitude": 77.962444}' http://localhost:8080/geolocation This command should give you an output similar to the following (pretty-printed for readability): { "latitude": 9.568012, "longitude": 77.962444, "userId": "f1196aac-470e-11e6-beb8-9e71128cae77", "timestamp": 1468203975 } To verify whether your entities were stored correctly, execute the following cURL command: curl http://localhost:8080/geolocation This should give you an output like this (pretty-printed for readability): [ { "latitude": 41.803488, "longitude": -88.14404, "userId": "f1196aac-470e-11e6-beb8-9e71128cae77", "timestamp": 1468203975 }, { "latitude": 9.568012, "longitude": 77.962444, "userId": "f1196aac-470e-11e6-beb8-9e71128cae77", "timestamp": 1468203975 } ] Whatever we have seen so far will give you a head start in building microservices with WildFly Swarm. Of course, there are tons of features that WildFly Swarm offers. Feel free to try them out based on your application needs. I strongly recommend going through the WildFly Swarm documentation for any advanced usages. Writing microservices with Dropwizard Dropwizard is a collection of libraries that help you build powerful applications quickly and easily. The libraries vary from Jackson, Jersey, Jetty, and so on. You can take a look at the full list of libraries on their website. This ecosystem of libraries that help you build powerful applications could be utilized to create microservices as well. As we saw earlier, it utilizes Jetty to expose its services. In this article, we will create the same GeoLocation API using Dropwizard and Jersey. To avoid confusion and dependency conflicts in our project, we will create the Dropwizard microservice as its own Maven project. This article is just here to help you get started with Dropwizard. When you are building your production-level application, it is your choice to either use Spring Boot, WildFly Swarm, Dropwizard, or SparkJava based on your needs. Getting ready Similar to how we created other Maven projects,  create a Maven JAR module with the groupId com.packt.microservices and name/artifactId geolocation-dropwizard. Feel free to use either your IDE or the command line. After the project is created, if you see that your project is using a Java version other than 1.8. Perform a Maven update for the change to take effect. How to do it… The first thing that you will need is the dropwizard-core Maven dependency. Add the following snippet to your project's pom.xml file: <dependencies> <dependency> <groupId>io.dropwizard</groupId> <artifactId>dropwizard-core</artifactId> <version>0.9.3</version> </dependency> </dependencies> Guess what? This is the only dependency you will need to spin up a simple Jersey-based Dropwizard microservice. Before we start configuring Dropwizard, we have to create the domain object, service class, and resource class: com.packt.microservices.geolocation.GeoLocation.java com.packt.microservices.geolocation.GeoLocationService.java com.packt.microservices.geolocation.GeoLocationImpl.java com.packt.microservices.geolocation.GeoLocationResource.java Let's see what each of these classes does. The GeoLocation.java class is our domain object that holds the geolocation information. The GeoLocationService.java class defines our interface, which is then implemented by the GeoLocationServiceImpl.java class. If you take a look at the GeoLocationServiceImpl.java class, we are using a simple collection to store the GeoLocation domain objects. In a real-time scenario, you will be persisting these objects in a database. But to keep it simple, we will not go that far. To be consistent with the previous, let's change the path of GeoLocationResource to /geolocation. To do so, replace @Path("/") with @Path("/geolocation") on line number 11 of the GeoLocationResource.java class. We have now created the service classes, domain object, and resource class. Let's configure Dropwizard. In order to make your project a microservice, you have to do two things: Create a Dropwizard configuration class. This is used to store any meta-information or resource information that your application will need during runtime, such as DB connection, Jetty server, logging, and metrics configurations. These configurations are ideally stored in a YAML file, which will them be mapped to your Configuration class using Jackson. In this application, we are not going to use the YAML configuration as it is out of scope for this article. If you would like to know more about configuring Dropwizard, refer to their Getting Started documentation page at http://www.dropwizard.io/0.7.1/docs/getting-started.html. Let's  create an empty Configuration class called GeoLocationConfiguration.java: package com.packt.microservices.geolocation; import io.dropwizard.Configuration; public class GeoLocationConfiguration extends Configuration { } The YAML configuration file has a lot to offer. Take a look at a sample YAML file from Dropwizard's Getting Started documentation page to learn more. The name of the YAML file is usually derived from the name of your microservice. The microservice name is usually identified by the return value of the overridden method public String getName() in your Application class. Now let's create the GeoLocationApplication.java application class: package com.packt.microservices.geolocation; import io.dropwizard.Application; import io.dropwizard.setup.Environment; public class GeoLocationApplication extends Application<GeoLocationConfiguration> { public static void main(String[] args) throws Exception { new GeoLocationApplication().run(args); } @Override public void run(GeoLocationConfiguration config, Environment env) throws Exception { env.jersey().register(new GeoLocationResource()); } } There are a lot of things going on here. Let's look at them one by one. Firstly, this class extends Application with the GeoLocationConfiguration generic. This clearly makes an instance of your GeoLocationConfiguraiton.java class available so that you have access to all the properties you have defined in your YAML file at the same time mapped in the Configuration class. The next one is the run method. The run method takes two arguments: your configuration and environment. The Environment instance is a wrapper to other library-specific objects such as MetricsRegistry, HealthCheckRegistry, and JerseyEnvironment. For example, we could register our Jersey resources using the JerseyEnvironment instance. The env.jersey().register(new GeoLocationResource())line does exactly that. The main method is pretty straight-forward. All it does is call the run method. Before we can start the microservice, we have to configure this project to create a runnable uber JAR. Uber JARs are just fat JARs that bundle their dependencies in themselves. For this purpose, we will be using the maven-shade-plugin. Add the following snippet to the build section of the pom.xml file. If this is your first plugin, you might want to wrap it in a <plugins> element under <build>: <plugin> <groupId>org.apache.maven.plugins</groupId> <artifactId>maven-shade-plugin</artifactId> <version>2.3</version> <configuration> <createDependencyReducedPom>true</createDependencyReducedPom> <filters> <filter> <artifact>*:*</artifact> <excludes> <exclude>META-INF/*.SF</exclude> <exclude>META-INF/*.DSA</exclude> <exclude>META-INF/*.RSA</exclude> </excludes> </filter> </filters> </configuration> <executions> <execution> <phase>package</phase> <goals> <goal>shade</goal> </goals> <configuration> <transformers> <transformer implementation="org.apache.maven.plugins.shade.resource.ServicesResourceTransformer" /> <transformer implementation="org.apache.maven.plugins.shade.resource.ManifestResourceTransformer"> <mainClass>com.packt.microservices.geolocation.GeoLocationApplication</mainClass> </transformer> </transformers> </configuration> </execution> </executions> </plugin> The previous snippet does the following: It creates a runnable uber JAR that has a reduced pom.xml file that does not include the dependencies that are added to the uber JAR. To learn more about this property, take a look at the documentation of maven-shade-plugin. It utilizes com.packt.microservices.geolocation.GeoLocationApplication as the class whose main method will be invoked when this JAR is executed. This is done by updating the MANIFEST file. It excludes all signatures from signed JARs. This is required to avoid security errors. Now that our project is properly configured, let's try to build and run it from the command line. To build the project, execute mvn clean package from the project's root directory in your terminal. This will create your final JAR in the target directory. Execute the following command to start your microservice: java -jar target/geolocation-dropwizard-0.0.1-SNAPSHOT.jar server The server argument instructs Dropwizard to start the Jetty server. After you issue the command, you should be able to see that Dropwizard has started the in-memory Jetty server on port 8080. If you see any warnings about health checks, ignore them. Your console logs should look something like this: We are now ready to test our application. Let's try to create two geolocations using the POST API and later try to retrieve them using the GET method. Execute the following cURL commands in your terminal one by one: curl -H "Content-Type: application/json" -X POST -d '{"timestamp": 1468203975, "userId": "f1196aac-470e-11e6-beb8-9e71128cae77", "latitude": 41.803488, "longitude": -88.144040}' http://localhost:8080/geolocation This should give you an output similar to the following (pretty-printed for readability): { "latitude": 41.803488, "longitude": -88.14404, "userId": "f1196aac-470e-11e6-beb8-9e71128cae77", "timestamp": 1468203975 } curl -H "Content-Type: application/json" -X POST -d '{"timestamp": 1468203975, "userId": "f1196aac-470e-11e6-beb8-9e71128cae77", "latitude": 9.568012, "longitude": 77.962444}' http://localhost:8080/geolocation This should give you an output like this (pretty-printed for readability): { "latitude": 9.568012, "longitude": 77.962444, "userId": "f1196aac-470e-11e6-beb8-9e71128cae77", "timestamp": 1468203975 } To verify whether your entities were stored correctly, execute the following cURL command: curl http://localhost:8080/geolocation It should give you an output similar to the following (pretty-printed for readability): [ { "latitude": 41.803488, "longitude": -88.14404, "userId": "f1196aac-470e-11e6-beb8-9e71128cae77", "timestamp": 1468203975 }, { "latitude": 9.568012, "longitude": 77.962444, "userId": "f1196aac-470e-11e6-beb8-9e71128cae77", "timestamp": 1468203975 } ] Excellent! You have created your first microservice with Dropwizard. Dropwizard offers more than what we have seen so far. Some of it is out of scope for this article. I believe the metrics API that Dropwizard uses could be used in any type of application. Writing your Dockerfile So far in this article, we have seen how to package our application and how to install Docker. Now that we have our JAR artifact and Docker set up, let's see how to Dockerize our microservice application using Docker. Getting ready In order to Dockerize our application, we will have to tell Docker how our image is going to look. This is exactly the purpose of a Dockerfile. A Dockerfile has its own syntax (or Dockerfile instructions) and will be used by Docker to create images. Throughout this article, we will try to understand some of the most commonly used Dockerfile instructions as we write our Dockerfile for the geolocation tracker microservice. How to do it… First, open your STS IDE and create a new file called Dockerfile in the geolocation project. The first line of the Dockerfile is always the FROM instruction followed by the base image that you would like to create your image from. There are thousands of images on Docker Hub to choose from. In our case, we would need something that already has Java installed on it. There are some images that are official, meaning they are well documented and maintained. Docker Official Repositories are very well documented, and they follow best practices and standards. Docker has its own team to maintain these repositories. This is essential in order to keep the repository clear, thus helping the user make the right choice of repository. To read more about Docker Official Repositories, take a look at https://docs.docker.com/docker-hub/official_repos/ We will be using the Java official repository. To find the official repository, go to hub.docker.com and search for java. You have to choose the one that says official. At the time of writing this, the Java image documentation says it will soon be deprecated in favor of the openjdk image. So the first line of our Dockerfile will look like this: FROM openjdk:8 As you can see, we have used version (or tag) 8 for our image. If you are wondering what type of operating system this image uses, take a look at the Dockerfile of this image, which you can get from the Docker Hub page. Docker images are usually tagged with the version of the software they are written for. That way, it is easy for users to pick from. The next step is creating a directory for our project where we will store our JAR artifact. Add this as your next line: RUN mkdir -p /opt/packt/geolocation This is a simple Unix command that creates the /opt/packt/geolocation directory. The –p flag instructs it to create the intermediate directories if they don't exist. Now let's create an instruction that will add the JAR file that was created in your local machine into the container at /opt/packt/geolocation. ADD target/geolocation-0.0.1-SNAPSHOT.jar /opt/packt/geolocation/ As you can see, we are picking up the uber JAR from target directory and dropping it into the /opt/packt/geolocation directory of the container. Take a look at the / at the end of the target path. That says that the JAR has to be copied into the directory. Before we can start the application, there is one thing we have to do, that is, expose the ports that we would like to be mapped to the Docker host ports. In our case, the in-memory Tomcat instance is running on port 8080. In order to be able to map port 8080 of our container to any port to our Docker host, we have to expose it first. For that, we will use the EXPOSE instruction. Add the following line to your Dockerfile: EXPOSE 8080 Now that we are ready to start the app, let's go ahead and tell Docker how to start a container for this image. For that, we will use the CMD instruction: CMD ["java", "-jar", "/opt/packt/geolocation/geolocation-0.0.1-SNAPSHOT.jar"] There are two things we have to note here. Once is the way we are starting the application and the other is how the command is broken down into comma-separated Strings. First, let's talk about how we start the application. You might be wondering why we haven't used the mvn spring-boot:run command to start the application. Keep in mind that this command will be executed inside the container, and our container does not have Maven installed, only OpenJDK 8. If you would like to use the maven command, take that as an exercise, and try to install Maven on your container and use the mvn command to start the application. Now that we know we have Java installed, we are issuing a very simple java –jar command to run the JAR. In fact, the Spring Boot Maven plugin internally issues the same command. The next thing is how the command has been broken down into comma-separated Strings. This is a standard that the CMD instruction follows. To keep it simple, keep in mind that for whatever command you would like to run upon running the container, just break it down into comma-separated Strings (in whitespaces). Your final Dockerfile should look something like this: FROM openjdk:8 RUN mkdir -p /opt/packt/geolocation ADD target/geolocation-0.0.1-SNAPSHOT.jar /opt/packt/geolocation/ EXPOSE 8080 CMD ["java", "-jar", "/opt/packt/geolocation/geolocation-0.0.1-SNAPSHOT.jar"] This Dockerfile is one of the simplest implementations. Dockerfiles can sometimes get bigger due to the fact that you need a lot of customizations to your image. In such cases, it is a good idea to break it down into multiple images that can be reused and maintained separately. There are some best practices to follow whenever you create your own Dockerfile and image. Though we haven't covered that here as it is out of the scope of this article, you still should take a look at and follow them. To learn more about the various Dockerfile instructions, go to https://docs.docker.com/engine/reference/builder/. Building your Docker image We created the Dockerfile, which will be used in this article to create an image for our microservice. If you are wondering why we would need an image, it is the only way we can ship our software to any system. Once you have your image created and uploaded to a common repository, it will be easier to pull your image from any location. Getting ready Before you jump right into it, it might be a good idea to get yourself familiar with some of the most commonly used Docker commands. In this article, we will use the build command. Take a look at this URL to understand the other commands: https://docs.docker.com/engine/reference/commandline/#/image-commands. After familiarizing yourself with the commands, open up a new terminal, and change your directory to the root of the geolocation project. Make sure your docker-machine instance is running. If it is not running, use the docker-machine start command to run your docker-machine instance: docker-machine start default If you have to configure your shell for the default Docker machine, go ahead and execute the following command: eval $(docker-machine env default) How to do it… From the terminal, issue the following docker build command: docker build –t packt/geolocation. We'll try to understand the command later. For now, let's see what happens after you issue the preceding command. You should see Docker downloading the openjdk image from Docker Hub. Once the image has been downloaded, you will see that Docker tries to validate each and every instruction provided in the Dockerfile. When the last instruction has been processed, you will see a message saying Successfully built. This says that your image has been successfully built. Now let's try to understand the command. There are three things to note here: The first thing is the docker build command itself. The docker build command is used to build a Docker image from a Dockerfile. It needs at least one input, which is usually the location of the Dockerfile. Dockerfiles can be renamed to something other than Dockerfile and can be referred to using the –f option of the docker build command. An instance of this being used is when teams have different Dockerfiles for different build environments, for example, using DockerfileDev for the dev environment, DockerfileStaging for the staging environment, and DockerfileProd for the production environment. It is still encouraged as best practice to use other Docker options in order to keep the same Dockerfile for all environments. The second thing is the –t option. The –t option takes the name of the repo and a tag. In our case, we have not mentioned the tag, so by default, it will pick up latest as the tag. If you look at the repo name, it is different from the official openjdk image name. It has two parts: packt and geolocation. It is always a good practice to put the Docker Hub account name followed by the actual image name as the name of your repo. For now, we will use packt as our account name, we will see how to create our own Docker Hub account and use that account name here. The third thing is the dot at the end. The dot operator says that the Dockerfile is located in the current directory, or the present working directory to be more precise. Let's go ahead and verify whether our image was created. In order to do that, issue the following command on your terminal: docker images The docker images command is used to list down all images available in your Docker host. After issuing the command, you should see something like this: As you can see, the newly built image is listed as packt/geolocation in your Docker host. The tag for this image is latest as we did not specify any. The image ID uniquely identifies your image. Note the size of the image. It is a few megabytes bigger than the openjdk:8 image. That is most probably because of the size of our executable uber JAR inside the container. Now that we know how to build an image using an existing Dockerfile, we are at the end of this article. This is just a very quick intro to the docker build command. There are more options that you can provide to the command, such as CPUs and memory. To learn more about the docker build command, take a look at this page: https://docs.docker.com/engine/reference/commandline/build/ Running your microservice as a Docker container We successfully created our Docker image in the Docker host. Keep in mind that if you are using Windows or Mac, your Docker host is the VirtualBox VM and not your local computer. In this article, we will look at how to spin off a container for the newly created image. Getting ready To spin off a new container for our packt/geolocation image, we will use the docker run command. This command is used to run any command inside your container, given the image. Open your terminal and go to the root of the geolocation project. If you have to start your Docker machine instance, do so using the docker-machine start command, and set the environment using the docker-machine env command. How to do it… Go ahead and issue the following command on your terminal: docker run packt/geolocation Right after you run the command, you should see something like this: Yay! We can see that our microservice is running as a Docker container. But wait—there is more to it. Let's see how we can access our microservice's in-memory Tomcat instance. Try to run a curl command to see if our app is up and running: Open a new terminal instance and execute the following cURL command in that shell: curl -H "Content-Type: application/json" -X POST -d '{"timestamp": 1468203975, "userId": "f1196aac-470e-11e6-beb8-9e71128cae77", "latitude": 41.803488, "longitude": -88.144040}' http://localhost:8080/geolocation Did you get an error message like this? curl: (7) Failed to connect to localhost port 8080: Connection refused Let's try to understand what happened here. Why would we get a connection refused error when our microservice logs clearly say that it is running on port 8080? Yes, you guessed it right: the microservice is not running on your local computer; it is actually running inside the container, which in turn is running inside your Docker host. Here, your Docker host is the VirtualBox VM called default. So we have to replace localhost with the IP of the container. But getting the IP of the container is not straightforward. That is the reason we are going to map port 8080 of the container to the same port on the VM. This mapping will make sure that any request made to port 8080 on the VM will be forwarded to port 8080 of the container. Now go to the shell that is currently running your container, and stop your container. Usually, Ctrl + C will do the job. After your container is stopped, issue the following command: docker run –p 8080:8080 packt/geolocation The –p option does the port mapping from Docker host to container. The port number to the left of the colon indicates the port number of the Docker host, and the port number to the right of the colon indicates that of the container. In our case, both of them are same. After you execute the previous command, you should see the same logs that you saw before. We are not done yet. We still have to find the IP that we have to use to hit our RESTful endpoint. The IP that we have to use is the IP of our Docker Machine VM. To find the IP of the docker-machine instance, execute the following command in a new terminal instance: docker-machine ip default. This should give you the IP of the VM. Let's say the IP that you received was 192.168.99.100. Now, replace localhost in your cURL command with this IP, and execute the cURL command again: curl -H "Content-Type: application/json" -X POST -d '{"timestamp": 1468203975, "userId": "f1196aac-470e-11e6-beb8-9e71128cae77", "latitude": 41.803488, "longitude": -88.144040}' http://192.168.99.100:8080/geolocation This should give you an output similar to the following (pretty-printed for readability): { "latitude": 41.803488, "longitude": -88.14404, "userId": "f1196aac-470e-11e6-beb8-9e71128cae77", "timestamp": 1468203975 } This confirms that you are able to access your microservice from the outside. Take a moment to understand how the port mapping is done. The following figure shows how your machine, VM, and container are orchestrated: This confirms that you are able to access your microservice from the outside. Summary We looked at an example of a geolocation tracker application to see how it can be broken down into smaller and manageable services. Next, we saw how to create the GeoLocationTracker service using the Spring Boot framework. Resources for Article: Further resources on this subject: Domain-Driven Design [article] Breaking into Microservices Architecture [article] A capability model for microservices [article]
Read more
  • 0
  • 0
  • 46772
article-image-fuzzy-logic-ai-characters-unity-3d-games
Kunal Chaudhari
01 Jun 2018
16 min read
Save for later

Implementing fuzzy logic to bring AI characters alive in Unity based 3D games

Kunal Chaudhari
01 Jun 2018
16 min read
Fuzzy logic is a fantastic way to represent the rules of your game in a more nuanced way. Perhaps more so than other concepts, fuzzy logic is a very math-heavy topic. Most of the information can be represented purely by mathematical functions. For the sake of teaching the important concepts as they apply to Unity, most of the math has been simplified and implemented using Unity's built-in features. In this tutorial, we will take a look at the concepts behind fuzzy logic systems and implement in your AI system. Implementing fuzzy logic will make your game characters more believable and depict real-world attributes. This article is an excerpt from a book written by Ray Barrera, Aung Sithu Kyaw, and Thet Naing Swe titled  Unity 2017 Game AI Programming - Third Edition. This book will help you leverage the power of artificial intelligence to program smart entities for your games. Defining fuzzy logic The simplest way to define fuzzy logic is by comparison to binary logic.  Generally, transition rules as looked at as true or false or 0 or 1 values. Is something visible? Is it at least a certain distance away? Even in instances where multiple values were being evaluated, all of the values had exactly two outcomes; thus, they were binary. In contrast, fuzzy values represent a much richer range of possibilities, where each value is represented as a float rather than an integer. We stop looking at values as 0 or 1, and we start looking at them as 0 to 1. A common example used to describe fuzzy logic is temperature. Fuzzy logic allows us to make decisions based on non-specific data. I can step outside on a sunny Californian summer's day and ascertain that it is warm, without knowing the temperature precisely. Conversely, if I were to find myself in Alaska during the winter, I would know that it is cold, again, without knowing the exact temperature. These concepts of cold, cool, warm, and hot are fuzzy ones. There is a good amount of ambiguity as to at what point we go from warm to hot. Fuzzy logic allows us to model these concepts as sets and determine their validity or truth by using a set of rules. When people make decisions, people have some gray areas. That is to say, it's not always black and white. The same concept applies to agents that rely on fuzzy logic. Say you hadn't eaten in a few hours, and you were starting to feel a little hungry. At which point were you hungry enough to go grab a snack? You could look at the time right after a meal as 0, and 1 would be the point where you approached starvation. The following figure illustrates this point: When making decisions, there are many factors that determine the ultimate choice. This leads into another aspect of fuzzy logic controllers—they can take into account as much data as necessary. Let's continue to look at our "should I eat?" example. We've only considered one value for making that decision, which is the time since the last time you ate. However, there are other factors that can affect this decision, such as how much energy you're expending and how lazy you are at that particular moment. Or am I the only one to use that as a deciding factor? Either way, you can see how multiple input values can affect the output, which we can think of as the "likeliness to have another meal." Fuzzy logic systems can be very flexible due to their generic nature. You provide input, the fuzzy logic provides an output. What that output means to your game is entirely up to you. We've primarily looked at how the inputs would affect a decision, which, in reality, is taking the output and using it in a way the computer, our agent, can understand. However, the output can also be used to determine how much of something to do, how fast something happens, or for how long something happens. For example, imagine your agent is a car in a sci-fi racing game that has a "nitro-boost" ability that lets it expend a resource to go faster. Our 0 to 1 value can represent a normalized amount of time for it to use that boost or perhaps a normalized amount of fuel to use. Picking fuzzy systems over binary systems With most things in game programming, we must evaluate the requirements of our game and the technology and hardware limitations when deciding on the best way to tackle a problem. As you might imagine, there is a performance cost associated with going from a simple yes/no system to a more nuanced fuzzy logic one, which is one of the reasons we may opt out of using it. Of course, being a more complex system doesn't necessarily always mean it's a better one. There will be times when you just want the simplicity and predictability of a binary system because it may fit your game better. While there is some truth to the old adage, "the simpler, the better", one should also take into account the saying, "everything should be made as simple as possible, but not simpler". Though the quote is widely attributed to Albert Einstein, the father of relativity, it's not entirely clear who said it. The important thing to consider is the meaning of the quote itself. You should make your AI as simple as your game needs it to be, but not simpler. Pac-Man's AI works perfectly for the game–it's simple enough. However, rules say that simple would be out of place in a modern shooter or strategy game. Using fuzzy logic Once you understand the simple concepts behind fuzzy logic, it's easy to start thinking of the many ways in which it can be useful. In reality, it's just another tool in our belt, and each job requires different tools. Fuzzy logic is great at taking some data, evaluating it in a similar way to how a human would (albeit in a much simpler way), and then translating the data back to information that is usable by the system. Fuzzy logic controllers have several real-world use cases. Some are more obvious than others, and while these are by no means one-to-one comparisons to our usage in game AI, they serve to illustrate a point: Heating ventilation and air conditioning (HVAC) systems: The temperature example when talking about fuzzy logic is not only a good theoretical approach to explaining fuzzy logic, but also a very common real-world example of fuzzy logic controllers in action. Automobiles: Modern automobiles come equipped with very sophisticated computerized systems, from the air conditioning system (again), to fuel delivery, to automated braking systems. In fact, putting computers in automobiles has resulted in far more efficient systems than the old binary systems that were sometimes used. Your smartphone: Ever notice how your screen dims and brightens depending on how much ambient light there is? Modern smartphone operating systems look at ambient light, the color of the data being displayed, and the current battery life to optimize screen brightness. Washing machines: Not my washing machine necessarily, as it's quite old, but most modern washers (from the last 20 years) make some use of fuzzy logic. Load size, water dirtiness, temperature, and other factors are taken into account from cycle to cycle to optimize water use, energy consumption, and time. If you take a look around your house, there is a good chance you'll find a few interesting uses of fuzzy logic, and I mean besides your computer, of course. While these are neat uses of the concept, they're not particularly exciting or game-related. I'm partial to games involving wizards, magic, and monsters, so let's look at a more relevant example. Implementing a simple fuzzy logic system For this example, we're going to use my good friend, Bob, the wizard. Bob lives in an RPG world, and he has some very powerful healing magic at his disposal. Bob has to decide when to cast this magic on himself based on his remaining health points (HPs). In a binary system, Bob's decision-making process might look like this: if(healthPoints <= 50) { CastHealingSpell(me); } We see that Bob's health can be in one of two states—above 50, or not. Nothing wrong with that, but let's have a look at what the fuzzy version of this same scenario might look like, starting with determining Bob's health status: Before the panic sets in upon seeing charts and values that may not quite mean anything to you right away, let's dissect what we're looking at. Our first impulse might be to try to map the probability that Bob will cast a healing spell to how much health he is missing. That would, in simple terms, just be a linear function. Nothing really fuzzy about that—it's a linear relationship, and while it is a step above a binary decision in terms of complexity, it's still not truly fuzzy. Enter the concept of a membership function. It's key to our system, as it allows us to determine how true a statement is. In this example, we're not simply looking at raw values to determine whether or not Bob should cast his spell; instead, we're breaking it up into logical chunks of information for Bob to use in order to determine what his course of action should be. In this example, we're comparing three statements and evaluating not only how true each one is, but which is the most true: Bob is in a critical condition Bob is hurt Bob is healthy If you're into official terminology, we call this determining the degree of membership to a set. Once we have this information, our agent can determine what to do with it next. At a glance, you'll notice it's possible for two statements to be true at a time. Bob can be in a critical condition and hurt. He can also be somewhat hurt and a little bit healthy. You're free to pick the thresholds for each, but, in this example, let's evaluate these statements as per the preceding graph. The vertical value represents the degree of truth of a statement as a normalized float (0 to 1): At 0 percent health, we can see that the critical statement evaluates to 1. It is absolutely true that Bob is critical when his health is gone. At 40 percent health, Bob is hurt, and that is the truest statement. At 100 percent health, the truest statement is that Bob is healthy. Anything outside of these absolutely true statements is squarely in fuzzy territory. For example, let's say Bob's health is at 65 percent. In that same chart, we can visualize it like this: The vertical line drawn through the chart at 65 represents Bob's health. As we can see, it intersects both sets, which means that Bob is a little bit hurt, but he's also kind of healthy. At a glance, we can tell, however, that the vertical line intercepts the Hurt set at a higher point in the graph. We can take this to mean that Bob is more hurt than he is healthy. To be specific, Bob is 37.5 percent hurt, 12.5 percent healthy, and 0 percent critical. Let's take a look at this in code; open up our FuzzySample scene in Unity. The hierarchy will look like this: The important game object to look at is Fuzzy Example. This contains the logic that we'll be looking at. In addition to that, we have our Canvas containing all of the labels and the input field and button that make this example work. Lastly, there's the Unity-generated EventSystem and Main Camera, which we can disregard. There isn't anything special going on with the setup for the scene, but it's a good idea to become familiar with it, and you are encouraged to poke around and tweak it to your heart's content after we've looked at why everything is there and what it all does. With the Fuzzy Example game object selected, the inspector will look similar to the following image: Our sample implementation is not necessarily something you'll take and implement in your game as it is, but it is meant to illustrate the previous points in a clear manner. We use Unity's AnimationCurve for each different set. It's a quick and easy way to visualize the very same lines in our earlier graph. Unfortunately, there is no straightforward way to plot all the lines in the same graph, so we use a separate AnimationCurve for each set. In the preceding screenshot, they are labeled Critical, Hurt, and Healthy. The neat thing about these curves is that they come with a built-in method to evaluate them at a given point (t). For us, t does not represent time, but rather the amount of health Bob has. As in the preceding graph, the Unity example looks at a HP range of 0 to 100. These curves also provide a simple user interface for editing the values. You can simply click on the curve in the inspector. That opens up the curve editing window. You can add points, move points, change tangents, and so on, as shown in the following screenshot: Unity's curve editor window Our example focuses on triangle-shaped sets. That is, linear graphs for each set. You are by no means restricted to this shape, though it is the most common. You could use a bell curve or a trapezoid, for that matter. To keep things simple, we'll stick to the triangle. You can learn more about Unity's AnimationCurve editor at http://docs.unity3d.com/ScriptReference/AnimationCurve.html. The rest of the fields are just references to the different UI elements used in code that we'll be looking at later in this chapter. The names of these variables are fairly self-explanatory, however, so there isn't much guesswork to be done here. Next, we can take a look at how the scene is set up. If you play the scene, the game view will look something similar to the following screenshot: A simple UI to demonstrate fuzzy values We can see that we have three distinct groups, representing each question from the "Bob, the wizard" example. How healthy is Bob, how hurt is Bob, and how critical is Bob? For each set, upon evaluation, the value that starts off as 0 true will dynamically adjust to represent the actual degree of membership. There is an input box in which you can type a percentage of health to use for the test. No fancy controls are in place for this, so be sure to enter a value from 0 to 100. For the sake of consistency, let's enter a value of 65 into the box and then press the Evaluate! button. This will run some code, look at the curves, and yield the exact same results we saw in our graph earlier. While this shouldn't come as a surprise (the math is what it is, after all), there are fewer things more important in game programming than testing your assumptions, and sure enough, we've tested and verified our earlier statement. After running the test by hitting the Evaluate! button, the game scene will look similar to the following screenshot: This is how Bob is doing at 65 percent health Again, the values turn out to be 0.125 (or 12.5 percent) healthy and 0.375 (or 37.5 percent) hurt. At this point, we're still not doing anything with this data, but let's take a look at the code that's handling everything: using UnityEngine; using UnityEngine.UI; using System.Collections; public class FuzzySample1 : MonoBehaviour { private const string labelText = "{0} true"; public AnimationCurve critical; public AnimationCurve hurt; public AnimationCurve healthy; public InputField healthInput; public Text healthyLabel; public Text hurtLabel; public Text criticalLabel; private float criticalValue = 0f; private float hurtValue = 0f; private float healthyValue = 0f; We start off by declaring some variables. The labelText is simply a constant we use to plug into our label. We replace {0} with the real value. Next, we declare the three AnimationCurve variables that we mentioned earlier. Making these public or otherwise accessible from the inspector is key to being able to edit them visually (though it is possible to construct curves by code), which is the whole point of using them. The following four variables are just references to UI elements that we saw earlier in the screenshot of our inspector, and the last three variables are the actual float values that our curves will evaluate into: private void Start () { SetLabels(); } /* * Evaluates all the curves and returns float values */ public void EvaluateStatements() { if (string.IsNullOrEmpty(healthInput.text)) { return; } float inputValue = float.Parse(healthInput.text); healthyValue = healthy.Evaluate(inputValue); hurtValue = hurt.Evaluate(inputValue); criticalValue = critical.Evaluate(inputValue); SetLabels(); } The Start() method doesn't require much explanation. We simply update our labels here so that they initialize to something other than the default text. The EvaluateStatements() method is much more interesting. We first do some simple null checking for our input string. We don't want to try and parse an empty string, so we return out of the function if it is empty. As mentioned earlier, there is no check in place to validate that you've input a numerical value, so be sure not to accidentally input a non-numerical value or you'll get an error. For each of the AnimationCurve variables, we call the Evaluate(float t) method, where we replace t with the parsed value we get from the input field. In the example we ran, that value would be 65. Then, we update our labels once again to display the values we got. The code looks similar to this: /* * Updates the GUI with the evluated values based * on the health percentage entered by the * user. */ private void SetLabels() { healthyLabel.text = string.Format(labelText, healthyValue); hurtLabel.text = string.Format(labelText, hurtValue); criticalLabel.text = string.Format(labelText, criticalValue); } } We simply take each label and replace the text with a formatted version of our labelText constant that replaces the {0} with the real value. To summarize, we learned how fuzzy logic is used in the real world, and how it can help illustrate vague concepts in a way binary systems cannot. We also learned to implement our own fuzzy logic controllers using the concepts of member functions, degrees of membership, and fuzzy sets.  If you enjoyed this excerpt, check out the book Unity 2017 Game AI Programming - Third Edition, to build exciting and richer games by mastering advanced Artificial Intelligence concepts in Unity. Unity Machine Learning Agents: Transforming Games with Artificial Intelligence Put your game face on! Unity 2018.1 is now available How to create non-player Characters (NPC) with Unity 2018
Read more
  • 0
  • 0
  • 46762

article-image-does-it-make-sense-to-talk-about-devops-engineers-or-devops-tools
Richard Gall
10 May 2019
6 min read
Save for later

Does it make sense to talk about DevOps engineers or DevOps tools?

Richard Gall
10 May 2019
6 min read
DevOps engineers are in high demand - the job represents an engineering unicorn, someone that understands both development and operations and can help to foster a culture where the relationship between the two is almost frictionless. But there's some debate as to whether it makes sense to talk about a DevOps engineer at all. If DevOps is a culture of a set of practices that improves agility and empowers engineers to take more ownership over their work, should we really be thinking about DevOps as a single job that someone can simply train for? The quotes in this piece are taken from DevOps Paradox by Viktor Farcic, which will be published in June 2019. The book features interviews with a diverse range of figures drawn from across the DevOps world. Is DevOps engineer a 'real' job or just recruitment spin? Nirmal Mehta (@normalfaults), Technology Consultant at Booz Allen Hamilton, says "There's no such thing as a DevOps engineer. There shouldn't even be a DevOps team, because to me DevOps is more of a cultural and philosophical methodology, a process, and a way of thinking about things and communicating within an IT organization..." Mehta is cyncical about organizations that put out job descriptions asking for DevOps engineers. It is, he argues, a way of cutting costs - a way of simply doing more with less. "A DevOps engineer is just a job posting that signals an organization wants to hire one less person to do twice as much work rather than hire both a developer and an operator." This view is echoed by other figures associated with the DevOps world. Mike Kail (@mdkail), CTO at Everest, says "I certainly don't view DevOps as a tool or a job title. In my view, at the core, it's a cultural approach to leveraging automation and orchestration to streamline both code development, infrastructure, application deployments and subsequently, the managing of those resources." Similarly, Damian Duportal (@DamienDuportal), Træfik's Developer Advocate, says "there is no such thing as a DevOps engineer or even a DevOps team.  The main purpose of DevOps is to focus on value, finding the optimal for the organization, and the value it will bring." For both Duportal and Kail, then, DevOps is primarily a cultural thing, something which needs to be embedded inside the practices of an organization. Is it useful to talk about a DevOps team? There are big question marks over the concept of a DevOps engineer. But what about a specific team? It's all well and good talking about organizational philosophy, but how do you actually affect change in a practical manner? Julian Simpson (@builddoctor), Neo4J's Global IT Manager is sceptical about the concept of a DevOps team: “Can we have something called a DevOps team? I don't believe so. You might spin up a team to solve a DevOps problem, but then I wouldn't even say we specifically have a DevOps problem. I'd say you just have a problem." DevOps consultant Chris Riley (@HoardingInfo) has a similar take, saying: “DevOps Engineer as a title makes sense to me, but I don't think you necessarily have DevOps departments, nor do you seek that out. Instead, I think DevOps is a principle that you spread throughout your entire development organization. Rather, you look to reform your organization in a way that supports those initiatives versus just saying that we need to build this DevOps unit, and there we go, we're done, we're DevOps. Because by doing that you really have to empower that unit and most organizations aren't willing to do that." However, Red Hat Solutions Architect Wian Vos  (@wianvos) has a different take. For Vos the idea of a DevOps team is actually crucial if you are to cultivate a DevOps mindset inside your organization: "Imagine... you and I were going to start a company. We're going to need a DevOps team because we have a burning desire to put out this awesome application. The questions we have when we're putting together a DevOps team is both ‘Who are we hiring?’ and ‘What are we hiring for? Are we going to hire DevOps engineers? No. In that team, we want the best application developers, the best tester, and maybe we want a great infrastructure guy and a frontend/backend developer. I want people with specific roles who fit together as a team to be that DevOps team." For Vos, it's not so much about finding and hiring DevOps engineers - people with a specific set of skills and experience - but rather building a team that's constructed in such a way that it can put DevOps principles into practice. Is there such a thing as a DevOps tool? One of the interesting things about DevOps is that the debate seems to lead you into a bit of a bind. It's almost as if the more concrete we try and make it - turning it into a job, or a team - the less useful it becomes. This is particularly true when we consider tooling. Surely thinking about DevOps technologically, rather than speculatively makes it more real? In general, it appears there is a consensus against the idea of DevOps tools. On this point Julian Simpson said "my original thinking about the movement from 2009 onwards, when the name was coined, was that it would be about collaboration and perhaps the tools would sort of come out of that collaboration." James Turnbull (@kartar), CEO of Rethink Robotics is critical of the notion of DevOps tools. He says "I don't think there are such things as DevOps tools. I believe there are tools that make the process of being a cross-functional team better... Any tool that facilitates building that cross-functionality is probably a DevOps tool to the point where the term is likely meaningless." When it comes to DevOps, everyone's still learning With even industry figures disagreeing on what terms mean, or which ones are relevant, it's pretty clear that DevOps will remain a field that's contested and debated. But perhaps this is important - if we expect it to simply be a solution to the engineering challenges we face, it's already failed as a concept. However, if we understand it as a framework or mindset for solving problems then that is when it acquires greater potency. Viktor Farcic is a Developer Advocate at CloudBees, a member of the Google Developer Experts and Docker Captains groups, and published author. His big passions are DevOps, Microservices, Continuous Integration, Delivery and Deployment (CI/CD) and Test-Driven Development (TDD).
Read more
  • 0
  • 0
  • 46702

article-image-why-dont-you-have-a-monorepo
Viktor Charypar
01 Feb 2019
27 min read
Save for later

Why don't you have a monorepo?

Viktor Charypar
01 Feb 2019
27 min read
You’ve probably heard that Facebook, Twitter, Google, Microsoft, and other tech industry behemoths keep their entire codebase, all services, applications and tools in a single huge repository - a monorepo. If you’re used to the standard way most teams manage their codebase - one application, service or tool per repository - this sounds very strange. Many people conclude it must only solve problems the likes of Google and Facebook have. This is a guest post by Viktor Charypar, Technical Director at Red Badger. But monorepos are not only useful if you can build a custom version control system to cope. They actually have many advantages even at a smaller scale that standard tools like Git handle just fine. Using a monorepo can result in fewer barriers in the software development lifecycle. It can allow faster feedback loops, less time spent looking for code,, and less time reporting bugs and waiting for them to be fixed. It also makes it much easier to analyze a huge treasure trove of interesting data about how your software is actually built and where problem areas are. We’ve used a monorepo at one of our clients for almost three years and it’s been great. I really don’t see why you wouldn’t. But roughly every two months I tend to have a conversation with someone who’s not used to working in this way and the entire idea just seems totally crazy to them. And the conversation tends to always follow the same path, starting with the sheer size and quickly moving on to dependency management, testing and versioning strategies. It gets complicated. It’s time I finally wrote down a coherent reasoning behind why I believe monorepos should be the default way we manage a codebase. Especially if you’re building something even vaguely microservices based, you have multiple teams and want to share common code. What do you mean “just one repo”? Just so we’re all thinking about the same thing, when I say monorepo, I’m talking about a strategy of storing all the code you as an organization are responsible for. This could be a project, a programme of work, or the entirety of a product and infrastructure code of your company in a single repository, under one revision history. Individual components (libraries, services, custom tools, infrastructure automation, ...) are stored alongside each other in folders. It’s analogous to the UNIX file tree which has a single root, as opposed to multiple, device based roots in Windows operating systems. People not familiar with the concept typically have a fairly strong reaction to the idea. One giant repo? Why would anyone do that? That cannot possibly scale! Many different objections come out, most of them often only tangentially related to storing all the code together. Occasionally, people get almost religious about it (I am talking about engineers after all). Despite being used by some of the largest tech companies, it is a relatively foreign concept and on the surface goes against everything you’ve been taught about not making huge monolithic things. It also seems like we’re fixing things that are not broken: everyone in the world is doing multiple repos, building and sharing artifacts (npm modules, JARs, ruby gems…), using SemVer to version and manage dependencies and long running branches to patch bugs in older versions of code, right? Surely if it’s industry standard it must be the right thing to do. Well, I don’t believe so. I personally think almost every single one of those things is harder, more laborious, more brittle, harder to test and generally just more wasteful than the equivalent approach you get as a consequence of a monorepo. And a few of the capabilities a monorepo enables can’t be replicated in a multi-repo situation even if you build a lot of infrastructure around it, basically because you introduce distributed computing problems and get on the bad side of CAP theorem (we’ll look at this closer below). Apart from letting you make dependency management easier and testing more reliable than it can get with multiple repos, monorepo will also give you a few simple, important, but easy to underestimate advantages. The biggest advantages of using a monorepo It's easier to find and discover your code in a monorepo With a monorepo, there is no question about where all the code is and when you have access to some of it, you can see all of it. It may come as a surprise, but making code visible to the rest of the organization isn’t always the default behavior. Human insecurities get in the way and people create private repositories and squirrel code away to experiment with things “until they are ready”. Typically, when the thing does get “ready”, it now has a Continuous Integration (CI) service attached, many hyperlinks lead to it from emails, chat rooms and internal wikis, several people have cloned the repo and it’s now quite a hassle to move the code to a more visible, obvious place and so it stays where it started. As a consequence, it is often quite hard work to find all the code for the project and gain access to it, which is hard and expensive for new joiners and hurts collaboration in general. You could say this is a matter of discipline and I will agree with you, but why leave to individual discipline what you can simply prevent by telling everyone that all the code belongs in the one repo, it’s completely ok to put even little experiments and pet projects there. You never know what they will grow into and putting them in the repo has basically no cost attached. Visibility aids understanding of how to use internal APIs (internal in the sense of being designed and built by your organisation). The ability to search the entire codebase from within your editor and find usages of the call you’re considering to use is very powerful. Code editors and languages can also be set up for cross-references to work, which means you can follow references into shared libraries and find usages of shared code across the codebase. And I mean the entire codebase. This also enables all kind of analyses to be done on the codebase and its history. Knowing the totality of the codebase and having a history of all the code lets you see dependencies, find parts of the codebase only committed to by a very limited group of people, hotspots changing suspiciously frequently or by a large number of people… Your codebase is the source of truth about what your engineering organization is producing, it contains an incredible amount of interesting information we typically just ignore. Monorepos give you more flexibility when moving code Conway’s Law famously states that “organizations which design systems (...) are constrained to produce designs which are copies of the communication structures of these organisations”. This is due to the level of communication necessary to produce a coherent piece of software. The further away in the organisation an owner of a piece of software is, the harder it is to directly influence it, so you design strict interfaces to insulate yourself from the effect of “their” changes. This typically affects the repository structure as well. There are two problems with this: the structure is generally chosen upfront, before we know what the right shape of the software is, and changing the structure has a cost attached. With each service and library being in a separate repository, the module boundaries are quite a lot stronger than if they are all in one repository. Extracting common pieces of code into a shared library becomes more difficult and involves a setup of a whole new repository - full with CI integration, pull request templates and labels, access control setup… hard work. In a monorepo, these boundaries are much more fluid and flexible: moving code between services and libraries, extracting new ones or inlining libraries back into their consumers all become as easy as general refactoring. There is no reason to use a completely different set of tools to change the small-scale and the large-scale structure of your codebase. The only real downside is tooling support for access control and declaring ownership. However, as monorepos get more popular, this support is getting better. GitHub now supports codeowners, for example. We will get there. A monorepo gives you a single history timeline While visibility and flexibility are quite convenient, the one feature of a monorepo which is very hard (if not impossible) to replicate is the single history timeline. We’ll go into why it’s so hard further below, but for now let’s look at the advantages it brings. Single history timelines gives us a reliable total order of changes to the codebase over time. This means that for any two contributions to the codebase, we can definitively and reliably decide which came first and which came second. It should never be ambiguous. It also means that each commit in a monorepo is a snapshot of the system as it was at that given moment. This enables a really interesting capability: it means cross-cutting changes can be made atomically, safely, in one go. Atomic cross-cutting commits Atomic cross-cutting commits make two specific scenarios much easier to achieve. First, externally forced global migrations are much easier and quicker. Let’s say multiple services use a common database and need the password and we need to rotate it. The password itself is (hopefully!) stored in a secure credential store, but at least the reference to it will be in several different places within the codebase. If the reference changes (let’s say the reference is generated every time), we can update every specific the mention of it at once, in one commit, with a search & replac. This will get everything working again. Second, and more important, we can change APIs and update both producer and all consumers at the same time, atomically. For example, we can add an endpoint to an API service and migrate consumers to use the new endpoint. In the next commit, we can remove the old API endpoint as it’s no longer needed. If you're trying to do this across multiple repositories with their own histories, the change will have to be split into several parallel commits. This leaves the potential for the two changes to overlap and happen in the wrong order. Some consumers can get migrated, then the endpoint gets removed, then the rest of the consumers get migrated. The mid-stage is an inconsistent state and an attempt to use the not-yet-migrated consumers will fail attempting to call an API endpoint that no longer exists. Monorepos remove inconsistencies in your dependencies Inconsistencies between dependent modules are the entire reason why dependency management and versioning exist. In a monorepo, the above scenario simply can’t happen. And that’s how the conversation about storing code in one place ends up being about versioning and dependency management. Monorepos essentially make the problem go away (which is my favourite kind of problem solving). Okay, this isn’t entirely true. There are still consequences to making breaking changes to APIs. For one, you need to update all the consumers, which is work, but you also need to build all of them, test everything works and deploy it all. This is quite hard for (micro)services that get individually deployed: making coordinated deployment of multiple services atomic is possible but not trivial. You can use a blue-green strategy, for example, to make sure there is no moment in time where only some services changed but not others. It gets harder for shared libraries. Building and publishing artifacts of new versions and updating all consumers to use the new version are still at least two commits, otherwise you’d be referring to versions that won’t exist until the builds finish. Now, things are getting inconsistent again and the view of what is supposed to work together is getting blurred in time again. And what if someone sneaks some changes in between the two commits. We are, once again, in a race. Unless... Building from the latest source Yes. What if instead of building and publishing shared code as prebuilt artifacts (binaries, jars, gems, npm modules), we build each deployable service completely from source. Every time a service changes, it is entirely rebuilt, including all dependencies. This is a fair bit of work for some compiled languages. However, it can be optimised with incremental build tools which skip work that’s already been done and cached. Some, like Go, solve it by simply having designed for a fast compiler. For dynamic languages, it’s just a matter of setting up include paths correctly and bundling all the relevant code. The added benefit here is you don’t need to do anything special if you’re working on a set of interdependent projects locally. No more `npm link`. The more interesting consequence is how this affects changing a shared dependency. When building from source, you have to make sure every time that happens, all the consumers get rebuilt using it. This is great, everyone gets the latest and greatest things immediately. ...right? Don’t worry, I can hear the alarm bells ringing in your head all the way from here. Your service depends on tens, if not hundreds of libraries. Any time anyone makes a mistake and breaks any of them, it breaks your code? Hell no. But hear me out. This is a problem of understanding dependencies and testing consumers. The important consequence of building from source is you now have a single handle on what is supposed to work together. There are no separate versions, you know what to test and it’s just one thing at any one time. Push dependency management In manual dependency update management - I will call it “pull” dependency management - you as a consumer are responsible for updating your dependencies as you see fit and making sure everything still works. If you find a bug, you simply don’t upgrade. Instead you report the bug to the maintainer and expect them to fix it. This can be months after the bug was introduced and the bug may have already been fixed in a newer version you haven’t yet upgraded to because things have moved on quite a bit while you were busy hitting a deadline and it would now be a sizable investment to upgrade. Now you’re a little stuck and all the ways out are a significant amount of work for someone, all that because the feedback loop is too long. Normally, as a library maintainer, you’re never quite certain how to make sure you’re not breaking anything. Even if you could run your consumers’ test suites, which consumers at what versions do you test against? And as a DevOps team doing 24/7 support for a system, how do you know which version or versions of a library is used across your services. What do you need to update to roll out that important bug fix to your customers? In push dependency management, quite a few things are the other way round. As a consumer, you’re not responsible for updating, it is done for you - effectively, you depended on the “latest” version of everything. Every time a maintainer of the library makes a change you are responsible for testing for regressions. No, not manually! You do have unit tests, right? Right?? Please have a solid regression test suite you trust, it’s 2019. So with your unit test suite, all you need to do is run it. Actually no, let’s let the maintainer run it. If they introduce a problem, they get immediate feedback from you, before their change ever hits the master branch. And this is the contract agreement in push dependency management: If you make a change and break anyone, you are responsible for fixing them. They are responsible for supplying a good enough, by their own standard, automated mechanism for you to verify things still work. The definition of “works” is the tests pass. Seriously though, you need to have a decent regression test suite! Continuous integration for push dependencies: the final piece of the monorepo puzzle The main missing piece of tooling around monorepos is support for push dependencies in CI systems. It’s quite straightforward to implement the strategy yourself, but it’s still hard enough to be worth some shared tooling. Unfortunately, the existing build tools geared towards monorepos like Bazel and Buck take over the entire build process from more familiar tools (like Maven or Babel) and you need to switch to them. Although to be fair, in exchange, you get very performant incremental builds. A lighter tooling, which lets you express dependencies between components in a monorepo, in a language agnostic way, and is only responsible for deciding which build jobs needs to be triggered given a set of changed components in a commit seems to be missing. So I built one. It’s far from perfect, but it should do the trick. Hopefully, someone with more time on their hands will eventually come up with something similarly cheap to introduce into your build system and the community will adopt it more widely. The main takeaway is that if we build from source in a monorepo we can set up a central Continuous Integration system responsible for triggering builds for all projects potentially affected by a change, intended to make sure you didn’t break anything with the work you did, whether it belongs to you or someone else. This is next to impossible in a multi-repo codebase because of the blurriness of history mentioned above. It’s interesting to me that we have this problem today in the larger ecosystem. Everywhere. And we stumble forward and somewhat successfully live with upstream changes occasionally breaking us, because we don’t really have a better choice. We don’t have the ability to test all the consumers in the world and fix things when we break them. But if we can, at least for our own codebase, why wouldn’t we do that? Along with a “you broke it you fix it” policy. Building from source in a monorepo allows that. It also makes it significantly easier to make breaking changes much harder to make. That said... About breaking changes There are two kinds of changes that break the consumer - the ones you introduce by accident, intending the keep backwards compatibility, and then the intentional ones. The first kind should not be too laborious to fix: once you find out what’s wrong, fix it in one place, make sure you didn’t break anything else, done. The second kind is harder. If you absolutely have to make an intentional breaking change, you will need to update all the consumers. Yes. That’s the deal. And that’s also fair. I’m not sure why we’re okay with breaking changes being intentionally introduced upstream on a whim. In any other area of human endeavour, a breach of contract makes people angry and they will expect you to make good by them. Yet we accept breaking changes in software as a fact of life. “It’s fine, I bumped the major version!” Semantic versioning: a bad idea It’s not fine. In fact, semantic versioning is just a bad idea. I know that’s a bold claim and this is a whole separate article (which I promise to write soon), but I’ll try to do it at least some justice here. Semantic versioning is meant to convey some meaning with the version number, but the chosen meanings it expresses are completely arbitrary. First of all, semver only talks about API contract, not behaviour. Adding side effects or changing performance characteristics of an API call for worse while keeping the data interface the same is a completely legal patch level change. And I bet you consider that a breaking change, because it will break your app. Second, does anyone really care about minor vs. patch? The promise is the API doesn’t break. So really we only care about major or other. Major is a dealbreaker, otherwise we’re ok. From a consumer perspective a major version bump spells trouble and potentially a lot of work. Making breaking changes is a mean thing to do to your consumers and you can and should avoid them. Just keep the old API around and working and add the new one next to it. As for version numbers, the most important meaning to convey seems to be “how old?” because code tends to rot, and so versioning by date might be a good choice. But, you say, I’ll have more and more code to maintain! Well yes, of course. And that’s the other problem with semver, the expectation that even old versions still get patches and fixes. It’s not very explicitly stated but it’s there. And because we typically maintain old versions on long-running branches in version control, it’s not even very visible in the codebase. What if you kept older APIs around, but deprecated them and the answer to bugs in them would be to migrate to the newer version of a particular call, which doesn’t have the bug? Would you care about having that old code? It just sits there in the codebase, until nobody uses it. It would also be much less work for the consumer, it’s just one particular call. Also, the bug is typically deeper inside your code, so it’s actually more likely you can fix it in one go for all the API surfaces, old or new. Doing the same thing in the branching model is N times the work (for N maintenance branches). There are technologies that follow this model out of necessity. One example is GraphQL, which was built to solve (among other things) the problem of many old API consumers in people’s hands and the need to support all of them for at least some time. In GraphQL, you deprecate data fields in your API and they become invisible in documentation and introspection calls, but they still work as they used to. Possibly forever. Or at least until barely anyone uses them. The other option you have if you want to keep an older version of a library around and maintain it in a monorepo is to make a copy of the folder and work on the two separately. It’s the same thing as cutting a long running branch, you’re just making a copy in “file space” not “branch space”. And it’s more visible and representative of reality - both versions exist as first-class components being maintained. There are many different versioning and maintenance strategies you could adopt, but in my opinion the preference should be to invest effort into the latest version, making breaking changes only when absolutely inevitable (and at that point, isn’t the new version just a new thing? Like Luxon, the next version of Moment.js) and making updates trivial for your consumers. And if it’s trivial you can do it for them. Ultimately it was your decision to break the API, so you should also do the work, it’s only fair and it makes you evaluate the cost-benefit trade-off of the change. In a monorepo with building from source, this versioning strategy happens naturally. You can, however, adopt others. You just lose some of the guarantees and make feedback loops longer. Versioning strategy is really an orthogonal concept to storing code in a single repository, but the relative costs do change if you use one. Versioning by using a single version that cuts across the system becomes a lot cheaper, which means breaking changes becomes more expensive. This tends to lead to more discussions about versioning. This is actually true for most of the things we covered above. You can, but don’t have to adopt these strategies with a monorepo. Pay as you go monorepos It’s totally possible to just store everything in a single repo and not do anything else. You’ll get the visibility of what exists and flexibility of boundaries and ownership. You can still publish individual build artifacts and pull-manage dependencies (but you’d be missing out). Add building from source and you get the single snapshot benefit - you now know what code runs in a particular version of your system and to an extent, you can think about it as a monolith, despite being formed of many different independent modules and services. Add dependency aware continuous integration and the feedback loop around issues introduced while working on the codebase gets much much shorter, allowing you to go faster and waste less time on carefully managing versions, reporting bugs, making big forklift upgrades, etc. Things tend to get out of control much less. It’s simpler. Best of all, you can mix and match strategies. If you have a hugely popular library in your monorepo and each change in it triggers a build of hundreds of consumers, it only takes a couple of those builds being flakey and it will make it very hard to get builds for the changes in the library to pass. This is really a CI problem to fix (and there are so many interesting strategies out there), but sometimes you can’t do that easily. You could also say the feedback loop is now too tight for the scale and start versioning the library’s intentional releases. This still doesn’t mean you have to publish versioned artifacts. You can have a stable copy of the library in the repo, which consumers depend on, and a development copy next to it which the maintainers work on. Releasing then means moving changes from the development folder to the release one and getting its builds to pass. Or, if you wish, you can publish artifacts and let consumers pull them on their own time and report bugs to you. And you still don’t need to promise fixes for older versions without upgrading to the latest (Libraries should really have a published “code of maintenance” outlining the promises and setting maintenance expectations). And if you have to, I would again recommend making a copy, not branching. In fact, in a monorepo, branching might just not be a very good idea. Temporary branches are still useful to work on proposed changes, but long-running branches just hide the full truth about the system. And so does relying on old commits. The copies of code being used exist either way, the are still relevant and you still need to consider them for testing and security patching, they are just hidden in less apparent dimensions of the codebase “space” - branch dimension or time dimension. These are hard to think about and visualise, so maybe it’s not a good idea to use them to keep relevant and current versions of the code and stick to them as change proposal and “time travel” mechanisms. Hopefully you can see that there’s an entire spectrum of strategies you can follow but don’t have to adopt wholesale. I’m sold, but... can’t we do all this with a multi-repo? Most of the things discussed above are not really strictly dependent on monorepos, they are more a natural consequence of adopting one. You can follow versioning strategies other than semver outside of a monorepo. You can probably implement an automated version bumping system which will upgrade all the dependents of a library and test them, logging issues if they don’t pass. What you can’t do outside of a monorepo, as far as I can tell, is the atomic snapshotting of history to have a clear view of the system. And have the same view of the system a year ago. And be able to reproduce it. As soon as multiple parallel version histories are established, this ability goes away and you introduce distributed state. It’s impossible to update all the “heads” in this multi history at the same time, consistently. In version control, like git, the histories are ordered by the “follows” relationship. Later version follows - points to - its predecessor. To get a consistent, canonical view of time, there needs to be a single entrypoint. Without this central entry point, it’s impossible to define a consistent order across the entire set, it depends on where we start looking. Essentially, you already chose Partition from the three CAP properties. Now you can pick either Consistency or Availability. Typically, availability is important and so you lose consistency. You could choose consistency instead, but that would mean you can’t have availability - in order to get a consistent snapshot of the state of all of the repos, write access would need to be stopped while the snapshot is taken. In a monorepo, you don’t have partitioning, and can therefore have consistency and availability. From a physics perspective, multiple repositories with their own history effectively create a kind of spacetime, where each repository is a place and references across repos represent information propagating across space. The speed of that propagation isn’t infinite - it’s not instant. If changes happen in two places close enough in time, from the perspective of those two places, they happen in a globally inconsistent order, first the local change, then the remote change. Neither of the views is better and more true and it’s impossible to decide which of the changes came first. Unless, that is, we introduce an agreed upon central point which references all the repositories that exist and every time one of them updates, the reference in this master gets updated and a revision is created. Congratulations, we created a monorepo, well done us. The benefits of going all-in when it comes to monorepos As I said at the beginning, adopting the monorepo approach fully will result in fewer barriers in the software development lifecycle. You get faster feedback loops - the ability to test consumers of libraries before checking in a change and immediate feedback. You will spend less time looking for code and working out how it gets assembled together. You won’t need to set up repositories or ask for permissions to contribute. You can spend more time-solving problems to help your customers instead of problems you created for yourself. It takes some time to get the tooling setup right, but you only do it once, all the later projects get the setup for free. Some of the tooling is a little lacking, but in our experience there are no show stoppers. A stable, reliable CI is an absolute must, but that’s regardless of monorepos. Monorepos should also help make builds repeatable. The repo does eventually get big, but it takes years and years and hundreds of people to get to a size where it actually becomes a real problem. The Linux kernel is a monorepo and it’s probably still at least an order of magnitude bigger than your project (it is bigger than ours anyway, despite having hundreds of engineers involved at this point). Basically, you’re not Google or Microsoft. And when you are, you’ll be able to afford optimising your version control system. The UX of your code review and source hosting tooling is probably the first thing that will break, not the underlying infrastructure. For smoother scaling, the one recommendation I have is to set a file size limit - accidentally committed large files are quite hard to remove, at least in git. After using a monorepo for over two years we’re still yet to have any big technical issues with it (plenty of political ones, but that’s a story for another day) and we see the same benefits as Google reported in their recent paper. I honestly don’t really know why you would start with any other strategy. Viktor Charypar is Technical Director at Red badger, a digital consultancy based in London. You can follow him on Twitter @charypar or read more of his writing on the Red Badger blog here.
Read more
  • 0
  • 0
  • 46680
article-image-google-open-sources-an-on-device-real-time-hand-gesture-recognition-algorithm-built-with-mediapipe
Sugandha Lahoti
21 Aug 2019
3 min read
Save for later

Google open sources an on-device, real-time hand gesture recognition algorithm built with MediaPipe

Sugandha Lahoti
21 Aug 2019
3 min read
Google researchers have unveiled a new real-time hand tracking algorithm that could be a new breakthrough for people communicating via sign language. Their algorithm uses machine learning to compute 3D keypoints of a hand from a video frame. This research is implemented in MediaPipe which is an open-source cross-platform framework for building multimodal (eg. video, audio, any time series data) applied ML pipelines. What is interesting is that the 3D hand perception can be viewed in real-time on a mobile phone. How real-time hand perception and gesture recognition works with MediaPipe? The algorithm is built using the MediaPipe framework. Within this framework, the pipeline is built as a directed graph of modular components. The pipeline employs three different models: a palm detector model, a handmark detector model and a gesture recognizer. The palm detector operates on full images and outputs an oriented bounding box. They employ a single-shot detector model called BlazePalm, They achieve an average precision of 95.7% in palm detection. Next, the hand landmark takes the cropped image defined by the palm detector and returns 3D hand keypoints. For detecting key points on the palm images, researchers manually annotated around 30K real-world images with 21 coordinates. They also generated a synthetic dataset to improve the robustness of the hand landmark detection model. The gesture recognizer then classifies the previously computed keypoint configuration into a discrete set of gestures. The algorithm determines the state of each finger, e.g. bent or straight, by the accumulated angles of joints. The existing pipeline supports counting gestures from multiple cultures, e.g. American, European, and Chinese, and various hand signs including “Thumb up”, closed fist, “OK”, “Rock”, and “Spiderman”. They also trained their models to work in a wide variety of lighting situations and with a diverse range of skin tones. Gesture recognition - Source: Google blog With MediaPipe, the researchers built their pipeline as a directed graph of modular components, called Calculators. Individual calculators like cropping, rendering , and neural network computations can be performed exclusively on the GPU. They employed TFLite GPU inference on most modern phones. The researchers are open sourcing the hand tracking and gesture recognition pipeline in the MediaPipe framework along with the source code. The researchers Valentin Bazarevsky and Fan Zhang write in a blog post, “Whereas current state-of-the-art approaches rely primarily on powerful desktop environments for inference, our method, achieves real-time performance on a mobile phone, and even scales to multiple hands. We hope that providing this hand perception functionality to the wider research and development community will result in an emergence of creative use cases, stimulating new applications and new research avenues.” People commended the fact that this algorithm can run on mobile devices and is useful for people who communicate via sign language. https://twitter.com/SOdaibo/status/1163577788764495872 https://twitter.com/anshelsag/status/1163597036442148866 https://twitter.com/JonCorey1/status/1163997895835693056 Microsoft Azure VP demonstrates Holoportation, a reconstructed transmittable 3D technology Terrifyingly realistic Deepfake video of Bill Hader transforming into Tom Cruise is going viral on YouTube. Google News Initiative partners with Google AI to help ‘deep fake’ audio detection research
Read more
  • 0
  • 0
  • 46578

article-image-windows-powershell-desired-state-configuration-video
Fatema Patrawala
16 Jul 2018
1 min read
Save for later

Scripting with Windows Powershell Desired State Configuration [Video]

Fatema Patrawala
16 Jul 2018
1 min read
https://www.youtube.com/watch?v=H3jqgto5Rk8&list=PLTgRMOcmRb3OpgM9tsUjuI3MgLCHDJ3oM&index=4 What is Desired State Configuration? Powershell Desired State Configuration (DSC) is really a powerful way of scripting. It is a declarative model of scripting, instead of you defining Powershell exactly each and every step to get from point A to point B. You only need to describe what point B is and Powershell takes care of it before anything. The biggest benefit is that we get to define our configuration, our infrastructures, our servers as a code. Desired State Configuration in Powershell can really be achieved through 3 simple steps: Create the Configuration Compile the Configuration into a MoF file Deploy the Configuration What will you need to run Powershell DSC? Thankfully we do not need a whole lot, Powershell comes with it built-in. So, for managing Windows systems with DSC you are going to need modern version of Powershell, that is: Windows 4.0, 5.0, 5.1 Powershell DSC for Linux is available Currently limited support for Powershell Core Exploring Windows PowerShell 5.0 Introducing PowerShell Remoting Managing Nano Server with Windows PowerShell and Windows PowerShell DSC    
Read more
  • 0
  • 0
  • 46474