How-To Tutorials

10 Jul 2017

6 min read

Thread of Execution

10 Jul 2017

In this article by Anton Polukhin Alekseevic, the author of the book Boost C++ Application Development Cookbook - Second Edition, we will see the multithreading concept. Multithreading means multiple threads of execution exist within a single process. Threads may share process resources and have their own resources. Those threads of execution may run independently on different CPUs, leading to faster and more responsible programs. Let's see how to create a thread of execution. (For more resources related to this topic, see here.) Creating a thread of execution On modern multicore compilers, to achieve maximal performance (or just to provide a good user experience), programs usually must use multiple threads of execution. Here is a motivating example in which we need to create and fill a big file in a thread that draws the user interface: #include <algorithm> #include <fstream> #include <iterator> bool is_first_run(); // Function that executes for a long time. void fill_file(char fill_char, std::size_t size, const char* filename); // ... // Somewhere in thread that draws a user interface: if (is_first_run()) { // This will be executing for a long time during which // users interface freezes. fill_file(0, 8 * 1024 * 1024, "save_file.txt"); } Getting ready This recipe requires knowledge of boost::bind or std::bind. How to do it... Starting a thread of execution never was so easy: #include <boost/thread.hpp> // ... // Somewhere in thread that draws a user interface: if (is_first_run()) { boost::thread(boost::bind( &fill_file, 0, 8 * 1024 * 1024, "save_file.txt" )).detach(); } How it works... The boost::thread variable accepts a functional object that can be called without parameters (we provided one using boost::bind) and creates a separate thread of execution. That functional object is copied into a constructed thread of execution and run there. We are using version 4 of the Boost.Thread in all recipes (defined BOOST_THREAD_VERSION to 4). Important differences between Boost.Thread versions are highlighted. After that, we call the detach() function, which does the following: The thread of execution is detached from the boost::thread variable but continues its execution The boost::thread variable start to hold a Not-A-Thread state Note that without a call to detach(), the destructor of boost::thread will notice that it still holds a OS thread and will call std::terminate. std::terminate terminates our program without calling destructors, freeing up resources, and without other cleanup. Default constructed threads also have a Not-A-Thread state, and they do not create a separate thread of execution. There's more... What if we want to make sure that file was created and written before doing some other job? In that case we need to join the thread in the following way: // ... // Somewhere in thread that draws a user interface: if (is_first_run()) { boost::thread t(boost::bind( &fill_file, 0, 8 * 1024 * 1024, "save_file.txt" )); // Do some work. // ... // Waiting for thread to finish. t.join(); } After the thread is joined, the boost::thread variable holds a Not-A-Thread state and its destructor does not call std::terminate. Remember that the thread must be joined or detached before its destructor is called. Otherwise, your program will terminate! With BOOST_THREAD_VERSION=2 defined, the destructor of boost::thread calls detach(), which does not lead to std::terminate. But doing so breaks compatibility with std::thread, and some day, when your project is moving to the C++ standard library threads or when BOOST_THREAD_VERSION=2 won't be supported; this will give you a lot of surprises. Version 4 of Boost.Thread is more explicit and strong, which is usually preferable in C++ language. Beware that std::terminate() is called when any exception that is not of type boost::thread_interrupted leaves boundary of the functional object that was passed to the boost::thread constructor. There is a very helpful wrapper that works as a RAII wrapper around the thread and allows you to emulate the BOOST_THREAD_VERSION=2 behavior; it is called boost::scoped_thread<T>, where T can be one of the following classes: boost::interrupt_and_join_if_joinable: To interrupt and join thread at destruction boost::join_if_joinable: To join a thread at destruction boost::detach: To detach a thread at destruction Here is a small example: #include <boost/thread/scoped_thread.hpp> void some_func(); void example_with_raii() { boost::scoped_thread<boost::join_if_joinable> t( boost::thread(&some_func) ); // 't' will be joined at scope exit. } The boost::thread class was accepted as a part of the C++11 standard and you can find it in the <thread> header in the std:: namespace. There is no big difference between the Boost's version 4 and C++11 standard library versions of the thread class. However, boost::thread is available on the C++03 compilers, so its usage is more versatile. There is a very good reason for calling std::terminate instead of joining by default! C and C++ languages are often used in life critical software. Such software is controlled by other software, called watchdog. Those watchdogs can easily detect that application has terminated but not always can detect deadlocks or detect them with bigger delays. For example for a defibrillator software it's safer to terminate, than hang on join() for a few seconds waiting for a watchdog reaction. Keep that in mind while designing such applications. See also All the recipes in this chapter are using Boost.Thread. You may continue reading to get more information about the library. The official documentation has a full list of the boost::thread methods and remarks about their availability in the C++11 standard library. The official documentation can be found at http://boost.org/libs/thread. The Interrupting a thread recipe will give you an idea of what the boost::interrupt_and_join_if_joinable class does. Summary We saw how to create a thread of execution using some easy techniques. Resources for Article: Further resources on this subject: Introducing the Boost C++ Libraries [article] Boost.Asio C++ Network Programming [article] Application Development in Visual C++ - The Tetris Application [article]

0
0
34258

article-image-when-do-we-use-r-over-python

Packt

10 Jul 2017

16 min read

When do we use R over Python?

Packt

10 Jul 2017

16 min read

0
0
5425

Packt

06 Jul 2017

10 min read

Exposure to RxJava

Packt

06 Jul 2017

10 min read

0
0
1778

Packt

06 Jul 2017

9 min read

Command-Line Tools

Packt

06 Jul 2017

9 min read

In this article by Aaron Torres, author of the book, Go Cookbook, we will cover the following recipes: Using command-line arguments Working with Unix pipes An ANSI coloring application (For more resources related to this topic, see here.) Using command-line arguments This article will expand on other uses for these arguments by constructing a command that supports nested subcommands. This will demonstrate Flagsets and also using positional arguments passed into your application. This recipe requires a main function to run. There are a number of third-party packages for dealing with complex nested arguments and flags, but we'll again investigate doing so using only the standard library. Getting ready You need to perform the following steps for the installation: Download and install Go on your operating system at https://golang.org/doc/install and configure your GOPATH. Open a terminal/console application. Navigate to your GOPATH/src and create a project directory, for example, $GOPATH/src/github.com/yourusername/customrepo. All code will be run and modified from this directory. Optionally, install the latest tested version of the code using the go get github.com/agtorre/go-cookbook/ command. How to do it... From your terminal/console application, create and navigate to the chapter2/cmdargs directory. Copy tests from https://github.com/agtorre/go-cookbook/tree/master/chapter2/cmdargs or use this as an exercise to write some of your own. Create a file called cmdargs.go with the following content: package main import ( "flag" "fmt" "os" ) const version = "1.0.0" const usage = `Usage: %s [command] Commands: Greet Version ` const greetUsage = `Usage: %s greet name [flag] Positional Arguments: name the name to greet Flags: ` // MenuConf holds all the levels // for a nested cmd line argument type MenuConf struct { Goodbye bool } // SetupMenu initializes the base flags func (m *MenuConf) SetupMenu() *flag.FlagSet { menu := flag.NewFlagSet("menu", flag.ExitOnError) menu.Usage = func() { fmt.Printf(usage, os.Args[0]) menu.PrintDefaults() } return menu } // GetSubMenu return a flag set for a submenu func (m *MenuConf) GetSubMenu() *flag.FlagSet { submenu := flag.NewFlagSet("submenu", flag.ExitOnError) submenu.BoolVar(&m.Goodbye, "goodbye", false, "Say goodbye instead of hello") submenu.Usage = func() { fmt.Printf(greetUsage, os.Args[0]) submenu.PrintDefaults() } return submenu } // Greet will be invoked by the greet command func (m *MenuConf) Greet(name string) { if m.Goodbye { fmt.Println("Goodbye " + name + "!") } else { fmt.Println("Hello " + name + "!") } } // Version prints the current version that is // stored as a const func (m *MenuConf) Version() { fmt.Println("Version: " + version) } Create a file called main.go with the following content: package main import ( "fmt" "os" "strings" ) func main() { c := MenuConf{} menu := c.SetupMenu() menu.Parse(os.Args[1:]) // we use arguments to switch between commands // flags are also an argument if len(os.Args) > 1 { // we don't care about case switch strings.ToLower(os.Args[1]) { case "version": c.Version() case "greet": f := c.GetSubMenu() if len(os.Args) < 3 { f.Usage() return } if len(os.Args) > 3 { if.Parse(os.Args[3:]) } c.Greet(os.Args[2]) default: fmt.Println("Invalid command") menu.Usage() return } } else { menu.Usage() return } } Run the go build command. Run the following commands and try a few other combinations of arguments: $./cmdargs -h Usage: ./cmdargs [command] Commands: Greet Version $./cmdargs version Version: 1.0.0 $./cmdargs greet Usage: ./cmdargs greet name [flag] Positional Arguments: name the name to greet Flags: -goodbye Say goodbye instead of hello $./cmdargs greet reader Hello reader! $./cmdargs greet reader -goodbye Goodbye reader! If you copied or wrote your own tests go up one directory and run go test, and ensure all tests pass. How it works... Flagsets can be used to set up independent lists of expected arguments, usage strings, and more. The developer is required to do validation on a number of arguments, parsing in the right subset of arguments to commands, and defining usage strings. This can be error prone and requires a lot of iteration to get it completely correct. The flag package makes parsing arguments much easier and includes convenience methods to get the number of flags, arguments, and more. This recipe demonstrates basic ways to construct a complex command-line application using arguments, including a package-level config, required positional arguments, multi-leveled command usage, and how to split these things into multiple files or packages if needed. Working with Unix pipes Unix pipes are useful when passing the output of one program to the input of another. Consider the following example: $ echo "test case" | wc -l 1 In a Go application, the left-hand side of the pipe can be read in using os.Stdin and acts like a file descriptor. To demonstrate this, this recipe will take an input on the left-hand side of a pipe and return a list of words and their number of occurrences. These words will be tokenized on white space. Getting ready Refer to the Getting Ready section of the Using command-line arguments recipe. How to do it... From your terminal/console application, create a new directory, chapter2/pipes. Navigate to that directory and copy tests from https://github.com/agtorre/go-cookbook/tree/master/chapter2/pipes or use this as an exercise to write some of your own. Create a file called pipes.go with the following content: package main import ( "bufio" "fmt" "os" ) // WordCount takes a file and returns a map // with each word as a key and it's number of // appearances as a value func WordCount(f *os.File) map[string]int { result := make(map[string]int) // make a scanner to work on the file // io.Reader interface scanner := bufio.NewScanner(f) scanner.Split(bufio.ScanWords) for scanner.Scan() { result[scanner.Text()]++ } if err := scanner.Err(); err != nil { fmt.Fprintln(os.Stderr, "reading input:", err) } return result } func main() { fmt.Printf("string: number_of_occurrencesnn") for key, value := range WordCount(os.Stdin) { fmt.Printf("%s: %dn", key, value) } } Run echo "some string" | go run pipes.go. You may also run: go build echo "some string" | ./pipes You should see the following output: $ echo "test case" | go run pipes.go string: number_of_occurrences test: 1 case: 1 $ echo "test case test" | go run pipes.go string: number_of_occurrences test: 2 case: 1 If you copied or wrote your own tests, go up one directory and run go test, and ensure that all tests pass. How it works... Working with pipes in go is pretty simple, especially if you're familiar with working with files. This recipe uses a scanner to tokenize the io.Reader interface of the os.Stdin file object. You can see how you must check for errors after completing all of the reads. An ANSI coloring application Coloring an ANSI terminal application is handled by a variety of code before and after a section of text that you want colored. This recipe will explore a basic coloring mechanism to color the text red or keep it plain. For a more complete application, take a look at https://github.com/agtorre/gocolorize, which supports many more colors and text types implements the fmt.Formatter interface for ease of printing. Getting ready Refer to the Getting Ready section of the Using command line arguments recipe. How to do it... From your terminal/console application, create and navigate to the chapter2/ansicolor directory. Copy tests from https://github.com/agtorre/go-cookbook/tree/master/chapter2/ansicolor or use this as an exercise to write some of your own. Create a file called color.go with the following content: package ansicolor import "fmt" //Color of text type Color int const ( // ColorNone is default ColorNone = iota // Red colored text Red // Green colored text Green // Yellow colored text Yellow // Blue colored text Blue // Magenta colored text Magenta // Cyan colored text Cyan // White colored text White // Black colored text Black Color = -1 ) // ColorText holds a string and its color type ColorText struct { TextColor Color Text string } func (r *ColorText) String() string { if r.TextColor == ColorNone { return r.Text } value := 30 if r.TextColor != Black { value += int(r.TextColor) } return fmt.Sprintf("33[0;%dm%s33[0m", value, r.Text) } Create a new directory named example. Navigate to example and then create a file named main.go with the following content. Ensure that you modify the ansicolor import to use the path you set up in step 1: package main import ( "fmt" "github.com/agtorre/go-cookbook/chapter2/ansicolor" ) func main() { r := ansicolor.ColorText{ansicolor.Red, "I'm red!"} fmt.Println(r.String()) r.TextColor = ansicolor.Green r.Text = "Now I'm green!" fmt.Println(r.String()) r.TextColor = ansicolor.ColorNone r.Text = "Back to normal..." fmt.Println(r.String()) } Run go run main.go. Alternatively, you may also run the following: go build ./example You should see the following with the text colored if your terminal supports the ANSI coloring format: $ go run main.go I'm red! Now I'm green! Back to normal... If you copied or wrote your own tests, go up one directory and run go test, and ensure that all the tests pass. How it works... This application makes use of a struct keyword to maintain state of the colored text. In this case, it stores the color of the text and the value of the text. The final string is rendered when you call the String() method, which will either return colored text or plain text depending on the values stored in the struct. By default, the text will be plain. Summary In this article, we demonstrated basic ways to construct a complex command-line application using arguments, including a package-level config, required positional arguments, multi-leveled command usage, and how to split these things into multiple files or packages if needed. We saw how to work with Unix pipes and explored a basic coloring mechanism to color text red or keep it plain. Resources for Article: Further resources on this subject: Building a Command-line Tool [article] A Command-line Companion Called Artisan [article] Scaffolding with the command-line tool [article]

0
0
18487

Packt

06 Jul 2017

12 min read

IIS 10 Fundamentals

Packt

06 Jul 2017

12 min read

In this article by Ashraf Khan, the author of the book Microsoft IIS 10 Cookbook, helps us to understand the following topics: Understanding IIS 10 Basic requirements of IIS 10 Understanding application pools on IIS 10 Installation of lower framework version Configuration of application pool on IIS 10 (For more resources related to this topic, see here.) Understanding IIS 10 In this recipe, we will understand how to work with IIS 10's new features. We will have an overview of the following new features added to IIS 10: HTTP/2 HTTP/2 requests are now faster than ever. This feature is active by default with IIS 10 on Windows Server 2016 and Windows 10. IIS 10 on Nano Server IIS 10 is easy and quick to install on Nano Server. You can manage IIS 10 remotely with PowerShell or the IIS Manager console. Nano Server is much faster and consumes less memory and disk space that the full-fledged Windows Server. Rebooting is also faster so that you can manage time effectively. Wildcard host headers IIS 10 support the subdomain feature for your parent domain name. This will really help you manage more subdomains with the same primary domain name. PowerShell 5 cmdlets IIS 10 adds a new, simplified PowerShell module for quick and easy management. You can use PowerShell to access server-management features remotely. It also supports existing WebAdministration cmdlets. FTP FTP is a simple protocol for transferring files. This system can transfer files inside your company LAN and WAN using the default port, 21. IIS 10 includes an FTP server that is easy to configure and manage. FTPS FTPS is the same as FTP, with the only difference that it is secure. FTPS transfers data with SSL. We are going to use HTTPS port 443. For this, we need to create and install an SSL certificate that encrypts and decrypts data securely. SSL ensures that all data passed between web server and browser remains private and consistent during upload and download over private or public networks. Multi-web hosting IIS 10 allows you to create multiple websites and multiple applications on the same server. You can easily manage and create a new virtual directory located in the default location or a custom location. Virtual directories IIS 10 makes it easy to manage and create the virtual directories you require. Understanding application pools on IIS 10 In this recipe, we are going to understand application pools. We can simply say that application pool is the heart of IIS 10. Application pools are logical groupings of web applications that will execute in a common process, thereby allowing greater granularity of which programs are clustered together in a single process. For example, if you required every Web Application to execute in a separate process, you simply go and create an Application Pool for each application of different frameworks versions. Let's say that we have more than one version of website, one which supports framework 2.0 and another one supporting framework 4.0 or different application like PHP or WordPress. All these website process are managed through application pool. Getting ready To step through this recipe , you will need a running IIS 10. You should also be having Administrative privilege. No other prerequisites are required. How to do it... Open the Server Manager on Windows Server 2016. Click on Tools menu and open the IIS Manager. Expand the IIS server (WIN2016IIS) this is the localhost server name WIN2016IIS. We get the listed application pools and sites. In Application Pools, you will get IIS 10 DefaultsAppPool as shown in above figure, also you get Actions panel in right side of the screen where you may add application pools. Click on DefaultAppPool, then you will get the Actions panel of DefaultAppPool. Here you will get an option for Application Pool Tasks highlighted in right side, where you may Start, Stop, and Recycle the services of IIS 10. In Edit Application Pool section, you can change the settings of application pool as Basic Settings..., Advanced Settings..., Rename the application pool and you may also do the Recycling... How it works... Let's take a look at what we explored in IIS Manager and Application Pools. We understood the basics of application pools and the property where we can get the changes done as per our requirement. IIS 10 default application pool framework is v4.0 which is supported upto v4.6, but we will get some more option for installing different versions of application pool. We can easily customize the application pool, which helps us to fulfill our typical web application requirement. We have several options for application pool in action pane. We can add new application pool, we can start, stop and recycle the application pool task. We can do the editing and automated recycling. Now we are going to learn in the next recipe more about application pools for Installation of lower framework version. Installation of lower framework version In this recipe, we are going to install framework 3.5 on Windows Server 2016. Default IIS 10 has the framework 4.0. We will install the lower version of framework which supports the web application of Version 2.0 to Version 3.5 .NET framework. Let's start now if you have your own web application which you had created a few years back and it was developed in v2.0 .NET framework. You want to run this application on IIS 10. We are going to cover this topic in this recipe. Getting ready To step through this recipe you need to install framework version3.5, v3.5 framework is based on v2.0 framework. You will need a Windows Server 2016. You should be having a Window Server 2016 Operating System media for framework 3.5 or Internet connected on Window server 2016. You should have Administrative privilege. No other prerequisites are required. How to do it.... Open the Server Manager on Windows Server 2016, click on highlighted Add roles and features option. Click on Next until you get the Select features wizard. You can see the next figure. Click on Features panel and click on the check box .NET Framework 3.5 Features. It will also install the 2.0 supported framework. Move to next wizard as shown in figure. There is a warning coming before the installation: Do you need to specify an alternate source path? One or more installation selection are missing source files on the destination. We have to provide the Installation media sourcesSxS folder path. Click on Specify an alternate source path. See next figure for more details. Here we have Windows Server 2016 media in D:drive. This is the media path in our case which I have downloaded but in your case it can be different path to locate where you already had downloaded. There is a folder which is called sources and sub folder(SxS). Inside framework 3, installation file is available. You may see the next figure. Now you know where the source folder is. Come to confirm Screen and click on Install. The next figure shows Installation progress on WIN2016IIS. Click on close when installation is completed, you have framework 3.5 available on your server. Now you have to check whether framework 3.5 has been installed or not. It should be available in features wizard. Open the Server Manager, click on Add roles and features. Click next and next until you get the Select Features wizard. You will see .NET framework 3.5 check box checked with gray color which is disabled. You can not check and uncheck the checkbox. You can see the next figure. As shown in the preceding figure, it has been confirmed that .NET framework 3.5 has been installed. This can be installed through PowerShell. We can install directly from Windows Update, you need Internet connectivity and running Windows Update service on Window Server. How it works... In this recipe, IIS administrator is installed in the framework v3.5. The version 3.5 framework on window server 2016 helps us to run built in application .NET framework v2.0 or v3.5. The framework v3.5 processes the application which is built in framework v3.5 or v2.0. We also find out where is the sourcesSxS folder, after installation we verified that this .NET framework v3.5 is available. We are going to create application pool which will support the .NET framework v3.5. Configuration of application pool on IIS 10 In this recipe, we will have an overview of application pool property. We will check the default configuration of Basic Settings, Recycling and Advanced Settings. This is very helpful for developer or system administrator as we can do the configuration of different property of different application pool based upon application requirement. Getting ready For this recipe, we need IIS 1o and .NET framework of any version which is installed on IIS 10. You must have Administrative privilege. No other prerequisites are required. How to do it... Open the Server Manager on Windows Server 2016. Click on Tools menu and open the IIS Manager. Expand the IIS server (WIN2016IIS). We get the listed Application Pools. You may see in the next figure. Now we have already created application pool which is displayed in Application Pools. We created 2and3.5AppPool, Asp.net and DefaultAppPool (Default one). In the Actions panel, we can add many application pools and we can set any one of the created application pool as default application pool. The default application pool helps us when we are creating a website. The default application pool will be selected as an application pool. Select the 2and3.5AppPool from application pools. You will see the Actions pane having a list of available properties in which you can do some changes if needed. The version of 2and3.5AppPool is v2.0, you can see in the next figure. See the Actions panel, Application Pool Tasks and Edit Application Pool which we selected. From the Application Tool Tasks we can Start, Stop and Recycle... the application pool. Now let's come to the basic property of application pool. Click on Basic Settings... from Edit Application Pool, see the next figure which will appear after clicking. Basic Settings... is nothing but a quick settings to change limited number of things. We can change the .NET framework version to framework v4.0 or framework v3.5(version 2.0 is updated version 3.5). We can change the Managed pipeline mode to Integrated or Classic, also we can check or uncheck the start option. Next is the Advanced Settings... which has more options for customization of relevant Application Pool. Click on Advanced Settings..., the next figure will open. We have more settings option available in Advanced Settings... window. You may change the .NET framework version, you can select 32 bit application support true or false. Queue Length is 1000 by default. You may reduce or increase as you need. Start Mode should be OnDemand or Always Running. We can also customize utilization of CPU which helps you to manage the load of each application and their performance. Process Model will help you to define task for application pool availability and accessibility. We can see more about application pool in next figure. Rapid-Fail Protection is generally used for fail over. We can setup the fail over server and configuration. Recycling is to refresh the application pool overlapped. We can set a default recycling value. We can do more specific settings through Recycling settings by clicking on Recycling.... You may see your recycling conditions window in the next figure. Recycling is based on conditions like virtual memory usage, private memory usage, specific time, regular time intervals and fixed number of request, also it will generate you a log file which will help you understand which one was executed at what time. Here you will set the fixed intervals based on time and based on number of request, or specific time and based on Memory utilization, virtual and private memory. Click Next. In the Recycling Events to Log window, we generate log on the recycling events. How it works.... In this recipe we have learned three properties of IIS Application - Basic property, Advanced property and Recycling. We can use these properties for web application which we will host in IIS server to process through the application pool. When we are hosting a web application, there is always some requirement which we need to configure in application pool settings. For example, our management decides that we need to limit the queue of 2and3.5apppool application. We can just go to advance settings and change it. In the next section, we are going to host v4.0 .NET framework website and we will make use of application pool v4.0. Summary In this article, we understood application pools in IIS 10 and how to install and configure them. We also understood how to install a lower framework version. Resources for Article: Further resources on this subject: Exploring Microsoft Dynamics NAV – An Introduction [article] The Microsoft Azure Stack Architecture [article] Setting up Microsoft Bot Framework Dev Environment [article]

0
0
11165

Packt

06 Jul 2017

11 min read

Spark Streaming

Packt

06 Jul 2017

11 min read

In this article by Romeo Kienzler, the author of the book Mastering Apache Spark 2.x - Second Edition, we will see Apache Streaming module is a stream processing-based module within Apache Spark. It uses the Spark cluster to offer the ability to scale to a high degree. Being based on Spark, it is also highly fault tolerant, having the ability to rerun failed tasks by check-pointing the data stream that is being processed. The following areas will be covered in this article after an initial section, which will provide a practical overview of how Apache Spark processes stream-based data: Error recovery and check-pointing TCP-based stream processing File streams Kafka stream source For each topic, we will provide a worked example in Scala, and will show how the stream-based architecture can be set up and tested. (For more resources related to this topic, see here.) Overview The following diagram shows potential data sources for Apache Streaming, such as Kafka, Flume, and HDFS: These feed into the Spark Streaming module, and are processed as Discrete Streams. The diagram also shows that other Spark module functionality, such as machine learning, can be used to process the stream-based data. The fully processed data can then be an output for HDFS, databases, or dashboards. This diagram is based on the one at the Spark streaming website, but we wanted to extend it for expressing the Spark module functionality: When discussing Spark Discrete Streams, the previous figure, again taken from the Spark website at http://spark.apache.org/, is the diagram we like to use. The green boxes in the previous figure show the continuous data stream sent to Spark, being broken down into a Discrete Streams (DStream). The size of each element in the stream is then based on a batch time, which might be two seconds. It is also possible to create a window, expressed as the previous red box, over the DStream. For instance, when carrying out trend analysis in real time, it might be necessary to determine the top ten Twitter-based hashtags over a ten minute window. So, given that Spark can be used for stream processing, how is a stream created? The following Scala-based code shows how a Twitter stream can be created. This example is simplified because Twitter authorization has not been included, but you get the idea. The Spark Stream Context (SSC) is created using the Spark Context sc. A batch time is specified when it is created; in this case, 5 seconds. A Twitter-based DStream, called stream, is then created from the Streamingcontext using a window of 60 seconds: val ssc = new StreamingContext(sc, Seconds(5) ) val stream = TwitterUtils.createStream(ssc,None).window( Seconds(60) ) The stream processing can be started with the stream context start method (shown next), and the awaitTermination method indicates that it should process until stopped. So, if this code is embedded in a library-based application, it will run until the session is terminated, perhaps with a Crtl + C: ssc.start() ssc.awaitTermination() This explains what Spark Streaming is, and what it does, but it does not explain error handling, or what to do if your stream-based application fails. The next section will examine Spark Streaming error management and recovery. Errors and recovery Generally, the question that needs to be asked for your application is; is it critical that you receive and process all the data? If not, then on failure you might just be able to restart the application and discard the missing or lost data. If this is not the case, then you will need to use check pointing, which will be described in the next section. It is also worth noting that your application's error management should be robust and self-sufficient. What we mean by this is that; if an exception is non-critical, then manage the exception, perhaps log it, and continue processing. For instance, when a task reaches the maximum number of failures (specified by spark.task.maxFailures), it will terminate processing. Checkpointing It is possible to set up an HDFS-based checkpoint directory to store Apache Spark-based streaming information. In this Scala example, data will be stored in HDFS, under /data/spark/checkpoint. The following HDFS file system ls command shows that before starting, the directory does not exist: [hadoop@hc2nn stream]$ hdfs dfs -ls /data/spark/checkpoint ls: `/data/spark/checkpoint': No such file or directory The Twitter-based Scala code sample given next, starts by defining a package name for the application, and by importing Spark Streaming Context, and Twitter-based functionality. It then defines an application object named stream1: package nz.co.semtechsolutions import org.apache.spark._ import org.apache.spark.SparkContext._ import org.apache.spark.streaming._ import org.apache.spark.streaming.twitter._ import org.apache.spark.streaming.StreamingContext._ object stream1 { Next, a method is defined called createContext, which will be used to create both the spark, and streaming contexts. It will also checkpoint the stream to the HDFS-based directory using the streaming context checkpoint method, which takes a directory path as a parameter. The directory path being the value (cpDir) that was passed into the createContext method: def createContext( cpDir : String ) : StreamingContext = { val appName = "Stream example 1" val conf = new SparkConf() conf.setAppName(appName) val sc = new SparkContext(conf) val ssc = new StreamingContext(sc, Seconds(5) ) ssc.checkpoint( cpDir ) ssc } Now, the main method is defined, as is the HDFS directory, as well as Twitter access authority and parameters. The Spark Streaming context ssc is either retrieved or created using the HDFS checkpoint directory via the StreamingContext method—getOrCreate. If the directory doesn't exist, then the previous method called createContext is called, which will create the context and checkpoint. Obviously, we have truncated our own Twitter auth.keys in this example for security reasons: def main(args: Array[String]) { val hdfsDir = "/data/spark/checkpoint" val consumerKey = "QQpxx" val consumerSecret = "0HFzxx" val accessToken = "323xx" val accessTokenSecret = "IlQxx" System.setProperty("twitter4j.oauth.consumerKey", consumerKey) System.setProperty("twitter4j.oauth.consumerSecret", consumerSecret) System.setProperty("twitter4j.oauth.accessToken", accessToken) System.setProperty("twitter4j.oauth.accessTokenSecret", accessTokenSecret) val ssc = StreamingContext.getOrCreate(hdfsDir, () => { createContext( hdfsDir ) }) val stream = TwitterUtils.createStream(ssc,None).window( Seconds(60) ) // do some processing ssc.start() ssc.awaitTermination() } // end main Having run this code, which has no actual processing, the HDFS checkpoint directory can be checked again. This time it is apparent that the checkpoint directory has been created, and the data has been stored: [hadoop@hc2nn stream]$ hdfs dfs -ls /data/spark/checkpoint Found 1 items drwxr-xr-x - hadoop supergroup 0 2015-07-02 13:41 /data/spark/checkpoint/0fc3d94e-6f53-40fb-910d-1eef044b12e9 This example, taken from the Apache Spark website, shows how checkpoint storage can be set up and used. But how often is checkpointing carried out? The metadata is stored during each stream batch. The actual data is stored with a period, which is the maximum of the batch interval, or ten seconds. This might not be ideal for you, so you can reset the value using the method: DStream.checkpoint( newRequiredInterval ) Where newRequiredInterval is the new checkpoint interval value that you require, generally you should aim for a value which is five to ten times your batch interval. Checkpointing saves both the stream batch and metadata (data about the data). If the application fails, then when it restarts, the checkpointed data is used when processing is started. The batch data that was being processed at the time of failure is reprocessed, along with the batched data since the failure. Remember to monitor the HDFS disk space being used for check pointing. In the next section, we will begin to examine the streaming sources, and will provide some examples of each type. Streaming sources We will not be able to cover all the stream types with practical examples in this section, but where this article is too small to include code, we will at least provide a description. In this article, we will cover the TCP and file streams, and the Flume, Kafka, and Twitter streams. We will start with a practical TCP-based example. This article examines stream processing architecture. For instance, what happens in cases where the stream data delivery rate exceeds the potential data processing rate? Systems like Kafka provide the possibility of solving this issue by providing the ability to use multiple data topics and consumers. TCP stream There is a possibility of using the Spark Streaming Context method called socketTextStream to stream data via TCP/IP, by specifying a hostname and a port number. The Scala-based code example in this section will receive data on port 10777 that was supplied using the Netcat Linux command. The code sample starts by defining the package name, and importing Spark, the context, and the streaming classes. The object class named stream2 is defined, as it is the main method with arguments: package nz.co.semtechsolutions import org.apache.spark._ import org.apache.spark.SparkContext._ import org.apache.spark.streaming._ import org.apache.spark.streaming.StreamingContext._ object stream2 { def main(args: Array[String]) { The number of arguments passed to the class is checked to ensure that it is the hostname and the port number. A Spark configuration object is created with an application name defined. The Spark and streaming contexts are then created. Then, a streaming batch time of 10 seconds is set: if ( args.length < 2 ) { System.err.println("Usage: stream2 <host> <port>") System.exit(1) } val hostname = args(0).trim val portnum = args(1).toInt val appName = "Stream example 2" val conf = new SparkConf() conf.setAppName(appName) val sc = new SparkContext(conf) val ssc = new StreamingContext(sc, Seconds(10) ) A DStream called rawDstream is created by calling the socketTextStream method of the streaming context using the host and port name parameters. val rawDstream = ssc.socketTextStream( hostname, portnum ) A top-ten word count is created from the raw stream data by splitting words by spacing. Then a (key,value) pair is created as (word,1), which is reduced by the key value, this being the word. So now, there is a list of words and their associated counts. Now, the key and value are swapped, so the list becomes (count and word). Then, a sort is done on the key, which is now the count. Finally, the top 10 items in the RDD, within the DStream, are taken and printed out: val wordCount = rawDstream .flatMap(line => line.split(" ")) .map(word => (word,1)) .reduceByKey(_+_) .map(item => item.swap) .transform(rdd => rdd.sortByKey(false)) .foreachRDD( rdd => { rdd.take(10).foreach(x=>println("List : " + x)) }) The code closes with the Spark Streaming start, and awaitTermination methods being called to start the stream processing and await process termination: ssc.start() ssc.awaitTermination() } // end main } // end stream2 The data for this application is provided, as we stated previously, by the Linux Netcat (nc) command. The Linux Cat command dumps the contents of a log file, which is piped to nc. The lk options force Netcat to listen for connections, and keep on listening if the connection is lost. This example shows that the port being used is 10777: [root@hc2nn log]# pwd /var/log [root@hc2nn log]# cat ./anaconda.storage.log | nc -lk 10777 The output from this TCP-based stream processing is shown here. The actual output is not as important as the method demonstrated. However, the data shows, as expected, a list of 10 log file words in descending count order. Note that the top word is empty because the stream was not filtered for empty words: List : (17104,) List : (2333,=) List : (1656,:) List : (1603,;) List : (1557,DEBUG) List : (564,True) List : (495,False) List : (411,None) List : (356,at) List : (335,object) This is interesting if you want to stream data using Apache Spark Streaming, based upon TCP/IP from a host and port. But what about more exotic methods? What if you wish to stream data from a messaging system, or via memory-based channels? What if you want to use some of the big data tools available today like Flume and Kafka? The next sections will examine these options, but first I will demonstrate how streams can be based upon files. Summary We could have provided streaming examples for systems like Kinesis, as well as queuing systems, but there was not room in this article. This article has provided practical examples of data recovery via checkpointing in Spark Streaming. It has also touched on the performance limitations of checkpointing and shown that that the checkpointing interval should be set at five to ten times the Spark stream batch interval. Resources for Article: Further resources on this subject: Understanding Spark RDD [article] Spark for Beginners [article] Setting up Spark [article]

0
0
19571

article-image-chart-model-and-draggable-and-droppable-directives

Packt

06 Jul 2017

9 min read

Chart Model and Draggable and Droppable Directives

Packt

06 Jul 2017

9 min read

In this article by Sudheer Jonna and Oleg Varaksin, the author of the book Learning Angular UI Development with PrimeNG, we will see how to work with the chart model and learn about draggable and droppable directives. (For more resources related to this topic, see here.) Working with the chart model The chart component provides a visual representation of data using chart on a web page. PrimeNG chart components are based on charts.js 2.x library (as a dependency), which is a HTML5 open source library. The chart model is based on UIChart class name and it can be represented with element name as p-chart. The chart components will work efficiently by attaching the chart model file (chart.js) to the project root folder entry point. For example, in this case it would be index.html file. It can be configured as either CDN resource, local resource, or CLI configuration. CDN resource configuration: <script src="https://cdnjs.cloudflare.com/ajax/libs/Chart.js/2.5.0/Chart.bu ndle.min.js"></script> Angular CLI configuration: "scripts": [ "../node_modules/chart.js/dist/Chart.js", //..others ] More about the chart configuration and options will be available in the official documentation of the chartJS library (http://www.chartjs.org/). Chart types The chart type is defined through the type property. It supports six types of charts with an options such as pie, bar, line, doughnut, polarArea, and radar. Each type has it's own format of data and it can be supplied through the data property. For example, in doughnut chart, the type should refer to doughnut and the data property should bind to the data options as shown here: <p-chart type="doughnut" [data]="doughnutdata"></p-chart> The component class has to define data the options with labels and datasets as follows: this.doughnutdata = { labels: ['PrimeNG', 'PrimeUI', 'PrimeReact'], datasets: [ { data: [3000, 1000, 2000], backgroundColor: [ "#6544a9", "#51cc00", "#5d4361" ], hoverBackgroundColor: [ "#6544a9", "#51cc00", "#5d4361" ] } ] }; Along with labels and data options, other properties related to skinning can be applied too. The legends are closable by default (that is, if you want to visualize only particular data variants then it is possible by collapsing legends which are not required). The collapsed legend is represented with a strike line. The respective data component will be disappeared after click operation on legend. Customization Each series is customized on a dataset basis but you can customize the general or common options via the options attribute. For example, the line chart which customize the default options would be as follows: <p-chart type="line" [data]="linedata" [options]="options"></p-chart> The component class needs to define chart options with customized title and legend properties as follows: this.options = { title: { display: true, text: 'PrimeNG vs PrimeUI', fontSize: 16 }, legend: { position: 'bottom' } }; As per the preceding example, the title option is customized with a dynamic title, font size, and conditional display of the title. Where as legend attribute is used to place the legend in top, left, bottom, and right positions. The default legend position is top. In this example, the legend position is bottom. The line chart with preceding customized options would results as a snapshot shown here: The Chart API also supports the couple of utility methods as shown here: refresh Redraws the graph with new data reinit Destroys the existing graph and then creates it again generateLegend Returns an HTML string of a legend for that chart Events The chart component provides a click event on data sets to process the select data using onDataSelect event callback. Let us take a line chart example with onDataSelect event callback by passing an event object as follows: <p-chart type="line" [data]="linedata" (onDataSelect)="selectData($event)"></p-chart> In the component class, an event callback is used to display selected data information in a message format as shown: selectData(event: any) { this.msgs = []; this.msgs.push({ severity: 'info', summary: 'Data Selected', 'detail': this.linedata.datasets[event.element._datasetIndex] .data[event.element._index] }); } In the preceding event callback (onDataSelect), we used an index of the dataset to display information. There are also many other options from an event object: event.element = Selected element event.dataset = Selected dataset event.element._datasetIndex = Index of the dataset in data event.element._index = Index of the data in dataset Learning Draggable and Droppable directives Drag and drop is an action, which means grabbing an object and dragging it to a different location. The components capable of being dragged and dropped enrich the web and make a solid base for modern UI patterns. The drag and drop utilities in PrimeNG allow us to create draggable and droppable user interfaces efficiently. They make it abstract for the developers to deal with the implementation details at the browser level. In this section, we will learn about pDraggable and pDroppable directives. We will introduce a DataGrid component containing some imaginary documents and make these documents draggable in order to drop them onto a recycle bin. The recycle bin is implemented as DataTable component which shows properties of dropped documents. For the purpose of better understanding the developed code, a picture comes first: This picture shows what happens after dragging and dropping three documents. The complete demo application with instructions is available on GitHub at https://github.com/ova2/angular-development-with-primeng/tree/master/chapter9/dragdrop. Draggable pDraggable is attached to an element to add a drag behavior. The value of the pDraggable attribute is required--it defines the scope to match draggables with droppables. By default, the whole element is draggable. We can restrict the draggable area by applying the dragHandle attribute. The value of dragHandle can be any CSS selector. In the DataGrid with available documents, we only made the panel's header draggable: <p-dataGrid [value]="availableDocs"> <p-header> Available Documents </p-header> <ng-template let-doc pTemplate="item"> <div class="ui-g-12 ui-md-4" pDraggable="docs" dragHandle=".uipanel- titlebar" dragEffect="move" (onDragStart)="dragStart($event, doc)" (onDragEnd)="dragEnd($event)"> <p-panel [header]="doc.title" [style]="{'text-align':'center'}"> <img src="/assets/data/images/docs/{{doc.extension}}.png"> </p-panel> </div> </ng-template> </p-dataGrid> The draggable element can fire three events when dragging process begins, proceeds, and ends. These are onDragStart, onDrag, and onDragEnd respectively. In the component class, we buffer the dragged document at the beginning and reset it at the end of the dragging process. This task is done in two callbacks: dragStart and dragEnd. class DragDropComponent { availableDocs: Document[]; deletedDocs: Document[]; draggedDoc: Document; constructor(private docService: DocumentService) { } ngOnInit() { this.deletedDocs = []; this.docService.getDocuments().subscribe((docs: Document[]) => this.availableDocs = docs); } dragStart(event: any, doc: Document) { this.draggedDoc = doc; } dragEnd(event: any) { this.draggedDoc = null; } ... } In the shown code, we used the Document interface with the following properties: interface Document { id: string; title: string; size: number; creator: string; creationDate: Date; extension: string; } In the demo application, we set the cursor to move when the mouse is moved over any panel's header. This trick provides a better visual feedback for draggable area: body .ui-panel .ui-panel-titlebar { cursor: move; } We can also set the dragEffect attribute to specifies the effect that is allowed for a drag operation. Possible values are none, copy, move, link, copyMove, copyLink, linkMove, and all. Refer the official documentation to read more details at https://developer.mozilla.org/en-US/docs/Web/API/DataTransfer/effectAllowed. Droppable pDroppable is attached to an element to add a drop behavior. The value of the pDroppable attribute should have the same scope as pDraggable. Droppable scope can also be an array to accept multiple droppables. The droppable element can fire four events. Event name Description onDragEnter Invoked when a draggable element enters the drop area onDragOver Invoked when a draggable element is being dragged over the drop area onDrop Invoked when a draggable is dropped onto the drop area onDragLeave Invoked when a draggable element leaves the drop area In the demo application, the whole code of the droppable area looks as follows: <div pDroppable="docs" (onDrop)="drop($event)" [ngClass]="{'dragged-doc': draggedDoc}"> <p-dataTable [value]="deletedDocs"> <p-header>Recycle Bin</p-header> <p-column field="title" header="Title"></p-column> <p-column field="size" header="Size (bytes)"></p-column> <p-column field="creator" header="Creator"></p-column> <p-column field="creationDate" header="Creation Date"> <ng-template let-col let-doc="rowData" pTemplate="body"> {{doc[col.field].toLocaleDateString()}} </ng-template> </p-column> </p-dataTable> </div> Whenever a document is being dragged and dropped into the recycle bin, the dropped document is removed from the list of all available documents and added to the list of deleted documents. This happens in the onDrop callback: drop(event: any) { if (this.draggedDoc) { // add draggable element to the deleted documents list this.deletedDocs = [...this.deletedDocs, this.draggedDoc]; // remove draggable element from the available documents list this.availableDocs = this.availableDocs.filter((e: Document) => e.id !== this.draggedDoc.id); this.draggedDoc = null; } } Both available and deleted documents are updated by creating new arrays instead of manipulating existing arrays. This is necessary in data iteration components to force Angular run change detection. Manipulating existing arrays would not run change detection and the UI would not be updated. The Recycle Bin area gets a red border while dragging any panel with document. We have achieved this highlighting by setting ngClass as follows: [ngClass]="{'dragged-doc': draggedDoc}". The style class dragged-doc is enabled when the draggedDoc object is set. The style class is defined as follows: .dragged-doc { border: solid 2px red; } Summary Initially we started with chart components. At first we started with chart Model API and then will learn how to create charts programmatically using various chart types such as pie, bar, line, doughnut, polar and radar charts. We also learned features of Draggable and Droppable. Resources for Article: Further resources on this subject: Building Components Using Angular [article] Get Familiar with Angular [article] Writing a Blog Application with Node.js and AngularJS [article]

0
0
17930

Packt

06 Jul 2017

11 min read

Continous Delivery

Packt

06 Jul 2017

11 min read

0
0
1854

Packt

06 Jul 2017

9 min read

Ruby Strings

Packt

06 Jul 2017

9 min read

In this article by Jordan Hudgens, the author of the book Comprehensive Ruby Programming, you'll learn about the Ruby String data type and walk through how to integrate string data into a Ruby program. Working with words, sentences, and paragraphs are common requirements in many applications. Additionally you learn how to: Employ string manipulation techniques using core Ruby methods Demonstrate how to work with the string data type in Ruby (For more resources related to this topic, see here.) Using strings in Ruby A string is a data type in Ruby and contains set of characters, typically normal English text (or whatever natural language you're building your program for), that you would write. A key point for the syntax of strings is that they have to be enclosed in single or double quotes if you want to use them in a program. The program will throw an error if they are not wrapped inside quotation marks. Let's walk through three scenarios. Missing quotation marks In this code I tried to simply declare a string without wrapping it in quotation marks. As you can see, this results in an error. This error is because Ruby thinks that the values are classes and methods. Printing strings In this code snippet we're printing out a string that we have properly wrapped in quotation marks. Please note that both single and double quotation marks work properly. It's also important that you do not mix the quotation mark types. For example, if you attempted to run the code: puts "Name an animal' You would get an error, because you need to ensure that every quotation mark is matched with a closing (and matching) quotation mark. If you start a string with double quotation marks, the Ruby parser requires that you end the string with the matching double quotation marks. Storing strings in variables Lastly in this code snippet we're storing a string inside of a variable and then printing the value out to the console. We'll talk more about strings and string interpolation in subsequent sections. String interpolation guide for Ruby In this section, we are going to talk about string interpolation in Ruby. What is string interpolation? So what exactly is string interpolation? Good question. String interpolation is the process of being able to seamlessly integrate dynamic values into a string. Let's assume we want to slip dynamic words into a string. We can get input from the console and store that input into variables. From there we can call the variables inside of a pre-existing string. For example, let's give a sentence the ability to change based on a user's input. puts "Name an animal" animal = gets.chomp puts "Name a noun" noun= gets.chomp p "The quick brown #{animal} jumped over the lazy #{noun} " Note the way I insert variables inside the string? They are enclosed in curly brackets and are preceded by a # sign. If I run this code, this is what my output will look: So, this is how you insert values dynamically in your sentences. If you see sites like Twitter, it sometimes displays personalized messages such as: Good morning Jordan or Good evening Tiffany. This type of behavior is made possible by inserting a dynamic value in a fixed part of a string and leverages string interpolation. Now, let's use single quotes instead of double quotes, to see what happens. As you'll see, the string was printed as it is without inserting the values for animal and noun. This is exactly what happens when you try using single quotes—it prints the entire string as it is without any interpolation. Therefore it's important to remember the difference. Another interesting aspect is that anything inside the curly brackets can be a Ruby script. So, technically you can type your entire algorithm inside these curly brackets, and Ruby will run it perfectly for you. However, it is not recommended for practical programming purposes. For example, I can insert a math equation, and as you'll see it prints the value out. String manipulation guide In this section we are going to learn about string manipulation along with a number of examples of how to integrate string manipulation methods in a Ruby program. What is string manipulation? So what exactly is string manipulation? It's the process of altering the format or value of a string, usually by leveraging string methods. String manipulation code examples Let's start with an example. Let's say I want my application to always display the word Astros in capital letters. To do that, I simply write: "Astros".upcase Now if I always a string to be in lower case letters I can use the downcase method, like so: "Astros".downcase Those are both methods I use quite often. However there are other string methods available that we also have at our disposal. For the rare times when you want to literally swap the case of the letters you can leverage the swapcase method: "Astros".swapcase And lastly if you want to reverse the order of the letters in the string we can call the reverse method: "Astros".reverse These methods are built into the String data class and we can call them on any string values in Ruby. Method chaining Another neat thing we can do is join different methods together to get custom output. For example, I can run: "Astros".reverse.upcase The preceding code displays the value SORTSA. This practice of combining different methods with a dot is called method chaining. Split, strip, and join guides for strings In this section, we are going to walk through how to use the split and strip methods in Ruby. These methods will help us clean up strings and convert a string to an array so we can access each word as its own value. Using the strip method Let's start off by analyzing the strip method. Imagine that the input you get from the user or from the database is poorly formatted and contains white space before and after the value. To clean the data up we can use the strip method. For example: str = " The quick brown fox jumped over the quick dog " p str.strip When you run this code, the output is just the sentence without the white space before and after the words. Using the split method Now let's walk through the split method. The split method is a powerful tool that allows you to split a sentence into an array of words or characters. For example, when you type the following code: str = "The quick brown fox jumped over the quick dog" p str.split You'll see that it converts the sentence into an array of words. This method can be particularly useful for long paragraphs, especially when you want to know the number of words in the paragraph. Since the split method converts the string into an array, you can use all the array methods like size to see how many words were in the string. We can leverage method chaining to find out how many words are in the string, like so: str = "The quick brown fox jumped over the quick dog" p str.split.size This should return a value of 9, which is the number of words in the sentence. To know the number of letters, we can pass an optional argument to the split method and use the format: str = "The quick brown fox jumped over the quick dog" p str.split(//).size And if you want to see all of the individual letters, we can remove the size method call, like this: p str.split(//) And your output should look like this: Notice, that it also included spaces as individual characters which may or may not be what you want a program to return. This method can be quite handy while developing real-world applications. A good practical example of this method is Twitter. Since this social media site restricts users to 140 characters, this method is sure to be a part of the validation code that counts the number of characters in a Tweet. Using the join method We've walked through the split method, which allows you to convert a string into a collection of characters. Thankfully, Ruby also has a method that does the opposite, which is to allow you to convert an array of characters into a single string, and that method is called join. Let's imagine a situation where we're asked to reverse the words in a string. This is a common Ruby coding interview question, so it's an important concept to understand since it tests your knowledge of how string work in Ruby. Let's imagine that we have a string, such as: str = "backwards am I" And we're asked to reverse the words in the string. The pseudocode for the algorithm would be: Split the string into words Reverse the order of the words Merge all of the split words back into a single string We can actually accomplish each of these requirements in a single line of Ruby code. The following code snippet will perform the task: str.split.reverse.join(' ') This code will convert the single string into an array of strings, for the example it will equal ["backwards", "am", "I"]. From there it will reverse the order of the array elements, so the array will equal: ["I", "am", "backwards"]. With the words reversed, now we simply need to merge the words into a single string, which is where the join method comes in. Running the join method will convert all of the words in the array into one string. Summary In this article, we were introduced to the string data type and how it can be utilized in Ruby. We analyzed how to pass strings into Ruby processes by leveraging string interpolation. We also learned the methods of basic string manipulation and how to find and replace string data. We analyzed how to break strings into smaller components, along with how to clean up string based data. We even introduced the Array class in this article. Resources for Article: Further resources on this subject: Ruby and Metasploit Modules [article] Find closest mashup plugin with Ruby on Rails [article] Building tiny Web-applications in Ruby using Sinatra [article]

0
0
16622

Packt

05 Jul 2017

9 min read

Azure Feature Pack

Packt

05 Jul 2017

9 min read

0
0
4100

Packt

05 Jul 2017

14 min read

OpenDaylight Fundamentals

Packt

05 Jul 2017

14 min read

0
0
42607

article-image-devops-concepts-and-assessment-framework

Packt

05 Jul 2017

21 min read

DevOps Concepts and Assessment Framework

Packt

05 Jul 2017

21 min read

0
1
18722

Packt

05 Jul 2017

10 min read

KVM Networking with libvirt

Packt

05 Jul 2017

10 min read

0
0
46272

article-image-managing-nano-server-windows-powershell-and-windows-powershell-dsc

Packt

05 Jul 2017

8 min read

Managing Nano Server with Windows PowerShell and Windows PowerShell DSC

Packt

05 Jul 2017

8 min read

In this article by Charbel Nemnom, the author of the book Getting Started with Windows Nano Server, we will cover the following topics: Remote server graphical tools Server manager Hyper-V manager Microsoft management console Managing Nano Server with PowerShell (For more resources related to this topic, see here.) Remote server graphical tools Without the Graphical User Interface (GUI), it’s not easy to carry out the daily management and maintenance of Windows Server. For this reason, Microsoft integrated Nano Server with all the existing graphical tools that you are familiar with such as Hyper-V manager, failover cluster manager, server manager, registry editor, File explorer, disk and device manager, server configuration, computer management, users and groups console, and so on. All those tools and consoles are compatible to manage Nano Server remotely. The GUI is always the easiest way to use. In this section, we will discuss how to access and set the most common configurations in Nano Server with remote graphical tools. Server manager Before we start managing Nano Server, we need to obtain the IP address or the computer name of the Nano Server to connect to and remotely manage a Nano instance either physical or virtual machine. Login to your management machine and make sure you have installed the latest Remote Server Administration Tools (RSAT) for Windows Server 2016 or Windows 10. You can download the latest RSAT tools from the following link: https://www.microsoft.com/en-us/download/details.aspx?id=45520 Launch server manager as shown in Figure 1, and add your Nano Server(s) that you would like to manage: Figure 1: Managing Nano Server using server manager You can refresh the view and browse all events and services as you expect to see. I want to point out that Best Practices Analyzer (BPA) is not supported in Nano Server. BPA is completely cmdlets-based and written in C# back during the days of PowerShell 2.0. It is also statically using some .NET XML library code that was not part of .NET framework at that time. So, do not expect to see Best Practices Analyzer in server manager. Hyper-V manager The next console that you probably want to access is Hyper-V Manager, right click on Nano Server name in server manager and select Hyper-V Manager console as shown in Figure 2: Figure 2: Managing Nano Server using Hyper-V manager Hyper-V Manager will launch with full support as you expect when managing full Windows Server 2016 Hyper-V, free Hyper-V server, server core and Nano Server with Hyper-V role. Microsoft management console You can use the Microsoft Management Console (MMC) to manage Nano Server as well. From the command line type mmc.exe. From the File menu, Click Add/Remove Snap-in…and then select Computer Management and click Add. Choose Another computer and add the IP address or the computer name of your Nano Server machine. Click Ok. As shown in Figure 3, you can expand System Tools and check the tools that you are familiar with like (Event Viewer, Local Users and Groups, Shares,and Services). Please note that some of these MMC tools such as Task Scheduler and Disk Management cannot be used against Nano Server. Also, for certain tools you need to open some ports in Windows firewall: Figure 3: Managing Nano Server using Microsoft Management Console Managing Nano Server with PowerShell For most IT administrators, the graphical user interface is the easiest way to use. But on the other hand, PowerShell can bring a fast and an automated process. That's why in Windows Server 2016, the Nano Server deployment option of Windows Server comes with full PowerShell remoting support. The purpose of the core PowerShell engine, is to manage Nano Server instances at scale. PowerShell remoting including DSC, Windows Server cmdlets (network, storage, Hyper-V, and so on), Remote file transfer, Remote script authoring and debugging, and PowerShell Web access. Some of the new features in Windows PowerShell version 5.1 on Nano Server supports the following: Copying files via PowerShell sessions Remote file editing in PowerShell ISE Interactive script debugging over PowerShell session Remote script debugging within PowerShell ISE Remote host process connects and debug PowerShell version 5.1 is available in different editions which denote varying feature sets and platform compatibility. Desktop Edition targeting Full Server, Server Core and Windows Desktop, Core Edition targeting Nano Server and Windows IoT. You can find a list of Windows PowerShell features not available yet in Nano Server here. As Nano Server is still evolving, we will see what the next cadence update will bring for unavailable PowerShell features. If you want to manage your Nano Server, you can use PowerShell Remoting or if your Nano Server instance is running in a virtual machine you can also use PowerShell Direct, more on that at the end of this section. In order to manage a Nano server installation using PowerShell remoting carry out the following steps: You may need to start the WinRM service on your management machine to enable remote connections. From the PowerShell console type the following command: net start WinRM If you want to manage Nano Server in a workgroup environment, open PowerShell console, and type the following command, substituting server name or IP with the right value using your machine-name is the easiest to use, but if your device is not uniquely named on your network, you can use the IP address instead: Set-Item WSMan:localhostClientTrustedHosts -Value "servername or IP" If you want to connect multiple devices, you can use comma and quotation marks to separate each device. Set-Item WSMan:localhostClientTrustedHosts -Value "servername or IP, servername or IP" You can also set it to allow to connect to a specific network subnet using the following command: Set-Item WSMan:localhostClientTrustedHosts -Value 10.10.100.* To test Windows PowerShell remoting against Nano Server and check if it’s working, you can use the following command: Test-WSMan -ComputerName"servername or IP" -Credential servernameAdministrator -Authentication Negotiate You can now start an interactive session with Nano Server. Open an elevated PowerShell console and type the following command: Enter-PSSession -ComputerName "servername or IP" -Credential servernameAdministrator In the following example, we will create two virtual machines on Nano Server Hyper-V host using PowerShell remoting. From your management machine, open an elevated PowerShell console or PowerShell scripting environment ,and run the following script (make sure to update the variables to match your environment): #region Variables $NanoSRV='NANOSRV-HV01' $Cred=Get-Credential"DemoSuperNano" $Session=New-PSSession-ComputerName$NanoSRV-Credential$Cred $CimSesion=New-CimSession-ComputerName$NanoSRV-Credential$Cred $VMTemplatePath='C:Temp' $vSwitch='Ext_vSwitch' $VMName='DemoVM-0' #endregion # Copying VM Template from the management machine to Nano Server Get-ChildItem-Path$VMTemplatePath-filter*.VHDX-recurse|Copy-Item-ToSession$Session-DestinationD: 1..2|ForEach-Object { New-VM-CimSession$CimSesion-Name$VMName$_-VHDPath"D:$VMName$_.vhdx"-MemoryStartupBytes1024GB` -SwitchName$vSwitch-Generation2 Start-VM-CimSession$CimSesion-VMName$VMName$_-Passthru } In this script, we are creating a PowerShell session and CIM session to Nano Server. A CIM session is a client-side object representing a connection to a local computer or a remote computer. Then we are copying VM Templates from the management machine to Nano Server over PowerShell remoting, when the copy is completed, we are creating two virtual machines as Generation 2 and finally starting them. After a couple of seconds, you can launch Hyper-V Manager console and see the new VMs running on Nano Server host as shown in Figure 4: Figure 4: Creating virtual machines on Nano Server host using PowerShell remoting If you have installed Nano Server in a virtual machine running on a Hyper-V host, you can use PowerShell direct to connect directly from your Hyper-V host to your Nano Server VM without any network connection by using the following command: Enter-PSSession -VMName <VMName> -Credential.Administrator So instead of specifying the computer name, we specified the VM Name, PowerShell Direct is so powerful, it’s one of my favorite feature, you can configure a bunch of VMs from scratch in just couple of seconds without any network connection. Moreover, if you have Nano Server running as a Hyper-V host as shown in the example earlier, you could use PowerShell remoting first to connect to Nano Server from your management machine, and then leverage PowerShell Direct to manage your virtual machines running on top of Nano Server. In this example, we used two PowerShell technologies (PS remoting and PS Direct).This is so powerful and open many possibilities to effectively manage Nano Server. To do that, you can use the following command: #region Variables $NanoSRV='NANOSRV-HV01'#Nano Server name or IP address $DomainCred=Get-Credential"DemoSuperNano" $VMLocalCred=Get-Credential"~Administrator" $Session=New-PSSession-ComputerName$NanoSRV-Credential$DomainCred #endregion Invoke-Command-Session$Session-ScriptBlock { Get-VM Invoke-Command-VMName (Get-VM).Name-Credential$Using;VMLocalCred-ScriptBlock { hostname Tzutil/g } } In this script, we have created a PowerShell session into Nano Server physical host, and then we used PowerShell Direct to list all VMs, including their hostnames and time zone. The result is shown in Figure 5: Figure 5. Nested PowerShell remoting Summary In this article, we discussed how to manage a Nano Server installation using remote server graphic tools, and Windows PowerShell remoting. Resources for Article: Further resources on this subject: Exploring Windows PowerShell 5.0 [article] Exchange Server 2010 Windows PowerShell: Mailboxes and Reports [article] Exchange Server 2010 Windows PowerShell: Managing Mailboxes [article]

0
0
58407

Packt

05 Jul 2017

16 min read

Lambda Functions

Packt

05 Jul 2017

16 min read

In this article, by Udita Gupta and Yohan Wadia, the authors of the book Mastering AWS Lambda, we are going to take things a step further by learning the anatomy of a typical Lambda Function and also how to actually write your own functions. We will cover the programming model for a Lambda function using simple functions as examples, the use of logs and exceptions and error handling. (For more resources related to this topic, see here.) The Lambda programming model Certain applications can be broken down into one or more simple nuggets of code called as functions and uploaded to AWS Lambda for execution. Lambda then takes care of provisioning the necessary resources to run your function along with other management activities such as auto-scaling of your functions, their availability, and so on. So what exactly are we supposed to do in all this? A developer basically has three tasks to perform when it comes to working with Lambda: Writing the code Packaging it for deployment Finally monitoring its execution and fine tuning In this section, we are going to explore the different components that actually make up a Lambda Function by understanding what AWS calls as a programming model or a programming pattern. As of date, AWS officially supports Node.js, Java, Python, and C# as the programming languages for writing Lambda functions, with each language following a generic programming pattern that comprises of certain concepts which we will see in the following sections. Handler The handler function is basically a function that Lambda calls first for execution. A handler function is capable of processing incoming event data that is passed to it as well as invoking other functions or methods from your code. We will be concentrating a lot of our code and development on Node.js; however, the programming model remains more or less the same for the other supported languages as well. A skeleton structure of a handler function is shown as follows: exports.myHandler = function(event, context, callback) { // Your code goes here. callback(); } Where, myHandler is the name of your handler function. By exporting it we make sure that Lambda knows which function it has to invoke first. The other parameters that are passed with the handler function are: event: Lambda uses this parameter to pass any event related data back to the handler. context: Lambda again uses this parameter to provide the handler with the function's runtime information such as the name of the function, the time it took to execute, and so on . callback: This parameter is used to return any data back to its caller. The callback parameter is the only optional parameter that gets passed when writing handlers. If not specified, AWS Lambda will call it implicitly and return the value as null. The callback parameter also supports two optional parameters in the form of error and result where error will return any of the function's error information back to the caller while result will return any result of your function's successful execution. Here are a few simple examples of invoking callbacks in your handler: callback() callback(null, 'Hello from Lambda') callback(error) The callback parameter is supported only in Node.js runtime v4.3. You will have to use the context methods in case your code supports earlier Node.js runtime (v0.10.42) Let us try out a simple handler example with a code: exports.myHandler = function(event, context, callback) { console.log("value = " + event.key); console.log("functionName = ", context.functionName); callback(null, "Yippee! Something worked!"); }; The following code snippet will print the value of an event (key) that we will pass to the function, print the function's name as part of the context object and finally print the success message Yippee! Something worked! if all goes well! Login to the AWS Management Console and select AWS Lambda from the dashboard. Select the Create a Lambda function option. From the Select blueprint page, select the Blank Function blueprint. Since we are not configuring any triggers for now, simple click on Next at the Configure triggers page. Provide a suitable Name and Description for your Lambda function and paste the preceding code snippet in the inline code editor as shown: Next, in the Lambda function handler and role section on the same page, type in the correct name of your Handler as shown. The handler name should match with the handler name in your function to work. Remember also to select the basic-lambda-role for your function's execution before selecting the Next button: In the Review page, select the Create function option. With your function now created, select the Test option to pass the sample event to our function. In the Sample event, pass the following event and select the Save and test option: { "key": "My Printed Value!!" } With your code execution completed, you should get a similar execution result as shown in the following figure. The important things to note here are the values for the event, context and callback parameters. You can note the callback message being returned back to the caller as the function executed successfully. The other event and context object values are printed in the Log output section as highlighted in the following figure: In case you end up with any errors, make sure the handler function name matches the handler name that you passed during the function's configuration. Context object The context object is a really useful utility when it comes to obtaining runtime information about your function. The context object can provide information such as the executing function's name, the time remaining before Lambda terminates your function's execution, the log name and stream associated with your function and much more. The context object also comes with its own methods that you can call to correctly terminate your function's executions such as context.succed(), context.fail(), context.done(), and so on. However, post April 2016, Lambda has transitioned the Node.js runtime from v0.10.42 to v4.3 which does support these methods however encourages to use the callback() for performing the same actions. Here are some of the commonly used context object methods and properties described as follows: getRemainingTimeInMillis(): This property returns the number of milliseconds left for execution before Lambda terminates your function. This comes in really handy when you want to perform some corrective actions before your function exits or gets timed out. callbackWaitsForEmptyEventLoop: This property is used to override the default behaviour of a callback() function, such as to wait till the entire event loop is processed and only then return back to the caller. If set to false, this property causes the callback() function to stop any further processing in the event loop even if there are any other tasks to be performed. The default value is set to true. functionName: This property returns the name of the executing Lambda function. functionVersion: The current version of the executing Lambda function. memoryLimitInMB: The amount of resource in terms of memory set for your Lambda function. logGroupName: This property returns the name of the CloudWatch Log Group that stores function's execution logs. logStreamName: This property returns the name of the CloudWatch Log Stream that stores function's execution logs. awsRequestID: This property returns the request ID associated with that particular function's execution. If you are using Lambda functions as mobile backend processing services, you can then extract additional information about your mobile application using the context of identity and clientContext objects. These are invoked using the AWS Mobile SDK. To learn more, click here http://docs.aws.amazon.com/lambda/latest/dg/nodejs-prog-model-context.html. Let us look at a simple example to understand the context object a bit better. In this example, we are using the context object callbackWaitsForEmptyEventLoop and demonstrating its working by setting the object's value to either yes or no on invocation: Login to the AWS Management Console and select AWS Lambda from the dashboard. Select the Create a Lambda function option. From the Select blueprint page, select the Blank Function blueprint. Since we are not configuring any triggers for now, simple click on Next at the Configure triggers page. Provide a suitable Name and Description for your Lambda function and paste the following code in the inline code editor: exports.myHandler = (event, context, callback) => { console.log('remaining time =', context.getRemainingTimeInMillis()); console.log('functionName =', context.functionName); console.log('AWSrequestID =', context.awsRequestId); console.log('logGroupName =', context.logGroupName); console.log('logStreamName =', context.logStreamName); switch (event.contextCallbackOption) { case "no": setTimeout(function(){ console.log("I am back from my timeout of 30 seconds!!"); },30000); // 30 seconds break break; case "yes": console.log("The callback won't wait for the setTimeout() n if the callbackWaitsForEmptyEventLoop is set to false"); setTimeout(function(){ console.log("I am back from my timeout of 30 seconds!!"); },30000); // 30 seconds break context.callbackWaitsForEmptyEventLoop = false; break; default: console.log("The Default code block"); } callback(null, 'Hello from Lambda'); }; Next, in the Lambda function handler and role section on the same page, type in the correct name of your Handler as shown. The handler name should match with the handler name in your function to work. Remember also to select the basic-lambda-role for your function's execution. The final change that we will do is change the Timeout value of our function from the default 3 seconds to 1 minute specifically for this example. Click Next to continue: In the Review page, select the Create function option. With your function now created, select the Test option to pass the sample event to our function. In the Sample event, pass the following event and select the Save and test option. You should see a similar output in the Log output window as shown: With the contextCallbackOption set to yes, the function does not wait for the 30 seconds setTimeout() function and will exit, however it prints the function's runtime information such as the remaining execution time, the function name, and so on. Now set the contextCallbackOption to no and re-run the test and verify the output. This time, you can see the setTimeout() function getting called and verify the same by comparing the remaining time left for execution with the earlier test run. Logging You can always log your code's execution and activities using simple log statements. The following statements are supported for logging with Node.js runtime: console.log() console.error() console.warn() console.info() The logs can be viewed using both the Management Console as well as the CLI. Let us quickly explore both the options. Using the Management Console We have already been using Lambda's dashboard to view the function's execution logs, however the logs are only for the current execution. To view your function's logs from the past, you need to view them using the CloudWatch Logs section: To do so, search and select CloudWatch option from the AWS Management Console. Next, select the Logs option to display the function's logs as shown in the following figure: You can use the Filter option to filter out your Lambda logs by typing in the log group name prefix as /aws/lambda. Select any of the present Log Groups and its corresponding Log Stream Name to view the complete and detailed execution logs of your function. If you do not see any Lambda logs listed out here it is mostly due to your Lambda execution role. Make sure your role has the necessary access rights to create the log group and log stream along with the capability to put log events. Using the CLI The CLI provides two ways using which you can view your function's execution logs: The first is using the Lambda function's invoke command itself. The invoke command when used with the --log-type parameter will print the latest 4 KB of log data that is written to CloudWatch Logs. To do so, first list out all available functions in your current region using the following command: # aws lambda list-functions Next, pick a Lambda function that you wish to invoke and substitute that function's name and payload with the following example snippet: # aws lambda invoke --invocation-type RequestResponse --function-name myFirstFunction --log-type Tail --payload '{"key1":"Lambda","key2":"is","key3":"awesome!"}' output.txt The second way is by using a combination of the context() object and the CloudWatch CLI. You can obtain your function's log group name and the log stream name using the context.logGroupName and the context.logStreamName. Next, substitute the data gathered from the output of these parameters in the following command: # aws logs get-log-events --log-group-name "/aws/lambda/myFirstFunction" --log-stream-name "2017/02/07/[$LATEST]1ae6ac9c77384794a3202802c683179a" If you run into the error The specified log stream does not exist in spite of providing correct values for the log group name and stream name; then make sure to add the "" escape character in the [$LATEST] as shown. Let us look at a few options that you can additionally pass with the get-log-events command: --start-time: The start of the log's time range. All times are in UTC. --end-time: The end of the log's time range. All times are in UTC. --next-token: The token for the next set of items to return. (You received this token from a previous call.) --limit: Used to set the maximum number of log events returned. By default the limit is set to either 10,000 log events. Alternatively, if you don't wish to use the context() objects in your code, you can still filter out the log group name and log stream name by using a combination of the following commands: # aws logs describe-log-groups --log-group-name-prefix "/aws/lambda/" The describe-log-groups command will list all the log groups that are prefixed with /aws/lambda. Make a note of your function's log group name from this output. Next, execute the following command to list your log group name's associated log stream names: # aws logs describe-log-streams --log-group-name "/aws/lambda/myFirstFunction" Make a note of the log stream name and substitute the same in the next and final command to view your log events for that particular log stream name: # aws logs get-log-events --log-group-name "/aws/lambda/myFirstFunction" --log-stream-name "2017/02/07/[$LATEST]1ae6ac9c77384794a3202802c683179a" Once again, make sure to add the backslash "" in the [$LATEST] to avoid the The specified log stream does not exist error. With the logging done, let's move on to the next piece of the programming model called exceptions. Exceptions and error handling Functions have the ability to notify AWS Lambda in case it failed to execute correctly. This is primarily done by the function passing the error object to Lambda which converts the same to a string and returns it to the user as an error message. The error messages that are returned also depend on the invocation type of the function; for example, if your function performs a synchronous execution (RequestResponse invocation type), then the error is returned back to the user and displayed on the Management Console as well as in the CloudWatch Logs. For any asynchronous executions (event invocation type), Lambda will not return anything. Instead it logs the error messages to CloudWatch Logs. Let us examine a function's error and exception handling capabilities with a simple example of a calculator function that accepts two numbers and an operand as the test events during invocation: Login to the AWS Management Console and select AWS Lambda from the dashboard. Select the Create a Lambda function option. From the Select blueprint page, select the Blank Function blueprint. Since we are not configuring any triggers for now, simple click on Next at the Configure triggers page. Provide a suitable Name and Description for your Lambda function and paste the following code in the inline code editor: exports.myHandler = (event, context, callback) => { console.log("Hello, Starting the "+ context.functionName +" Lambda Function"); console.log("The event we pass will have two numbers and an operand value"); // operand can be +, -, /, *, add, sub, mul, div console.log('Received event:', JSON.stringify(event, null, 2)); var error, result; if (isNaN(event.num1) || isNaN(event.num2)) { console.error("Invalid Numbers"); // different logging error = new Error("Invalid Numbers!"); // Exception Handling callback(error); } switch(event.operand) { case "+": case "add": result = event.num1 + event.num2; break; case "-": case "sub": result = event.num1 - event.num2; break; case "*": case "mul": result = event.num1 * event.num2; break; case "/": case "div": if(event.num2 === 0){ console.error("The divisor cannot be 0"); error = new Error("The divisor cannot be 0"); callback(error, null); } else{ result = event.num1/event.num2; } break; default: callback("Invalid Operand"); break; } console.log("The Result is: " + result); callback(null, result); }; Next, in the Lambda function handler and role section on the same page, type in the correct name of your Handler. The handler name should match with the handler name in your function to work. Remember also to select the basic-lambda-role for your function's execution. Leave the rest of the values to their defaults and click Next to continue. In the Review page, select the Create function option. With your function now created, select the Test option to pass the sample event to our function. In the Sample event, pass the following event and select the Save and test option. You should see a similar output in the Log output window as shown: { "num1": 3, "num2": 0, "operand": "div" } So what just happened there? Well first, we can print simple user friendly error messages with the help of the console.error() statement. Additionally, we can also print the stackTrace array of the error by passing the error in the callback() as shown: error = new Error("The divisor cannot be 0"); callback(error, null); You can also view the custom error message and the stackTrace JSON array both from the Lambda dashboard as well as from the CloudWatch Logs section. Next, give this code a couple of tries with some different permutations and combinations of events and check out the results. You can even write your own custom error messages and error handlers that can perform some additional task when an error is returned by the function. With this we come towards the end of a function's generic programming model and its components. Summary We deep dived into the Lambda programming model and understood each of its sub components (handlers, context objects, errors and exceptions) with easy to follow examples.

0
0
51421

Thread of Execution

When do we use R over Python?

Exposure to RxJava

Command-Line Tools

IIS 10 Fundamentals

Spark Streaming

Chart Model and Draggable and Droppable Directives

Continous Delivery

Ruby Strings

Azure Feature Pack

Trending Topics

OpenDaylight Fundamentals

DevOps Concepts and Assessment Framework

KVM Networking with libvirt

Managing Nano Server with Windows PowerShell and Windows PowerShell DSC

Lambda Functions