Data virtualization using polybase

Editor's Notes

• #6: Hello and welcome to another SQL Night I am Antonios Chatzipavlis I am a Data Solutions Consultant and Trainer and I have been in the Information Technology Industry since 1988 I have been an MCT since 2000 and Microsoft Data platform MVP since 2010. I started using SQL Server since version 6.0 this means I have more than 25 of experience with this product in large scale environments. I have more than 60 (sixty) of certifications mostly in MS products Finally I am the founder of SQLschool.gr
• #8: SQLschool.gr is a community for Greek professionals who use the Microsoft Data Platform. In this you will find Articles, Webinars, Videos, Resources, news about Microsoft Data Platform. You can join us as a member or follow us in social media to keep up with our community This year SQLschool.gr became 10 years old and I would like to thank you all for your participation and support.
• #9: SQLschool.gr is a community for Greek professionals who use the Microsoft Data Platform. In this you will find Articles, Webinars, Videos, Resources, news about Microsoft Data Platform. You can join us as a member or follow us in social media to keep up with our community This year SQLschool.gr became 10 years old and I would like to thank you all for your participation and support.
• #17: There are two components selected aside from Database Engine Services: the PolyBase Query Service for External Data and the Java connector for HDFS data sources. The Java connector for HDFS data sources provides us support for connecting to Hadoop and Azure Blob Storage, which were the two endpoints available with PolyBase in SQL Server 2016 and SQL Server 2017; I refer to this throughout the book as PolyBase V1. SQL Server 2019 also adds the PolyBase Query Service for External Data component, which includes support for services like Oracle, Teradata, MongoDB, Cosmos DB, and even other SQL Server instances. In order to install this component, SQL Server’s installer will also install the Microsoft Visual C++ 2017 Redistributable.
• #18: Before you begin installation, it is important to know whether you want to install PolyBase as a standalone service or as part of a Scale-Out Group because you will not be able to switch between the two afterward without uninstalling and reinstalling the PolyBase features. If you are using SQL Server on Linux, the only option available to you at this time is to install standalone; SQL Server on Windows allows for both installation methods. All other things equal, a Scale-Out Group is preferable to a standalone installation. The reason for this is that PolyBase is a Massively Parallel Processing (MPP) technology. This means we can scale PolyBase horizontally, improving performance by adding additional servers. But that only works if you incorporate your machine as part of a Scale-Out Group, however; as a standalone installation, your SQL Server instance will not be able to enlist the support of other SQL Server instances when using PolyBase to perform queries. The preceding text makes sense when all other things are equal, but installing PolyBase as part of a Scale-Out Group has some requirements which standalone PolyBase does not. To wit, in order to install PolyBase as part of a Scale-Out Group, all of the following must be true:
• #19: The first option is to install the Azul Zulu Open JRE. This is a distribution of Oracle’s Open Java Runtime Environment which Azul Systems supports. Your license for SQL Server includes support for this particular distribution of Open JRE, meaning that you could contact Microsoft support for issues related to the JRE. The link on the installation page includes more information on this licensing agreement. If you are already a licensed Oracle Standard Edition (SE) customer, you can of course install the Oracle SE version of the Java Runtime Environment. To do so, select the “Provide the location of a different version that has been installed to on this computer” option and navigate to your already-installed version of the Java Runtime Environment. SQL Server 2016 and 2017 supported JRE version 7 update 51 and later, as well as JRE version 8. SQL Server 2019 supports later versions of the Java Runtime Environment, including version 11. If you are not a licensed Oracle SE customer, you can also install Oracle’s Open JRE. The downside to this is that your support options are limited to public forum access.
• #23: Configuration.sql
• #25: PolybaseBlob.sql
• #26: PolybaseSQL.sql
• #27: Linked servers are a classic technique database administrators and developers can use to query another server’s data from the local server. On the plus side, there is extensive OLEDB driver support, and linked servers can reach out to technologies like Oracle, Apache Hive, other SQL Server instances, and even Excel. On the minus side, linked servers have an oft-deserved reputation for bringing over too much data from the remote server during queries and a somewhat undeserved reputation for being a security issue. Still, introducing the idea of an alternative for linked servers should excite many a DBA. Here is where I have mixed news for you: PolyBase can be superior to linked servers in some circumstances, but you will not want to replace all of your linked servers with external tables, as there are some cases where linked servers will be superior. Instead, think of these as two complementary technologies with considerable overlap. Object Scopes Linked servers are scoped at the instance level, which means that when you create a linked server, any database on that instance has access to the linked server. Furthermore, on the remote side, linked servers allow you to query any table or view on any database where the remote login has rights. The advantage to the linked server model is its flexibility: you can use linked servers for any number of queries across an indefinite number of remote tables or views. The biggest disadvantage of this approach is that it promotes the idea that perhaps you ought to make that cross-server join of two very large tables. By contrast, PolyBase requires more deliberation: a database administrator or developer needs to create the external table link on a table-by-table or view-by-view basis before anybody can use it. This additional effort should make the creator think about whether a cross-server link is really necessary and can provide a bit of extra documentation about which tables the staff intend to use for cross-server queries. The downside to this is, if you have a large number of tables to query, it means writing a large number of external table definitions and also maintaining these definitions across table changes. This makes PolyBase a better choice for more stable data models and linked servers for more dynamic data models. Operational Intent Linked servers allow for reads as well as inserts, updates, and deletes. With PolyBase V1, we were able to read and insert but could not update or delete data. For the PolyBase V2 types, we are able to read but the engine prohibits any data modification, including inserts. If you attempt a data modification statement against a PolyBase V2 external table, you will get an error message similar to that Msg 46519 – DML Operations are not supported with external tables Scale-Out Capabilities Linked servers offer no ability to scale out. One SQL Server instance may read from one SQL Server instance. If you experience performance problems, there is no way to add additional SQL Server instances to the mix to share the load. PolyBase, meanwhile, offers Scale-Out Groups for cases when three or four servers are better than one. In this regard, PolyBase is strictly superior. Data Sizes Tying in with scale-out capabilities, linked servers and PolyBase have different expectations for ideal data size. If you intend to pull back one row or a few rows from a small table, linked servers will generally be a superior option because there are fewer moving parts. As you get more complicated queries with larger data sets, PolyBase tends to do at least as well and often better. Over the rest of this chapter, we will test the performance of PolyBase vs. linked servers in several scenarios to see when PolyBase succeeds and when linked servers come out ahead.
• #28: There are 13 Dynamic Management Views available in SQL Server 2019 which relate to PolyBase. In this section, we will review each of these at a high level, starting with basic metadata resources, followed by the DMVs which help with service and node setup, and finishing with DMVs for query troubleshooting.
• #29: External Data Sources Returns one row per external data source External File Formats Shows each of the most important settings for an external file External Tables Inherits several columns from sys.objects. Contains PolyBase-specific columns. External table The useful external table columns include external data source and external file format IDs, allowing us to tie these three tables together. For PolyBase V2 tables, the file format ID will be 0, as we do not use external file formats for these data sources.
• #30: Compute Nodes returns one row for the head node and one row for each PolyBase compute node, including the server name and port, as well as its IP address. If you have a standalone installation of PolyBase, you will get two rows back: one for the head and one for the local instance’s compute node. If you are using a scale-out cluster, you will get back the two rows in a standalone installation as well as one row for each scale-out compute node you have in the cluster. The sys.dm_exec_compute_node_status DMV connects to each compute node in order to determine if it is available. It retrieves server-level information such as allocated and available memory (in bytes), process and total CPU utilization (in ticks), the last communication time per node, and the latest error to have occurred as well. Figure 10-5 shows an example of some of the columns in this DMV. When it comes to errors, however, we can see the value of all of the columns while on-premises by querying sys.dm_exec_compute_node_errors. This DMV holds a history of error messages and is a good place to look when troubleshooting failures on a system. In addition to its unique ID data type, the data in sys.dm_exec_compute_node_errors will persist even after we restart the SQL Server services. Most Dynamic Management Views—for example, wait stat measures—reset when the database engine restarts, but compute node errors will stick around.
• #31: The first of these is sys.dm_exec_dms_services . This view returns one row per compute node—including one row for the head instance’s compute node—and the status for each of these nodes. Figure 10-7 shows the output of this DMV. We also have the ability to see the outputs of data movement service operations using the sys.dm_exec_dms_workers DMV. This gives us one row for each execution ID and execution step and includes performance measures, including bytes and rows, total elapsed time, CPU utilization time, and more To clear up potential confusion, the total elapsed time and query time values are in milliseconds, whereas CPU time is in ticks, where 10,000 ticks add up to a millisecond. Therefore, to get a clearer measure across the board, we want to divide the CPU time column by 10,000 to get a better picture of just how much CPU time we are actually using in relation to total elapsed time. In addition to these measures, we are also able to see the source SQL query for these operations, as well as the error ID if an operation fails. Unlike the compute node errors DMV, the DMS workers Dynamic Management View resets every time you restart the PolyBase engine service.
• #32: The final five Dynamic Management Views help us learn more about the SQL queries users run on our instances. Like the server and node resource DMVs we just looked at, these are all instance-level DMVs, meaning we will get the same results when running in any database. These views break down into two types, based on their names: external work and distributed requests. The external work results reset each time we restart the PolyBase engine, whereas the distributed requests DMVs persist even after service restarts. First up in our set of views is sys.dm_exec_external_work. This Dynamic Management View returns one row for each of the last 1000 PolyBase queries we have run since the last time the PolyBase engine started, as well as any active queries currently running. This DMV contains information on the current status of each execution, including the latest step for each compute node and Data Movement Service step. We can see the type of operation, which is “File Split” for PolyBase V1 queries and “ODBC Data Split” for PolyBase V2 queries. The input name tells us which file, folder, or table we are reading—for the SQL Server example on the first line, the input name is sqlserver://sqlcontrol/PolyBaseRevealed.dbo.Person. If we are reading from a file, the read_location field gives us the starting offset from 0 bytes. In the three cases in Figure 10-9, we read the file starting from the beginning. We can see the actual ODBC command next in the read_command column, which is a new field for SQL Server 2019. Finally, there are some columns containing top-level metrics, including bytes processed, file length (when reading files), start and end dates, the total elapsed time in milliseconds, and the status of each request. This status will be one of the following values: Pending, Processing, Done, Failed, or Aborted. If you perform a predicate pushdown operation against a Hadoop cluster, the sys.dm_exec_external_operations Dynamic Management View will give you a rundown of these pushdown operations. Figure 10-10 shows an example of a pushdown MapReduce job which failed—we can see that the map and reduce progress values are both at 0%. The sys.dm_exec_distributed_requests view returns one line per distributed operation. It provides us with one extremely helpful piece of information: a SQL handle, which we can use to return query text or an execution plan for our PolyBase queries. Figure 10-11 shows several rows from this table, including QID2260, which failed in the prior figure. The sys.dm_exec_distributed_request_steps view returns one row per execution ID and step. It is particularly useful when you already know an execution ID and want to understand what happened at each step along the way. Figure 10-12 gives us a glimpse at some of the most important columns here. The sys.dm_exec_distributed_sql_requests Dynamic Management View is our final DMV of note. It contains one row per SQL-related step on each compute node and for each distribution. Figure 10-13 shows an example of this for execution ID QID2148. This view makes clear the distributed nature of PolyBase: each distributed request step has eight separate SPIDs running on a single compute node. As with the distributed request steps DMV, we will look at this DMV in some greater detail next.
• #33: The first step creates the name of our temp table, TEMP_ID_XX. This appears to be an incrementing value, and the operation runs on the head node. The second step has us create a temporary table on each compute node named TEMP_ID_XX . The shape of this table is the set of columns that we will need for our query: population type, year, and population. The third step adds an extended property named IS_EXTERNAL_STREAMING_TABLE to each of the temp tables, presumably to make it easier to track which temp tables are used for loading external data. The step 4 runs a statistics update, updating statistics and telling SQL Server that we expect the temp table will have 566 rows. Our fifth step (i.e., step index 4) runs on the head node once more and is a MultiStreamOperation. There is no official documentation on this step, but it takes up 848 of the 919 total milliseconds of elapsed time and appears to be the operation which causes our compute nodes to do work. From there, we see a HadoopShuffleOperation on the Data Movement Service. This returns all 13,607 rows in the population table. We can see from the cleaned-up query in Listing 10-3 that this is a simple query of all rows from our population table. While we shuffle data across our compute nodes’ Data Movement Services, the next step runs, a StreamingReturnOperation . We can tell these are running concurrently because the shuffle operation takes 845 milliseconds and the streaming return operation 804 milliseconds, yet our entire query finished in under a second. This streaming query, which again runs on each of the compute nodes, queries TEMP_ID_73 and performs the aggregation we requested. Of interest is the fact that this query does not follow exactly the same shape as what we sent the database engine.
• #38: Polybase Log Files C:\Program Files\Microsoft SQL Server\MSSQL15.MSSQLSERVER\MSSQL\Log\Polybase DMS Errors The DMS error log gives stack traces when an exception occurs in the data movement service. One of the more common errors you might find when reading through this log is System.Data.SqlClient.SqlException: Operation cancelled by user. This exception occurs when a user or application stops a query, such as when a user hits the “Stop” button in Azure Data Studio. You can safely ignore this error. This particular log file tends to give you a high-level view of when errors occur but little information on the root cause or even the specific error. One of the more common errors I tend to see in this log is Internal Query Processor Error: The query processor encountered an unexpected error during the processing of a remote query phase. This phrase will not help me diagnose the problem, but this log file does tend to include information like the query ID and plan ID, which I can use to figure out which queries are failing. DMS Movement The data movement service writes a good amount of information to the DMS Movement log and includes detailed information on what data moves over from Azure Blob Storage or Hadoop to SQL Server. This includes the SQL queries the PolyBase data movement service generates, configuration settings such as the number of readers the DMS will use to migrate data, and detailed operation at each step. Combined with the DMS error log, we can start to piece together our errors. DWEngine Errors Like the DMS error log, the DWEngine error log gives a higher-level overview of when errors occur, as well as stack traces. This file can help you pinpoint when an error occurs. The errors in this file tend to be a bit more descriptive than the ones in the DMS error log. For example, we can find errors relating to the maximum reject threshold in this file: Query aborted-- the maximum reject threshold (1 rows) was reached while reading from an external source: 2 rows rejected out of total 2 rows processed. DWEngine Movement This log provides us with more detail on queries and errors which the DWEngine error log captures. In some cases, this file has enough information to drive to the root cause. In Figure 5-3, we see an example of a clear error message where I defined a column in an ORC file as a string data type but am trying to use an integer data type to access it via PolyBase. DWEngine Server The DWEngine Server log contains a few pieces of useful information. One of the most useful is that it contains the create statements for external data sources, file formats, and tables. We can use this log to determine what our external resources looked like at the time of exception, just in case somebody changed one of them during troubleshooting. This log also contains information on failed external table access attempts. If you have firewall or connection problems, this should be your first log to review. Figure 5-4 shows an example of a common HDFS bridge error whose root cause is insufficient permissions granted to the PolyBase pdw_user account. DMS PolyBase The DMS PolyBase log shows us something extremely important: any data translation failure. Figure 5-5 gives us three examples of data translation errors, including column conversion errors, data length errors, and string delimiter errors. We can also find cases where values are NULL, but the external table requires a non-nullable field, invalid date conversion attempts, and more. DWEngine PolyBase This file is much less interesting than most of the other logs. In my work, I have not seen it stretch to more than a few lines, and the most interesting thing in this log is the location of new Hadoop clusters as you create external data sources.
• #39: Structural Mismatch The first common data problem is structural mismatch—that is, when you define your external table one way but the data does not comport to that structure. For example, you might define an external table as having eight columns, but the underlying data set has seven or nine columns. In that case, the PolyBase engine will reject rows because they do not fit the expected structure. Caution In production Hadoop systems, developers are liable to change the structure of files and leave old files as is. For example, a report with eight columns might suddenly populate with nine columns on a certain date. The PolyBase engine cannot support multiple data structures for the same external table and will reject at least one of the two structures. This might cause a previously working external table query suddenly and unexpectedly to fail. Aside from column totals, there are several other mismatch problems which can cause queries to fail. For example, text files might have different schemas or delimiters: one type might be comma-delimited and another pipe-delimited. Some text files might use the quotation mark as a string delimiter, and others might use brackets or tildes. Any lack of consistency will cause the PolyBase engine to fail processing. If you do run into this scenario, an easy solution would be to create several external tables—one for each distinct file structure—and use a view to combine them together as one logical unit. Unsupported Characters or Formats PolyBase supports only a limited number of date formats. The safest route is to limit your text file dates to use supported formats. You can find these on Microsoft Docs (https://docs.microsoft.com/en-us/sql/t-sql/statements/create-external-file-format-transact-sql). PolyBase also struggles with newlines in text fields, so strip those out before trying to load data. Even within a quoted delimiter, newlines will cause the PolyBase engine to think it is starting a new record. PolyBase Data Limitations PolyBase also has limits to what data it can support. From Microsoft Docs (https://docs.microsoft.com/en-us/sql/relational-databases/polybase/polybase-versioned-feature-summary), we can see that the maximum row size cannot exceed 32KB for SQL Server or 1MB for Azure Synapse Analytics. In addition, if you save your data in ORC format, you might receive Java out-of-memory exceptions due to data size. For text-heavy files, it might be best to keep them as delimited files rather than ORC files. The maximum possible row size, which includes the full length of variable length columns, can't exceed 32 KB in SQL Server or 1 MB in Azure Synapse Analytics. When data is exported into an ORC file format from SQL Server or Azure Synapse Analytics, text-heavy columns might be limited. They can be limited to as few as 50 columns because of Java out-of-memory error messages. To work around this issue, export only a subset of the columns. PolyBase can't connect to a Hortonworks instance if Knox is enabled. If you use Hive tables with transactional = true, PolyBase can't access the data in the Hive table's directory.