Transforming Business Intelligence Testing
- 1. Method360SECTION NAME GOES HERE IN CAPSHeader goes here © 2015 Method360, Inc. All rights reserved.1 Transforming BI Testing Steps to Achieve Significant Cost Savings Sunny Hiran Director Quality Assurance
- 2. Method360SECTION NAME GOES HERE IN CAPSHeader goes here © 2015 Method360, Inc. All rights reserved.2 Method360COMPANY OVERVIEWAbout Us Leading Information Technology Consulting Services Company • Headquartered in San Francisco, CA Local offices in San Diego, Chicago, Dallas, Pittsburgh, and Calgary • Celebrating Our 15th Year of Operations • 100+ Employees Throughout the US • Focus on Data, Analytics and Quality Assurance • Strong Emphasis on Consultant Quality and Client Relationship • Onsite, Offsite and Nearshore Capabilities • Certified SAP Services Partner
- 3. Method360SECTION NAME GOES HERE IN CAPSHeader goes here © 2015 Method360, Inc. All rights reserved.3 Method360TRANSFORMING BI TESTINGIntroduction Sunny Hiran Director Quality Assurance Connect with me on LinkedIn: sunnyhiran Sunny has worked both strategically and tactically with a broad selection of fortune 500 companies touching upon multiple vertical industries including retail, manufacturing, banking, finance, and high tech companies. Bringing both project and program management experience along with PMP certification, Sunny brings considerable testing expertise to clients.
- 4. Method360SECTION NAME GOES HERE IN CAPSHeader goes here © 2015 Method360, Inc. All rights reserved.4 Method360Session Overview You will gain an understanding of: • The need for rigorous BI testing – strategy, planning, and execution • Why the prioritization of BI scope including data structures, report fields, calculations, and filters is essential • The importance of enabling your BI testing • How a test strategy addresses multiple levels of BI-specific report testing that includes report layout, field data, DSO, and query results validation and sign off • How targeted test automation makes a significant impact • The need to safeguard business/end user response times and ensure solution scalability TRANSFORMING BI TESTING
- 5. Method360SECTION NAME GOES HERE IN CAPSHeader goes here © 2015 Method360, Inc. All rights reserved.5 Method360TRANSFORMING BI TESTING
- 6. Method360SECTION NAME GOES HERE IN CAPSHeader goes here © 2015 Method360, Inc. All rights reserved.6 Method360The Cost of Quality • Quality Assurance and testing cannot be avoided – pay now or pay later • Ineffective testing will result in poor quality deliverables, late delivery, and overspend • Failure is not an option (potential loss of revenue and competitiveness, lost customer confidence, and bad publicity) TRANSFORMING BI TESTING Testing costs anywhere from 25-35% of a projects budget
- 7. Method360SECTION NAME GOES HERE IN CAPSHeader goes here © 2015 Method360, Inc. All rights reserved.7 Method360Common BI Testing Challenges • There are inherently complex challenges associated with BI testing: Extensive manual test labor required to perform complex validations at significant cost and different levels of testing • BI reconciliation. (Do things match – front end to BW, ECC) • BO – multiple distribution channels such as email burst, online desktop vs. mobile report, etc. • ETL processes • Data warehouses (OLAP cubes and queries) • Report/Query filters, and data validation • Test data and environment management • Performance and response-time validation • Dashboards and scorecards TRANSFORMING BI TESTING
- 8. Method360SECTION NAME GOES HERE IN CAPSHeader goes here © 2015 Method360, Inc. All rights reserved.8 Method360Advantages of BI Risk-based Testing Focus on functionality that: Is most important to end users and key decision makers Most commonly used Has an impact to revenue generation • Achieve cost savings by utilizing the 80/20 rule • Provides for better impact analysis (of adjacent functionality) • Early identification of issues by utilizing: Smoke testing Automated/accelerated validation based on priority • Root cause analysis TRANSFORMING BI TESTING It’s impossible to test the entire solution, nor should you. Cut costs by addressing appropriate risk areas and improve quality.
- 9. Method360SECTION NAME GOES HERE IN CAPSHeader goes here © 2015 Method360, Inc. All rights reserved.9 Method360Advantages of BI Risk-based Testing (Cont.) • Key reports • Data model • Indexes • Transformations TRANSFORMING BI TESTING The areas of focus/risk tend to be around the following: • Data capture points in ECC • Extractors • Queries (calculation change, formatting change) • Distribution channels
- 10. Method360SECTION NAME GOES HERE IN CAPSHeader goes here © 2015 Method360, Inc. All rights reserved.10 Method360TRANSFORMING BI TESTING
- 11. Method360SECTION NAME GOES HERE IN CAPSHeader goes here © 2015 Method360, Inc. All rights reserved.11 Method360Identifying Solution Scope • Gain a clear understanding of scope, priority, and impact • Assess the impact of failure points within the infrastructure • Document all items (by priority) within a matrix TRANSFORMING BI TESTING Extractors ECC data capture points Key reports Data model Transformation
- 12. Method360SECTION NAME GOES HERE IN CAPSHeader goes here © 2015 Method360, Inc. All rights reserved.12 Method360Demystifying BI Testing TRANSFORMING BI TESTING
- 13. Method360SECTION NAME GOES HERE IN CAPSHeader goes here © 2015 Method360, Inc. All rights reserved.13 Method360Demystifying BI Testing (Cont.) • Do we have the correct focus? Are we testing the right things to validate the organization’s most important business priorities? • Have we assessed potential business impacts and failure points in our strategy? • Have we employed the right test approach to validate the different technologies and address integrations and customizations? • Do we understand our stakeholders reporting needs? • Given the budget and scope, do we have the correct resourcing, test cycles, strategies and tools? • Are the acceptance criteria well defined? • Do we have buy-in from the business? • Does our testing address current and future business needs? TRANSFORMING BI TESTING A good test strategy should answer the following questions
- 14. Method360SECTION NAME GOES HERE IN CAPSHeader goes here © 2015 Method360, Inc. All rights reserved.14 Method360Where to Start 1. An assessment/roadmap will lay a foundation 2. Create a test plan that clarifies scope, risk, timelines, resourcing, goals and objectives of testing, automation priorities, etc. 3. Identify stakeholders and ensure sign off 4. Identify a formal test strategy/framework/methodology that details progressive levels of testing (data, queries, report layout, etc.) 5. Document priority (risk-based) items along with acceptance criteria 6. Create a test matrix that identifies all the in scope test objects as shown in the diagram on the following slide 7. Create a requirements traceability matrix of all prioritized functionality including queries, calculations, and reports TRANSFORMING BI TESTING
- 15. Method360SECTION NAME GOES HERE IN CAPSHeader goes here © 2015 Method360, Inc. All rights reserved.15 Method360Where to Start • Dedicated QA resources utilizing targeted test strategies • Automation, performance (query and ETL), DR • Clear communication on testing progress to all stakeholders • Release Management/Change Control • Quality gating criteria • Standard best practices around environments • I.e., Dev, QA, and Prod • QA environment configuration • Representative data sources/test data TRANSFORMING BI TESTING There are many BI test considerations that will enable the success of your projects and further mitigate risks
- 16. Method360SECTION NAME GOES HERE IN CAPSHeader goes here © 2015 Method360, Inc. All rights reserved.16 Method360Supporting Successful Testing (Cont.) • Data alignment and data models • Engaging business teams early to validate the test scope, strategy, and priorities will achieve several key objectives: Business buy-in Confidence in the solution Quality outcomes and results Knowledge transfer • System-wide refresh strategy • Formal risk management and risk-reassessment • Return on Investment (ROI) based test automation to expand test coverage and accelerate validation activities • Validation of solution scalability and response times vs. SLA TRANSFORMING BI TESTING
- 17. Method360SECTION NAME GOES HERE IN CAPSHeader goes here © 2015 Method360, Inc. All rights reserved.17 Method360Incorporating Test Automation • Do your research when selecting a tool – any vendor should demonstrate the claims they make on a tool • Get educated about the value your project/organization will gain from Test Automation • Maximize ROI by performing validations across multiple environments, i.e., Dev and QA • Have a clear roadmap that will guide your automation efforts • Business buy-in/sign off will build confidence and accuracy • The diagram on the following slide shows a recommended approach for implementing Test Automation TRANSFORMING BI TESTING Test Automation when utilized successfully will significantly expand test coverage and accelerate validation activities
- 18. Method360SECTION NAME GOES HERE IN CAPSHeader goes here © 2015 Method360, Inc. All rights reserved.18 Method360Incorporating Test Automation (Cont.) TRANSFORMING BI TESTING
- 19. Method360SECTION NAME GOES HERE IN CAPSHeader goes here © 2015 Method360, Inc. All rights reserved.19 Method360Realizing Value from Performance Testing • Having business buy-in on what to simulate will ensure an accurate and meaningful activity • Aim for a mirror of production • Not all tools are equal • Accurately simulating the entire infrastructure is a requirement • Monitoring is essential as is an understanding of monitoring components and sources of performance failures TRANSFORMING BI TESTING Many organizations struggle with performance and response time issues of business critical reports; some in spite of having internal performance testing. Obviously there is a problem!
- 20. Method360SECTION NAME GOES HERE IN CAPSHeader goes here © 2015 Method360, Inc. All rights reserved.20 Method360Why Metrics Matter • Provide project/program visibility to stakeholders and key decision makers • Communication tool • Metrics provide a clear way of determining progress and readiness for go- live/release • Clear measure as to the level of conformance to requirements/ specifications • Justification for: QA efforts/resourcing Inherent quality or the lack thereof Pinpointing weaknesses across people, process, and tools Understanding when the solution is complete TRANSFORMING BI TESTING
- 21. Method360SECTION NAME GOES HERE IN CAPSHeader goes here © 2015 Method360, Inc. All rights reserved.21 Method360Why Metrics Matter (Cont.) TRANSFORMING BI TESTING
- 22. Method360SECTION NAME GOES HERE IN CAPSHeader goes here © 2015 Method360, Inc. All rights reserved.22 Method3607 Key Takeaways • BI testing should not be underestimated and requires a detailed strategy and planning in order to avoid unsatisfactory outcomes • Subject BI projects to the same rigor/quality standards as any other project • Risk-based testing with business buy-in is the key • Dedicated QA resourcing will address ownership and accountability issues around testing • Use a formal test planning and management tool as your test system of record • Use test automation – will make a significant difference • Performance testing is becoming more critical to the success of organizations with extensive BI capabilities TRANSFORMING BI TESTING
- 23. Method360SECTION NAME GOES HERE IN CAPSHeader goes here © 2015 Method360, Inc. All rights reserved.23 Method360BI Testing White Paper All attendees will receive an electronic copy of Sunny’s BI Testing white paper via email TRANSFORMING BI TESTING
- 24. Method360SECTION NAME GOES HERE IN CAPSHeader goes here © 2015 Method360, Inc. All rights reserved.24 Method360Questions TRANSFORMING BI TESTING
- 25. Method360SECTION NAME GOES HERE IN CAPSHeader goes here © 2015 Method360, Inc. All rights reserved.25 Method360More webinars… Check out our YouTube channel to view more webinars on-demand TRANSFORMING BI TESTING
- 26. Method360SECTION NAME GOES HERE IN CAPSHeader goes here © 2015 Method360, Inc. All rights reserved.26 Method360PRACTICE LEADERSHIPContact Information P R E S E N T E R P R A C T I C E S ANALYTICS BLAKE DUGGAN 415.548.0762 [email protected] ENTERPRISE INFORMATION MANAGEMENT JIM HANCK 415.694.8229 [email protected] FINANCIAL MANAGEMENT STEPHEN STONE 408.206.2830 [email protected] SALES & MARKETING GREG CLARKE 714.380.2753 [email protected] MANAGED SERVICES, PMO AND DELIVERY ERIK AXTON 925.389.6708 [email protected] STRATEGIC ACCOUNTS GREG STIMPSON 925.640.8785 [email protected] QUALITY ASSURANCE SUNNY HIRAN 415.535.5453 [email protected]
- 27. Method360SECTION NAME GOES HERE IN CAPSHeader goes here © 2015 Method360, Inc. All rights reserved.27 Method360TRANSFORMING BI TESTINGThank You E N D - T O - E N D S E R V I C E S . H I G H V A L U E S O L U T I O N S . Follow / like us on