Building Scalable FME Solutions: Best Practices for Workspaces, Automation and Beyond
0 likes724 views
The document discusses best practices for building scalable FME solutions, emphasizing the importance of effective project organization, workspace management, and automation setups. It highlights the challenges of dynamic project requirements and promotes strategies for documentation and communication within teams. Additionally, it provides insights into the upcoming Peak of Data Integration event, encouraging participation and certification opportunities.
Building Scalable FME Solutions: Best Practices for Workspaces, Automation and Beyond
- 1. Building Scalable FME Solutions: Best Practices for Workspaces, Automation and Beyond
- 2. Darren Fergus Data Solutions Locus Mark Ireland Knowledge & Community Specialist Safe Software Meet the Presenters
- 3. Welcome to Livestorm. A few ways to engage with us during the webinar: Audio issues? Click this for 4 simple troubleshooting steps.
- 4. Peak of Data Integration Seattle, WA | May 6–8, 2025 ● Register at the best price! ○ Early Bird Pricing live til Feb 7, 2025. ● 3 days of data disruption ○ Be among the first to hear ground breaking updates and launches! ● *New* - Get FME certified for free at the event! ○ 2 exams offered on Wednesday, May 7.
- 5. Agenda 1 Introduction 2 Foundations of Effective Project Organization 3 Building Tidy and Scalable Workspaces 4 Managing FME Flow 5 Conclusion 6 Resources & Next Steps 7 Q&A Agenda
- 6. Poll: How do you kick off a new FME Project?
- 8. Today, learn how to bring the principles of scalability and collaboration to life in your own workflows.
- 9. One platform, two technologies FME Form FME Flow Data Movement and transformations (“ETL”) workflows are built here. Brings life to FME Form workflows FME Flow Hosted Safe Software managed FME Flow fme.safe.com/platform FME Enterprise Integration Platform Safe & FME
- 10. With 500+ supported data types in FME. Unrivalled Data Support GIS CAD Database XML Raster 3D BIM Web Point Cloud Cloud Big Data IOT Gaming BI Indoor Mapping AR/VR Generative AI Cloud Native Tabular
- 11. Key Considerations ● What problem are we solving? ● Managing dependencies ● Governance and collaboration across teams. ● Example Integration: Showcase initial planning and setup of the example project.
- 12. The Why of Best Practices Projects in development are not static! ● Requirements change ● Your skills and knowledge change
- 13. Live projects are never static! ● Requirements are added ● Data changes ● FME changes ● Web services change ● Staff change The Why of Best Practices Projects in development are not static! ● Requirements change ● Your skills and knowledge change
- 14. Live projects are never static! ● Requirements are added ● Data changes ● FME changes ● Web services change ● Staff change The Why of Best Practices Projects in development are not static! ● Requirements change ● Your skills and knowledge change If you don’t plan ahead and aren’t organized, these changes will cause chaos. ● The project will decay, not grow ● Processes will be unreliable ● Support to end users will be slow ● Authors and admins will be frustrated ● The ROI will be substantially reduced
- 15. The Why of Best Practices We understand! FME is meant to be fun and exciting. ● Planning is not as exciting as jumping straight into a workspace! ● You don’t want to stop to restructure a workspace when you’re in full flow! ● But now the workspace is done, you’re too tired to clean it up. ● It’s hard to find motivation to revisit prior workspaces and make updates. ● Documentation is boring and it’s a chore to keep it up to date.
- 16. The Why of Best Practices We understand! FME is meant to be fun and exciting. ● Planning is not as exciting as jumping straight into a workspace! ● You don’t want to stop to restructure a workspace when you’re in full flow! ● But now the workspace is done, you’re too tired to clean it up. ● It’s hard to find motivation to revisit prior workspaces and make updates. ● Documentation is boring and it’s a chore to keep it up to date. It’s important to find the balance between avoiding best practices completely, and documenting every single workspace component.
- 18. Demo
- 19. Core Principles & Practices to Note ● Workspace and web connection naming conventions. ● Using bookmarks, tunnels, and organizing attribute managers. ● Adding fields, conditional values, and inline documentation. ● Example Application: Demonstrate each principle by iteratively building out the example project. ● Error trapping is important! ● Expect the unexpected! ● Don’t let end users see your failures!
- 20. Efficient Workflow Design ● Choosing and using transformers effectively. ● Minimizing clicks and optimizing for updates and debugging. ● Example Progression of use case: Show transformations and adjustments in the project workspace.
- 22. ● Setting Up Automations: Flow organization and workspace design ● Using webhooks and handling payloads. ● Managing database and web connections ● Building flexibility into workflows for future changes. ● Example project: Incorporate FME Flow features into the project, highlighting benefits and challenges. FME Flow in Action
- 23. FME Flow automations are made up of a series of components.
- 24. Slide Title An event from a webhook or similar notification. Event Automating FME Flow FME Community Example: A user posts a new question.
- 25. Slide Title An event from a webhook or similar notification. Event Automation Automating FME Flow A tool in FME Flow to carry out a series of actions. FME Community Example: An automation carries out an action in response to a new question.
- 26. Slide Title An event from a webhook or similar notification. Event Automation Automating FME Flow A tool in FME Flow to carry out a series of actions. An object in an automation to trigger an action. Trigger FME Community Example: An automation trigger detects that a new question was asked.
- 27. Slide Title An event from a webhook or similar notification. Event Automation Automating FME Flow A tool in FME Flow to carry out a series of actions. An object in an automation to trigger an action. A workspace or similar action. Trigger Workspace FME Community Example: A workspace is run to analyze and assess each new question.
- 28. Slide Title An event from a webhook or similar notification. Event Automation Automating FME Flow A tool in FME Flow to carry out a series of actions. An object in an automation to trigger an action. A workspace or similar action. Trigger Workspace Task A task required to be carried out. FME Community Example: The user is awarded a badge if the question meets certain criteria.
- 29. Slide Title Event Automation Automating FME Flow Trigger Workspace Task There can be multiple relationships between components. These relationships can be: ● 1:1 1 Trigger runs 1 Workspace ● 1:Many 1 Trigger runs Many Workspaces ● Many:1 Many Triggers run 1 Workspace ● Many:Many Many Triggers run Many Workspaces
- 30. Because of these complex relationships, organizing FME Flow components is as important as - maybe more important than - organizing FME Workspaces!
- 31. Demo
- 33. Slide Title Events Trigger Automations Task Task Task Workspace The “Exploded” Approach Every task has its own workspace inside its own Automation.
- 35. Slide Title Events Trigger Automations Task “Hybrid” Approaches Tasks Tasks Task Tasks Task A workspace can be run by multiple Automations.
- 36. You - as the administrator/author - need to decide which structure is going to work best for your scenario.
- 37. Payload is key!You (mostly) don’t want the same workspace to have to deal with different event payloads. You - as the administrator/author - need to decide which structure is going to work best for your scenario. ● Start by creating a list of all the separate tasks you want to do. ● Determine which events need to trigger which tasks. ● Work out which tasks can be part of the same workspace. ● Divide your incoming events by payload type/content.
- 38. Why? ● Each Gainsight Event Webhook has its own separate Automation. ● Each Automation can run several workspaces. ● Each workspace can do several similar tasks. FME Community Setup
- 39. ● Each Gainsight Event Webhook has its own separate Automation. ● Each Automation can run several workspaces. ● Each workspace can do several similar tasks. FME Community Setup Why? ● Authors know to put new tools into an existing Automation. ● Authors can make use of existing triggers. ● Authors can decide whether a new workspace is needed or not. But…
- 40. You have to document these requirements! Otherwise other authors won’t be aware of them.
- 41. Demo
- 42. ● Balance the load between Form and Flow (between Workspaces/Automations) ● Create a structure that is flexible and helps your project to grow ● Document authoring rules that can’t be inferred from looking at FME Flow ● Don’t overload any one component with too many tasks ● Consider how to handle different payloads as efficiently as possible Things to consider…
- 43. 8 Resources
- 44. Get our Ebook Spatial Data for the Enterprise fme.ly/gzc Guided learning experiences at your fingertips academy.safe.com FME Academy Resources Check out how-to’s & demos in the knowledge base support.safe.com Knowledge Base Webinars Upcoming & on-demand webinars safe.com/webinars
- 45. 9 Next Steps
- 46. We’d love to help you get started. Get in touch with us at [email protected] Experience the FME Accelerator Contact Us A world where data is not just a commodity but a catalyst for real change. fme.safe.com/accelerator Next Steps
- 47. ClaimYour Community Badge & Dive into the new Community! ● Get community badges for watching webinars ● community.safe.com ● Today’s code: 0RGAWX Join the Community today! Next Steps
- 48. Peak of Data Integration Seattle, WA | May 6–8, 2025 ● Register at the best price! ○ Early Bird Pricing live til Feb 7, 2025. ● 3 days of data disruption ○ Be among the first to hear ground breaking updates and launches! ● *New* - Get FME certified for free at the event! ○ 2 exams offered on Wednesday, May 7.
- 49. 10 Q&A
- 50. ThankYou Recap of Next Steps 1 Follow us on LinkedIn! 2 Contact us 3 Experience the FME Accelerator Please fill out our webinar survey