2024-09-28 –, AI/SRE
In today’s data-driven world, the ability to manage, monitor, and optimize large-scale data platforms is crucial for organizations aiming to stay competitive. This talk will delve into the architecture and operations of Air India’s data platform, showcasing how DataOps, observability, and FinOps are leveraged to achieve a robust and efficient big data environment. Attendees will gain insights into best practices and practical approaches for building and maintaining resilient data platforms.
- Introduction to Air India’s Data Platform:
• Overview of the data platform’s architecture.
• Key use cases and data-driven applications.
• Importance of a resilient and scalable data platform for a global airline.- DataOps: Streamlining Data Pipeline Management:
• Definition and principles of DataOps.
• Tools and technologies used for automating data pipelines.
• Strategies for ensuring data quality, consistency, and reliability.
• Case studies on DataOps implementation and its impact on operational efficiency. - Observability: Enhancing Visibility and Monitoring:
• Importance of observability in big data platforms.
• Techniques and tools for monitoring data workflows and system health.
• Implementing real-time alerts and dashboards.
• Ensuring proactive issue detection and resolution.
• Case studies on how observability has improved Air India’s data operations. - FinOps: Optimizing Costs in Big Data Environments:
• Introduction to Financial Operations (FinOps) in the context of big data.
• Strategies for cost management and optimization.
• Tools and frameworks for tracking and controlling expenses.
• Balancing performance with cost efficiency.
• Real-world examples of cost-saving measures implemented at Air India. - Challenges and Lessons Learned:
• Common challenges faced in managing a large-scale data platform.
• Lessons learned from Air India’s journey.
• Best practices for overcoming obstacles and ensuring continuous improvement. - Q&A and Interactive Discussion:
• Engaging the audience with an open floor for questions.
• Encouraging discussion on experiences and solutions from other participants.
- DataOps: Streamlining Data Pipeline Management:
As an Azure Data Platform Architect , I have a passion for developing and implementing innovative data solutions that drive business success. With a deep understanding of the Azure ecosystem and a wealth of experience in designing, building, and maintaining complex data systems