Lufthansa and Data Analytics. Interview with James Dixon.
“Lufthansa is now able to aggregate and feed data into a management cockpit to analyze collected data for key decision-making purposes in the future. Users get instantly notified of transmission errors, enabling the company to detect patterns on large amounts of data at a rapid speed. There is also an automatic alarm messages sent out to IT product management, and partner airlines are informed of errors right away in the case of transmission errors between different IT systems for passenger data. Lufthansa is now able to comprehensively monitor one of its most important core processes in real-time for quality management: the handover of passenger data between different airlines” – James Dixon.
On the state of the market for Big Data Analytics I have Interviewed James Dixon, co-founder and Chief Geek / CTO, Pentaho Corporation.
Q1. What is In your opinion the expected realistic Market Demand for Big Data analytics?
James Dixon: Big. Until recently it has not been possible to perform analysis of sub-transactional and detailed operational data for a reasonable price-tag. Systems such as Hadoop and the NoSQL repositories such as MongoDB and Cassandra make it possible to economically store and process large amount of data. The first use of this data is often to answer operational and tactical questions. Shortly after that comes the desire to answer managerial and strategic questions, and this where Big Data Analytics comes in. I estimate that 90% of all Big Data repositories will have some form of reporting/visualization/analysis requirement applied to it.
Q2. Aren’t we too early with respect to the maturity of the Big Data Analytics technology and the market acceptance?
James Dixon: We are early, but not too early. There is significant market acceptance in certain domains already – financial services, SaaS application providers, and media companies to name a few. As these initial markets mature we will see common use cases emerge and public endorsements of these technologies, this will help to increase acceptance in other markets. We’ve seen a significant uptake in commercial deals over the last few quarters whereas 2011 was more tire-kicking and exploratory.
Q3. Pentaho has worked with Lufthansa to improve their passenger handling. Could you please tell us more about this? In particular what requirement and technical challenges did you have for this project? And how did you solve them?
James Dixon: Lufthansa needed a solution that would make the core processes of Inter Airline Through Check In (IATCI) accessible, measurable and available for real-time operational monitoring. They also wanted to deliver consolidated management reporting dashboards to inform decision making out of this information. This was implemented by Pentaho’s services organization with onsite training and consulting. Our Pentaho Business Analytics suite was used for the front-end for real-time data analysis and report generation. In the back-end, Pentaho Data Integration (aka Kettle) retrieves, transforms and loads the message data streams into the data warehouse on a continuous basis.
Q4. And what results did you obtain so far?
James Dixon: Lufthansa is now able to aggregate and feed data into a management cockpit to analyze collected data for key decision-making purposes in the future. Users get instantly notified of transmission errors, enabling the company to detect patterns on large amounts of data at a rapid speed. There is also an automatic alarm messages sent out to IT product management, and partner airlines are informed of errors right away in the case of transmission errors between different IT systems for passenger data. Lufthansa is now able to comprehensively monitor one of its most important core processes in real-time for quality management: the handover of passenger data between different airlines. With Pentaho, Lufthansa is now instantly aware if they are dealing with a single occurrence of an error or if there is a pattern. They can immediately take action in order to minimize the impact on their passengers.
Q5. What is special about Pentaho’s big data analytic platform? How does it differ with respect to other vendors?
James Dixon: We have an end-to-end offering that encompasses data integration/orchestration across Big Data and regular data stores/sources, data transformation, desktop and web-based reporting, slice-and-dice analysis tools, dash boarding, and predictive analytics. Very few vendors have the breadth of technology that we do, and those that do are mainly pushing hardware and services. We enable the creation of hybrid solutions that allow companies to use the most appropriate data storage technology for every part of their system – we don’t force you to load all your data into Hadoop, for example. From an architecture perspective our ability to run our data integration engine inside of MapReduce tasks on the data nodes is a unique capability. And we provide analytics directly on top of big data tech that gives users instant results via our schema-on-read approach – you don’t have to predefine ETL or Schemas or Data Marts – we do it on the fly.
Q6. What are the technical challenges in creating and viewing Analytics on the iPad?
James Dixon: The navigation concepts are different on mobile devices, so the overall user experience of the analysis software needs to be adapted for the iPad. Vendors need to be sensitive to the interaction techniques that the touch screens provide. We have changed the way that all of our end-user web-based interfaces work so that experience on the iPad is similar. It is possible to allow ad-hoc analysis and content authoring on the iPad, and Pentaho provides that with our recent V4.8 release.
Q7. Big Data and Mobile: what are the challenges and opportunities?
James Dixon: There are some use cases that are easy to identify. Report bursting to mobile and non-mobile devices is a technique that is easy to do today. Real-time analysis of Big Data combined with the alerting and notification capability of mobile devices is an interesting combination.
Q8. How are you supposed to view complex analytics with the limited display of a mobile phone?
James Dixon: Even with a desktop computer and a large monitor, analysis of Big Data requires lots of aggregation and/or lots of filtering. If you could display all the raw data from a Big Data repository, you would not be able to interpret it. As the display gets smaller the amount of aggregation and filtering has to go up, and the complexity has to come down. It is possible to do reasonably complex analysis on a tablet, but it is certainly a challenge on the smaller devices.
Q9. What are the main technical and business challenges that customers face when they want to use Cloud analytics deployments?
James Dixon: Moving large amounts of data around is a hurdle for some organizations. For this reason cloud analytics is not very appealing to companies with established data centers. However young companies that exclusively use hosted applications do not have their data on-premise. As these companies grow and mature we will see the market for cloud analytics increase.
Q10. Pentaho has announced in July this year a technical integration of their analytics platform with Cloudera. What is the technical and business meaning of this? What are results obtained so far?
James Dixon:We are working closely with Coudera on a technical and business level. For example we worked with Cloudera to test their new Impala database with Pentaho’s analytics, so that we could demo the integration on the day that Impala was announced. We also have joint marketing campaigns and sales field engagement, as customers of Cloudera find tremendous benefit in engaging with Pentaho and vice-versa. Our tech makes it much easier and 20x faster to get Hadoop productive so their customers gravitate to us naturally.
James Dixon, Founder and Chief Geek / CTO, Pentaho Corporation
As “Chief Geek” (CTO) at Pentaho, James Dixon is responsible for Pentaho’s architecture and technology roadmap. James has over 15 years of professional experience in software architecture, development and systems consulting. Prior to Pentaho, James held key technical roles at AppSource Corporation (acquired by Arbor Software which later merged into Hyperion Solutions) and Keyola (acquired by Lawson Software). Earlier in his career, James was a technology consultant working with large and small firms to deliver the benefits of innovative technology in real-world environments.
- Big Data: Challenges and Opportunities.
Roberto V. Zicari, October 5, 2012.
Abstract: In this presentation I review three current aspects related to Big Data:
1. The business perspective, 2. The Technology perspective, and 3. Big Data for social good.
Presentation (89 pages) | Intermediate| English | DOWNLOAD (PDF)| October 2012|
- ODBMS.org: Big Data and Analytical Data Platforms.
Blog Posts | Free Software | Articles | PhD and Master Thesis |
You can follow ODBMS.org on Twitter : @odbmsorg.