Query-centric failure recovery for distributed stream processing engines

Research output: Contribution to journalConference articleResearchpeer-review

Abstract

Correlated failures that usually involve a number of nodes failing simultaneously have significant effect on systems' availability, especially for streaming applications that require real-Time analysis. Most state-of-The-Art distributed stream processing engines focus on recovering individual operator failure. By analyzing the existing recovery techniques, we identify the challenges and propose a fault-Tolerance framework that can tolerate both individual and correlated failures with minimum overhead during the system's normal execution. Our progressive and query-centric recovery paradigm carefully schedules the recovery of failed operators based on the current availability of resources, such that the outputs of queries can be recovered as early as possible. We also formulate the new problem of recovery scheduling under correlated failures and design algorithms to optimize the recovery latency with a performance guarantee.

Original languageEnglish
JournalProceedings - International Conference on Data Engineering
Volume2018
Pages (from-to)1280-1283
Number of pages4
ISSN1084-4627
DOIs
Publication statusPublished - 24 Oct 2018
Event34th IEEE International Conference on Data Engineering, ICDE 2018 - Paris, France
Duration: 16 Apr 201819 Apr 2018

Conference

Conference34th IEEE International Conference on Data Engineering, ICDE 2018
Country/TerritoryFrance
CityParis
Period16/04/201819/04/2018

Keywords

  • Correlated Failure
  • Distributed Stream Processing
  • Fault Tolerance

Cite this