No Repetition: Fast and Reliable Sampling with Highly Concentrated Hashing

Anders Aamand, Debarati Das, Evangelos Kipouridis, Jakob B.T. Knudsen, Peter M.R. Rasmussen, Mikkel Thorup

Research output: Contribution to journalConference articleResearchpeer-review

1 Citation (Scopus)
85 Downloads (Pure)

Abstract

Stochastic sample-based estimators are among the most fundamental and universally applied tools in statistics. Such estimators are particularly important when processing huge amounts of data, where we need to be able to answer a wide range of statistical queries reliably, yet cannot afford to store the data in its full length. In many applications we need the sampling to be coordinated which is typically attained using hashing. In previous work, a common strategy to obtain reliable sample-based estimators that work within certain error bounds with high probability has been to design one that works with constant probability, and then boost the probability by taking the median over r independent repetitions. Aamand et al. (STOC’20) recently proposed a fast and practical hashing scheme with strong concentration bounds, Tabulation-1Permutation, the first of its kind. In this paper, we demonstrate that using such a hash family for the sampling, we achieve the same high probability bounds without any need for repetitions. Using the same space, this saves a factor r in time, and simplifies the overall algorithms. We validate our approach experimentally on both real and synthetic data. We compare Tabulation-1Permutation with other hash functions such as strongly universal hash functions and various other hash functions such as MurmurHash3 and BLAKE3, both with and without resorting to repetitions. We see that if we want reliability in terms of small error probabilities, then Tabulation-1Permutation is significantly faster.

Original languageEnglish
JournalProceedings of the VLDB Endowment
Volume15
Issue number13
Pages (from-to)3989-4001
ISSN2150-8097
DOIs
Publication statusPublished - 2022
Event48th International Conference on Very Large Data Bases, VLDB 2022 - Sydney, Australia
Duration: 5 Sep 20229 Sep 2022

Conference

Conference48th International Conference on Very Large Data Bases, VLDB 2022
Country/TerritoryAustralia
CitySydney
Period05/09/202209/09/2022

Bibliographical note

Publisher Copyright:
© 2022, VLDB Endowment. All rights reserved.

Cite this