Result: Justin: Hybrid CPU/Memory Elastic Scaling for Distributed Stream Processing ⋆

Title:
Justin: Hybrid CPU/Memory Elastic Scaling for Distributed Stream Processing ⋆
Authors:
Schmitz, Donatien, Rosinosky, Guillaume, Rivière, Etienne
Contributors:
Université Catholique de Louvain = Catholic University of Louvain (UCL), Laboratoire des Sciences du Numérique de Nantes (LS2N), Institut National de Recherche en Informatique et en Automatique (Inria)-Centre National de la Recherche Scientifique (CNRS)-IMT Atlantique (IMT Atlantique), Institut Mines-Télécom [Paris] (IMT)-Institut Mines-Télécom [Paris] (IMT)-NANTES UNIVERSITÉ - École Centrale de Nantes (Nantes Univ - ECN), Nantes Université (Nantes Univ)-Nantes Université (Nantes Univ)-Nantes université - UFR des Sciences et des Techniques (Nantes univ - UFR ST), Nantes Université - pôle Sciences et technologie, Nantes Université (Nantes Univ)-Nantes Université (Nantes Univ)-Nantes Université - pôle Sciences et technologie, Nantes Université (Nantes Univ), Software Stack for Massively Geo-Distributed Infrastructures (STACK), Centre Inria de l'Université de Rennes, Institut National de Recherche en Informatique et en Automatique (Inria)-Institut National de Recherche en Informatique et en Automatique (Inria)-Orange S.A.-Laboratoire des Sciences du Numérique de Nantes (LS2N), Nantes Université (Nantes Univ)-Centre National de la Recherche Scientifique (CNRS)-IMT Atlantique (IMT Atlantique), IMT Atlantique (IMT Atlantique), Institut Mines-Télécom [Paris] (IMT), Win2Wal GEPICIAD (Belgium), Daniel Balouek, Ibéria Medeiros, GRID5000
Source:
DAIS 2025 - 25th International Conference on Distributed Applications and Interoperable Systems. :1-17
Publisher Information:
CCSD, 2025.
Publication Year:
2025
Collection:
collection:CNRS
collection:INRIA
collection:EC-NANTES
collection:INRIA-RENNES
collection:INRIA_TEST
collection:GRID5000
collection:UNAM
collection:TESTALAIN1
collection:INRIA2
collection:LS2N
collection:LS2N-STACK
collection:INRIA-RENGRE
collection:INSTITUTS-TELECOM
collection:NANTES-UNIVERSITE
collection:NANTES-UNIV
collection:SLICES-FR
Subject Terms:
Distributed Stream Processing, Resource Management, Elastic Scaling, Hybrid Scaling, Apache Flink, [INFO.INFO-DC]Computer Science [cs], Distributed, Parallel, and Cluster Computing [cs.DC]
Subject Geographic:
Lille, France
Original Identifier:
HAL: hal-05081993
Document Type:
Conference conferenceObject<br />Conference papers
Language:
English
Access URL:
https://hal.science/hal-05081993
https://hal.science/hal-05081993v1/document
https://hal.science/hal-05081993v1/file/paper.pdf
Rights:
info:eu-repo/semantics/OpenAccess
URL: http://hal.archives-ouvertes.fr/licences/copyright/
Accession Number:
edshal.hal.05081993v1
Database:
HAL

Further Information

Distributed Stream Processing (DSP) engines analyze continuous data via queries expressed as a graph of operators. Auto-scalers adjust the number of parallel instances of these operators to support a target rate. Current auto-scalers couple CPU and memory scaling, allocating resources as one-size-fits-all packages. This contrasts with operators' high diversity of requirements. We present Justin, an auto-scaler that enables hybrid CPU and memory scaling of DSP operators. Justin monitors both CPU usage and the performance of operators' storage operations. Its mechanisms enable finegrain memory allocation for tasks upon a query reconfiguration. The Justin policy identifies individual operators' memory pressure and decides between adjusting parallelism and/or memory assignment. We implement Justin in Apache Flink, extending the Flink Kubernetes Operator and the DS2 CPU-only auto-scaler. Using the Nexmark benchmark, our evaluation shows that Justin identifies suitable resource allocation in as many or fewer reconfiguration steps as DS2 and supports a target rate with significantly fewer CPU and memory resources.