Logo

Batch Processing

Pasted image 20241030174951.png

[It involves:]

  1. Collecting and storing data over a period of time (hours, days, or even weeks).
  2. Processing this data in bulk at scheduled intervals.
  3. Producing some output data.

[Full Process]

  1. Data Collection (File, Buffer, Warehouse, DB)
  2. Pre-Process
  3. Execution -> Usually in parts
  4. Post Process (mark ACK)

Can Use: Hadoop (Map Reduce) | AWS Batch (AWS) | CRON (Custom)

Stream Processing

Pasted image 20241030175127.png

[Full Process]

  1. Ingestion (Kafka Topics, Buffer)
  2. Processing: Fillter -> Aggregate -> Windowing (stream - batching)

Can Use: Apache Kafka / Flink | AWS Kinesis

© 2025 All rights reservedBuilt with DataHub Cloud

Built with LogoDataHub Cloud