Batch data flow chart

외부 데이터(API, 웹스크래핑) → EMR(Spark) → S3 (Data Lake) → EMR(Spark) → Redshift (Data Warehouse) → EMR(Spark) → postgreSQL (production DB)

Real-time Data flow chart

앱 생성 데이터(UserInfo, Log) → Kafka → Kinesis Firehose → S3 (Data Lake) → Redshift (Data Warehouse) → postgreSQL (production DB) → Web (Spring Boot) → User

2023-09-03 배치 데이터 로직