استخدام data engineer

توضیحات آگهی

About the Role
We are hiring a Senior Data Engineer to build and run our open source Lakehouse platform on Apache Iceberg and S3 compatible object storage serving tens of terabytes across thousands of tables
The platform separates storage from compute and runs CDC ingestion transformation ad hoc analysis and BI serving on scalable engines mostly on Kubernetes It serves hundreds of data analysts and BI engineers
This is a hands on high ownership role You will design core pipelines and data models set engineering standards and shape how the platform scales
Key Responsibilities
Ingestion and storage
Build and run streaming pipelines Spark Structured Streaming or Flink that consume CDC events from Kafka and merge them into lakehouse tables reliably and idempotently
Design large Iceberg tables partitioning and partition evolution file sizing compaction snapshot expiry and orphan file cleanup
Build real time pipelines with Flink Kafka Streams or ksqlDB for joins aggregations and enrichment
Set lifecycle and retention policies on object storage
Catalog governance and versioning
Run the Iceberg REST catalog shared by all engines
Design namespaces warehouses and access policies so each team gets isolated governed access
Use branches tags and time travel for safe changes backfills and fast rollback
Transformation and data modeling
Build layered tested and documented models with dbt or Spark Declarative Pipelines
Design dimensional and aggregate models in OLAP engines for fast dashboards
Own data contracts and quality checks so analysts can trust every published dataset
Serving and performance
Tune query and OLAP engines for high concurrency materialized views caching resource groups and workload isolation
Work with BI engineers to expose clean datasets and metrics in BI tools
Platform and Kubernetes
Run data services on Kubernetes with Helm charts and operators
Plan capacity scaling and upgrades for stateful and compute workloads
Engineering excellence
Set standards for code review CI CD testing data quality checks and SLAs
Build monitoring and alerting for pipeline freshness data quality and cluster health
Mentor engineers and analysts on Spark dbt table formats and distributed query engines
Requirements
4 years in data engineering including 2 years building distributed data platforms in production
Strong Apache Spark experience PySpark Scala or Java including Structured Streaming and tuning large jobs Experience with Spark 4 and Declarative Pipelines is a plus
Hands on production experience with an open table format Iceberg Hudi Paimon or Delta Lake MERGE schema and partition evolution compaction and table maintenance
Stream processing with Apache Flink Kafka Streams or ksqlDB state windows watermarks and exactly once delivery
Solid Kafka and Kafka Connect knowledge and experience processing CDC events ideally from Debezium with correct ordering deletes and late data
Experience with Iceberg catalogs REST JDBC or Hive Metastore and with designing access control
dbt or SQLMesh in production model layering incremental models tests macros and documentation
At least one distributed query or OLAP engine in depth Trino Presto StarRocks Doris Druid Pinot or similar
Expert SQL and strong data modeling skills dimensional modeling slowly changing dimensions aggregate design
Strong Kubernetes experience running stateful and data workloads Helm charts and operators
Comfort with S3 compatible object storage Linux and Docker
Working knowledge of Git CI CD and an orchestrator such as Airflow or Dagster
Clear communication and experience mentoring engineers and working with analysts
شرکت دیجی کالا Digikala
دسته بندی وب برنامه نویسی و نرم افزار
نحوه همکاری تمام وقت
نوع همکاری حضوری
مدرک تحصیلی کارشناسی
سابقه کار سه تا شش سال
حقوق توافقی
جنسیت مهم نیست
مهارت ها Python SQL Data engineer
شهر تهران تهران