استخدام dataops data engineer

توضیحات آگهی

We are looking for a
DataOps Data Engineer to design build and maintain data pipelines and the infrastructure that powers them You will work hands on with distributed data systems such as Qdrant OpenSearch Elasticsearch Apache Flink Apache Spark and Apache Kafka both in Kubernetes based and Docker Compose based environments This role blends Data Engineering with Platform Infrastructure Operations DataOps so you should be comfortable both writing pipeline code and operating tuning and troubleshooting the systems it runs on
Key Responsibilities
Design build and maintain scalable batch and streaming data pipelines
Develop and optimize stream processing jobs using Apache Flink and Apache Spark
Build and manage data ingestion integration flows using Apache Kafka topics partitions consumer groups schema management
Ensure data quality consistency and reliability across pipelines
Operate maintain and fine tune core data platforms Qdrant OpenSearch Elasticsearch Flink Spark and Kafka
Own performance tuning indexing strategy and configuration optimization for search and vector databases OpenSearch Elasticsearch Qdrant
Manage and maintain a Kafka cluster running on both Kubernetes and Docker Compose environments
Monitor system health set up alerting and proactively troubleshoot performance or reliability issues
Fine tune and optimize data related workloads running on Kubernetes resource limits storage configuration Helm chart values etc while the DevOps team owns cluster provisioning and lifecycle management
Work closely with DevOps backend engineers and other data teams to ensure workloads are deployed scaled and configured correctly on shared infrastructure
Troubleshoot data platform related issues and provide timely resolutions
Requirements
Hands on experience with Kafka Flink Spark Elasticsearch OpenSearch Qdrant or a similar vector database
Solid understanding of distributed systems concepts partitioning replication consistency scaling
Experience with Docker Docker Compose for service deployment
Strong scripting programming skills Python and or Bash working knowledge of Java Scala is a plus for Flink Spark jobs
Comfortable with Linux system administration basics and reading logs metrics to debug production issues
Familiarity with monitoring observability stacks Prometheus Grafana or similar
Strong problem solving skills and the ability to work in a fast paced environment
Experience running workloads on Kubernetes Deployments StatefulSets ConfigMaps Helm you don t need to manage the cluster itself but you should be comfortable configuring and tuning what runs on it
Experience with vector search embeddings and how they re used in Qdrant or similar systems
Familiarity with CI CD pipelines and infrastructure config change management
Knowledge of Infrastructure as Code IaC tools like Terraform or Ansible
Experience with open table formats such as Apache Iceberg or Delta Lake Hudi
شرکت دیجی کالا Digikala
دسته بندی وب برنامه نویسی و نرم افزار
نحوه همکاری تمام وقت
نوع همکاری حضوری
مدرک تحصیلی کارشناسی
سابقه کار بیش از شش سال
حقوق توافقی
جنسیت مهم نیست
مهارت ها Develop dataops
شهر تهران تهران