Data Engineer
Role Overview
This mid-level Data Engineer role involves designing and building scalable data pipelines using technologies like Kafka, Scylla, PostgreSQL, and MongoDB to provide actionable insights for business strategy. The engineer will collaborate with cross-functional teams, optimize data architecture, and ensure data integrity through rigorous quality checks and ETL frameworks. Working in an Agile Scrum environment, the hire will impact the company by enabling seamless data integration and driving operational efficiency through advanced data solutions.
Perks & Benefits
This is a fully remote position, offering flexibility in work location with likely asynchronous collaboration to accommodate different time zones. The role emphasizes career growth through exposure to modern cloud-based data technologies, DevOps practices, and opportunities to build LLM-based applications. The company culture values innovation, strong communication, and collaboration with leadership to deliver tailored business solutions.
Full Job Description
Key Responsibilities :
1. Design & Develop Scalable Data Pipelines: Architect and build efficient data ingestion pipelines that empower the business with timely and actionable insights to drive strategy and operations.
2. Collaborate with Cross-Functional Teams: Work closely with technical and business teams to gather requirements and translate them into technical specifications and solutions.
3. Optimize Data Architecture: Continuously refine and improve the performance, reliability, and scalability of data pipelines to ensure seamless integration of data across systems.
4. Ensure Data Integrity: Perform rigorous data quality checks and uphold best practices to maintain the accuracy, consistency, and integrity of data.
5. Build ETL Solutions: Develop ETL frameworks to collect, transform, and integrate data from diverse sources such as Kafka, Scylla, PostgreSQL, MongoDB, APIs, and various file formats.
6. Adopt Best Practices: Implement best-in-class engineering practices around reporting and analysis, ensuring data integrity, testing, validation, and comprehensive documentation.
Basic Qualifications :
1. Bachelor’s degree in Computer Science, Information Systems, or a related technical field, or equivalent work experience.
2. 3+ years of hands-on experience with cloud-based data technologies, including message queues, event grids, relational databases, NoSQL databases, data warehouses, and big data technologies (e.g., Spark).
3. Proficiency in Spark (Java, Python, SQL): Expertise in developing and optimizing Spark-based applications for large-scale data processing.
4. Advanced SQL Skills: Ability to create complex queries, manage database structures, and ensure optimal performance.
5. Experience with Docker and Kubernetes: Familiarity with deploying applications using modern containerisation and orchestration tools.
6. DevOps and CI/CD: Solid understanding of modern DevOps practices, including Git, continuous integration, and continuous deployment pipelines.
7. Agile Development Experience: Comfortable working in an Agile environment, particularly using Scrum methodology.
Preferred Qualifications
1. API Frameworks & OOP: Experience with API frameworks and object-oriented programming to integrate services and improve data workflows.
2. Business Acumen: Ability to collaborate closely with leadership to deliver innovative solutions tailored to evolving business needs.
3. Strong Communication Skills: Excellent verbal and written communication abilities, with a knack for explaining complex technical concepts to both technical and non-technical stakeholders.
4. Ability to leverage tools like langchain, llama index ..etc to build llm based applications.
Similar jobs
Found 6 similar jobs
