Senior Data Engineer

SENIOR DATA ENGINEER

(Lisbon office x1 day a week *OR* fully remote in Portugal)

Xcede are delighted to be working with an organisation dedicated to using state-of-the-art AI technologies to drive continuous improvements in our products and business strategies. Their AI division, encompassing teams focused on research, platforms, and product innovation, serves as the backbone of this mission. They deliver AI-powered tools for various purposes, including fraud detection, personalized recommendations for gaming, AI-enabled co-pilots, and CRM enhancements.

Our client cultivate teams that are autonomous, collaborative, and self-directed. As a member of these teams, you'll work alongside with Data Scientists, ML Engineers, Data Engineers, Product Analysts, Agile Delivery Leads, and Product Owners to bring impactful AI solutions to life.

As a Senior Data Engineer, your work will center on designing and implementing robust data infrastructure to transform raw data into high-quality features used in machine learning models.

Responsibilities:

• Build and optimize scalable pipelines for processing large datasets in distributed production environments.
• Create Python-based microservices that deliver processed data and features for production use.
• Develop and maintain internal systems that support CI/CD workflows, experimental tracking, and versioning of data.
• Apply data quality measures to ensure accuracy and reliability in data processing workflows.

Requirements:

• Demonstrated expertise in developing software with Python in large-scale, high-performance environments.
• Strong proficiency in designing and managing models for both structured and unstructured datasets.
• Practical experience utilizing Apache Kafka for distributed data streaming applications.
• Solid familiarity with cloud-based ecosystems and infrastructure solutions.
• Advanced knowledge of distributed system design and architecture.
• Comprehensive hands-on experience with Apache Spark and PySpark for data processing

Preferred Qualifications:

• Skilled in leveraging the Databricks platform for data and analytics tasks.
• Experience working with technologies for processing real-time and streaming data.
• Familiarity with NoSQL database systems like Neo4j, Redis, etc