Data Engineer
TFG
Western Cape, South Africa
Salary not specified - Full-time
Job Description
About the Role
The Data Engineer will be responsible for designing, implementing, and maintaining the data infrastructure necessary for our Insights team to analyze data and derive valuable insights. The Data Engineer will work with Engineering Leads, Data Architects and other stakeholders to ensure that our data is properly sourced, stored, and accessible for analysis. They will also contribute to our mission by extracting maximum predictive value from data assets.
Responsibilities
- Design, develop, test and maintain data pipelines to extract, transform, and load (ETL) data from various sources into a data warehouse or data lake
- Implement data quality and validation checks to ensure that data is accurate, complete and consistent.
- Optimize and scale data processing and storage to handle large volumes of data and high concurrency.
- Troubleshoot and resolve data pipeline issues and perform root cause analysis
- Develop and maintain documentation for data pipelines and processes.
- Building and maintaining data pipelines using lakehouse architecture (medallion)
- Extracting data from multiple sources, transforming it, and loading it into the target database
- Monitoring and troubleshooting ETL jobs to ensure they complete successfully and on time.
- Design and implement data models and database schemas for storing and querying data.
- Optimize database schema and indexing for efficient data retrieval
- Modernise legacy data environments, migrating from on-premises solutions to cloud-native platforms such as Databricks
- Ensure database security, availability, and scalability
- Perform database tuning and optimization to improve query performance
- Develop and maintain data connectors and integrations with various data sources (e.g., databases, APIs, flat files)
- Implement data streaming and real-time data processing technologies.
- Designing and implementing complex reports that meet business requirements
- Developing dashboards and visualizations using tools like Pyramid or other data visualization tools
- Develop and maintain BI reports and dashboards
- Analysing data to identify trends, patterns, and insights
- Providing insights to business stakeholders based on the analysis of data
- Ensure compliance with data privacy regulations according TFG’s Data privacy policies.
- Develop and maintain data governance policies, procedures, and standards
- Ensure that data access and authorization policies are in place and adhered to
- Monitor and audit data access to ensure data security and prevent data breaches
- Implement disaster recovery and business continuity plans for data infrastructure
- Work collaboratively with Engineering Leads, Architects, Analysts, Scrum Masters and other stakeholders to understand data requirements and provide data solutions.
- Communicate data pipeline issues, status, and progress to stakeholders and project teams.
- Participate in cross-functional meetings and initiatives to ensure alignment with data strategies and goals
- Must have at least 3 years of relevant IT experience or related qualification.
- 3+ years experience with Data pipeline design and development experience.
- Proficiency in Python and/or SQL.
- RDBMS experience in technology such as SQL Server, MySQL and PostgreSQL.
- Ability to use common scripting languages such as Powershell, Bash, or Python for automation.
- Experience using unit and integration tests.
- Experience working with production BI environments and tools such as PowerBI, Tableau or Pyramid Analytics.
- Strong analytical/problem solving skills.
Requirements
- Databricks Certified Data Engineer Associate certification.
- Hands-on experience with Azure DevOps, Git, CI/CD practices, Docker and containerization
- Exposure to .net entity framework.
- Exposure to Databricks, Delta Lake and PySpark.
- Experience with data streaming or distributed processing technologies such as Kafka or Hadoop.
- Manages Complexity - interprets and simplifies complex and contradictory information when resolving organisational problems
- Strategic Mindset - thinks and plans strategically, focusing on the long-term goals and objectives of the organisation
- Tech Savvy - leverages new technology to enhance productivity, improve problem solving, and support business growth
- Cultivates Innovation - creates an environment that fosters and nurtures a culture of creativity which drives success
- Drives Engagement - inspires, motivates and empowers individuals to go above and beyond for the benefit of the team and the organisation
Preferred Qualifications
- Experience in Azure data technologies; a relevant Azure certification would be advantageous
- Experience delivering data projects in Retail, Finance, Customer or HR domains would be advantageous
- Experience mentoring junior engineers would be advantageous
