Tóm lược
- Yêu cầu kỹ thuật:
- AWS ,
- Apache Kafka ,
- Apache Airflow ,
- MS SQL ,
- Apache Spark
Mô tả công việc
Tóm tắt công việc
We are seeking a Senior Data Engineer to play a pivotal role in building our data platform from the ground up. You will be responsible for designing, developing, and deploying a scalable, secure, and open-source based data platform on the AWS cloud. This is a unique opportunity to shape the future of our data infrastructure and empower data-driven decision making across the organization.
- Design and implement a robust data platform architecture on AWS, leveraging open-source technologies whenever possible.
- Evaluate and select appropriate open-source tools and services for data ingestion, transformation, storage, and analytics.
- Develop and maintain data pipelines for efficient movement and processing of large datasets.
- Ensure data quality and consistency throughout the data lifecycle.
- Implement data security best practices to protect sensitive information.
- Collaborate with data scientists, analysts, and other stakeholders to understand data needs and translate them into technical requirements.
- Automate data platform tasks and processes for improved efficiency and reliability.
- Monitor and troubleshoot data platform performance to identify and resolve issues proactively.
- Stay up-to-date on the latest trends and technologies in the data engineering space.
- Document data platform design, architecture, and processes for future reference.
Yêu cầu công việc
- Proven experience (4+ years) as a Senior Data Engineer or similar role.
- Strong understanding of data engineering principles and best practices.
- Expertise in designing and building scalable data pipelines.
- Proficient in working with open-source data technologies (e.g., Apache Spark, Apache Airflow, Kafka, etc.).
- Experience with cloud platforms, preferably AWS.
- Solid knowledge of SQL and experience with relational databases.
- Excellent written and verbal communication skills.
- Ability to work independently and as part of a cross-functional team.
Bonus Points
- Experience with machine learning and data science workflows.
- Experience with data governance and data security frameworks.
- Experience with DevOps principles and practices.
- Experience building data platforms for similar industries.
Ngôn ngữ
-
English
Nói: Intermediate - Đọc: Intermediate - Viết: Intermediate
Yêu cầu kỹ thuật
- AWS
- Apache Kafka
- Apache Airflow
- MS SQL
- Apache Spark
NĂNG LỰC
- Communication Skills
- Working Independently
- Teamwork
Thông tin doanh nghiệp
Got It is a library for Technology and Entrepreneurship.
It’s an on-demand marketplace that instantly connects you, a smart phone user, to a live, paid expert for a 10-minute chat session about a problem you want to solve or a topic you want to learn about. The goal is to enable people to teach each other about topics in which they are knowledgeable.
You will work with the best software engineers from both Vietnam and Silicon Valley to build the next generation of Got It platform that will handle a massive number of sessions between askers and experts in multiple verticals. In addition, you will help implement and integrate advanced machine learning algorithms and technologies to make the platform smart, efficient, and highly personalized.