Data Engineer - Kafka / PySpark / Hadoop Job at Acestack, Ontario, CA

WnM2UjB3ejBFV0M0aEU5Zit0TEtZVUtjQUE9PQ==
  • Acestack
  • Ontario, CA

Job Description

  • Job Title: Data Engineer Kafka / PySpark / Hadoop
  • Location: Toronto, ON
  • Work Model: Onsite
  • Job Type: Full Time (FTE)
Job Description

We are seeking an experienced Data Engineer with strong hands-on expertise in Kafka, PySpark, Python, and Hadoop to design, develop, and support scalable batch and real-time data pipelines. The ideal candidate will have strong experience working with large-scale distributed data processing environments and enterprise data integration solutions.

Key Responsibilities
  • Design, develop, and maintain scalable batch and real-time data pipelines .
  • Develop data processing applications using Python and PySpark/Apache Spark .
  • Build and support Kafka-based data ingestion and streaming pipelines .
  • Work with Hadoop and related technologies to process large volumes of data.
  • Develop and maintain ETL/ELT pipelines for data ingestion, transformation, cleansing, and integration.
  • Perform data validation, reconciliation, and quality checks.
  • Troubleshoot pipeline failures, data discrepancies, and performance issues.
  • Optimize Spark/PySpark jobs and SQL queries for performance and scalability.
  • Monitor data pipelines and resolve production issues.
  • Collaborate with data architects, developers, analysts, and business teams.
  • Participate in Agile development, testing, deployment, and production support activities.
Required Skills
  • Strong hands-on experience with Python for data engineering and automation.
  • Strong expertise in PySpark / Apache Spark .
  • Hands-on experience with Apache Kafka for real-time data ingestion and streaming.
  • Strong experience with the Hadoop ecosystem and distributed data processing.
  • Strong SQL skills and experience working with large datasets.
  • Experience developing and maintaining ETL/ELT data pipelines .
  • Strong understanding of distributed computing and data processing concepts.
  • Experience with data ingestion, transformation, cleansing, and integration.
  • Strong troubleshooting and performance optimization skills.
Good to Have
  • Hive
  • Databricks
  • AWS, Azure, or GCP
  • Git and CI/CD
  • Unix/Linux
  • Airflow or Autosys
  • Relational and NoSQL databases

Job Tags

Full time

Similar Jobs

Confidential employer

Hotel Housekeeper Job at Confidential employer

A Hotel Housekeeper is responsible for maintaining guest rooms and common areas in a clean, orderly, and welcoming condition. Typical duties include changing linens, making beds, dusting surfaces, vacuuming carpets, cleaning bathrooms, restocking basic supplies, and reporting... 

Commonwealth Health

Clinical Pharmacist Job at Commonwealth Health

 ...Graduate from an accredited school of Pharmacy with BS in Pharmacy or a PharmD. Current licensure as Registered Pharmacist by passing examination of State Board of Pharmacy in Pennsylvania. 6- 8-week training period to become acquainted with specific hospital procedures... 

Cornerstone Caregiving

Overnight Aide Job at Cornerstone Caregiving

We are committed to providing exceptional care to individuals in need. Our caregivers are highly trained and compassionate, with a passion for making a positive impact in the lives of others. With flexible scheduling, caring environment, competitive pay, bonus opportunities...

Hillside Resort

Marina Dockhand Associate Job at Hillside Resort

 ...Job Description Job Title: Marina DockHand & Guest Services Associate Company: Hillside Resort Location: Wentworth, SD (Lake Madison) Job Type: Part-Time / Seasonal (Summer) Job Description: Hillside Resort is looking for a reliable, hardworking Marina... 

Geisinger Health

Biostatistician Job at Geisinger Health

Location: Henry Hood Center For Health Research Shift: Days (United States of America) Scheduled Weekly Hours: 40 Worker Type: Regular Exemption Status: Yes Job Summary: The Biostatistics Core at Geisinger collaborates closely with researchers...