[Remote] AWS Data Engineer

Note: The job is a remote job and is open to candidates in USA. Diverse Lynx is a company seeking an AWS Data Engineer to support their data engineering initiatives. The role involves developing and managing data pipelines, implementing event-driven architectures, and optimizing cloud infrastructure on AWS.

Responsibilities

• Develop a comprehensive plan for migrating near real-time fraud detection campaigns from on-premises systems to AWS
• Design and implement event-driven architectures to process inbound dialer data (fraud events) using services such as Amazon EventBridge, Kafka, Kinesis Data Streams, and Kinesis Firehose
• Build and manage scalable data pipelines using AWS Glue (ETL jobs, Crawlers), PySpark, and Python for data ingestion, transformation, and processing
• Configure and manage Glue Crawlers to automatically Client schemas and update the Data Catalog
• Store and optimize data using Parquet format and enable analytics through Amazon Athena for efficient querying
• Develop integrations between Customer Profiles and messaging platforms to automatically trigger profile updates and downstream processes
• Implement automation to trigger fraud-related outbound calls based on updates in customer profiles
• Design and orchestrate workflows using AWS Step Functions to manage complex processing pipelines
• Provision and manage cloud infrastructure using Terraform (Infrastructure as Code)
• Optimize system architecture for scalability, reliability, cost-efficiency, and ensure data integrity and security
• Conduct end-to-end testing of the entire framework to validate functionality, performance, and reliability
• Deploy, automate, and manage resources using CI/CD pipelines
• Continuously monitor system performance and implement optimizations post-deployment
• Maintain detailed documentation of architecture, workflows, and operational processes

Skills

• Strong expertise in AWS services including: Lambda, S3, EventBridge, Kinesis (Data Streams & Firehose), Glue (ETL + Crawlers), Step Functions, Amazon Connect, Athena, Macie
• Proficient in Python
• Proficient in PySpark
• Experience in Glue Crawlers for schema discovery and cataloging
• Experience in Parquet-based data storage
• Experience in building scalable data pipelines
• Strong understanding of event-driven architectures (Pub/Sub model)
• Hands-on experience with Terraform (Infrastructure as Code)
• Familiarity with CI/CD tools and automation pipelines
• Experience with messaging platforms like Kafka and Amazon EventBridge
• Experience in designing real-time data processing systems
• Experience in using Glue Crawlers + Athena for data lake architectures

Benefits

• Benefits

Company Overview

Diverse Lynx is a WBENC- and NMSDC-certified partner, helping organizations turn diversity goals into measurable impact through staffing and contingent workforce solutions. It was founded in 2002, and is headquartered in Princeton, New Jersey, US, with a workforce of 1001-5000 employees. Its website is http://www.diverselynx.com.

Company H1B Sponsorship

Diverse Lynx has a track record of offering H1B sponsorships, with 1 in 2024, 1 in 2021. Please note that this does not guarantee sponsorship for this specific role.