Note: The job is a remote job and is open to candidates in USA. Horizon Industries Limited is a dynamic IT and Management Consulting firm based in the Washington, DC area. They are seeking a Databricks Data Engineer to build end-to-end ETL/ELT pipelines and support clients in the data engineering domain.
Responsibilities
- Build end-to-end implementation of multiple ETL/ELT pipelines, demonstrating efficient data transformation and ingest patterns to move raw data from data producers to an enterprise data ecosystem, with a focus on performance and reliability
- Assess and understand the ETL jobs, workflows, BI tools, and reports
- Address technical inquiries concerning customization, integration, enterprise architecture and general feature / functionality of data products
- Experience in crafting database / data warehouse solutions in the cloud (Preferably AWS, Azure, Alternatively GCP)
- Key must have skill sets – Python, SQL, Databricks, AWS Data Services
- Experience with message queuing, stream processing, and highly scalable ‘big data’ data stores
- Experience manipulating, processing, and extracting value from large, disconnected datasets
- Experience manipulating structured and unstructured data for analysis
- Experience with data modeling tools and processes
- Experience aggregating and transforming data from multiple datasets to create data products
- Support an Agile software development lifecycle
Skills
- Ability to hold a position of public trust with the US government
- B.S. in Computer Science or equivalent
- 4+ years' experience in data engineering and big data. At least 2 years' of professional services experience interacting directly with clients
- Big data tools: Hadoop, Spark, Kafka, etc
- Relational SQL and NoSQL databases and experience working with relational databases
- AWS cloud services: EC2, S3, RDS, Glue, Step Functions, Lambda, EMR, DynamoDB, DocumentDB, Redshift, Aurora, Athena
- Data Platforms: Databricks
- Data streaming systems: Batch, Kafka, Storm, Spark-Streaming, etc
- Languages: Python, R, Scala, Go
- Ability to inspect existing data pipelines, discern their purpose and functionality, and re-implement them efficiently in Databricks
- Extensive knowledge of data warehousing concepts and hands-on experience deploying pipelines using Databricks **a must
- Data modeling and database design skills and knowledge of version control
- Excellent verbal and written communication skills
- Experience architecting scalable and fault-tolerant data solutions across Azure, AWS, and Databricks
- Databricks Data Engineer Professional certification a plus
- Preference for candidates with Databricks Professional certifications
Benefits
- A comprehensive benefits package including healthcare (medical, dental, vision and disability)
- A 401k program where you are 100% vested from day one with an employer match after 90 days.
- An Educational Assistance program.
- A Student Loan Repayment Program
- Gym Reimbursement Program.
- Paid Time off
- Dynamics, passionate, multi-disciplinary team of creative minds to work with, and many more.
Company Overview