Note: The job is a remote job and is open to candidates in USA. AllStates Consulting Services is seeking a Data Automation Engineer to design and implement innovative data automation solutions for an enterprise-scale Microsoft Azure-based analytics and reporting platform. The role involves building scalable data pipelines, automating workflows, and supporting AI/ML initiatives while collaborating with various stakeholders to optimize cloud-based data environments.
Responsibilities
- Design, develop, and maintain high-performance data pipelines using:
- Azure Data Factory
- Azure Synapse Pipelines
- Apache Spark Notebooks
- Python
- SQL
- Stored Procedures
- Translate business requirements into scalable data engineering and AI-driven solutions
- Continuously improve automation tools for reliability, scalability, and adaptability
- Research and implement AI/ML and Generative AI solutions to automate data processes and eliminate workflow bottlenecks
- Collaborate with implementation specialists, engineering teams, and customers to develop data-driven solutions
- Design and implement data ingestion, transformation, integration, and processing solutions
- Support advanced analytics, reporting, visualization, and AI/ML initiatives
- Implement:
- Data migration
- Data quality
- Data integrity
- Metadata management
- Data security functions
- Monitor, troubleshoot, and optimize data pipeline performance
- Execute ETL performance testing and validate benchmark results
- Analyze:
- Pipeline runtime
- Throughput
- Latency
- Resource utilization
- Participate in performance testing for:
- Azure Data Factory (ADF)
- Azure Synapse
- Databricks
- Support performance tuning activities including:
- Query optimization
- Partitioning
- Indexing
- Validate data consistency and completeness after performance testing
- Collaborate with DevOps and infrastructure teams on compute, memory, and scaling optimization
- Document test results, findings, and recommendations
- Support Agile DevOps processes including Program Increment planning
- Maintain strict versioning and configuration control processes
Skills
- 2+ years of experience with two or more of the following: SQL, T-SQL, MDX/DAX, Python, PySpark
- Experience designing and building ETL and data engineering solutions
- Experience with Azure Data Lake Services
- Experience with Azure Synapse Analytics
- Experience with Azure Data Factory
- Experience with Integration Runtime
- Experience with Microsoft data and BI technologies including: SQL Server, Stored Procedures, SSIS, SSRS, SSAS (Cubes), Power BI
- Experience automating data processes using: Azure CLI, AWS CLI, Bash, PowerShell
- Experience with Azure DevOps Repos
- Experience with GitHub
- Experience with Pipeline versioning
- Experience with Release management
- Experience supporting Production environments
- Experience supporting Development environments
- Experience supporting Testing environments
- Experience supporting Integration environments
- Knowledge of Agile development methodologies
- Strong analytical, troubleshooting, and problem-solving skills
- Ability to support multiple projects simultaneously
- Strong communication and collaboration skills
- Bachelor's degree in: Computer Science, Related technical field
- 2+ years of relevant professional experience
- U.S. Citizenship required
- Ability to successfully obtain and maintain a Public Trust clearance
- Demonstrated commitment to continuous learning and professional development
- Generative AI development experience
- Generative AI for Data Analytics experience
- Microsoft certifications including: Azure Fundamentals, Azure Data Engineer, Power BI, Azure AI
- AWS Certified Data Engineer certification
- Experience with Databricks
- Experience with REST APIs
- Experience with Docker
- Experience with Enterprise ETL toolsets
- Performance tuning experience including: Indexing, Execution plans, Query analytics, Data profiling
- Knowledge of: Data encryption, Cloud virtual networks, Routing, Firewalls, Log Analytics, Monitoring tools
- Experience with ARM templates
- Experience with Bicep templates
- Experience with RBAC access controls
- Data lineage and impact analysis experience using: Microsoft Purview, Synapse Pipeline Tracing
Benefits
- Health insurance
- Health savings account
- Dental insurance
- Vision insurance
- Flexible spending accounts
- Life insurance
- Retirement plan
Company Overview
AllStates Consulting Services offers technical staffing solutions and IT professional services. It was founded in 1945, and is headquartered in Plano, Texas, USA, with a workforce of 201-500 employees. Its website is https://www.allstatesconsulting.net/.