Databricks Data Engineer – Sales Incentive Compensation | New York, NY

Databricks Data Engineer – Sales Incentive Compensation

📍 Location: New York, New York
🏢 Work Arrangement: Hybrid
📄 Contract Length: Six Months
💰 Hourly Rate: Approximately $75 per hour
💼 Employment Type: Contract
🚫 Visa Sponsorship: Not Available
🚚 Relocation Assistance: Not Available
🏦 Industry: Insurance and Financial Services

Build the Data Infrastructure Behind Enterprise Sales Compensation

A leading global organization is seeking an experienced Databricks Data Engineer to design, develop and optimize enterprise data solutions supporting analytics, performance reporting and Sales Incentive Compensation.

This is not purely a pipeline-development position. The successful candidate will combine strong Databricks engineering expertise with an understanding of incentive compensation processes, including quota allocation, attainment calculations, commission processing and payout reporting.

You will work closely with Sales Operations, Finance, Business Intelligence and technology teams to translate complex business rules into scalable, accurate and auditable data solutions.

Why This Opportunity Stands Out

This opportunity will allow you to:

• Own the design and development of enterprise Databricks solutions

• Influence the wider Databricks architecture and technical roadmap

• Build data pipelines supporting business-critical compensation processes

• Work with Delta Lake, Delta Live Tables, Unity Catalog and Databricks Workflows

• Translate complex incentive-plan rules into reliable transformation logic

• Improve automation, integration, data quality and platform performance

• Partner directly with senior Finance, Sales Operations and technology stakeholders

• Deliver solutions that directly influence commission and compensation accuracy

What You’ll Be Doing

Databricks Architecture and Engineering

• Design, develop and implement scalable Databricks-based data solutions

• Build and maintain data pipelines supporting ingestion, transformation, reporting and analytics

• Develop data models, Delta Lake tables, notebooks and reporting datasets

• Take ownership of the overall Databricks architecture

• Ensure new pipelines and enhancements integrate effectively with existing workflows

• Maintain and optimize Databricks workspaces, pipelines and orchestration processes

• Identify opportunities to improve automation, performance and data integration

• Design reliable ETL and ELT processes across multiple enterprise data sources

Sales Incentive Compensation

• Build data pipelines supporting Sales Incentive Compensation processes

• Translate incentive-plan rules into accurate and auditable data transformations

• Support quota allocation and quota-management processes

• Develop logic for attainment tracking and performance calculations

• Support commission calculations and payout processing

• Produce reliable datasets for compensation reporting and reconciliation

• Work closely with Sales Operations and Finance to understand plan structures and business rules

• Ensure calculation logic is transparent, documented and capable of being audited

• Integrate incentive-compensation platforms with enterprise data and analytics solutions

Performance and Data Quality

• Test, debug and optimize Databricks pipelines and notebooks

• Investigate and resolve data-quality, performance and integration issues

• Improve the reliability and efficiency of existing workflows

• Implement appropriate controls and validation processes

• Monitor data pipelines and troubleshoot failures or inconsistencies

• Deliver a high-quality experience for analysts and other data consumers

Stakeholder and Technical Collaboration

• Work with business stakeholders to understand complex data requirements

• Translate functional requirements into scalable technical architectures

• Challenge requirements constructively when a more effective solution is available

• Partner with Finance, Sales Operations, IT and Business Intelligence teams

• Provide technical support to analysts and end users

• Communicate technical concepts clearly to both technical and non-technical audiences

• Manage multiple priorities within a fast-paced enterprise environment

Documentation and Platform Development

• Document data architectures, models, pipelines and configuration decisions

• Produce clear technical specifications and development documentation

• Maintain documentation to support knowledge transfer and long-term platform ownership

• Stay current with Databricks capabilities and platform enhancements

• Identify opportunities to use features such as Unity Catalog, Delta Live Tables and Photon

What You’ll Bring

Essential Databricks and Data Engineering Experience

• At least seven years of professional data-engineering experience

• A minimum of three years of hands-on Databricks experience

• Extensive experience building and managing production Databricks solutions

• Strong knowledge of:

• Delta Lake

• Delta Live Tables

• Unity Catalog

• Databricks Workflows

• Databricks notebooks and workspaces

• Advanced proficiency in Python and/or Scala

• Strong SQL development expertise

• Experience with data ingestion, transformation, ETL/ELT and data modelling

• Experience designing and orchestrating enterprise data pipelines

• Strong understanding of data quality, scalability and performance optimization

Sales Incentive Compensation Experience

• Functional and technical experience with Sales Incentive Compensation processes

• Understanding of incentive-plan design and compensation rules

• Experience supporting quota allocation and management

• Knowledge of sales-attainment calculations

• Experience with commission calculation and payout processing

• Ability to model complex incentive rules within data pipelines and transformation logic

• Experience partnering with Sales Operations and Finance stakeholders

Cloud and Integration Experience

• Experience integrating Databricks with at least one major cloud platform:

• Microsoft Azure

• Amazon Web Services

• Google Cloud Platform

• Experience working with integration and ingestion technologies such as:

• Azure Data Factory

• Azure Event Hubs

• Apache Kafka

• Comparable cloud-based data-integration tools

• Ability to create holistic, end-to-end data solutions rather than isolated pipelines

Professional Capabilities

• Strong stakeholder-management and consulting skills

• Ability to translate complex business requirements into scalable data architectures

• Excellent analytical and problem-solving capabilities

• Strong written and verbal communication skills

• Ability to work effectively with cross-functional teams

• Resourceful and comfortable working with minimal direction

• Strong technical-documentation skills

• Ability to manage competing priorities and deadlines

• Bachelor’s degree in Computer Science, Data Engineering, Information Technology or a related discipline

Preferred Experience

• Experience integrating Databricks with platforms such as:

• Anaplan

• Oracle Incentive Compensation

• Varicent

• Comparable Sales Performance Management platforms

• Databricks Certified Data Engineer Associate certification

• Databricks Certified Data Engineer Professional certification

• Experience using Photon to improve query and workload performance

• Previous experience within insurance, financial services or another large regulated enterprise

• Experience delivering auditable financial or compensation-related data solutions

The Ideal Candidate

The ideal candidate will be an experienced Databricks engineer who understands that compensation data requires exceptional accuracy, traceability and control.

You will be comfortable moving between detailed technical development and strategic conversations with Finance, Sales Operations and technology leaders. You should be willing to challenge requirements constructively, propose stronger architectural solutions and take ownership of delivery from initial design through testing, deployment and production support.

Most importantly, you will be able to combine enterprise data-engineering expertise with a practical understanding of how incentive plans, quotas, attainment calculations, commissions and payouts operate.

Contract Details

• Initial Contract: Six months

• Hourly Rate: Approximately $75 per hour

• Location: New York, New York

• Working Arrangement: Hybrid

• Visa Sponsorship: Not available

• Relocation Assistance: Not available

Ready to Build Business-Critical Data Solutions?

If you have deep Databricks expertise and experience translating complex Sales Incentive Compensation processes into scalable, accurate and auditable data solutions, this position offers the opportunity to make an immediate impact within a major enterprise environment.

Senior Data Engineer - USA, Remote - $110,560 to $155,840

Senior Data Engineer

USA, Remote

$110,560 to $155,840

Job Description

You are a driven and motivated problem solver ready to pursue meaningful work. You strive to make an impact every day & not only at work, but in your personal life and community too. If that sounds like you, then you've landed in the right place.

The Data Science AI Factory team is committed to exploring new ways to use data and analytics to solve business problems.  The team utilizes a variety of data sources, with a strong focus on unstructured and semi-structured text using NLP to enhance outcomes related to claim, underwriting, operations and the customer experience. 

As a Sr. Data Engineer, you will be an established thought leader through close partnerships with expert resources to design, develop, and implement data assets for a wide range of new initiatives across multiple lines of business. The role involves heavy data exploration, proficiency with SQL and Python, knowledge of service-based deployments and APIs, and the ability to discover and learn quickly through collaboration.  There is a need to think analytically and outside of the box while questioning current processes and continuing to build on the individual’s business acumen.

There will be a combination of team collaboration and independent work efforts.  We seek candidates with strong quantitative background and excellent analytical and problem-solving skills. This position combines business and technical skills involving interaction with business customers, data science partners, internal and external data suppliers and information technology partners.

Responsibilities

• Identify and validate internal and external data sources for availability and quality. Work with SME’s to describe and understand data lineage and suitability for a use case.

• Create data assets and build data pipelines that align to modern software development principles for further analytical consumption. Perform data analysis to ensure quality of data assets.

• Create summary statistics/reports from data warehouses, marts, and operational data stores.

• Extract data from source systems, and data warehouses, and deliver in a pre-defined format using standard database query and parsing tools.

• Understand ways to link or compare information already in our systems with new information.

• Perform preliminary exploratory analysis to evaluate nulls, duplicates and other issues with data sources.

• Work with data scientists and knowledge engineers to understand the requirements and propose and identify data sources and alternatives.

• Produce code artifacts and documentation using Github for reproducible results and hand-off to other data science teams.

• Propose ways to improve and standardize processes to enable new data and capability assessment and to enable pivoting to new projects.

• Understand data classification and adhere to the information protection and privacy restrictions on data.

• Collaborate closely with data scientists, business partners, data suppliers, and IT resources.

Experience & Skills

Candidates must have the technical skills to transform, manipulate and store data, the analytical skills to relate the data to the business processes that generates it, and the communication skills to document & disseminate information regarding the availability, quality, and other characteristics of the data to a diverse audience. These varied skills may be demonstrated through the following:

• Bachelor’s degree or equivalent experience in a related quantitative field

• 5 + years experience accessing and retrieving data from disparate large data sources, by creating and tuning SQL queries. Understanding of data modeling concepts, data warehousing tools and databases (e.g. Oracle, AWS, Snowflake, Spark/PySpark, ETL, Big Data, and Hive) 

• Demonstrated ability to create and deliver high quality Python code using software engineering best practices. Experience with object-oriented programming and software development a plus. Proficiency with Github and Linux highly desired.

• Ability to analyze data sources and provide technical solutions. Strong exploratory and problem-solving skills to check for data quality issues.

• Determine business recommendations and translate into actionable steps 

• Self-starter with curiosity and a willingness to become a data expert

• Demonstrate a passion to both learn new skills and lead discovery of the data research 

• Results oriented with the ability to multi-task and adjust priorities when necessary 

• Ability to work both independently and in a team environment with internal customers 

• Ability to articulate and train technical concepts regarding data to both data scientists and partners