Vacancy caducado!
Project OverviewThe details of the different data integrations that will be enabled over the duration of the project are outlined below.1) Datasets informationa) 6 plus Westchester - 4 dataset files for each customerb) Big 5 government - 4 dataset files for each customerc) energy projects - 4 dataset files2) Reviewing and confirming data requirements (schema, format, frequency, etc.) for the different data flows3)Provisioning the project on client Enterprise Data Platform4)Initial one-time data profiling of datasets to provide visibility on Data Quality and help identify required Data Quality rules5) Storing the processed data and merging with historical data set6) Enabling a nightly automated process to ingest the data, refresh the tables and create the required output files7) Setting up monitoring jobs using client's monitoring service to send email alerts on any job / validation failures8) Performing testing and UAT activities to validate input and output tables and columnsJob Functions & Responsibilities
- Collaborate with all developers and business users to gather required data and execute all ETL programs and scripts on systems and implement all data warehouse activities.
- Collaborate and perform tests and provide update to all ETL activities within schedule and provide support to all large data volumes and assist in data processing.
- Documents all technical and system specifications documents for all ETL processes and perform unit tests on all processes and prepare required programs and scripts.
- Analyze and interpret all complex data on all target systems and analyze and provide resolutions to all data issues and coordinate with data analyst to validate all requirements.
- Perform tests and validate all data flows and prepare all ETL processes according to business requirements and incorporate all business requirements into all design specifications.
- Develop all data warehouse models and prepare reports for all meta data integration into systems and draft all ETL scripts and prepare required reports for all end users.
- Provide support to all ETL schedule and maintain compliance to same and develop and maintain various standards to perform ETL codes and maintain an effective project life cycle on all ETL processes.
- Document all test procedures for systems and processes and coordinate with business analysts and users to resolve all requirement issues and maintain quality for same.
- Monitor all business requirements and validate all designs and schedule all ETL processes and prepare documents for all data flow diagrams.
- Pentaho Data Integrator tool (PDI)
- Python & Java scripting
- Write SQL complex queries and Store Procedures
- PySpark language
- AWS RDS, API and S3 buckets
Vacancy caducado!