Data Storage Architect

The RDH is the digital center of excellence responsible for the digital transformation of the RCAF. Included in this mission is enabling data-driven decision making for the RCAF through the provision of timely, accurate, and reliable data. As DND produces greater amounts of data each day, there are increasing needs to harness this data and transform it into insights for decision making. The RDH needs to leverage a variety of tools and methods to extract, transform, and load (ETL) data onto DND tools where it can be visualized and analyzed. This process relies heavily on the highly skilled and technical knowledge of storage architects.

Mandatory requirements

• A Bachelor Degree in Information Technology, Computer Science, Computer Engineering, Software Engineering, or a related Engineering or Science stream
• has a minimum of two (2) years of experience within the past four (4) years, at the time of bid closing performing all of the following:
  A) Modelling data for storage and consumption in Amazon Web Services (AWS) S3 buckets;
  B) Running and monitoring data pipelines using Amazon Web Services (AWS) Glue;
  C) Querying data using Structured Query Language (SQL) and PostgreSQL;
  D) Implementing automated extract, transform, and load (ETL) pipelines using Amazon Web Services (AWS) Glue.
• Minimum of five (5) years of experience within the past eight (8) years, at the time of bid closing, conducting database engineering including all of the following:
  A) Designing and developing cloud-based data solutions to meet business requirements;
  B) Exposing data and services using Application Programming Interfaces (APIs);
  C) Implementing data observability within the data pipeline;
  D) Optimizing data modeling through indexing, partitioning, and performance tuning;
  E) Ensuring data integrity through validation, verification, error handling and logging, testing, version control and change management.
• Has a minimum of two (2) years of experience within the past four (4) years, at the time of bid closing, in all of the following:
  A) Providing technical leadership to junior members conducting extract, transform, and load (ETL) processes;
  B) Troubleshooting ongoing technical problems related to databases and data pipelines;
  C) Developing training material on extract, transform, and load (ETL) procedures;
  D) Ensuring data security through all of the following:
  i.data backups
  ii.encryption
  iii.access controls
  iv.anonymization OR masking
  v.adherence to relevant security policies.
  E) Continually optimizing data pipeline performance through performance tuning, troubleshooting, continuing support and capacity estimation
• Minimum of two (2) years of experience within the past four (4) years, at the time of bid closing, manipulated data using Structured Query Language coding languages for storage and extract, transform, and load (ETL) processes

Optional requirements

• Minimum of one (1) year experience working with cloud-based extract, transform, and load (ETL) OR extract, load, and transform (ELT) processes.
• 3 completed data projects within the Department of Defence
• Any of the following certifications:
  Professional Level AWS Solutions Architect – Professional
  Specialty Level
  • AWS Advanced Networking – Specialty
  • AWS Security – Specialty
  • AWS Machine Learning – Specialty
  • AWS Advanced Networking – Specialty
  Associate Level
  • AWS Developer – Associate
  • AWS Data Engineer – Associate
  • AWS Solutions Architect – Associate
• Minimum of one (1) year of experience designing, implementing, and optimizing cloud-based data storage solutions.

General Tasks

• Data Platform Development consisting of the following tasks:
  Provide advice to TA on ways to implement new initiatives;
  Provide Subject Matter Expert (SME) advice/recommendations for required data platform requirements such as database tools, data connectivity, and other digital application requirements.
• Perform Data Pipeline Implementation consisting of the following tasks:
• Design and implement extract, transform, and load (ETL) processes using tools including Glue, Fabric, DataBricks, and Informatica;
  Develop data pipelines to ingest, process, and transform data for data platforms;
  Profile data, develop metadata understanding and define data quality rules;
  Develop data quality checks and data observability measures. At the TA’s request provide a report with completed steps and measures implemented;
  Optimize the ETL process through performance tuning, troubleshooting, continuing support and capacity estimation.
• Database and Storage Solutions consisting of the following tasks:
  Design and implement scalable data and database solutions using appropriate services, tools, libraries and platforms;
  Optimize data modeling through indexing, partitioning, query optimization, and tuning considering data velocity, volume, and variety.
  Ensure the safe creation of data pipelines integrating data backups, encryption, access controls, anonymization, masking, and adherence to Director Information Management (DIM) Secure policies.
• Data Engineering/ Backend Development consisting of the following tasks:
  Design and deploy backend systems for historical data storage and auditing;
  Integrate backends with existing systems;
  Ensure data integrity through validation, verification, error handling/logging, testing, version control and change management;
  Design, construct, and implement automated data pipelines that handle extraction transformation, cleaning, and loading of data. Utilize scheduling and triggers to manage execution, and incorporate workflow management for seamless orchestration.
• Data Governance and Compliance consisting of the following tasks:
  Ensure compliance with DND data security standards and metadata directives such as Director Information Management Security (DIM Secur) policy, DAOD 6000 and the Chief Data Officer Directive on Metadata;
  Conduct security audits and update policies as required according to DND directives.
• Technical Solutions consisting of the following tasks:
  Provide ongoing technical support and troubleshooting for deployed solutions;
  Provide solutions and technical support that enables RCAF data pipelines to provide accurate, timely, and relevant data to RCAF stakeholders.
• Ongoing Technical Support consisting of the following tasks:
  Provide advice to junior team members on the proper application of existing infrastructure through knowledge transfer sessions;
  Develop training material such as videos, wiki posts, how-to’s, FAQs, or documents
• Project Reporting consisting of the following tasks:
  Report on project progress, ensuring alignment with RDH project objectives, and report weekly on issues.