Observability Engineer Level 2 at Derivco

Responsibilities
System Administration of Observability Solutions

• Create and maintain roles, environments and machines in the CMDB
• Create and maintain alerting from all logging technologies including but not limited to ElasticSearch, InfluxDB, Prometheus, SCOM, Graphite, Azure Monitor
• Ensure retention policies are effectively managing data retention in ElasticSearch, InfluxDB, Prometheus, SCOM, Graphite, Azure Monitor
• Ensure roles, environments and machines are kept in sync between CMDB, various logging platforms and the Ansible deployment services.

Automation

• Deployment and administration of new and existing tools
• Implement standards and procedures set out for the Automated Deployment solution using automation where feasible
• Prototype and develop new tools
• Research new technology trends and logging technologies tools to remain abreast of current technology
• Troubleshoot and fix bugs reported in existing tools

Documentation/User Guides     

• In conjunction with key stakeholders it responsible for the development of user guides and training documentation for systems.
• Documents functions and changes to new or modified modules and test activities/results.
• Maintain Observability of Infrastructure and Applications in Dev and Production
• Apply updates to logging software when necessary to ensure logging technologies can take advantage of the latest features and bug fixes
• Ensure Maintenance Mode and Change Controls are logged when making changes to servers
• Install all logging, monitoring and alerting tools are installed on newly provisioned Production server hardware.
• Keep abreast of latest software updates available for logging and metrics software
• Maintain the Development and Testing deployment software installations for ElasticSearch, InfluxDB, Prometheus, SCOM, Graphite, Azure Monitor
• Maintain the Production deployment software installations for ElasticSearch, InfluxDB, Prometheus, SCOM, Graphite, Azure Monitor
• Monitor and troubleshoot offline agents to ensure monitoring and observability are not affected
• Monitor and troubleshoot agents requiring upgrades to ensure monitoring and observability are not affected

Provide Support

• Assists in support across the business for all application logs and metrics.
• Provide guidance and expertise to stakeholders on Enterprise Observability related issues
• Provide support to Development teams to ensure all observability solutions are designed with user experience, performance, and operability in mind.
• Assist developers and IT deployment engineers with Logging related troubleshooting
• Assist developers with setting up of new methods for logging events for any new projects
• Monitor failed agent deployments and provide support to ensure observability is maintained
• Provide an overview and/or tutorial for teams who are new to Automated Deployments and Logging Technologies.
• Provide guidance on standards and procedures set out for the Logging Technologies

Standards, Policies and Procedures     

• Adheres to and participates in the formulation and implementation of the Enterprise Observability Strategy.
• Adheres to standards and procedures.
• Reviews modules for quality assurance.

Maintenance     

• Assists in establishing requirements, methods and procedures for routine maintenance.
• Plans and performs ongoing routine logging and metrics related application maintenance tasks.

Coaching and Mentoring      

• Provides technical coaching and mentoring to less-experienced team members.

Technology Evaluation and Research     

• Evaluates new application packages and tools and performs research on best practices.to provide recommendations for the solutionsprovide recommendations for the solutions.

Vendor Management

• Work with vendors to resolve problems and develop solutions.

Deployment     

• Builds automated deployments using configuration management technology.
• Deploys new modules, upgrades and fixes to logging and metrics technologies across the dev and production datacenters
• Documents and completes knowledge transfer to support.
• Validates deployments.

Development     

• Performs script maintenance and updates due to changes in requirements or implementations
• Assists with setup and maintenance of all logging and metrics environments for all Derivco (dev and production data centers) and its operators / customers.
• Codes and documents custom observability frameworks.
• Develops and/or implements reusable components.
• Develops/ builds IT solutions to meet business requirements.
• Installs and configures solutions.
• Integrates solutions with other applications and platforms outside of the framework.