Optimize Log Performance with Opentrends & Elastic
Opentrends aimed at centralizing the processing and exploitation of different server logs and TMB operations, by designing in-house a platform leveraging Elastic Stack.
The project consisted of identifying the different log sources, defining a process of intake, enrichment and design of different command tables for the exploitation and visualization of the logs. We designed an architecture capable of sustaining the load of logs to ingest, and to scale and mutate according to the needs that appear.
A LOG PROCESSING PLATFORM MANAGED BY ELASTIC STACK (ELK)
Different tasks have been carried out in order to set up an ecosystem for the ingestion and exploitation of logs. It is a project in continuous evolution, where flexibility allows the solution to evolve and adapt. Additionally, other tools that may fit the ecosystem have been researched and evaluated.
- Logstash configurations for the different logs.
- Use of nxLog and Filebeat for the ingestion of logs.
- Use of Redis as broker.
- Definition of dashboards in Kibana and Grafana.
- Implementation of services in NODE.JS for the enrichment of the logs
(add info to the base logs to improve exploitation)
- Lifecycle management of indexed data in Elasticsearch (index optimization, cleaning policies, backups, among others)
- Alerts definition
We have worked hand in hand with the technical department of TMB. In weekly meetings, the completed work has been reviewed and the pending tasks have been defined following the Elastic best practices.
The following technologies have been used:
- Elastic Stack (Elasticsearch, Logstash, Kibana, Filebeat, Metricbeat)
With the new log processing platform by Opentrends, TMB has achieved performance optimization of its infrastructure, as well as a global economization of their system, now capable of facing high levels of log processing.
Based on the results, the main characteristics include:
- The fixed architecture service that provides flexibility and continuous improvement of the solution.
- Governance and management of data's life cycle.
- User-friendly display of the main customer indicators.