capçalera

Case StudyCaso de EstudioCas d'Estudi

Optimize Log Performance with Opentrends & Elastic

Opentrends aimed at centralizing the processing and exploitation of different server logs and TMB operations, by designing in-house a platform leveraging Elastic Stack.

Challenge

the challenge was to identify the different log sources, define a process of intake, enrichment and design of different command tables for the exploitation and visualization of the logs. We designed an architecture capable of sustaining the load of logs to ingest, scale and mutate according to the needs that appear.

Solution

A Log Processing Platform for Performance Optimization

Different tasks have been carried out in order to set up an ecosystem for the ingestion and exploitation of logs managed by Elastic Stack (ELK). It is a project in continuous evolution, where flexibility allows the solution to evolve and adapt. Additionally, other tools that may fit the ecosystem have been researched and evaluated.

  • Logstash configurations for the different logs.
  • Use of nxLog and Filebeat for the ingestion of logs.
  • Use of Redis as broker. 
  • Definition of dashboards in Kibana and Grafana.
  • Implementation of services in NODE.JS for the enrichment of the logs
    (add info to the base logs to improve exploitation)
  • Lifecycle management of indexed data in Elasticsearch (index optimization, cleaning policies, backups, among others)
  • Alerts definition
Arquitectura

METHODOLOGY

We have worked hand in hand with the technical department of TMB. In weekly meetings, the completed work has been reviewed and the pending tasks have been defined following the Elastic best practices.

best practices Elastic

TECHNOLOGY

The following technologies have been used:

  • Elastic Stack (Elasticsearch, Logstash, Kibana, Filebeat, Metricbeat)
  • nxLog
  • NodeJs
  • Redis
  • Kafka
  • Grafana
  • Auth0

 

Tecnología
Results

With the new log processing platform by SEIDOR Opentrends, TMB has achieved performance optimization of its infrastructure, as well as a global economization of their system, now capable of facing high levels of log processing.

Based on the results, the main characteristics include:

  • The fixed architecture service that provides flexibility and continuous improvement of the solution.
  • Governance and management of data's life cycle.
  • User-friendly display of the main customer indicators.
Contact us