Improved Search Performance and Data Accuracy Through a Multi-Source Ingestion Tool

Feb 03, 2023

About the client

The client is a British publishing, business intelligence, and exhibitions group. Headquartered in London, it has its presence in 43 countries.

Challenge

The client faced a slew of challenges that have been highlighted below:

Given that the existing windows-based content authoring tool was developed 20 years ago, scalability was an issue.
The issue of the tool’s poor performance impacted productivity.
There was a lot of manual dependency, leading to errors.
The client was not able to cover the existing public domain comprising:

1. 45,000 identified sources.

2. Structured and unstructured data coverage.

3. 310,000+ clinical trials, 440,000 investigators, 77,000+ drugs, 165,000+ sites, and 255+ diseases in eight major therapeutic areas across 166 countries.

LTM’s Solution

LTM’s team worked with the client and executed a comprehensive solution to address their needs. Here are the solution highlights:
It is an AWS native solution with best practices incorporated.
The solution involves a multi-source data ingestion tool to automatically source and ingest structured and unstructured data from various sources.
The ingested data is curated processed and stored in the Central Data Repository.
Availability of AWS OpenSearch to increase search efficiency.
Uses microservices on AWS EKS and ReactJS to create, edit and, view structured data and provides the functionality for workflow management.
Task and queue workflows incorporated to simplify their business process.

Business Benefits

LTM’s solution led to several benefits for the client, including:

Timely availability of data in almost real-time. The lag in searched data reduced by six times.
Average performance of complex search improved by 5-10 times. In some cases, it has improved by 24 times.
Highly available, scalable, and easy to maintain.
The auto ingestion design allows large scale coverage with more than 80 data sources/registries.
Highly responsive content authoring tool and user-friendly workflow for the content team.
Easy to onboard new products.