This project was a challenge for me due to the volatility of the requirements, as it was a startup in the process of launching that aimed to find its place in the market.

I built the technological stack from scratch, mainly focused on NodeJS.

The project was very ambitious, capturing press content of any type of format: PDF, Digital, Audio, Video, Streaming and even official state documents. All this structured, analyzed and categorized to be served in a feed architecture similar to that of Twitter, for the consumption of the end user (communication cabinets of companies and professionals) based on complex conditions specified by them.

The main challenges were:

• Various newspaper formats (PDF) that needed to be parsed, using OCR algorithms, page layout detection and NLP.
• Capturing video streaming channels from the main television and radio platforms.
• Categorization of news content using NLP and a condition system adapted to the model structure.
• Data management, with a daily intake of more than 150k documents needed to be indexed.

The product started out consisting of a typical MERN stack, and then implemented improvements with:

• Django (ORM)
• ManticoreDB and MeiliSearch (Search Engine)
• Redis (Caching and queue management)

I carried out all these tasks myself from scratch iteratively while maintaining the productive service for the clients as required by the circumstances.