Social Sentiment Worker: Initial Implementation

[evahteev]

Summary

This PR introduces a complete social sentiment analysis worker, including all supporting scripts, configuration, and documentation, under the ai/ directory. The worker fetches tweets, processes them for token mentions, analyzes sentiment, and stores results in ClickHouse, enabling downstream analytics and reporting.

---

Key Features

• Social Sentiment Core Worker (social_sentiment_core_worker.py):

  • Fetches tweets for a configurable list of authors using the Twitter (X) API via RapidAPI.
  • Stores raw tweet data in ClickHouse, ensuring deduplication.
  • Processes tweets to identify token mentions (from a configurable token list).
  • For each mention, queries a MindsDB agent to extract sentiment and generate a summary.
  • Stores processed sentiment and summary data in a dedicated ClickHouse table.
  • Supports Camunda external task integration for orchestration.

• Configuration & Data Files:

  • social_sentiment_worker_kol_list.csv: List of authors (KOLs) to monitor.
  • social_sentiment_worker_tokens_list.csv: List of tokens/symbols/fullnames to detect in tweets.
  • social_sentiment_pipeline.png: Visual overview of the sentiment analysis pipeline.
  • social_sentiment_core_worker.md: Documentation and usage instructions.

• Settings Integration:

  • Adds all required ClickHouse and MindsDB settings to the main configuration, with sensible defaults and environment variable support.

---

How It Works

1. Fetch Raw Tweets:
   The worker fetches recent tweets for each author in the KOL list, storing new tweets in ClickHouse.

2. Process Sentiment:
   For each unprocessed tweet, the worker:

   • Identifies token mentions.
   • Calls a MindsDB agent to analyze sentiment and summarize the tweet.
   • Stores the results in a sentiment table and marks the tweet as processed.

3. Orchestration:
   The worker is designed to be triggered and managed via Camunda BPM, supporting external task handling.

---

How to Test

1. Configure your .env file with the required ClickHouse, RapidAPI, and MindsDB credentials.
2. Run the worker and trigger tasks via Camunda or manually.
3. Verify that:
   • Tweets are fetched and stored in ClickHouse.
   • Sentiment and summary data are generated and stored.
   • No errors occur due to missing configuration.

---