|
| 1 | +# CLAUDE.md |
| 2 | + |
| 3 | +This file provides guidance to Claude Code (claude.ai/code) when working with code in this repository. |
| 4 | + |
| 5 | +## Project Overview |
| 6 | + |
| 7 | +This is a Python backend for libmap.org that monitors social media for specific keywords and provides an API to serve the collected data. The system primarily handles Mastodon toots and Twitter tweets containing specific map-related keywords. |
| 8 | + |
| 9 | +## Development Setup |
| 10 | + |
| 11 | +### Prerequisites |
| 12 | +```bash |
| 13 | +python -m venv venv |
| 14 | +. venv/bin/activate |
| 15 | +pip install -r requirements.txt |
| 16 | +``` |
| 17 | + |
| 18 | +### Configuration |
| 19 | +Copy `config.template.json` to `config.json` and configure with your API credentials for Twitter and Mastodon. |
| 20 | + |
| 21 | +## Key Commands |
| 22 | + |
| 23 | +### Run Services |
| 24 | +- **API Server**: `./runApi.py` - Starts Flask web server on port 5000 |
| 25 | +- **Live Listener**: `./runListener.py` - Monitors social media streams |
| 26 | + - `./runListener.py` - Mastodon (own instance), fedi.buzz firehose, and Bluesky (default) |
| 27 | + - `./runListener.py --bluesky-only` - Only Bluesky stream |
| 28 | + - `./runListener.py --mastodon-only` - Only own-instance Mastodon stream |
| 29 | + - `./runListener.py --platforms mastodon,fedibuzz,bluesky` - Custom platform selection |
| 30 | + - fedi.buzz (https://fedi.buzz/api/v1/streaming/public) is an unauthenticated Mastodon-compatible firehose aggregated across many instances; posts arriving from multiple streams are deduplicated by ActivityPub `uri` (status `id`s are instance-local) |
| 31 | +- **Historical Data**: `./runInit.py` - Fetches old tweets for initialization |
| 32 | + |
| 33 | +### Python Environment |
| 34 | +- Activate venv (bash): `. venv/bin/activate` |
| 35 | +- Activate venv (fish): `source venv/bin/activate.fish` |
| 36 | +- Install dependencies: `pip install -r requirements.txt` |
| 37 | + |
| 38 | +## Architecture |
| 39 | + |
| 40 | +### Core Components |
| 41 | + |
| 42 | +**Entry Points:** |
| 43 | +- `runApi.py` - Flask web application entry point |
| 44 | +- `runListener.py` - Mastodon streaming listener service |
| 45 | +- `runInit.py` - Historical data fetcher using GetOldTweets3 |
| 46 | + |
| 47 | +**Core Libraries (`lib/`):** |
| 48 | +- `shared.py` - Configuration loading and shared paths |
| 49 | +- `TootForest.py` - Data structure for organizing toots/tweets |
| 50 | +- `mastodon_auth.py`, `mastodon_base.py` - Mastodon API integration |
| 51 | +- `Toot.py` - Individual toot/tweet data model |
| 52 | +- `toots_aux.py` - Auxiliary functions for data processing |
| 53 | + |
| 54 | +**API Module (`api/`):** |
| 55 | +- `__init__.py` - Flask app factory with authentication middleware |
| 56 | +- `modules/toots.py` - Toot/tweet endpoints |
| 57 | +- `modules/auth.py` - Authentication decorators |
| 58 | +- `modules/log.py` - Logging endpoints |
| 59 | + |
| 60 | +### Data Flow |
| 61 | + |
| 62 | +1. **Live Monitoring**: `runListener.py` connects to Mastodon streaming API, filters content by keywords, saves matching toots to filesystem |
| 63 | +2. **Data Processing**: `TootForest` class organizes toots/tweets and generates API JSON files |
| 64 | +3. **API Access**: Flask app serves processed data through authenticated endpoints |
| 65 | +4. **Historical Data**: `runInit.py` can fetch older tweets to populate initial dataset |
| 66 | + |
| 67 | +### Configuration System |
| 68 | + |
| 69 | +The system uses a JSON configuration file with sections for: |
| 70 | +- Authentication tokens (authToken, secretKey) |
| 71 | +- Mastodon instance settings (instance_url, client credentials, access token) |
| 72 | +- Twitter API credentials |
| 73 | +- Fetch settings with different profiles (test, live) containing search keywords and output paths |
| 74 | + |
| 75 | +### Authentication |
| 76 | + |
| 77 | +The Flask app requires token-based authentication via `/authenticate/<token>` before accessing protected endpoints. Sessions persist for 100 days. |
| 78 | + |
| 79 | +### File Structure |
| 80 | + |
| 81 | +- `data/toots/` - Storage for collected social media content |
| 82 | +- `log/` - Application logs (listener.log, api.log, init.log) |
| 83 | +- Configuration is centralized in `lib/shared.py` |
| 84 | + |
| 85 | +## Development Notes |
| 86 | + |
| 87 | +- No formal test framework detected in codebase |
| 88 | +- No linting configuration found |
| 89 | +- Uses Python virtual environment for dependency management |
| 90 | +- Logging is configured per service with both file and console output |
| 91 | +- The system can switch between different monitoring profiles via config |
0 commit comments