Offers Parse — CS2 Skins Offers Parser

🇺🇦 Українська версія

Six daemons that parse real offers (listings) from skin marketplaces — actual sell offers with prices, float values, paint seeds, etc. This is the core data source for arbitrage, order books, and price charts.

cd cs2_skins_trading_offers_parse
npm install
cp .env.example .env   # fill in API keys

# 1. Initialize databases first
cd ../database && npm i && node offers_dmarket.js && node offers_whitemarket.js && cd ../offers_parse

# 2. Make sure items catalog is populated
# (items_parse daemons should have run at least once)

# 3. Start offer parsers
pm2 start offers.config.cjs   # DMarket + WhiteMarket

# Or start individually
node dmarket.js
node whitemarket.js
node bitskins.js
node waxpeer.js
node lootfarm.js
node skinport.js

1. Go to dmarket.com/settings
2. Navigate to Trading API section
3. Click Generate Keys
4. Copy Public Key (hex with 0x prefix) and Secret Key (hex)
5. Paste into .env as DMARKET_PUBLIC_KEY and DMARKET_SECRET_KEY

Signing: ed25519 via tweetnacl library. Each request is signed with method + path + body + timestamp.

1. Go to white.market/profile/api
2. Generate Partner Token
3. Paste into .env as WHITE_PARTNER_TOKEN

Auth flow: Partner Token → JWT access token (cached 23 hours).

BitSkins, Waxpeer, Loot.Farm, Skinport — public APIs, no authentication required.

API: https://api.dmarket.com/exchange/v1/offers-by-title

How it works:

1. Takes a batch of items from items_dmarket.db queue (offers_parsed=0)
2. For each item, fetches all offers with pagination (up to MAX_PAGES × 100)
3. Stores each offer: id, price_usd, float_val, paint_seed, paint_index, inspect_url
4. Tracks changes via history table:
   • New offer → seen event
   • Price changed → price_changed event
   • Offer gone → removed event + DELETE
5. Marks item as offers_parsed=1
6. When queue empty → resets all to offers_parsed=0, starts over
7. Runs incremental_vacuum(1000) after each batch

Speed: 100 items/batch, 200ms delay. Full cycle ~12,600 items in ~8 hours.

API: https://api.white.market/graphql/partner (GraphQL)

How it works:

1. Takes batch of 500 items from items_whitemarket.db queue
2. For each item, fetches offers via GraphQL with pagination (up to 10,000 per item)
3. Extracts: id, price_usd, float_val, paint_seed, paint_index, phase, asset_id, seller_id
4. Upsert + history tracking (same as DMarket)
5. Parallel: 2 concurrent workers (configurable)

Speed: 500 items/batch, 2 parallel. Full cycle ~27,000 items in ~5 minutes.

API: https://api.bitskins.com/market/insell/730

Single request → all items with min price and quantity. Cycle every 5 minutes.

API: https://api.waxpeer.com/v1/prices?game=csgo

Single request → all items. Price in 1/1000 USD.

API: https://loot.farm/fullprice.json

Single request → all items. Price in 1/1000 USD.

API: https://api.skinport.com/v1/items?app_id=730

Single request → all items with min price and quantity.

• History tracking — every price change or offer removal is recorded
• auto_vacuum = INCREMENTAL — prevents DB file bloat (was 66 GB before fix)
• Hot-reload .env — change parameters without restart
• Retry with exponential backoff — handles 429 and network errors
• Queue-based parsing — offers_parsed field ensures even coverage of all items

# Check daemon status
pm2 logs dmarket-offers-daemon --lines 20
pm2 logs whitemarket-offers-daemon --lines 20

# Check offers count
sqlite3 ../database/offers_dmarket.db "SELECT COUNT(DISTINCT name), COUNT(*) FROM offers_dmarket;"
sqlite3 ../database/offers_whitemarket.db "SELECT COUNT(DISTINCT name), COUNT(*) FROM offers_whitemarket;"

# Check parsing queue
sqlite3 ../database/items_dmarket.db "SELECT SUM(offers_parsed=0) as pending, SUM(offers_parsed=1) as done FROM items_dmarket;"

# Check DB size (should stay reasonable with auto_vacuum)
ls -lh ../database/offers_*.db

# Check 429 errors
pm2 logs whitemarket-offers-daemon --err --lines 50 | grep 429

# Force re-parse all items
sqlite3 ../database/items_dmarket.db "UPDATE items_dmarket SET offers_parsed=0;"

offers_parse/
├── dmarket.js          # DMarket offers daemon (auth: ed25519)
├── whitemarket.js      # WhiteMarket offers daemon (auth: Partner Token)
├── bitskins.js         # BitSkins offers daemon (public API)
├── waxpeer.js          # Waxpeer offers daemon (public API)
├── lootfarm.js         # Loot.Farm offers daemon (public API)
├── skinport.js         # Skinport offers daemon (public API)
├── env.js              # Configuration with hot-reload
├── offers.config.cjs   # PM2 ecosystem config
├── .env.example        # Environment template
├── package.json
├── README.md           # English
└── README.uk.md        # Ukrainian

• better-sqlite3 — SQLite driver
• dotenv — Environment variables
• tweetnacl — Ed25519 signing for DMarket API