Compare commits

...

9 Commits

Author SHA1 Message Date
94e8d7bf84 Demote Signal receipt/typing/sync spam from INFO to DEBUG
signalbot logs every received envelope at INFO via `[Raw Message] {json}`. On a
busy account the delivery/read receipts, typing indicators, and read-sync
messages flood journald hard enough to rotate the journal down to ~1 minute,
burying real traffic and making failures impossible to debug after the fact.

Add a logging filter on the `signalbot` logger that suppresses content-free
envelopes (anything without a `dataMessage`) at INFO, while still emitting them
if the logger is explicitly set to DEBUG. Real message content stays at INFO.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-16 17:03:55 -04:00
d16c025753 Fix YouTube downloads: bump yt-dlp, require Node 22, drop YT cookies
Two YouTube regressions stacked up:

- yt-dlp 2026.3.17 hit "HTTP Error 403: Forbidden" on the actual media
  download. Bumping to 2026.6.9 fixes the data-download 403, but its EJS
  n-sig challenge solver now requires Node >= 22, and Debian 13 only ships
  Node 20 ("node-20.x (unsupported)" -> only image formats resolve). bot.py
  now points --js-runtimes at a standalone /opt/node22 via the new
  JS_RUNTIME constant.

- With a cookies.txt present, yt-dlp skips the cookieless android_vr/android
  clients (whose URLs work) and falls back to the `tv` client, whose media
  URLs 403. New _cookie_args(url) withholds cookies for YouTube hosts only,
  while still sending them to Instagram/TikTok/X.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-16 16:12:17 -04:00
76ac4ea57a Match Instagram /reels/ (plural), /tv/, and /share/ link forms
The share sheet in the Instagram app now emits instagram.com/reels/<id>
(plural), which INSTAGRAM_URL_PATTERN didn't match — only /reel/ and /p/.
A non-matching URL is silently dropped (regex never matches -> no download,
no error reply), so reels links appeared to do nothing. Broaden the pattern
to (?:share/)?(?:reels?|p|tv) to also cover IGTV (/tv/) and the /share/
prefix. Verified against should-match / should-not-match URL sets; other
platform patterns unchanged.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-15 21:41:57 -04:00
d6bb39922a Run /force downloads at low CPU + idle IO priority (nice/ionice)
Wrap the forced yt-dlp download (and its node/ffmpeg children) and the rsync
upload with 'nice -n 19 ionice -c 3' so the known-big, slow /force job yields the
CPU and disk to everything else under load while still finishing fast when idle.
Normal short downloads stay at full priority. Best-effort: prefix is empty if the
tools are absent.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-15 21:24:17 -04:00
12b1f2f03d Add /force: host oversized videos as temporary 24h links
When a posted video is too long (>1h, probed up front) or a normal download
fails as too-big/timeout, the bot offers /force. Replying /force downloads the
full video off-loop (separate slot, no size cap, --max-filesize 10G), uploads it
via rsync/SSH to a web docroot, and posts a tokenized link auto-deleted after 24h
by an hourly sweep (also evicts oldest past 10 GB; remote-clock TTL; only ever
removes its own token dirs).

Hosting target is env-configured (FORCE_REMOTE/FORCE_REMOTE_DIR/FORCE_BASE_URL/
FORCE_SSH_PORT/FORCE_SSH_KEY) so published code carries no infra details; the
feature self-disables when unset (probe is skipped too, so normal posting is
unchanged). Staged on disk (not tmpfs), self-heals orphaned staging dirs, and
re-arms the offer on transient failure. See signal-bot.service.example.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-15 20:42:37 -04:00
fd0da1a59e Persist URL dedup cache to SQLite (bounded, ~KB); keep video blobs in memory
Switch storage backend from in-memory to SQLite (silences the library warning).
recent_urls (the edit-redelivery dedup map) now survives restarts: it's tiny and
TTL-bounded, stored via the library KV store, with wall-clock timestamps and
prune-on-load. last_video is deliberately NOT persisted — each value is a base64
video up to ~133 MB, so persisting it would bloat the DB; it stays in memory.
DB lives at bot-state.db (gitignored).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-15 19:26:12 -04:00
c7eb101ff0 Fix bug pass: async offload, resource leaks, media/parse correctness, deploy hardening
bot.py:
- Run all yt-dlp/ffmpeg/ffprobe/file-IO/base64 work off the event loop via
  asyncio.to_thread, bounded by a Semaphore(2); the loop no longer freezes
  bot-wide during downloads/encodes.
- download_attachments=False; VideoTracker fetches only video attachments lazily
  instead of the library base64-ing every attachment of every message.
- last_video keyed per (group, sender) with group-latest fallback so /speed and
  /rev stop silently targeting a stranger's video; proactive TTL sweeps free the
  big base64 blobs and bound recent_urls growth.
- _reencode guards 0/NaN ffprobe duration and adds -maxrate/-bufsize; /rev now
  size-checks + re-encodes + faststart like the other paths.
- Twitter URL regex no longer merges space-separated links (keeps i/web/status);
  dedupe repeated URLs within a message; mark-handled only on success/no-media so
  a corrective edit can retry; 'unsupported url' surfaced instead of silently dropped.
- All sends wrapped (catch SendMessageError); base64 decode guarded; edit/sync
  attachment envelopes handled; /cookies temp file created 0600.

deploy:
- Pin signalbot==1.1.0, yt-dlp floor, add missing yt-dlp-ejs.
- Pin signal-cli-rest-api by digest + add /v1/health healthcheck.
- Restart=always (the library never exits non-zero when wedged).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-15 18:43:41 -04:00
26a5ecb2d0 Add /clip duration override and /help command
/clip <seconds> in a link's message overrides the default 60s clip
window (capped at 600s); with a ?t= it sets the window length, without
one it clips from the start. /help lists every command with examples.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-15 16:35:28 -04:00
f9e73333ba Add timestamp-aware clipping for shared ?t= links
When a YouTube link carries a t=/start= offset, download only a 60s
window around it via yt-dlp --download-sections instead of the whole
video. Hour-long uploads shared at a timestamp previously failed the
100 MB Signal limit even after re-encoding.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-15 16:31:09 -04:00
6 changed files with 959 additions and 234 deletions
Vendored
+4
View File
@@ -8,6 +8,10 @@ signal-cli-data.bak.*
# Local service file (contains your phone number) — use signal-bot.service.example # Local service file (contains your phone number) — use signal-bot.service.example
signal-bot.service signal-bot.service
# Runtime state (SQLite dedup cache) — created next to bot.py at runtime
bot-state.db
bot-state.db-*
# Python # Python
venv/ venv/
__pycache__/ __pycache__/
+5 -1
View File
@@ -50,7 +50,11 @@ Signal ──► signal-cli-rest-api (json-rpc, :8080) ──► bot.py (sig
python3 -m venv venv python3 -m venv venv
venv/bin/pip install -r requirements.txt venv/bin/pip install -r requirements.txt
``` ```
YouTube also needs a system `node` on PATH (for `yt-dlp-ejs`). YouTube also needs **Node >= 22** on PATH (for `yt-dlp-ejs`'s challenge
solver). If your distro only ships an older Node, install a standalone Node 22+
and point `bot.py`'s `_NODE22` at it. Note: YouTube cookies are deliberately
withheld (`_cookie_args`) — sending account cookies forces yt-dlp onto a player
client whose media URLs 403; cookieless YouTube requests work.
3. **Cookies:** copy `cookies.txt.example` to `cookies.txt` and fill in real 3. **Cookies:** copy `cookies.txt.example` to `cookies.txt` and fill in real
exported cookies for the sites you want auth'd. (gitignored) exported cookies for the sites you want auth'd. (gitignored)
+907 -215
View File
File diff suppressed because it is too large Load Diff
+11 -1
View File
@@ -1,6 +1,10 @@
services: services:
signal-cli-rest-api: signal-cli-rest-api:
image: bbernhard/signal-cli-rest-api:latest # Pinned by digest so a re-pull/recreate can't silently swap in a build with
# a protocol change (the getServerGuid-style breakage). This digest is
# signal-cli-rest-api with signal-cli 0.100 (build 2). Upgrade deliberately:
# `docker pull bbernhard/signal-cli-rest-api:latest`, test, then update this.
image: bbernhard/signal-cli-rest-api@sha256:2399d449123cdad56c4d859277e3b9127e1a00c4d2ab4601c239882609286cf8
container_name: signal-cli-rest-api container_name: signal-cli-rest-api
restart: unless-stopped restart: unless-stopped
environment: environment:
@@ -9,3 +13,9 @@ services:
- "127.0.0.1:8080:8080" - "127.0.0.1:8080:8080"
volumes: volumes:
- ./signal-cli-data:/home/.local/share/signal-cli - ./signal-cli-data:/home/.local/share/signal-cli
healthcheck:
test: ["CMD", "curl", "-fsS", "http://127.0.0.1:8080/v1/health"]
interval: 30s
timeout: 5s
retries: 3
start_period: 30s
+8 -4
View File
@@ -1,5 +1,9 @@
signalbot signalbot==1.1.0
yt-dlp # yt-dlp is kept current for site/extractor changes; bump the floor deliberately.
# 2026.6.9 fixes a YouTube 403-on-data-download regression; it also requires the
# JS challenge to run on Node >= 22 (see below).
yt-dlp>=2026.6.9
# Needed for YouTube: yt-dlp wraps URLs in a JS "n-sig" challenge that a JS # Needed for YouTube: yt-dlp wraps URLs in a JS "n-sig" challenge that a JS
# runtime must solve. Requires a system `node` on PATH plus this package. # runtime must solve. Requires Node >= 22 on PATH (bot.py points at a standalone
yt-dlp-ejs # /opt/node22 since Debian 13 only ships Node 20) plus this package.
yt-dlp-ejs>=0.8.0
+12 -1
View File
@@ -12,8 +12,19 @@ Environment=SIGNAL_PHONE_NUMBER=+15551234567
Environment=SIGNAL_SERVICE=127.0.0.1:8080 Environment=SIGNAL_SERVICE=127.0.0.1:8080
# Optional: comma-separated numbers allowed to run /cookies (admin command). # Optional: comma-separated numbers allowed to run /cookies (admin command).
# Environment=BOT_ADMINS=+15551234567 # Environment=BOT_ADMINS=+15551234567
# Optional: enable /force, which hosts oversized videos as temporary links.
# The bot rsyncs the file (over SSH) into a web-served directory and posts the URL;
# an hourly job deletes anything older than 24h and evicts oldest past 10 GB.
# Leave unset to disable /force. All three of REMOTE/REMOTE_DIR/BASE_URL are required.
# Environment=FORCE_REMOTE=user@webhost # ssh target that can write the docroot
# Environment=FORCE_REMOTE_DIR=/var/www/site/media # docroot on that host
# Environment=FORCE_BASE_URL=https://site/media # public URL serving that docroot
# Environment=FORCE_SSH_PORT=22 # ssh port (default 22)
# Environment=FORCE_SSH_KEY=/home/YOUR_USER/.ssh/id_ed25519 # ssh identity (optional)
ExecStart=/home/YOUR_USER/signal-bot/venv/bin/python bot.py ExecStart=/home/YOUR_USER/signal-bot/venv/bin/python bot.py
Restart=on-failure # always (not on-failure): the bot's library swallows exceptions in its own run
# loop and never exits non-zero even when wedged, so on-failure would never bounce it.
Restart=always
RestartSec=10 RestartSec=10
[Install] [Install]