From 3738533260a8eb914e211b8d909e92a14c0c266c Mon Sep 17 00:00:00 2001 From: "Anderson A. Giacomolli" Date: Tue, 10 Nov 2026 11:41:53 -0300 Subject: [PATCH 01/16] Rewrite the README as a user guide The README is a guide for the user: setup of the data directory, login, Telegram, WhatsApp, the user service, and backups, then the client commands, what the agent does, the log, and development. It drops the internal files, formats, locks, queues, recovery details, and limits that each feature had added, which the doc comments of their code hold. It shrinks from 703 to 240 lines. --- README.rst | 737 ++++++++++------------------------------------------- 1 file changed, 137 insertions(+), 600 deletions(-) diff --git a/README.rst b/README.rst index 7ca3d74..8002ea0 100644 --- a/README.rst +++ b/README.rst @@ -3,31 +3,23 @@ ######### OneChat is one AI agent in a single chat that never ends. A daemon owns -the agent and its files, and a terminal client and a Telegram bot talk -to it. It is in early development. Linux is supported. +the agent and its memory, and you talk to it from a terminal client, +Telegram, and through your WhatsApp chats. It is in early development. +Linux is supported. -**************** - Data directory -**************** - -OneChat keeps its data and its token file in one directory. It uses the -first of these: - -#. ``ONECHAT_DIR``, when it is set. -#. ``$XDG_DATA_HOME/onechat``, when ``XDG_DATA_HOME`` is an absolute - path. A relative value is ignored. -#. ``~/.local/share/onechat``. With no home directory, OneChat exits - with an error. +******* + Setup +******* -OneChat creates the directory with mode 0700. +Data directory +============== -*************** - Configuration -*************** +OneChat keeps all its data in one directory, with mode 0700. It uses +``ONECHAT_DIR`` when it is set, else ``$XDG_DATA_HOME/onechat``, else +``~/.local/share/onechat``. The development shell sets ``ONECHAT_DIR`` +to ``.devenv/onechat`` in the checkout. -``config.json`` in the data directory holds the settings. OneChat does -not create it. A missing file or field takes the default, and an unknown -field is an error. +Optional settings go in ``config.json`` in the data directory: .. code:: json @@ -36,155 +28,60 @@ field is an error. "compactor": {"model": "gpt-6.1-sol", "effort": "high"} } -``endpoint`` - ``chatgpt`` (the default) sends model calls to the ChatGPT backend - with the subscription login. ``api`` sends them to the OpenAI API - with the key in ``OPENAI_API_KEY``, which must then be set. With - ``chatgpt``, OneChat ignores ``OPENAI_API_KEY``. +``endpoint`` is ``chatgpt`` (the ChatGPT subscription, the default) or +``api`` (the OpenAI API, with ``OPENAI_API_KEY`` set). ``compactor`` is +the model and effort of the memory summaries. -``compactor`` - The model and the reasoning effort of the summaries. The defaults are - ``gpt-6.1-sol`` and ``high``. - -******* - Login -******* - -The ``chatgpt`` endpoint needs a login with a ChatGPT account: +Login +===== .. code:: console onechat login -The command shows a link and a code. Open the link on any device, enter -the code, and approve the login. Device code login must be allowed in -the ChatGPT security settings. OneChat writes the tokens to -``auth.json`` in the data directory, with mode 0600, and refreshes them -before they expire. The file belongs to OneChat only. Do not share it -with the Codex CLI, because each program rotates the refresh token and -logs the other out. +Open the link on any device and enter the code. Device code login must +be allowed in the ChatGPT security settings. Do not share the login with +the Codex CLI: each program logs the other out. -******* - Agent -******* +Telegram +======== + +#. Make a bot with BotFather and keep its token. +#. Find your numeric Telegram user ID. +#. Write both to ``telegram.json`` in the data directory, with mode + 0600: -Start the daemon, and then the client in another terminal: +.. code:: json -.. code:: console + {"token": "123456:ABC-token-from-BotFather", "owner": 123456789} - onechat daemon - onechat +Start the daemon and write to the bot in a private chat. It answers only +you. -``onechat`` with no command is the same as ``onechat chat``. ``onechat -help`` lists the commands. - -The daemon owns the root agent in ``agent/`` in the data directory and -listens on the Unix socket ``onechat.sock`` there. In the client, each -line is a message to the agent, the line ``/stop`` interrupts the -running turn, the line ``/stop NAME`` stops the subagent NAME, and -Ctrl-D ends the client. On terminal output, the client shows ``> `` -after each event while idle and clears it before the next event. An -event can interrupt a line while the user types. The client shows the -replies, the tool calls, and the first lines of each tool result as they -happen. Replies and subagent reports render Markdown with styles and -wrapped tables only when the output is a terminal, and stay unchanged in -a pipe or file. It also shows each message that the user sends from -another client, with the client as a mark, such as ``[telegram] text``. -When a subagent starts, the client shows ``[NAME] started``, and it -shows the whole report of the subagent, which starts with ``[NAME]``, -but not the tool calls of the subagent. The daemon is the only process -that uses the token file while it runs, so ``onechat login`` and -``onechat probe`` refuse to start until it stops. - -The agent runs commands with bash as the user in its working directory, -with no sandbox. It has the rights of the user, also on the token file. - -When the user asks for it, the agent starts subagents for long tasks. -Each subagent has a name, such as ``GreenOtter``, and runs in the -background while the agent answers other messages. It gets the view of -the agent and its task, and it runs its tools as the agent does, in -``agent/subagents/NAME/work``. A task in a repository gives the path of -the repository. The steps of a subagent go to its own log in -``agent/subagents/NAME``, not to the chat, and its final reply comes -back to the agent as one message that starts with ``[NAME]``. The agent -can send a subagent more messages: a running subagent reads them between -its tool calls, and a finished one goes on from its log, and its reply -comes back in the same way. When the user stops a subagent, the chat -gets a note, and the report of the subagent is logged, but it starts no -turn. Each model call of a subagent has the name in the ``agent`` field -of ``agent/usage.jsonl``. ``agent/work.json`` holds the names of the -running subagents. A subagent that runs when the daemon stops goes on at -the next start: the chat gets the note that the daemon restarted, and -the subagent gets the same note and goes on from its log. - -``agent/config.json`` holds the settings of the agent. The first start -writes it with the defaults and a random ID. Do not change the ID: the -prompt cache key comes from it. +WhatsApp +======== -.. code:: json +Link your WhatsApp account as a linked device from the terminal client +or Telegram: - { - "name": "OneChat", - "id": "0123456789abcdef0123456789abcdef", - "model": "gpt-6.1-sol", - "effort": "medium", - "workdir": "/home/user" - } +``/whatsapp pair PHONE`` + Gives a pairing code. On the phone, open Linked devices, Link a + device, Link with phone number instead, and enter the code. -``workdir`` is an absolute path, and its default is the home directory. -``agent/instructions.md`` holds the user's instructions: who they are, -how their files are organized, and how they want work done. The prompt -ends with them, and a change applies at the next call. The agent can -propose a change to the file, and it makes the change only after the -user agrees. - -A permanent child agent lives in ``agent/children/NAME/`` with the same -files as the root. The root's ``agent_create(name, responsibility)`` -tool creates and starts it with the root's model, effort, and working -directory, and a separate random ID. A name describes the area, such as -``House`` or ``Calendar``. It is one word of 2 to 32 ASCII letters, -starts with a capital letter, and cannot name the root, an existing -child, or a subagent. The daemon opens all children at startup. A -missing or invalid child config stops startup. Only one level of -children is supported. Remove a child by hand while the daemon is -stopped. The root's turn state lists each child and its responsibility. -The root passes area messages from mixed chats with sender, date, and -words, and asks the user before it sends a child's draft to WhatsApp. - -Each child has its own memory, model client, usage file, subagents, -schedules, and turn loop. Its prompt limits work to its area, asks for -only reports that need the parent or the user, and permits an empty -reply. Children have no WhatsApp tools and draft replies instead of -sending them. Clients address only the root. A child's saved schedules -and subagents can resume at startup. The ready log line lists the child -names. - -A message from a parent to its own child enters the child's log as -``user`` and wakes it. A child's final reply reaches the parent as -``work`` with the prefix ``[NAME] `` and wakes it. An empty reply sends -nothing. A busy recipient takes the message between tool calls or in its -next turn. ``agent/messages.json``, with mode 0600, keeps messages until -their recipient finishes its turn. A daemon stop preserves queued -messages and reports for the next start without a second log entry. An -interrupted turn can repeat its model calls and tools after restart. -Backups lock this queue with the agent files, including children made -during the daemon run. The root uses ``tell`` for both children and -subagents. ``zoom("Name")`` gives a child's view or a subagent's whole -log. ``spawn`` starts short-lived subagents, not permanent children. - -************** - User service -************** - -The Nix package includes ``share/systemd/user/onechat.service``. It runs -``onechat daemon`` from that package, starts at login when enabled, and -restarts 5 seconds after a failure. A stop sends SIGTERM to the daemon -and allows up to 120 seconds for cleanup. Its output goes to the user -journal. The service has the rights of the user, as the terminal daemon -does. - -Build a package with a separate result link. Keep this link so that Nix -does not collect the package while the service uses it: +``/whatsapp status`` + Shows the link and the routes of chats to child agents. + +``/whatsapp unlink`` + Ends the link. + +The session file gives full access to the account. If it leaks, remove +the device in Linked devices on the phone. + +User service +============ + +Run the daemon as a systemd user service. Keep the ``result-service`` +link, so that Nix keeps the package: .. code:: console @@ -192,112 +89,41 @@ does not collect the package while the service uses it: systemctl --user link --force "$PWD/result-service/share/systemd/user/onechat.service" systemctl --user edit onechat.service -In the editor, set the data directory with this drop-in. Replace -``DATA_DIR`` with an absolute path to the data directory. Set the same -``ONECHAT_DIR`` for each terminal client. Without a drop-in, the data -directory rules above apply to the environment of the user manager, not -to the development shell. +In the editor, set the data directory, and the tools that the agent may +use in its shell: .. code:: ini [Service] Environment="ONECHAT_DIR=DATA_DIR" + Environment="PATH=TOOLS_PATH" -The service does not read the environment of an open terminal. If the -agent needs tools from a shell, set ``Environment="PATH=TOOLS_PATH"`` in -the same drop-in, with ``TOOLS_PATH`` replaced by the complete search -path, including bash. For the API endpoint, use -``EnvironmentFile=ENV_FILE``, with ``ENV_FILE`` replaced by an absolute -path to a mode 0600 file that sets ``OPENAI_API_KEY``. Keep secrets out -of the unit and Nix sources. - -Use the existing data directory, or copy the development data before the -first service start. For a copy, first stop the terminal daemon with -Ctrl-C and wait for it to exit. Replace ``DATA_DIR`` below with a new, -empty destination. If the development shell uses a custom -``ONECHAT_DIR``, replace ``.devenv/onechat`` with that source. Copy the -whole directory so that ``auth.json``, ``telegram.json``, -``config.json``, ``agent/``, and ``whatsapp/`` stay together, including -the WhatsApp session and its database files. The copy preserves the file -modes and the source as a backup. Do not copy or use the same data from -two running daemons. - -.. code:: console - - install -d -m 0700 DATA_DIR - cp -a .devenv/onechat/. DATA_DIR/ - -After the terminal daemon exits and the data is ready, enable and start -the service: +Then start it, and let it run while you are logged out: .. code:: console systemctl --user daemon-reload systemctl --user enable --now onechat.service - systemctl --user status onechat.service - ONECHAT_DIR=DATA_DIR ./result-service/bin/onechat - journalctl --user -u onechat.service -f + loginctl enable-linger "$USER" -After a pull, update the package and unit with: - -.. code:: console +After a ``git pull``, ``bin/service-update`` builds and restarts it. +Never run two daemons on the same data. - git pull - bin/service-update - -The script builds with the same ``result-service`` link, reloads the -user manager, and restarts the service only if it is active. The unit -link follows ``result-service``, so no new link is needed. - -The service starts with the user manager at login, not at boot without a -login. To stop it, or also remove its start at login: +Backups +======= .. code:: console - systemctl --user stop onechat.service - systemctl --user disable onechat.service - -******************** - Backup and restore -******************** + onechat backup BACKUP_DIR/onechat.tar.gz + ONECHAT_DIR=EMPTY_DIR onechat restore BACKUP_DIR/onechat.tar.gz -Back up the running daemon to one mode 0600 ``.tar.gz`` archive. The -command prints its path and size in bytes. With no daemon, it fails. - -.. code:: console +A backup runs while the daemon runs. Restore needs a stopped daemon and +an empty directory. **An archive holds your logins, the bot token, and +the WhatsApp link**, so keep it private. - onechat backup BACKUP_FILE.tar.gz - ONECHAT_DIR=EMPTY_DATA_DIR onechat restore BACKUP_FILE.tar.gz - -The daemon flushes memory and holds its write locks during the agent -copy. It also flushes and locks the child agents and all subagent logs. -It copies the WhatsApp session with SQLite ``VACUUM INTO``, not as raw -database files. It then compresses the copy and installs the archive -through a temporary file and a rename. Use a backup path outside the -data directory. - -The archive includes configuration, instructions, the log, tree, -indexes, views, layout, schedules, subagents, children, Telegram offset, -tokens, and WhatsApp state and session. It excludes sockets, lock files, -``probe/``, and the ``agent-backup-*`` import copies. Links and special -files fail, rather than enter the archive. **The archive holds -secrets**, including login tokens, the Telegram bot token, and the -WhatsApp link. Keep it private. There is no encryption or remote copy. - -Restore needs a stopped daemon and an empty destination. It checks the -token lock and validates the archive before it installs files, with -their modes. It rejects unsafe paths, links, and special files. A failed -restore can leave ``auth.lock`` in the destination. Use another empty -directory for a retry. Do not run the source and restored WhatsApp links -at the same time. Check the restored log and view before you use it in -place of the source. - -For a daily local archive with the last 14 kept, save this script as -``BACKUP_SCRIPT``. Replace ``BACKUP_DIR`` with a private archive -directory outside the data directory. Replace ``ONECHAT_BINARY`` with -the absolute path through the stable ``result-service/bin/onechat`` -package link. Make the archive directory with mode 0700 before you -enable the timer. +For a daily backup with the last 14 kept, save this script, with +``BACKUP_DIR`` replaced by a private directory outside the data +directory: .. code:: sh @@ -305,399 +131,110 @@ enable the timer. set -eu umask 077 cd BACKUP_DIR - ONECHAT_BINARY backup "onechat-$(date +%F).tar.gz" - find . -maxdepth 1 -type f -name 'onechat-????-??-??.tar.gz' -printf '%f\n' | - sort -r | tail -n +15 | - while IFS= read -r file; do - rm -- "$file" - done - -Save these units as ``onechat-backup.service`` and -``onechat-backup.timer`` in the systemd user unit directory. Replace -``DATA_DIR`` and ``BACKUP_SCRIPT`` with absolute paths. The service uses -the daemon unit from User service above. + onechat backup "onechat-$(date +%F).tar.gz" + ls -1 onechat-????-??-??.tar.gz | sort -r | tail -n +15 | xargs -r rm -- -.. code:: ini +and run it from a ``Type=oneshot`` user service with the same +``ONECHAT_DIR`` and an ``OnCalendar=daily`` timer. - [Unit] - Description=Back up OneChat data - Requires=onechat.service - After=onechat.service - - [Service] - Type=oneshot - Environment="ONECHAT_DIR=DATA_DIR" - UMask=0077 - ExecStart=/bin/sh BACKUP_SCRIPT - -.. code:: ini - - [Unit] - Description=Back up OneChat each day - - [Timer] - OnCalendar=daily - Persistent=true - - [Install] - WantedBy=timers.target - -Enable the timer with the user manager. A failed backup leaves the -existing archives in place and skips removal of old archives. +********* + Talking +********* .. code:: console - systemctl --user daemon-reload - systemctl --user enable --now onechat-backup.timer - systemctl --user list-timers onechat-backup.timer - -************ - Daemon log -************ - -The daemon writes short events with the part prefix and ordered -``key=value`` attributes. An error cause comes last. Normal events use -info, failures that can be retried or lose nothing use warning, and -failures that lose data or stop work use error. - -With ``JOURNAL_STREAM`` set, each line carries a syslog priority without -a time stamp or level word. The user journal can show only warnings and -errors: - -.. code:: console - - journalctl --user -u onechat -p warning - -The summary warning names its node. A schedule run has ``late`` only -when it ran late: - -.. code:: text - - <4>onechat: summary failed level=0 index=0: fake failure - <6>onechat: schedule ran id=1 - <6>onechat: schedule ran id=2 late=1m0s + onechat -Outside the journal, each line starts with the time stamp and level. +``onechat`` connects to the running daemon. Each line is a message, and +replies show with Markdown styles and tables. These lines are commands: -********** - Telegram -********** +``/stop`` + Interrupts the running turn. -The daemon also connects the agent to a Telegram bot when -``telegram.json`` exists in the data directory. It polls the Bot API for -new messages, so it needs no open port. +``/stop NAME`` + Stops the subagent NAME. -#. Make a bot with BotFather and keep its token. +``/whatsapp ...`` + The WhatsApp commands above. -#. Find your numeric Telegram user ID. +In Telegram, write as in the client. ``onechat help`` lists all +commands. -#. Write the token and the ID to ``telegram.json``, and give the file - mode 0600, because the token is a secret. The daemon refuses the file - with any other mode. +********************* + What the agent does +********************* -#. Start the daemon, open a private chat with the bot, and write. +Memory + The agent remembers the whole chat as summaries that it can open + again, so it never starts over. ``agent/instructions.md`` holds your + lasting preferences. The agent can propose a change to it and makes + the change only after you agree. -.. code:: json +Tools + The agent runs shell commands as you, in its working directory, with + no sandbox. - {"token": "123456:ABC-token-from-BotFather", "owner": 123456789} +Subagents + For long tasks it starts subagents in the background, such as + ``GreenOtter``, and gets their reports as messages. -The bot reads only the private chat of the owner and ignores all other -updates. Each text message goes to the agent as a user message, the line -``/stop`` interrupts the running turn, and the line ``/stop NAME`` stops -the subagent NAME. The bot sends the replies in parts of at most 4096 -characters, with their Markdown as Telegram formats: bold, italic, -strikethrough, code, code blocks, links, and headings as bold text. When -Telegram refuses the formats of a part, the bot sends the part again as -plain text. Each error goes as a short message in plain text. It shows -the typing action while a turn runs. It does not send the tool calls. -For a subagent it sends only ``[NAME] started`` and ``[NAME] finished``, -``[NAME] stopped``, or ``[NAME] failed``, because the reply of the agent -carries the content of the report. It sends each message that the user -types in the terminal client, with the mark ``[terminal]``. - -``agent/telegram-offset.json`` holds the ID of the next update. The bot -moves it past an update only after the message is durable in the log, so -a restart of the daemon does not lose a message. A crash just after the -write of a message can make the message repeat once. A network failure -or a rate limit makes the bot wait and try again, and does not stop the -daemon. - -********** - WhatsApp -********** - -The daemon connects to the WhatsApp account of the user as a linked -device. The terminal client and Telegram take three commands: +Child agents + A child is a permanent agent for one area of your life, such as + ``House``, with its own memory. The root creates it, talks to it, and + hears from it only what needs you. Children never send WhatsApp + messages; they draft replies for you to approve. -``/whatsapp pair PHONE`` - Links the account of PHONE, the number with its country code, such as - ``+15550001111``. The daemon answers with the steps and then with a - pairing code in its own message: in WhatsApp on the phone, open - Linked devices, tap Link a device, then tap Link with phone number - instead, and enter the code. The code expires in about 2 minutes, and - the same command gives a new code. When the link succeeds, the daemon - answers with the linked number. Linked devices on the phone shows the - device as OneChat. The name applies at the pairing, so an existing - link keeps its old name until ``/whatsapp unlink`` and a new - ``/whatsapp pair PHONE``. +WhatsApp + The agent reads your chats in batches and wakes for a private message + or a mention of you. It finds chats and sends messages only when you + ask. It proposes to route a chat that is only about one area to the + child of that area, and sets the route after you agree. -``/whatsapp status`` - Shows the link: not linked, linked and connected, or linked and - disconnected, with the times of the connection and the WhatsApp - routes. +Schedules + Ask for reminders and timed tasks, such as "every weekday at 8, tell + me what my groups talked about". Each run uses the quota, so runs are + at least 5 minutes apart. -``/whatsapp unlink`` - Ends the link and deletes the session. When WhatsApp is disconnected, - the phone does not get the logout, so remove the device in Linked - devices on the phone. - -The answers go only to the client that asked, and the commands do not go -to the agent, so the agent never gets the phone number or the code. The -chat gets only the notes that WhatsApp is linked or unlinked. - -The session is ``whatsapp/session.db`` in the data directory, with mode -0600 in a directory with mode 0700. It gives full access to the WhatsApp -account, so the daemon refuses the file when its mode is not 0600. The -shell of the agent runs as the user, so it can read the session. If the -session leaks, remove the linked device in WhatsApp on the phone, and -the session stops working. Without the file, WhatsApp is not linked. - -The daemon reads the text messages of the private chats and the groups, -from other people and from the user. It does not read media, reactions, -edits, deletions, view-once and disappearing messages, or the history -from before the link. The messages of one chat form one batch until the -chat is quiet for 20 seconds, for 2 minutes or 50 messages at the most. -Each batch goes to the chat of the agent as one work message that starts -with ``[WhatsApp]``, with the name and the ID of the chat, and the time, -the sender, and the text of each message. The messages of the user show -as ``You``. ``whatsapp/state.json``, with mode 0600, keeps the batches -that are not in the chat yet, so they survive a restart, and the records -of the sends. A crash just after a batch goes to the chat can make the -batch repeat once. - -A batch of a private chat with a message of another person starts a -turn, and so does a batch of a group with a message that mentions the -user or replies to a message of the user. The other batches go to the -chat with no turn, and the agent sees them at its next turn. A batch -that starts a turn ends with a line that tells the agent to end its turn -with an empty reply when nothing in the batch needs the user. No client -shows an empty reply, so such a batch gives no message in Telegram. The -terminal client shows each batch, and Telegram shows none. - -The phone number of the user never goes to the chat: a mention of the -user shows as ``@you``, the number shows as ``OWN``, and the chat of the -user with itself is not read. But the IDs of the other chats go to the -chat and to the model, because the agent needs them to send. The ID of a -private chat holds the phone number of the contact, such as -``15550002222@s.whatsapp.net``, and the ID of a group can hold the -number of the person who made it. - -While the daemon is stopped or disconnected, the WhatsApp servers keep -the messages for the linked device and send them at the next connection. -They keep them only for a limited time, so a long stop can lose -messages. These messages form batches by their original times, as live -messages do. The daemon waits for the end of this catch-up, or one -minute at the most, and then gives all its unrouted batches to the root -in one turn, not one turn for each chat. Routed batches wake their -child. The batches of ``whatsapp/state.json`` at the start wait for the -first catch-up in the same way. - -The agent finds a chat with the tool ``whatsapp_find``: it gives the -contacts and the groups whose name holds a text, with no regard to case -and accents, with their IDs. The names of the contacts come from the -session, and the names of the groups from WhatsApp, kept in memory for -one hour. The names arrive from the phone some minutes after the link, -and until then the tool says that they are not ready. The agent sends a -text message with the tool ``whatsapp_send``, only when the user asks -for it. Each send has an ID that the agent gives: a repeated call with -the same ID gives the answer of the first call and sends nothing. The -state file keeps the last 1000 send records. When the daemon stops -during a send, the result is not known, and the tool tells the agent to -ask the user to look at the chat in WhatsApp. Without a link, both tools -tell the agent to ask the user to run ``/whatsapp pair PHONE``. Only the -agent has the WhatsApp tools, not its subagents. - -Each batch of a group goes to the chat, also with no turn, and the -memory tree summarizes it with model calls, which use the quota. - -The root manages source routes with ``route(source, agent)``, -``unroute(source)``, and ``routes()``, only after the user agrees. A -source is ``whatsapp:CHAT-ID``, with the ID from ``whatsapp_find`` or a -``[WhatsApp]`` message, and the agent must be its child. The root keeps -the map in ``agent/routes.json`` with mode 0600. Children and subagents -have no route tools. There is no command to change routes. Each routed -batch enters only its child's log and wakes it, including a group batch -with no mention. The durable agent queue prevents a second log entry -after a restart. An unrouted source or a missing child uses the root. A -queued batch whose child is removed while stopped goes to the root. - -When the session holds a linked device, the daemon connects at its -start. If the first connection fails, it tries again after 2 seconds, -then doubles the wait up to 5 minutes. The inbox and commands stay -active during each wait. An unlink or a logout stops these retries. -After a network failure of an open connection, the library connects -again by itself. When a disconnection lasts 10 minutes, all clients get -one notice, and one more when the connection comes back. When WhatsApp -ends the connection for good, for example because another program uses -the same session, all clients get a notice at once. When the user -removes the device in the phone, the daemon deletes the session and -tells all clients one time. - -************************* - Import WhatsApp history -************************* - -Stop the daemon and finish all subagents before an import. Export each -chat from WhatsApp as a ZIP with ``_chat.txt``. Keep the chat name in -the ZIP name, such as ``WhatsApp Chat - Example.zip``. Read the plan -first: +Import + Old WhatsApp chats can join the memory in date order. Stop the + daemon, export the chat as a ZIP, and run the plan before the import. + The import rebuilds the target's memory with model calls: .. code:: console - onechat import -n CHAT.zip OTHER.zip - onechat import CHAT.zip OTHER.zip - onechat import -n --agent House CHAT.zip OTHER.zip - onechat import --agent House CHAT.zip OTHER.zip - -The target is the root unless ``--agent NAME`` selects an existing -direct child. A missing child is refused. The rebuild uses the target's -log, prompt, cache key, and usage records. The root keeps its history -when the target is a child. An import or plan is refused while the -target has pending agent deliveries. Start the daemon and let those -turns finish before you stop it and try again. This keeps their saved -log IDs valid. - -``-n`` makes no model call and prints the target name, chats, chat-day -batches, local dates, cutoffs, text bytes, merged message count, and -estimated calls. The import keeps multiline text, system lines, and -omitted-media markers. It maps the session's profile name to ``You``. -Without that name, it keeps the senders as written and says so in the -plan. An exact, unambiguous local contact or batch name supplies the -chat ID when known. No network lookup is made. - -Each chat and local day becomes one work message marked ``(imported)``. -For each chat, the cutoff is the first message time of its earliest live -WhatsApp batch across the root and target logs. Only messages before -that cutoff enter the import. Live times have minute precision, so the -cutoff excludes that whole minute. A chat-day already imported into the -target is skipped. An imported day held only by the root does not skip -that day for a child. Overlapping chat-days in the supplied ZIPs are -refused. Review the exports before import: their text goes to the model -as chat history. - -The import merges the full log in date order and builds its tree with -real compactor calls, the agent prompt, and the agent cache key. It uses -a limit of 1,500 calls and the probe's 15-quota-point limit, and prints -the same usage report. Its estimate is about 1.3 calls per merged -message, not a cost guarantee. A stopped or failed run leaves the agent -unchanged. A new run starts over. Success keeps the other agent files -and renames the old directory to ``agent-backup-TIMESTAMP`` before it -installs the new one. A child's staging directory and backup stay in the -root agent directory, outside ``children/``. Its backup is -``NAME-backup-TIMESTAMP``. Keep the backup until the new memory is -checked, then start the daemon. Message IDs change to their positions in -the merged log. - -*********** - Schedules -*********** - -The agent sets reminders and timed tasks when the user asks for them, -such as "remind me tomorrow at 9" or "every weekday at 8, tell me what -my WhatsApp groups talked about". A schedule has a text, the task for -the agent, and either one time or a cron expression. The agent also -lists the schedules with their next times, and removes one when the user -asks. The root and its children have the schedule tools, not their -subagents. Each schedule records its owner and wakes only that agent. - -A cron expression has five fields: minute, hour, day of month, month, -and day of week, where 0 and 7 are Sunday. A field is a comma list of -``*``, ``N``, and ``N-M``, and ``*`` and ``N-M`` can have a step ``/S``, -as in ``0 8 * * 1-5`` or ``*/15 9-17 * * *``. When both day fields start -with something other than ``*``, a day matches when either field -matches. Names such as ``MON``, macros such as ``@daily``, the -characters ``? L W #``, and ``N/S`` are refused. The runs must be at -least 5 minutes apart, because each run is a full turn that uses the -quota, so ``*/5 * * * *`` is the shortest schedule. - -The times use the time zone of the machine. A time that the clock skips -at the start of daylight saving time runs later by the length of the -skip. A time that the clock repeats at its end runs once, at its first -occurrence. - -At its time, a schedule goes to the chat as a work message that starts -with ``[Schedule ID]``, with its time and its text, and it starts a -turn. The message ends with a line that tells the agent to end its turn -with an empty reply when nothing needs the user. The terminal client -shows the message, and Telegram shows only the reply. After its run, a -one-time schedule goes, and a repeating schedule gets its next time. - -``agent/schedules.json``, with mode 0600, holds the schedules. Each -change writes it before the tool answers, so the schedules survive a -restart. A schedule that came due while the daemon was stopped runs once -at the next start, with its original time and a line that says that the -run is late. A repeating schedule that missed several times runs once. A -crash just after a run goes to the chat can make the run repeat once. - -******* - Probe -******* + onechat import -n [--agent NAME] CHAT.zip + onechat import [--agent NAME] CHAT.zip -``onechat probe LOG COUNT [BUILDS]`` measures the compactor on a log. It -appends the first COUNT messages of LOG, one JSON object ``{"kind", -"text"}`` on each line, to a new chat in ``probe/`` in the data -directory, and builds its tree with real model calls. The calls use the -quota of the account. The run stops at the first failed call, after 660 -calls, or when the used percent of a quota window grows by more than 15 -points. At the end it reports the calls per message, the cached share of -the input tokens, the retries, the line sizes, and the quota at the -first and the last call. BUILDS, from 1 to 8, limits the model calls at -once in the run. +***** + Log +***** -A test writes a generated log of readable made-up messages: +The daemon writes short events with ``key=value`` attributes to the user +journal, at three levels: info, warning for a failure that is retried or +loses nothing, and error for a failure that loses data or stops work. .. code:: console - ONECHAT_PROBE_LOG=probe.jsonl go test -run TestProbeLogHasTheSyntheticMix + journalctl --user -u onechat -f + journalctl --user -u onechat -p warning ************* Development ************* Nix supplies the Go toolchain and all tools. Enter the development shell -with direnv: - -.. code:: console - - direnv allow - -Or enter it directly: - -.. code:: console - - nix develop - -The shell sets ``ONECHAT_DIR`` to ``.devenv/onechat`` in the checkout, -unless it is already set. Development data goes there, and Git ignores -``.devenv``. - -Build and run: +with ``direnv allow`` or ``nix develop``. Then: .. code:: console nix build - ./result/bin/onechat - -``onechat version`` prints ``yyyy-mm-dd-gitshort`` for a Nix build, adds -``-dirty`` for a dirty tree, and prints ``dev`` for a direct Go build. - -Format and check all files: - -.. code:: console - + ./result/bin/onechat version bin/format bin/check nix flake check -``bin/check`` and ``nix build`` run a short replay of the synthetic log. -``nix flake check`` also runs the full replay of 100,000 messages. +``onechat version`` prints ``yyyy-mm-dd-gitshort`` for a Nix build and +``dev`` for a direct Go build. ``bin/check`` runs the linters and a +short replay of a synthetic log, and ``nix flake check`` also runs the +full replay of 100,000 messages. ``onechat probe LOG COUNT`` measures +the memory summaries on a log with real model calls. -- 2.53.0 From 36229ef68a30c5a117599037d79fa1fc708fbfef Mon Sep 17 00:00:00 2001 From: "Anderson A. Giacomolli" Date: Tue, 10 Nov 2026 11:35:12 -0300 Subject: [PATCH 02/16] Store shared private images by content hash Keep one immutable JPEG, PNG, or WebP file per image in the shared data store. Validate size and format before atomic publication, and preserve image files and modes in backups. The WebP config decoder needs x/image because the standard library has no WebP decoder. --- agent.go | 10 +++ backup_archive.go | 2 +- go.mod | 1 + go.sum | 2 + image.go | 174 +++++++++++++++++++++++++++++++++++++++++++ image_helper_test.go | 27 +++++++ image_test.go | 112 ++++++++++++++++++++++++++++ nix/package.nix | 2 +- 8 files changed, 328 insertions(+), 2 deletions(-) create mode 100644 image.go create mode 100644 image_helper_test.go create mode 100644 image_test.go diff --git a/agent.go b/agent.go index dbcabd2..97d6685 100644 --- a/agent.go +++ b/agent.go @@ -80,6 +80,16 @@ type Agent struct { config AgentConfig } +// Images returns the shared data-directory image store, also for a child. +func (a *Agent) Images() *ImageStore { + root := a.dir + if a.config.Parent != "" { + root = filepath.Dir(filepath.Dir(root)) + } + + return &ImageStore{dir: filepath.Join(filepath.Dir(root), imageDirName)} +} + // CacheKey returns the prompt cache key of every model call of the agent. func (a *Agent) CacheKey() string { return "onechat-" + a.config.ID diff --git a/backup_archive.go b/backup_archive.go index 9560cd1..2ec9352 100644 --- a/backup_archive.go +++ b/backup_archive.go @@ -270,7 +270,7 @@ func backupExcluded(name string) bool { return true } - if filepath.Base(name) == "lock" { + if filepath.Base(name) == "lock" || (first == imageDirName && strings.HasPrefix(filepath.Base(name), ".image-")) { return true } diff --git a/go.mod b/go.mod index fcf4681..4643a53 100644 --- a/go.mod +++ b/go.mod @@ -5,6 +5,7 @@ go 1.26.5 require ( github.com/stretchr/testify v1.12.0 go.mau.fi/whatsmeow v0.0.0-20261107160454-79dc2a8c33df + golang.org/x/image v0.47.0 golang.org/x/sys v0.49.0 golang.org/x/text v0.43.0 modernc.org/sqlite v1.62.0 diff --git a/go.sum b/go.sum index 13d902b..16298eb 100644 --- a/go.sum +++ b/go.sum @@ -52,6 +52,8 @@ golang.org/x/crypto v0.58.0 h1:COkYLr4k7nDI4r6QtMML2AfzeIswWHAJthLS6K/M/54= golang.org/x/crypto v0.58.0/go.mod h1:Xh+kl5A+M0gMiWDytH2Hhr5Z7WbNCpv/eB/EFgfSk+w= golang.org/x/exp v0.0.0-20261014143641-f0251eb4aa3e h1:HgI6W1OolKK6tcV74is+rl9SMT0njsHlp0TbQiGXI/A= golang.org/x/exp v0.0.0-20261014143641-f0251eb4aa3e/go.mod h1:DBoxwWhjaW8DJJw/YeM4zpeOKMC1FMXSblsYy+1mFSw= +golang.org/x/image v0.47.0 h1:za/8gVz++PPZqmZJBNSISICN5Rrj5zixP/8K++fgmqo= +golang.org/x/image v0.47.0/go.mod h1:IkN5o4A7vq55vXN8FWjjnhg9Rs2GOtffqsSi9mWAKYE= golang.org/x/mod v0.42.0 h1:ICeL3t2L30ljSPHHxDnR4fqVcBt/zR2zJAb9juavmQg= golang.org/x/mod v0.42.0/go.mod h1:u5fzIrRInkL8MaO1cbRkUHT64v303YIT5KEqpNr7jRs= golang.org/x/net v0.61.0 h1:RR+6j/BTBCrggOLugxa+66R8FvoN6MHlzDlZIbN9YFw= diff --git a/image.go b/image.go new file mode 100644 index 0000000..d6a0b20 --- /dev/null +++ b/image.go @@ -0,0 +1,174 @@ +package main + +import ( + "bytes" + "crypto/sha256" + "encoding/hex" + "errors" + "fmt" + "image" + _ "image/jpeg" + _ "image/png" + "io" + "os" + "path/filepath" + "strings" + + _ "golang.org/x/image/webp" +) + +const ( + imageDirName = "images" + imageMaxBytes = 20 << 20 +) + +// ImageStore holds immutable images shared by the root, children, and +// subagents. Names are the SHA-256 of the bytes with the decoded format as +// extension. Publication is atomic and does not replace an existing image. +type ImageStore struct { + dir string +} + +// Save validates and stores data, and returns its filename. Only JPEG, PNG, and +// WebP images up to 20 MB are accepted. +func (s *ImageStore) Save(data []byte) (name string, err error) { + format, err := imageFormat(data) + if err != nil { + return "", fmt.Errorf("check image: %w", err) + } + + sum := sha256.Sum256(data) + name = hex.EncodeToString(sum[:]) + "." + format + err = os.MkdirAll(s.dir, 0o700) + if err != nil { + return "", fmt.Errorf("create image directory: %w", err) + } + + file, err := os.CreateTemp(s.dir, ".image-*") + if err != nil { + return "", fmt.Errorf("create image file: %w", err) + } + + defer func() { + err = errors.Join(err, os.Remove(file.Name())) + }() + + _, err = file.Write(data) + if err == nil { + err = file.Sync() + } + + err = errors.Join(err, file.Close()) + if err != nil { + return "", fmt.Errorf("write image: %w", err) + } + + err = os.Link(file.Name(), filepath.Join(s.dir, name)) + if err != nil && !errors.Is(err, os.ErrExist) { + return "", fmt.Errorf("publish image: %w", err) + } + + return name, nil +} + +// Read returns the bytes of a stored image after checking its name and content. +func (s *ImageStore) Read(name string) ([]byte, error) { + if !imageName(name) { + return nil, errors.New("invalid image name") + } + + data, err := imageReadFile(filepath.Join(s.dir, name)) + if err != nil { + return nil, fmt.Errorf("read stored image: %w", err) + } + + format, err := imageFormat(data) + if err != nil { + return nil, fmt.Errorf("check stored image: %w", err) + } + + sum := sha256.Sum256(data) + if name != hex.EncodeToString(sum[:])+"."+format { + return nil, errors.New("image content does not match its name") + } + + return data, nil +} + +func imageFormat(data []byte) (string, error) { + if len(data) > imageMaxBytes { + return "", errors.New("image exceeds 20 MB") + } + + config, format, err := image.DecodeConfig(bytes.NewReader(data)) + if err != nil || config.Width <= 0 || config.Height <= 0 { + return "", errors.New("use a JPEG, PNG, or WebP image") + } + + switch format { + case "jpeg": + return "jpg", nil + case "png", "webp": + return format, nil + default: + return "", errors.New("use a JPEG, PNG, or WebP image") + } +} + +func imageName(name string) bool { + hash, ext, found := strings.Cut(name, ".") + return found && len(hash) == 64 && strings.Trim(hash, "0123456789abcdef") == "" && (ext == "jpg" || ext == "png" || ext == "webp") +} + +func imageReadFile(path string) (data []byte, err error) { + file, err := os.Open(path) + if err != nil { + return nil, fmt.Errorf("open image: %w", err) + } + + defer func() { + err = errors.Join(err, file.Close()) + }() + + data, err = io.ReadAll(io.LimitReader(file, imageMaxBytes+1)) + if err != nil { + return nil, fmt.Errorf("read image: %w", err) + } + + if len(data) > imageMaxBytes { + return nil, errors.New("image exceeds 20 MB") + } + + return data, nil +} + +// imageFiles reads only complete marker lines with valid store filenames. +func imageFiles(text string) []string { + var names []string + + for line := range strings.SplitSeq(text, "\n") { + name, found := strings.CutPrefix(line, "[image ") + if !found || !strings.HasSuffix(name, "]") { + continue + } + + name = strings.TrimSuffix(name, "]") + if imageName(name) { + names = append(names, name) + } + } + + return names +} + +func imageText(names []string, caption string) string { + var text strings.Builder + + for _, name := range names { + text.WriteString("[image " + name + "]\n") + } + + text.WriteString(caption) + + return strings.TrimSuffix(text.String(), "\n") +} diff --git a/image_helper_test.go b/image_helper_test.go new file mode 100644 index 0000000..94788a2 --- /dev/null +++ b/image_helper_test.go @@ -0,0 +1,27 @@ +package main + +import ( + "bytes" + "image" + "image/color" + "image/png" + "testing" + + "github.com/stretchr/testify/require" +) + +func imageTestPNG(t *testing.T, width, height int) []byte { + t.Helper() + + picture := image.NewRGBA(image.Rect(0, 0, width, height)) + picture.Set(0, 0, color.RGBA{ + R: 255, + A: 255, + }) + + var data bytes.Buffer + + require.NoError(t, png.Encode(&data, picture)) + + return data.Bytes() +} diff --git a/image_test.go b/image_test.go new file mode 100644 index 0000000..24007d7 --- /dev/null +++ b/image_test.go @@ -0,0 +1,112 @@ +package main + +import ( + "bytes" + "crypto/sha256" + "encoding/hex" + "os" + "path/filepath" + "sync" + "testing" + + "github.com/stretchr/testify/require" +) + +func TestImagesArePrivateSharedAndWrittenOnce(t *testing.T) { + root, err := openAgent(filepath.Join(t.TempDir(), "agent")) + require.NoError(t, err) + + child := &Agent{ + dir: filepath.Join(root.dir, "children", "House"), + config: AgentConfig{Parent: root.config.Name}, + } + require.Equal(t, root.Images().dir, child.Images().dir) + + data := imageTestPNG(t, 3, 2) + sum := sha256.Sum256(data) + expected := hex.EncodeToString(sum[:]) + ".png" + store := root.Images() + name, err := store.Save(data) + require.NoError(t, err) + require.Equal(t, expected, name) + + info, err := os.Stat(filepath.Join(store.dir, name)) + require.NoError(t, err) + require.Equal(t, os.FileMode(0o600), info.Mode().Perm()) + + var workers sync.WaitGroup + + for range 8 { + workers.Go(func() { + saved, saveErr := child.Images().Save(data) + require.NoError(t, saveErr) + require.Equal(t, name, saved) + }) + } + + workers.Wait() + + again, err := os.Stat(filepath.Join(store.dir, name)) + require.NoError(t, err) + require.True(t, os.SameFile(info, again)) + + files, err := os.ReadDir(store.dir) + require.NoError(t, err) + require.Len(t, files, 1) + + actual, err := store.Read(name) + require.NoError(t, err) + require.Equal(t, data, actual) +} + +func TestImageStoreRejectsLargeUnsupportedAndUnsafeImages(t *testing.T) { + store := &ImageStore{dir: filepath.Join(t.TempDir(), imageDirName)} + _, err := store.Save(bytes.Repeat([]byte{0}, imageMaxBytes+1)) + require.ErrorContains(t, err, "exceeds 20 MB") + + _, err = store.Save([]byte("not an image")) + require.ErrorContains(t, err, "JPEG, PNG, or WebP") + + _, err = store.Read("../secret.png") + require.ErrorContains(t, err, "invalid image name") + + _, err = os.Stat(store.dir) + require.ErrorIs(t, err, os.ErrNotExist) +} + +func TestBackupKeepsImagesAndTheirModes(t *testing.T) { + dir := t.TempDir() + store := &ImageStore{dir: filepath.Join(dir, imageDirName)} + data := imageTestPNG(t, 2, 2) + name, err := store.Save(data) + require.NoError(t, err) + require.NoError(t, os.WriteFile(filepath.Join(store.dir, ".image-incomplete"), []byte("partial"), 0o600)) + + stage := t.TempDir() + archive := filepath.Join(t.TempDir(), "images.tar.gz") + require.NoError(t, backupCopy(t.Context(), dir, stage, archive)) + + _, err = backupArchive(t.Context(), stage, archive) + require.NoError(t, err) + + restored := t.TempDir() + require.NoError(t, backupRestore(t.Context(), archive, restored)) + + actual, err := (&ImageStore{dir: filepath.Join(restored, imageDirName)}).Read(name) + require.NoError(t, err) + require.Equal(t, data, actual) + + info, err := os.Stat(filepath.Join(restored, imageDirName, name)) + require.NoError(t, err) + require.Equal(t, os.FileMode(0o600), info.Mode().Perm()) + + _, err = os.Stat(filepath.Join(restored, imageDirName, ".image-incomplete")) + require.ErrorIs(t, err, os.ErrNotExist) +} + +func TestImageMarkersKeepCaptionAndOnlySafeNames(t *testing.T) { + name := "aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa.png" + text := imageText([]string{name}, "Read the label.") + require.Equal(t, "[image "+name+"]\nRead the label.", text) + require.Equal(t, []string{name}, imageFiles(text+"\n[image ../secret.png]\ntext [image "+name+"]")) +} diff --git a/nix/package.nix b/nix/package.nix index bbd5fb6..41cd24f 100644 --- a/nix/package.nix +++ b/nix/package.nix @@ -10,7 +10,7 @@ buildGoModule { subPackages = [ "." ]; ldflags = [ "-X main.version=${version}" ]; - vendorHash = "sha256-HEYbx27w3s+x29/gDLUOCrkBJu9Ke1ue+Ms++LdHpyQ="; + vendorHash = "sha256-nRDUAGSXAK5/L2xyfDh3NbMOZmrGGEIm4vtgfWsxHYU="; # The flake check runs the full replay. checkFlags = [ "-short" ]; -- 2.53.0 From 4343ca5fb6cc86eb887951f4fe16063bab29a238 Mon Sep 17 00:00:00 2001 From: "Anderson A. Giacomolli" Date: Tue, 10 Nov 2026 11:42:27 -0300 Subject: [PATCH 03/16] Carry new images through model request chains Attach stored pixels to new user messages for the whole turn chain. Later turns keep text markers and descriptions, which keeps the request prefix stable. Leaf compaction sees pixels once per build and preserves visible details. Single-message zoom returns text and image parts while the durable log keeps only text. --- build_run_test.go | 2 +- chain.go | 10 +- compactor.go | 37 +++++--- compactor_test.go | 2 +- daemon_agent.go | 2 +- image_model.go | 90 ++++++++++++++++++ image_model_test.go | 199 ++++++++++++++++++++++++++++++++++++++++ memory.go | 5 +- model.go | 25 +++-- probe.go | 2 +- prompt.go | 22 +++-- prompt_test.go | 5 +- rebuild_test.go | 6 +- subagent_run.go | 2 + tools.go | 26 +++++- tree.go | 6 +- turn.go | 14 ++- whatsapp_import.go | 2 +- whatsapp_import_test.go | 4 +- 19 files changed, 406 insertions(+), 55 deletions(-) create mode 100644 image_model.go create mode 100644 image_model_test.go diff --git a/build_run_test.go b/build_run_test.go index b45745e..ef9c3c0 100644 --- a/build_run_test.go +++ b/build_run_test.go @@ -215,7 +215,7 @@ func newTestBuildRun(t *testing.T, server *ModelFake) *BuildRun { root := t.TempDir() usagePath := filepath.Join(root, "usage.jsonl") client := newModelClient(server.URL, ModelAPIKey(modelTestKey), "onechat-"+probeName, usagePath) - memory, err := openMemory(t.Context(), root, newTestClock().Now, newCompactor(client, probePrompt, defaultConfig().Compactor, promptTools()), discardLog()) + memory, err := openMemory(t.Context(), root, newTestClock().Now, newCompactor(client, probePrompt, defaultConfig().Compactor, promptTools(), &ImageStore{dir: t.TempDir()}), discardLog()) require.NoError(t, err) return &BuildRun{ diff --git a/chain.go b/chain.go index c4a4302..23c7dec 100644 --- a/chain.go +++ b/chain.go @@ -32,6 +32,7 @@ type Chain struct { tools *Toolbox request ModelRequest take func() []Message + images *ImageStore } // Run runs the chain from input, the input of its first request, and returns @@ -122,7 +123,12 @@ func (c *Chain) handle(ctx context.Context, input []any, response ModelResponse) input = append(input, outputs...) for _, message := range c.take() { - input = append(input, modelUserMessage(message.Text)) + item, err := imageUserMessage(c.images, message.Text) + if err != nil { + return nil, "", fmt.Errorf("read queued images: %w", err) + } + + input = append(input, item) } return input, "", nil @@ -141,7 +147,7 @@ type ChainFunctionCall struct { type ChainFunctionCallOutput struct { Type string `json:"type"` CallID string `json:"call_id"` - Output string `json:"output"` + Output any `json:"output"` } // chainInputItem returns an output item as an input item for the next request. diff --git a/compactor.go b/compactor.go index 8a16c6e..2b23901 100644 --- a/compactor.go +++ b/compactor.go @@ -4,7 +4,6 @@ import ( "context" "errors" "fmt" - "slices" "strings" "unicode/utf8" ) @@ -20,12 +19,13 @@ var compactorRuler = strings.Repeat("-", nodeBytes) // client with model. instructions gives the system prompt of the agent at each // build. tools are the tools of that agent, so a build sends the same prompt // and tools as its turns. -func newCompactor(client *ModelClient, instructions func() (string, error), model ModelConfig, tools []ModelTool) *Compactor { +func newCompactor(client *ModelClient, instructions func() (string, error), model ModelConfig, tools []ModelTool, images *ImageStore) *Compactor { return &Compactor{ client: client, model: model, instructions: instructions, tools: tools, + images: images, } } @@ -37,17 +37,19 @@ func newCompactor(client *ModelClient, instructions func() (string, error), mode // // When the line is over nodeBytes, the next try sends the same message, plus // one more user message with a note with the size of the newest line and its -// cut at the limit. A try never sends the model's earlier reply or reasoning -// back, so every try begins with the same view and task and has about the same -// size. After at most compactorTries tries, the shortest line is the node. A -// few bytes over the limit is fine, because the view measures the real sizes. A -// call that fails before any line fails the build, and the core tries it again -// at the next message. +// cut at the limit. Only the first call of an image leaf sends its pixels. A +// try never sends the model's earlier reply or reasoning back, so every try +// begins with the same view and task and has about the same size. After at most +// compactorTries tries, the shortest line is the node. A few bytes over the +// limit is fine, because the view measures the real sizes. A call that fails +// before any line fails the build, and the core tries it again at the next +// message. type Compactor struct { client *ModelClient model ModelConfig instructions func() (string, error) tools []ModelTool + images *ImageStore } // Summarize returns the line of the node of task. @@ -62,7 +64,13 @@ func (c *Compactor) Summarize(ctx context.Context, task Task) (string, error) { return "", err } - first := []any{modelUserMessage(task.Context + "\n" + text)} + source := task.Context + "\n" + text + parts, err := imageParts(c.images, source, task.Images) + if err != nil { + return "", fmt.Errorf("read compaction images: %w", err) + } + + first := []any{modelUserParts(parts)} var lines []string @@ -70,7 +78,7 @@ func (c *Compactor) Summarize(ctx context.Context, task Task) (string, error) { for len(lines) < compactorTries { input := first if note != "" { - input = append(slices.Clip(first), modelUserMessage(note)) + input = []any{modelUserMessage(source), modelUserMessage(note)} } var response ModelResponse @@ -111,12 +119,17 @@ func (c *Compactor) Summarize(ctx context.Context, task Task) (string, error) { // compactorTask returns the task of a build, with its input. func compactorTask(task Task) (string, error) { if task.Level == 0 && len(task.Input) == 1 { - return fmt.Sprintf(`Compaction: compress message %d into one line of at most %d bytes + text := fmt.Sprintf(`Compaction: compress message %d into one line of at most %d bytes (about 70 words), the length of this ruler: %s %s -`, task.Index, nodeBytes, compactorRuler, task.Input[0]), nil +`, task.Index, nodeBytes, compactorRuler, task.Input[0]) + if len(task.Images) > 0 { + text += "\nKeep each [image FILE] name and describe its kind, visible text, amounts, names, and context." + } + + return text, nil } if task.Level > 0 && len(task.Input) == 2 { diff --git a/compactor_test.go b/compactor_test.go index 8540515..baa7f57 100644 --- a/compactor_test.go +++ b/compactor_test.go @@ -208,7 +208,7 @@ func newTestCompactor(t *testing.T, server *ModelFake) *Compactor { return newCompactor(client, compactorAdaPrompt, ModelConfig{ Model: "gpt-test", Effort: "high", - }, promptTools()) + }, promptTools(), &ImageStore{dir: t.TempDir()}) } // compactorInputTexts returns the text of each user message of call. Each diff --git a/daemon_agent.go b/daemon_agent.go index 662291c..227d23b 100644 --- a/daemon_agent.go +++ b/daemon_agent.go @@ -55,7 +55,7 @@ func openDaemonAgents(ctx context.Context, dir, address string, auth ModelAuthor // one agent. All model calls use its cache key and usage file. func openDaemonAgent(ctx context.Context, agent *Agent, address string, auth ModelAuthorizer, compactor ModelConfig, clock func() time.Time, logger *slog.Logger) (*DaemonAgent, error) { client := newModelClient(address, auth, agent.CacheKey(), filepath.Join(agent.dir, "usage.jsonl")) - memory, err := openMemory(ctx, agent.dir, clock, newCompactor(client, agent.Prompt, compactor, agent.Tools()), logger) + memory, err := openMemory(ctx, agent.dir, clock, newCompactor(client, agent.Prompt, compactor, agent.Tools(), agent.Images()), logger) if err != nil { return nil, fmt.Errorf("open memory: %w", err) } diff --git a/image_model.go b/image_model.go new file mode 100644 index 0000000..6965142 --- /dev/null +++ b/image_model.go @@ -0,0 +1,90 @@ +package main + +import ( + "encoding/base64" + "errors" + "fmt" + "strings" +) + +// imageUserMessage attaches pixels only to messages selected as new by a turn. +func imageUserMessage(store *ImageStore, text string) (any, error) { + names := imageFiles(text) + if len(names) == 0 { + return modelUserMessage(text), nil + } + + parts, err := imageParts(store, text, names) + if err != nil { + return nil, fmt.Errorf("make image message: %w", err) + } + + return modelUserParts(parts), nil +} + +// imageParts is the shared wire format for image messages and zoom results. +// Tool results use this list directly as function_call_output.output. +func imageParts(store *ImageStore, text string, names []string) ([]ModelInputPart, error) { + parts := []ModelInputPart{{ + Type: "input_text", + Text: text, + }} + for _, name := range names { + if store == nil { + return nil, errors.New("image store is not available") + } + + data, err := store.Read(name) + if err != nil { + return nil, fmt.Errorf("read image part: %w", err) + } + + mime := "image/" + strings.TrimPrefix(name[64:], ".") + if strings.HasSuffix(name, ".jpg") { + mime = "image/jpeg" + } + + parts = append(parts, ModelInputPart{ + Type: "input_image", + ImageURL: "data:" + mime + ";base64," + base64.StdEncoding.EncodeToString(data), + }) + } + + return parts, nil +} + +// imageToolOutput leaves all results as text except zoom of one image message. +// Only the text goes to the log, never the data URLs. +func imageToolOutput(box *Toolbox, name, arguments, text string) (any, error) { + if name != "zoom" || strings.HasPrefix(text, "error:") { + return text, nil + } + + var args ToolZoomArguments + + err := toolDecodeArguments(arguments, &args) + if err != nil { + return nil, fmt.Errorf("decode image zoom: %w", err) + } + + if args.Name != nil || args.N != 1 { + return text, nil + } + + message, err := box.chat.Message(args.ID) + if err != nil { + return nil, fmt.Errorf("read image zoom: %w", err) + } + + names := imageFiles(message.Text) + if len(names) == 0 { + return text, nil + } + + parts, err := imageParts(box.images, text, names) + if err != nil { + return nil, fmt.Errorf("make image zoom: %w", err) + } + + return parts, nil +} diff --git a/image_model_test.go b/image_model_test.go new file mode 100644 index 0000000..53b0f31 --- /dev/null +++ b/image_model_test.go @@ -0,0 +1,199 @@ +package main + +import ( + "encoding/base64" + "encoding/json" + "net/http" + "path/filepath" + "strings" + "testing" + + "github.com/stretchr/testify/require" +) + +func TestNewTurnKeepsImagesInEveryChainRequestButNotLaterTurns(t *testing.T) { + server := newModelFake(t) + server.reply = func(w http.ResponseWriter, call ModelFakeCall) { + if call.number == 0 { + modelWriteEvents(w, modelCompletedEvent([]any{modelFunctionCallItem("fc_1", "call_1", "date", `{"id":0}`)}, 100, 0, 5)) + + return + } + + modelWriteEvents(w, modelCompletedEvent([]any{modelMessageItem("A red square.")}, 100, 0, 5)) + } + + test := newTestTurnLoop(t, server) + data := imageTestPNG(t, 3, 3) + name, err := test.agent.Images().Save(data) + require.NoError(t, err) + + marker := imageText([]string{name}, "What is this?") + _, err = test.loop.Submit(terminalSource, marker) + require.NoError(t, err) + test.events.WaitFor(t, "idle", 1) + require.Equal(t, 2, server.Calls()) + + first := server.Call(0).body["input"].([]any) + second := server.Call(1).body["input"].([]any) + require.Equal(t, first, second[:len(first)]) + + expected := "data:image/png;base64," + base64.StdEncoding.EncodeToString(data) + require.Contains(t, string(server.Call(0).raw), expected) + require.Contains(t, string(server.Call(1).raw), expected) + + _, err = test.loop.Submit(terminalSource, "Continue with text.") + require.NoError(t, err) + test.events.WaitFor(t, "idle", 2) + require.NotContains(t, string(server.Call(2).raw), "input_image") + require.NotContains(t, strings.Join(test.Log(t), "\n"), "base64") +} + +func TestImageArrivingDuringAChainJoinsItsNextRequest(t *testing.T) { + server := newModelFake(t) + arrived := make(chan struct{}) + release := make(chan struct{}) + server.reply = func(w http.ResponseWriter, call ModelFakeCall) { + if call.number == 0 { + close(arrived) + <-release + modelWriteEvents(w, modelCompletedEvent([]any{modelFunctionCallItem("fc_1", "call_1", "date", `{"id":0}`)}, 100, 0, 5)) + + return + } + + modelWriteEvents(w, modelCompletedEvent([]any{modelMessageItem("done")}, 100, 0, 5)) + } + + test := newTestTurnLoop(t, server) + _, err := test.loop.Submit(terminalSource, "Start.") + require.NoError(t, err) + <-arrived + + name, err := test.agent.Images().Save(imageTestPNG(t, 2, 2)) + require.NoError(t, err) + + _, err = test.loop.Submit(terminalSource, imageText([]string{name}, "Read this.")) + require.NoError(t, err) + close(release) + test.events.WaitFor(t, "idle", 1) + require.NotContains(t, string(server.Call(0).raw), "input_image") + require.Contains(t, string(server.Call(1).raw), "input_image") +} + +func TestLeafCompactionDescribesImagesOnceAndMergesOnlyText(t *testing.T) { + server := newModelFake(t) + store := &ImageStore{dir: filepath.Join(t.TempDir(), imageDirName)} + name, err := store.Save(imageTestPNG(t, 2, 2)) + require.NoError(t, err) + + server.reply = func(w http.ResponseWriter, call ModelFakeCall) { + text := "receipt, Ada, $12, red square" + if call.number == 0 { + text = strings.Repeat("detail ", 100) + } + + modelWriteEvents(w, modelCompletedEvent([]any{modelMessageItem(text)}, 100, 0, 5)) + } + + compactor := newTestCompactor(t, server) + compactor.images = store + task := Task{ + Level: 0, + Index: 0, + Context: "\n", + Input: []string{"user: " + imageText([]string{name}, "Read the receipt.")}, + Images: []string{name}, + } + text, err := compactor.Summarize(t.Context(), task) + require.NoError(t, err) + require.Equal(t, "receipt, Ada, $12, red square", text) + require.Contains(t, string(server.Call(0).raw), "input_image") + require.Contains(t, string(server.Call(0).raw), "kind, visible text, amounts, names, and context") + require.NotContains(t, string(server.Call(1).raw), "input_image") + + _, err = compactor.Summarize(t.Context(), Task{ + Level: 1, + Index: 0, + Context: "\n", + Input: []string{text, "user: next"}, + }) + require.NoError(t, err) + require.NotContains(t, string(server.Call(2).raw), "input_image") + + memory, err := openMemory(t.Context(), t.TempDir(), newTestClock().Now, compactor, discardLog()) + require.NoError(t, err) + t.Cleanup(func() { + require.NoError(t, memory.Close()) + }) + + _, err = memory.Append("user", imageText([]string{name}, "A short caption.")) + require.NoError(t, err) + require.NoError(t, memory.Settle(t.Context())) + require.Equal(t, 4, server.Calls()) + require.Contains(t, string(server.Call(3).raw), "input_image") + require.Contains(t, memory.View(), "receipt, Ada, $12") +} + +func TestZoomReturnsImagePartsButLogsOnlyText(t *testing.T) { + server := newModelFake(t) + server.reply = func(w http.ResponseWriter, call ModelFakeCall) { + if call.number == 0 { + modelWriteEvents(w, modelCompletedEvent([]any{modelFunctionCallItem("fc_1", "call_1", "zoom", `{"id":0,"n":1,"name":null}`)}, 100, 0, 5)) + + return + } + + modelWriteEvents(w, modelCompletedEvent([]any{modelMessageItem("A red square.")}, 100, 0, 5)) + } + + test := newTestTurnLoop(t, server) + name, err := test.agent.Images().Save(imageTestPNG(t, 2, 2)) + require.NoError(t, err) + + _, err = test.memory.Append("user", imageText([]string{name}, "Old image.")) + require.NoError(t, err) + + _, err = test.loop.Submit(terminalSource, "Zoom the old image.") + require.NoError(t, err) + test.events.WaitFor(t, "idle", 1) + require.NotContains(t, string(server.Call(0).raw), "input_image") + + input := server.Call(1).body["input"].([]any) + result := input[len(input)-1].(map[string]any) + require.Equal(t, "function_call_output", result["type"]) + + parts := result["output"].([]any) + require.Len(t, parts, 2) + require.Equal(t, "input_text", parts[0].(map[string]any)["type"]) + require.Equal(t, "input_image", parts[1].(map[string]any)["type"]) + require.Contains(t, parts[0].(map[string]any)["text"], "Old image.") + require.NotContains(t, strings.Join(test.Log(t), "\n"), "base64") +} + +func TestImageUserMessageKeepsTextPartsAndMarkerOrder(t *testing.T) { + store := &ImageStore{dir: t.TempDir()} + first, err := store.Save(imageTestPNG(t, 1, 2)) + require.NoError(t, err) + + second, err := store.Save(imageTestPNG(t, 2, 1)) + require.NoError(t, err) + + text := imageText([]string{first, second}, "Compare.") + item, err := imageUserMessage(store, text) + require.NoError(t, err) + + data, err := json.Marshal(item) + require.NoError(t, err) + + var message struct { + Role string `json:"role"` + Content []ModelInputPart `json:"content"` + } + + require.NoError(t, json.Unmarshal(data, &message)) + require.Equal(t, "user", message.Role) + require.Len(t, message.Content, 3) + require.Equal(t, text, message.Content[0].Text) + require.NotEqual(t, message.Content[1].ImageURL, message.Content[2].ImageURL) +} diff --git a/memory.go b/memory.go index a240b6c..602b86c 100644 --- a/memory.go +++ b/memory.go @@ -607,6 +607,7 @@ func (m *Memory) task(key NodeKey) (Task, error) { } task.Input = []string{leafSource(message)} + task.Images = imageFiles(message.Text) return task, nil } @@ -781,12 +782,14 @@ func (m *Memory) buildBuiltByModel(key NodeKey) (bool, error) { // Task is the build of the node (Level, Index). Context is the compaction view // for the build. Input holds the source "kind: text" of a leaf, or the two -// child lines of a merge with their newlines as spaces. +// child lines of a merge with their newlines as spaces. Images holds the leaf +// image filenames and is empty for merges. type Task struct { Level int Index int Context string Input []string + Images []string } // MemoryViewFile is the content of view.json. diff --git a/model.go b/model.go index 507fbb1..6ddd300 100644 --- a/model.go +++ b/model.go @@ -73,22 +73,22 @@ func newModelClient(url string, auth ModelAuthorizer, cacheKey, usagePath string // modelUserMessage returns a user input item with one text part for each text. func modelUserMessage(texts ...string) any { - type part struct { - Type string `json:"type"` - Text string `json:"text"` - } - - content := make([]part, len(texts)) + content := make([]ModelInputPart, len(texts)) for n, text := range texts { - content[n] = part{ + content[n] = ModelInputPart{ Type: "input_text", Text: text, } } + return modelUserParts(content) +} + +// modelUserParts keeps text and image parts in one user message. +func modelUserParts(content []ModelInputPart) any { return struct { - Role string `json:"role"` - Content []part `json:"content"` + Role string `json:"role"` + Content []ModelInputPart `json:"content"` }{ Role: "user", Content: content, @@ -321,6 +321,13 @@ type ModelOutputItem struct { } `json:"content"` } +// ModelInputPart is a text or image part in a user message or tool result. +type ModelInputPart struct { + Type string `json:"type"` + Text string `json:"text,omitempty"` + ImageURL string `json:"image_url,omitempty"` +} + // ModelRequest is one model call. Input holds the input items in order, such as // modelUserMessage values or output items of an earlier response. Node names // the node that a compaction builds, or is nil. Turn names the request of a diff --git a/probe.go b/probe.go index b004829..384d0f8 100644 --- a/probe.go +++ b/probe.go @@ -103,7 +103,7 @@ func probe(ctx context.Context, args []string, out io.Writer) (err error) { client.LimitCalls(probeCallLimit) summarizer := ProbeSlotSummarizer{ - inner: newCompactor(client, probePrompt, config.Compactor, promptTools()), + inner: newCompactor(client, probePrompt, config.Compactor, promptTools(), &ImageStore{dir: filepath.Join(dir, imageDirName)}), slots: make(chan struct{}, builds), } diff --git a/prompt.go b/prompt.go index 36fda32..df6f17d 100644 --- a/prompt.go +++ b/prompt.go @@ -41,10 +41,10 @@ var promptEfforts = []string{"low", "medium", "high", "xhigh"} // one added paragraph on the user's instructions file. The reference names its // own agent, so promptText has the name of the agent and the kind reply in its // place. It describes only what OneChat can do, so it leaves out the spec text -// on images, computers, and computer tasks. Its paragraph on persistent mode -// tells the model that a final answer ends the turn. prompt adds the clause on -// the instructions of the user only when there are instructions. It holds no -// date and no state, so turns and compactions share its prompt cache. +// on computers and computer tasks. Its paragraph on persistent mode tells the +// model that a final answer ends the turn. prompt adds the clause on the +// instructions of the user only when there are instructions. It holds no date +// and no state, so turns and compactions share its prompt cache. const promptText = `You are {name}, an AI agent that works for one user in a single chat that never ends. Each call to you is a turn or a compaction: the view below is followed by the user's new message, or by a task starting "Compaction:". @@ -76,6 +76,9 @@ Tools: - zoom("Name") gives an agent's whole chat - date(id) gives the date and time of message id +An [image FILE] line names a stored image. New images include pixels for this +turn only. Use zoom(id, 1) to see an older image again. + # Turns Do the user's tasks yourself, with your tools. @@ -155,10 +158,10 @@ Set or remove a route only after the user agrees.` // promptSubagentText is the system prompt of a subagent, from the subagent // prompt of the reference design: its head, then the doc of the view, without -// the parts on devices and images. A subagent gets the view of the agent, then -// its whole log, so it goes on from its log after a tell or a restart. -// promptSubagent adds the clause on the instructions of the user only when -// there are instructions. +// the parts on devices. A subagent gets the view of the agent, then its whole +// log, so it goes on from its log after a tell or a restart. promptSubagent +// adds the clause on the instructions of the user only when there are +// instructions. const promptSubagentText = `You are a subagent of {name}, an AI agent that works for one user in a single chat that never ends. {name} gave you a task. Do it yourself, with your tools. @@ -189,7 +192,8 @@ was made from; zoom(id, 1) gives message id in full. Zoom whenever a summary only mentions something you need, such as what your last reply said, a decision, a past attempt or where a file is, before you act, guess or ask. date(id) gives the date and time of message id. zoom("Name") gives an agent's -whole chat.` +whole chat. An [image FILE] line names a stored image. Use zoom(id, 1) +to see its pixels.` // prompt returns the system prompt of the reference with one added paragraph on // the user's instructions file at instructionsPath. It uses the agent name and diff --git a/prompt_test.go b/prompt_test.go index c9cadc2..e9e87f3 100644 --- a/prompt_test.go +++ b/prompt_test.go @@ -18,6 +18,7 @@ func TestPromptNamesTheAgent(t *testing.T) { require.Contains(t, text, `- work: an agent's report, starting "[Name]"`) require.Contains(t, text, "- zoom(id, 1) gives message id whole\n- zoom(\"Name\") gives an agent's whole chat\n") require.NotContains(t, text, promptName) + require.Contains(t, text, "[image FILE]") // The old agent name is built from parts, so the source does not spell // it. @@ -26,7 +27,7 @@ func TestPromptNamesTheAgent(t *testing.T) { func TestPromptLeavesOutAbsentFeatures(t *testing.T) { text := prompt("Ada", "/AGENT/instructions.md", "") - for _, absent := range []string{"image", "computer", "device", "instructions at the end"} { + for _, absent := range []string{"computer", "device", "instructions at the end"} { require.NotContains(t, text, absent) } } @@ -50,7 +51,7 @@ func TestSubagentPromptNamesTheAgentAndEndsWithTheInstructions(t *testing.T) { require.True(t, strings.HasSuffix(text, "\n\n# The user's instructions\n\nI am Lena.")) require.NotContains(t, text, promptName) - for _, absent := range []string{"image", "computer", "device", "Compaction"} { + for _, absent := range []string{"computer", "device", "Compaction"} { require.NotContains(t, text, absent) } diff --git a/rebuild_test.go b/rebuild_test.go index fc4ff7e..104ec88 100644 --- a/rebuild_test.go +++ b/rebuild_test.go @@ -63,7 +63,7 @@ func TestRebuildMergesDatesAndKeepsAgentFiles(t *testing.T) { }, } - backup, err := rebuild(t.Context(), agent, added, client, newCompactor(client, agent.Prompt, defaultConfig().Compactor, agent.Tools()), &report) + backup, err := rebuild(t.Context(), agent, added, client, newCompactor(client, agent.Prompt, defaultConfig().Compactor, agent.Tools(), agent.Images()), &report) require.NoError(t, err) require.Equal(t, before, rebuildTestSnapshot(t, backup)) @@ -143,7 +143,7 @@ func TestRebuildFailureAndCancellationKeepOriginal(t *testing.T) { Kind: "work", Text: strings.Repeat("long ", 200), Date: "2025-01-01T00:00:00Z", - }}, client, newCompactor(client, agent.Prompt, defaultConfig().Compactor, agent.Tools()), &strings.Builder{}) + }}, client, newCompactor(client, agent.Prompt, defaultConfig().Compactor, agent.Tools(), agent.Images()), &strings.Builder{}) require.Error(t, err) require.Empty(t, backup) require.Equal(t, before, rebuildTestSnapshot(t, agent.dir)) @@ -179,7 +179,7 @@ func TestRebuildKeepsOriginalWhenAnInvisibleMergeFails(t *testing.T) { } client := newModelClient(server.URL, ModelAPIKey(modelTestKey), agent.CacheKey(), filepath.Join(t.TempDir(), "usage.jsonl")) - _, err := rebuild(t.Context(), agent, nil, client, newCompactor(client, agent.Prompt, defaultConfig().Compactor, agent.Tools()), &strings.Builder{}) + _, err := rebuild(t.Context(), agent, nil, client, newCompactor(client, agent.Prompt, defaultConfig().Compactor, agent.Tools(), agent.Images()), &strings.Builder{}) require.ErrorContains(t, err, "summary tree has unfinished nodes") require.Equal(t, before, rebuildTestSnapshot(t, agent.dir)) } diff --git a/subagent_run.go b/subagent_run.go index 8084c39..891766c 100644 --- a/subagent_run.go +++ b/subagent_run.go @@ -515,11 +515,13 @@ func (r *SubagentRuns) chain(ctx context.Context, run *SubagentRun) (string, err input = append(input, modelUserMessage(r.state()), modelUserMessage(log)) chain := &Chain{ client: r.client, + images: r.agent.Images(), tools: &Toolbox{ log: run.subagent, chat: r.memory, subagents: r.subagents, workdir: run.subagent.WorkDir(), + images: r.agent.Images(), }, request: ModelRequest{ Model: r.agent.config.Model, diff --git a/tools.go b/tools.go index a75514f..96cbb97 100644 --- a/tools.go +++ b/tools.go @@ -68,13 +68,15 @@ type Toolbox struct { messages *DaemonMessages children *DaemonChildren routes *Routes + images *ImageStore } // Call logs the call of the tool name with its JSON arguments, makes the log -// durable, runs the tool, and logs its clipped output, which it returns. A -// failed tool gives its error as the output. An error of Call itself is an -// error of the log. When ctx ends before the flush, the tool does not run. -func (b *Toolbox) Call(ctx context.Context, name, arguments string) (string, error) { +// durable, runs the tool, and logs its clipped text. It returns that text or +// the text and image parts of a single-message zoom. Pixels never enter the +// log. A failed tool gives its error as the output. An error of Call itself is +// an error of the log. When ctx ends before the flush, the tool does not run. +func (b *Toolbox) Call(ctx context.Context, name, arguments string) (any, error) { _, err := b.log.Append("tool", toolCallText(name, arguments)) if err != nil { return "", fmt.Errorf("append tool call: %w", err) @@ -90,12 +92,26 @@ func (b *Toolbox) Call(ctx context.Context, name, arguments string) (string, err output = toolClip(b.run(ctx, name, arguments)) } + var ( + result any = output + resultErr error + ) + + if err == nil { + result, resultErr = imageToolOutput(b, name, arguments, output) + } + + if resultErr != nil { + output = "error: " + resultErr.Error() + result = output + } + _, err = b.log.Append("echo", output) if err != nil { return "", fmt.Errorf("append tool result: %w", err) } - return output, nil + return result, nil } // run runs one tool and returns its output, or its error as text. diff --git a/tree.go b/tree.go index 69558c5..079969c 100644 --- a/tree.go +++ b/tree.go @@ -55,11 +55,11 @@ func leafSource(message Message) string { return message.Kind + ": " + message.Text } -// freeLeaf returns the leaf of message when its source fits in a node. Such a -// leaf holds the source word for word and needs no model call. +// freeLeaf returns a text-only leaf when its source fits in a node. Images need +// a model description even when their text fits. func freeLeaf(message Message) (Node, bool) { text := leafSource(message) - if len(text) > nodeBytes { + if len(text) > nodeBytes || len(imageFiles(message.Text)) > 0 { return Node{}, false } diff --git a/turn.go b/turn.go index e0bd489..2c1d9c2 100644 --- a/turn.go +++ b/turn.go @@ -60,6 +60,7 @@ func newTurnLoop(agent *Agent, memory *Memory, subagents *Subagents, whatsapp *W schedules: schedules, workdir: agent.config.Workdir, agentName: agent.config.Name, + images: agent.Images(), } return l, nil @@ -423,10 +424,19 @@ func (l *TurnLoop) chain(ctx context.Context) (string, error) { Instructions: instructions, Tools: l.agent.Tools(), }, - take: l.take, + take: l.take, + images: l.agent.Images(), } - reply, err := chain.Run(ctx, turnInput(view, l.state(), messages)) + input := turnInput(view, l.state(), messages) + for n, message := range messages { + input[len(input)-len(messages)+n], err = imageUserMessage(l.agent.Images(), message.Text) + if err != nil { + return "", fmt.Errorf("read turn images: %w", err) + } + } + + reply, err := chain.Run(ctx, input) if err != nil { return "", fmt.Errorf("run turn chain: %w", err) } diff --git a/whatsapp_import.go b/whatsapp_import.go index df871c1..1092b49 100644 --- a/whatsapp_import.go +++ b/whatsapp_import.go @@ -143,7 +143,7 @@ func whatsappImport(ctx context.Context, args []string, out io.Writer) (err erro } client := newModelClient(address, auth, agent.CacheKey(), usagePath) - backup, err := rebuild(ctx, agent, plan.added, client, newCompactor(client, agent.Prompt, config.Compactor, agent.Tools()), out) + backup, err := rebuild(ctx, agent, plan.added, client, newCompactor(client, agent.Prompt, config.Compactor, agent.Tools(), agent.Images()), out) if err != nil { return fmt.Errorf("rebuild imported memory: %w", err) } diff --git a/whatsapp_import_test.go b/whatsapp_import_test.go index 1f9c9af..4bd0a1f 100644 --- a/whatsapp_import_test.go +++ b/whatsapp_import_test.go @@ -563,7 +563,7 @@ func TestWhatsAppImportChildRebuildUsesOwnPromptAndCache(t *testing.T) { } client := newModelClient(server.URL, ModelAPIKey(modelTestKey), child.CacheKey(), filepath.Join(t.TempDir(), "usage.jsonl")) - backup, err := rebuild(t.Context(), child, plan.added, client, newCompactor(client, child.Prompt, defaultConfig().Compactor, child.Tools()), &strings.Builder{}) + backup, err := rebuild(t.Context(), child, plan.added, client, newCompactor(client, child.Prompt, defaultConfig().Compactor, child.Tools(), child.Images()), &strings.Builder{}) require.NoError(t, err) require.Equal(t, before, rebuildTestSnapshot(t, backup)) require.Equal(t, root.dir, filepath.Dir(backup)) @@ -613,7 +613,7 @@ func TestWhatsAppImportChildFailureKeepsConfiguredChildren(t *testing.T) { Kind: "work", Text: strings.Repeat("The plan is ready. ", 80), Date: "2025-01-01T08:00:00Z", - }}, client, newCompactor(client, child.Prompt, defaultConfig().Compactor, child.Tools()), &strings.Builder{}) + }}, client, newCompactor(client, child.Prompt, defaultConfig().Compactor, child.Tools(), child.Images()), &strings.Builder{}) require.ErrorContains(t, err, "rebuild") require.Equal(t, before, rebuildTestSnapshot(t, root.dir)) -- 2.53.0 From b2fc2d6a737dd5bab25e57ba822dd07258de93e6 Mon Sep 17 00:00:00 2001 From: "Anderson A. Giacomolli" Date: Tue, 10 Nov 2026 11:57:46 -0300 Subject: [PATCH 04/16] Accept owner photos and image files in Telegram Download the largest supplied photo up to 1568 pixels, or an image file, into the shared store and submit its marker and exact caption. Refuse unsupported or large images with a short notice. Retry network failures without moving the offset, and stop on storage failures so images are not lost. Keep tokens and pixels out of daemon logs. --- README.rst | 8 +- image.go | 24 ++-- telegram.go | 89 +++++++++--- telegram_api.go | 88 +++++++++++- telegram_fake_helper_test.go | 39 ++++++ telegram_image.go | 71 ++++++++++ telegram_image_test.go | 259 +++++++++++++++++++++++++++++++++++ turn_test.go | 2 +- 8 files changed, 546 insertions(+), 34 deletions(-) create mode 100644 telegram_image.go create mode 100644 telegram_image_test.go diff --git a/README.rst b/README.rst index 8002ea0..cb47c98 100644 --- a/README.rst +++ b/README.rst @@ -157,8 +157,8 @@ replies show with Markdown styles and tables. These lines are commands: ``/whatsapp ...`` The WhatsApp commands above. -In Telegram, write as in the client. ``onechat help`` lists all -commands. +In Telegram, write as in the client, also with photos and image files. +``onechat help`` lists all commands. ********************* What the agent does @@ -190,6 +190,10 @@ WhatsApp ask. It proposes to route a chat that is only about one area to the child of that area, and sets the route after you agree. +Images + Send a photo, and the agent sees it. Its memory keeps what the image + shows, and it can look at an older image again. + Schedules Ask for reminders and timed tasks, such as "every weekday at 8, tell me what my groups talked about". Each run uses the quota, so runs are diff --git a/image.go b/image.go index d6a0b20..4b23f7c 100644 --- a/image.go +++ b/image.go @@ -22,6 +22,11 @@ const ( imageMaxBytes = 20 << 20 ) +var ( + errImageSize = errors.New("image exceeds 20 MB") + errImageFormat = errors.New("use a JPEG, PNG, or WebP image") +) + // ImageStore holds immutable images shared by the root, children, and // subagents. Names are the SHA-256 of the bytes with the decoded format as // extension. Publication is atomic and does not replace an existing image. @@ -97,12 +102,12 @@ func (s *ImageStore) Read(name string) ([]byte, error) { func imageFormat(data []byte) (string, error) { if len(data) > imageMaxBytes { - return "", errors.New("image exceeds 20 MB") + return "", errImageSize } config, format, err := image.DecodeConfig(bytes.NewReader(data)) if err != nil || config.Width <= 0 || config.Height <= 0 { - return "", errors.New("use a JPEG, PNG, or WebP image") + return "", errImageFormat } switch format { @@ -111,7 +116,7 @@ func imageFormat(data []byte) (string, error) { case "png", "webp": return format, nil default: - return "", errors.New("use a JPEG, PNG, or WebP image") + return "", errImageFormat } } @@ -136,7 +141,7 @@ func imageReadFile(path string) (data []byte, err error) { } if len(data) > imageMaxBytes { - return nil, errors.New("image exceeds 20 MB") + return nil, errImageSize } return data, nil @@ -162,13 +167,14 @@ func imageFiles(text string) []string { } func imageText(names []string, caption string) string { - var text strings.Builder - + lines := make([]string, 0, len(names)+1) for _, name := range names { - text.WriteString("[image " + name + "]\n") + lines = append(lines, "[image "+name+"]") } - text.WriteString(caption) + if caption != "" { + lines = append(lines, caption) + } - return strings.TrimSuffix(text.String(), "\n") + return strings.Join(lines, "\n") } diff --git a/telegram.go b/telegram.go index f2c687a..a595b68 100644 --- a/telegram.go +++ b/telegram.go @@ -133,13 +133,13 @@ func (c TelegramConfig) validate() error { } // TelegramBot connects the private chat of the owner with the agent. It polls -// the Bot API for new messages and sends each text message of the owner to the -// agent. It ignores every other update. Its sender sends the replies, the -// errors, the notices of the bot, and the user messages of the other clients to -// the chat, and shows the typing action while a turn runs. Only the replies -// have a format: their Markdown goes as rich messages, which Telegram renders -// itself, and as message entities only when Telegram refuses a rich message. -// The rest is plain text. Tool calls do not go to the chat. +// the Bot API for new messages and sends owner text and images to the agent. It +// ignores every other update. Its sender sends the replies, the errors, the +// notices of the bot, and the user messages of the other clients to the chat, +// and shows the typing action while a turn runs. Only the replies have a +// format: their Markdown goes as rich messages, which Telegram renders itself, +// and as message entities only when Telegram refuses a rich message. The rest +// is plain text. Tool calls do not go to the chat. // // The offset file holds the ID of the next update. It moves past an update // only after the message of the update is durable in the log, so a restart @@ -193,8 +193,8 @@ func (b *TelegramBot) Notify(event Event) { } // Run polls for the messages of the owner and sends the queued events to the -// chat until ctx ends. It returns an error only when a message or the offset -// cannot be written, because then the bot would lose or repeat messages. +// chat until ctx ends. It returns an error only when a message or the offset or +// image cannot be written, because then the bot would lose or repeat messages. func (b *TelegramBot) Run(ctx context.Context, l *TurnLoop, whatsapp *WhatsApp) error { ctx, cancel := context.WithCancel(ctx) defer cancel() @@ -258,9 +258,9 @@ func (b *TelegramBot) send(ctx context.Context) { } } -// poll gets the updates from the stored offset on and handles each one, until -// ctx ends. Each update is handled to its end also when ctx ends during it, so -// a stop does not repeat its message. +// poll gets updates from the stored offset and handles them until ctx ends. A +// failed image download retries from the same offset. A submitted message is +// flushed even if ctx ends, and only then can the offset advance. func (b *TelegramBot) poll(ctx context.Context, l *TurnLoop, whatsapp *WhatsApp) error { offset, err := telegramLoadOffset(b.offsetPath) if err != nil { @@ -273,7 +273,16 @@ func (b *TelegramBot) poll(ctx context.Context, l *TurnLoop, whatsapp *WhatsApp) back: "receive back", } + imageOutage := &TelegramOutage{ + logger: b.logger, + loss: "image receive failed", + back: "image receive back", + } + + imageWait := b.minWait wait := b.minWait + +poll: for ctx.Err() == nil { updates, err := b.api.GetUpdates(ctx, offset) if ctx.Err() != nil { @@ -290,16 +299,27 @@ func (b *TelegramBot) poll(ctx context.Context, l *TurnLoop, whatsapp *WhatsApp) outage.End() wait = b.minWait + for _, update := range updates { if ctx.Err() != nil { return nil } - err = b.handle(context.WithoutCancel(ctx), l, whatsapp, update) + err = b.handle(ctx, l, whatsapp, update) if err != nil { + retry, asked := telegramRetryable(err) + if retry { + imageWait = b.pause(ctx, imageOutage, err, imageWait, asked) + continue poll + } + return err } + imageOutage.End() + + imageWait = b.minWait + offset = update.UpdateID + 1 err = telegramSaveOffset(b.offsetPath, offset) if err != nil { @@ -373,9 +393,9 @@ func (b *TelegramBot) pause(ctx context.Context, outage *TelegramOutage, err err // to the agent, and handle returns when it is durable in the log. The line // /stop interrupts the turn, the line /stop NAME stops the subagent NAME, and a // line that starts with /whatsapp runs a WhatsApp command on whatsapp, whose -// answers go to the chat only. Any other message of the owner gets the answer -// that only text is supported. Every other update is ignored, and its content -// is not logged. +// answers go to the chat only. Photos and image documents enter the shared +// image store with their caption. Other media keep the text-only answer. Every +// other update is ignored, and its content is not logged. func (b *TelegramBot) handle(ctx context.Context, l *TurnLoop, whatsapp *WhatsApp, update TelegramUpdate) error { message := update.Message if message == nil || message.Chat.Type != "private" || message.Chat.ID != b.owner || message.From == nil || message.From.ID != b.owner { @@ -387,6 +407,41 @@ func (b *TelegramBot) handle(ctx context.Context, l *TurnLoop, whatsapp *WhatsAp name, stop := stopSubagentName(message.Text) args, isWhatsApp := whatsappCommandArgs(message.Text) switch { + case len(message.Photo) > 0 || message.Document != nil: + text, err := telegramImageText(ctx, b.api, l.agent.Images(), message) + if err != nil { + retry, _ := telegramRetryable(err) + if retry { + return fmt.Errorf("receive image: %w", err) + } + + switch { + case errors.Is(err, errTelegramOtherMedia): + b.notice(telegramTextOnly) + case errors.Is(err, errImageSize), errors.Is(err, errImageFormat), errors.Is(err, errTelegramPhotoSize): + b.notice(err.Error()) + default: + var answer *TelegramError + + if !errors.As(err, &answer) { + return fmt.Errorf("receive Telegram image: %w", err) + } + + b.notice("Cannot download image.") + } + + return nil + } + + _, err = l.Submit(telegramSource, text) + if err != nil { + return fmt.Errorf("submit Telegram image: %w", err) + } + + err = l.Flush(context.WithoutCancel(ctx)) + if err != nil { + return fmt.Errorf("flush Telegram image: %w", err) + } case message.Text == "": b.notice(telegramTextOnly) case strings.TrimSpace(message.Text) == stopCommand: @@ -410,7 +465,7 @@ func (b *TelegramBot) handle(ctx context.Context, l *TurnLoop, whatsapp *WhatsAp return err } - err = l.Flush(ctx) + err = l.Flush(context.WithoutCancel(ctx)) if err != nil { return fmt.Errorf("flush the message from Telegram: %w", err) } diff --git a/telegram_api.go b/telegram_api.go index 07d2bf1..32ad9a9 100644 --- a/telegram_api.go +++ b/telegram_api.go @@ -9,6 +9,7 @@ import ( "io" "net/http" "net/url" + "path" "strings" "time" ) @@ -102,6 +103,66 @@ func (b *TelegramAPI) GetUpdates(ctx context.Context, offset int64) ([]TelegramU return updates, err } +// GetImage downloads a file through the authenticated file endpoint. Errors +// never include its token or URL, and the body is limited before storage. +func (b *TelegramAPI) GetImage(ctx context.Context, id string) (data []byte, err error) { + var file struct { + Path string `json:"file_path"` + Size int64 `json:"file_size"` + } + + err = b.call(ctx, b.sendTimeout, "getFile", map[string]any{"file_id": id}, &file) + if err != nil { + return nil, fmt.Errorf("get image file: %w", err) + } + + if file.Size > imageMaxBytes { + return nil, errImageSize + } + + if file.Path == "" || path.Clean(file.Path) != file.Path || strings.HasPrefix(file.Path, "/") || strings.HasPrefix(file.Path, "../") || strings.ContainsAny(file.Path, "?%#\\") { + return nil, errors.New("invalid image file path") + } + + ctx, cancel := context.WithTimeout(ctx, b.sendTimeout) + defer cancel() + + request, err := http.NewRequestWithContext(ctx, http.MethodGet, b.base+"/file/bot"+b.token+"/"+file.Path, nil) + if err != nil { + return nil, fmt.Errorf("make image request: %s", b.redact(err)) + } + + response, err := b.http.Do(request) + if err != nil { + return nil, b.networkError("image download failed", err, b.sendTimeout) + } + + defer func() { + closeErr := response.Body.Close() + if closeErr != nil { + err = errors.Join(err, b.networkError("close image download", closeErr, b.sendTimeout)) + } + }() + + if response.StatusCode != http.StatusOK { + return nil, &TelegramError{ + Status: response.StatusCode, + Description: http.StatusText(response.StatusCode), + } + } + + data, err = io.ReadAll(io.LimitReader(response.Body, imageMaxBytes+1)) + if err != nil { + return nil, b.networkError("image cut off", err, b.sendTimeout) + } + + if len(data) > imageMaxBytes { + return nil, errImageSize + } + + return data, nil +} + // SendMessage sends text to chat with the format of entities, with no parse // mode, so the text needs no escaping. With no entities, the text is plain. A // call that takes longer than telegramSendTimeout fails with a @@ -291,12 +352,29 @@ type TelegramAnswer struct { } `json:"parameters"` } -// TelegramMessage is a message to the bot. Text is empty for a message that is -// not text, such as a photo. +// TelegramMessage holds the text, caption, and image sizes of a bot message. type TelegramMessage struct { - Chat TelegramChat `json:"chat"` - From *TelegramUser `json:"from"` - Text string `json:"text"` + Chat TelegramChat `json:"chat"` + From *TelegramUser `json:"from"` + Text string `json:"text"` + Caption string `json:"caption"` + Photo []TelegramPhoto `json:"photo"` + Document *TelegramDocument `json:"document"` +} + +// TelegramPhoto is one supplied size of a photo. +type TelegramPhoto struct { + FileID string `json:"file_id"` + Width int `json:"width"` + Height int `json:"height"` + FileSize int64 `json:"file_size"` +} + +// TelegramDocument is a file sent without photo compression. +type TelegramDocument struct { + FileID string `json:"file_id"` + MimeType string `json:"mime_type"` + FileSize int64 `json:"file_size"` } // TelegramChat is the chat of a message. Type is private for a chat with one diff --git a/telegram_fake_helper_test.go b/telegram_fake_helper_test.go index 3056aa1..6a89054 100644 --- a/telegram_fake_helper_test.go +++ b/telegram_fake_helper_test.go @@ -45,6 +45,7 @@ type TelegramFake struct { stall func(call TelegramFakeCall) bool added chan struct{} stopped chan struct{} + images map[string][]byte } // Add makes update available to getUpdates. @@ -86,7 +87,36 @@ func (f *TelegramFake) Sent() []string { return texts } +func (f *TelegramFake) Image(id string, data []byte) { + f.mutex.Lock() + defer f.mutex.Unlock() + + if f.images == nil { + f.images = make(map[string][]byte) + } + + f.images[id] = data +} + func (f *TelegramFake) serve(w http.ResponseWriter, r *http.Request) { + id, file := strings.CutPrefix(r.URL.Path, "/file/bot"+telegramTestToken+"/") + if file { + f.mutex.Lock() + + data, exists := f.images[id] + f.mutex.Unlock() + + if !exists { + http.NotFound(w, r) + + return + } + + _, _ = w.Write(data) + + return + } + method, found := strings.CutPrefix(r.URL.Path, "/bot"+telegramTestToken+"/") if !found { http.Error(w, `{"ok":false,"error_code":404,"description":"Not Found"}`, http.StatusNotFound) @@ -146,6 +176,15 @@ func (f *TelegramFake) serve(w http.ResponseWriter, r *http.Request) { call.ok = true f.record(call) + if method == "getFile" { + _ = json.NewEncoder(w).Encode(map[string]any{ + "ok": true, + "result": map[string]any{"file_path": params["file_id"]}, + }) + + return + } + if method != "getUpdates" { _, _ = io.WriteString(w, `{"ok":true,"result":true}`) return diff --git a/telegram_image.go b/telegram_image.go new file mode 100644 index 0000000..9b73341 --- /dev/null +++ b/telegram_image.go @@ -0,0 +1,71 @@ +package main + +import ( + "context" + "errors" + "fmt" + "strings" +) + +var ( + errTelegramOtherMedia = errors.New("only text messages are supported") + errTelegramPhotoSize = errors.New("no photo size fits 1568 pixels") +) + +// telegramImageText selects a supplied photo size, or an image document, and +// stores it before returning marker text with the unchanged caption. +func telegramImageText(ctx context.Context, api *TelegramAPI, store *ImageStore, message *TelegramMessage) (string, error) { + var ( + id string + size int64 + ) + + if len(message.Photo) > 0 { + var area int + + for _, photo := range message.Photo { + if photo.Width <= 0 || photo.Height <= 0 || max(photo.Width, photo.Height) > 1568 { + continue + } + + if photo.Width*photo.Height > area { + id = photo.FileID + size = photo.FileSize + area = photo.Width * photo.Height + } + } + + if id == "" { + return "", errTelegramPhotoSize + } + } else if message.Document != nil { + document := message.Document + switch document.MimeType { + case "image/jpeg", "image/png", "image/webp": + id = document.FileID + size = document.FileSize + default: + if strings.HasPrefix(document.MimeType, "image/") { + return "", errImageFormat + } + + return "", errTelegramOtherMedia + } + } + + if size > imageMaxBytes { + return "", errImageSize + } + + data, err := api.GetImage(ctx, id) + if err != nil { + return "", fmt.Errorf("download image: %w", err) + } + + name, err := store.Save(data) + if err != nil { + return "", fmt.Errorf("store image: %w", err) + } + + return imageText([]string{name}, message.Caption), nil +} diff --git a/telegram_image_test.go b/telegram_image_test.go new file mode 100644 index 0000000..837a64e --- /dev/null +++ b/telegram_image_test.go @@ -0,0 +1,259 @@ +package main + +import ( + "bytes" + "image" + "image/jpeg" + "net/http" + "os" + "path/filepath" + "strings" + "testing" + "time" + + "github.com/stretchr/testify/require" +) + +func TestTelegramSelectsLargestPhotoAtMost1568Pixels(t *testing.T) { + server := newModelFake(t) + test := newTestTurnLoop(t, server) + telegram := newTelegramFake(t) + bot, logs := telegramStartTestBot(t, telegram) + picture := imageTestPNG(t, 8, 6) + telegram.Image("selected", picture) + + message := telegramOwnerMessage("") + message.Caption = "Read this label.\n" + message.Photo = []TelegramPhoto{ + { + FileID: "small", + Width: 100, + Height: 200, + }, + { + FileID: "too-large", + Width: 1600, + Height: 1200, + }, + { + FileID: "selected", + Width: 1568, + Height: 1176, + }, + { + FileID: "medium", + Width: 800, + Height: 600, + }, + } + require.NoError(t, bot.handle(t.Context(), test.loop, nil, TelegramUpdate{ + UpdateID: 1, + Message: message, + })) + test.events.WaitFor(t, "idle", 1) + + files, err := os.ReadDir(test.agent.Images().dir) + require.NoError(t, err) + require.Len(t, files, 1) + + logged, err := test.memory.Message(0) + require.NoError(t, err) + require.Equal(t, imageText([]string{files[0].Name()}, message.Caption), logged.Text) + require.True(t, strings.HasSuffix(logged.Text, "\n")) + require.Contains(t, string(server.Call(0).raw), "input_image") + require.NotContains(t, logs.String(), "base64") + require.NotContains(t, logs.String(), "Read this label") + + for _, call := range telegram.Calls() { + if call.method == "getFile" { + require.Equal(t, "selected", call.params["file_id"]) + } + } +} + +func TestTelegramImageDocumentsUseContentFormatAndRejectOtherMedia(t *testing.T) { + telegram := newTelegramFake(t) + png := imageTestPNG(t, 2, 2) + + var jpegBytes bytes.Buffer + + require.NoError(t, jpeg.Encode(&jpegBytes, image.NewRGBA(image.Rect(0, 0, 2, 2)), nil)) + + // A made-up one-pixel lossless WebP image. + webp := []byte{'R', 'I', 'F', 'F', 22, 0, 0, 0, 'W', 'E', 'B', 'P', 'V', 'P', '8', 'L', 10, 0, 0, 0, 0x2f, 0, 0, 0, 0, 0x07, 0x10, 0x11, 0x11, 0x88} + + telegram.Image("png", png) + telegram.Image("jpeg", jpegBytes.Bytes()) + telegram.Image("webp", webp) + telegram.Image("invalid", []byte("not an image")) + telegram.Image("large", bytes.Repeat([]byte{0}, imageMaxBytes+1)) + + api := newTelegramAPI(telegram.URL, telegramTestToken) + + store := &ImageStore{dir: filepath.Join(t.TempDir(), imageDirName)} + testCases := []struct { + name string + document TelegramDocument + suffix string + failure string + }{ + { + name: "accepts PNG documents", + document: TelegramDocument{ + FileID: "png", + MimeType: "image/png", + }, + suffix: ".png]", + }, + { + name: "accepts JPEG documents", + document: TelegramDocument{ + FileID: "jpeg", + MimeType: "image/jpeg", + }, + suffix: ".jpg]", + }, + { + name: "accepts WebP documents", + document: TelegramDocument{ + FileID: "webp", + MimeType: "image/webp", + }, + suffix: ".webp]", + }, + { + name: "other documents keep text answer", + document: TelegramDocument{ + FileID: "pdf", + MimeType: "application/pdf", + }, + failure: "only text messages are supported", + }, + { + name: "other images refused", + document: TelegramDocument{ + FileID: "gif", + MimeType: "image/gif", + }, + failure: "JPEG, PNG, or WebP", + }, + { + name: "invalid image refused", + document: TelegramDocument{ + FileID: "invalid", + MimeType: "image/png", + }, + failure: "JPEG, PNG, or WebP", + }, + { + name: "large download refused", + document: TelegramDocument{ + FileID: "large", + MimeType: "image/png", + }, + failure: "exceeds 20 MB", + }, + { + name: "large metadata refused", + document: TelegramDocument{ + FileID: "not-downloaded", + MimeType: "image/png", + FileSize: imageMaxBytes + 1, + }, + failure: "exceeds 20 MB", + }, + } + for _, tc := range testCases { + t.Run(tc.name, func(t *testing.T) { + text, err := telegramImageText(t.Context(), api, store, &TelegramMessage{Document: &tc.document}) + if tc.failure != "" { + require.ErrorContains(t, err, tc.failure) + + return + } + + require.NoError(t, err) + require.True(t, strings.HasSuffix(text, tc.suffix)) + }) + } +} + +func TestTelegramRetriesImageDownloadBeforeAdvancingOffset(t *testing.T) { + telegram := newTelegramFake(t) + telegram.Image("photo", imageTestPNG(t, 2, 2)) + + tries := 0 + telegram.fail = func(call TelegramFakeCall) (int, string) { + if call.method == "getFile" { + tries++ + if tries == 1 { + return http.StatusServiceUnavailable, "try again" + } + } + + return 0, "" + } + bot, logs := telegramStartTestBot(t, telegram) + bot.minWait = time.Millisecond + bot.maxWait = 5 * time.Millisecond + server := newModelFake(t) + socket := daemonStartTestWithBot(t, server, bot) + terminal := terminalStartTestClient(t, socket) + terminal.WaitFor(t, "[connected to OneChat]") + + message := telegramOwnerMessage("") + message.Photo = []TelegramPhoto{{ + FileID: "photo", + Width: 2, + Height: 2, + }} + telegram.Add(TelegramUpdate{ + UpdateID: 7, + Message: message, + }) + terminal.WaitFor(t, "[done]") + require.Eventually(t, func() bool { + offset, err := telegramLoadOffset(bot.offsetPath) + return err == nil && offset == 8 + }, 10*time.Second, time.Millisecond) + require.Contains(t, logs.String(), "receive failed") + require.Contains(t, logs.String(), "receive back") + require.NotContains(t, logs.String(), telegramTestToken) + require.NotContains(t, logs.String(), "base64") +} + +func TestTelegramRefusesPhotoWhenEverySizeIsTooLarge(t *testing.T) { + telegram := newTelegramFake(t) + _, err := telegramImageText(t.Context(), newTelegramAPI(telegram.URL, telegramTestToken), &ImageStore{dir: t.TempDir()}, &TelegramMessage{Photo: []TelegramPhoto{{ + FileID: "too-big", + Width: 1600, + Height: 1000, + }}}) + require.ErrorContains(t, err, "1568 pixels") + require.Empty(t, telegram.Calls()) +} + +func TestTelegramStorageFailureDoesNotAcceptTheImage(t *testing.T) { + telegram := newTelegramFake(t) + telegram.Image("photo", imageTestPNG(t, 2, 2)) + + test := newTestTurnLoop(t, newModelFake(t)) + require.NoError(t, os.WriteFile(test.agent.Images().dir, []byte("blocked"), 0o600)) + + bot, _ := telegramStartTestBot(t, telegram) + message := telegramOwnerMessage("") + message.Photo = []TelegramPhoto{{ + FileID: "photo", + Width: 2, + Height: 2, + }} + err := bot.handle(t.Context(), test.loop, nil, TelegramUpdate{ + UpdateID: 1, + Message: message, + }) + require.ErrorContains(t, err, "create image directory") + require.Equal(t, 0, test.memory.Len()) + + _, found := bot.next() + require.False(t, found) +} diff --git a/turn_test.go b/turn_test.go index 949c0b8..3b86f0c 100644 --- a/turn_test.go +++ b/turn_test.go @@ -388,7 +388,7 @@ func turnInputTexts(t *testing.T, call ModelFakeCall) []string { func newTestTurnLoopWith(t *testing.T, server *ModelFake, summarizer Summarizer) *TurnTestLoop { t.Helper() - agent, err := openAgent(t.TempDir()) + agent, err := openAgent(filepath.Join(t.TempDir(), "agent")) require.NoError(t, err) agent.config.Workdir = t.TempDir() -- 2.53.0 From d28e3b2858db322899a81b59a30400c1acad2b42 Mon Sep 17 00:00:00 2001 From: "Anderson A. Giacomolli" Date: Tue, 10 Nov 2026 12:03:38 -0300 Subject: [PATCH 05/16] Send terminal images by local path Accept /image PATH with an optional caption and quoted paths with spaces. Resolve relative paths in the client and let the daemon store the image before it submits the marker and caption. The socket carries only the path, so image bytes never enter its events or the chat log. --- README.rst | 8 ++- daemon.go | 21 ++++++ main.go | 2 + socket.go | 4 +- terminal.go | 18 ++++- terminal_image.go | 46 ++++++++++++ terminal_image_test.go | 157 +++++++++++++++++++++++++++++++++++++++++ 7 files changed, 250 insertions(+), 6 deletions(-) create mode 100644 terminal_image.go create mode 100644 terminal_image_test.go diff --git a/README.rst b/README.rst index cb47c98..1c344f4 100644 --- a/README.rst +++ b/README.rst @@ -154,6 +154,9 @@ replies show with Markdown styles and tables. These lines are commands: ``/stop NAME`` Stops the subagent NAME. +``/image PATH [caption]`` + Sends a JPEG, PNG, or WebP image. Quote a path with spaces. + ``/whatsapp ...`` The WhatsApp commands above. @@ -191,8 +194,9 @@ WhatsApp child of that area, and sets the route after you agree. Images - Send a photo, and the agent sees it. Its memory keeps what the image - shows, and it can look at an older image again. + Send a photo in Telegram or with ``/image``, and the agent sees it. + Its memory keeps what the image shows, and it can look at an older + image again. Schedules Ask for reminders and timed tasks, such as "every weekday at 8, tell diff --git a/daemon.go b/daemon.go index 2a93f1d..137cc6f 100644 --- a/daemon.go +++ b/daemon.go @@ -493,6 +493,27 @@ func daemonRunRequest(l *TurnLoop, whatsapp *WhatsApp, request SocketRequest, an _, err := l.Submit(terminalSource, request.Text) return err + case "image": + if !filepath.IsAbs(request.Path) { + return errors.New("use /image PATH [caption]") + } + + data, err := imageReadFile(request.Path) + if err != nil { + return fmt.Errorf("read terminal image: %w", err) + } + + name, err := l.agent.Images().Save(data) + if err != nil { + return fmt.Errorf("store terminal image: %w", err) + } + + _, err = l.Submit(terminalSource, imageText([]string{name}, request.Text)) + if err != nil { + return fmt.Errorf("submit terminal image: %w", err) + } + + return nil case "interrupt": if !l.Interrupt() { return errors.New("no turn runs") diff --git a/main.go b/main.go index 843bf90..10ac995 100644 --- a/main.go +++ b/main.go @@ -28,6 +28,8 @@ Commands: help show this help Chat needs a running daemon. ONECHAT_DIR sets the data directory. +In chat, /image PATH [caption] sends a JPEG, PNG, or WebP image up to 20 MB. +Quote PATH with single or double quotes when it contains spaces. ` // version is set by the package build and stays dev for direct Go builds. diff --git a/socket.go b/socket.go index 7b2da2b..0dffeb0 100644 --- a/socket.go +++ b/socket.go @@ -33,8 +33,10 @@ func checkSocketPath(path string) error { // message, "interrupt", "stop" with the name of a subagent in Text, or // "whatsapp" with the text after /whatsapp of a WhatsApp command, or "backup" // with the absolute output path in Text. The backup answer is a "backup" event -// with the path and size, or an "error" event. +// with the path and size, or an "error" event. An "image" request has an +// absolute Path and the caption in Text. type SocketRequest struct { Type string `json:"type"` Text string `json:"text,omitempty"` + Path string `json:"path,omitempty"` } diff --git a/terminal.go b/terminal.go index 9945620..da0d78a 100644 --- a/terminal.go +++ b/terminal.go @@ -23,9 +23,10 @@ var errTerminalDaemonClosed = errors.New("the daemon closed the connection") // terminal runs "onechat" and "onechat chat": the terminal client of the // daemon. Each line of input is a user message, the line /stop interrupts the // running turn, the line /stop NAME stops the subagent NAME, and a line that -// starts with /whatsapp is a WhatsApp command. The client shows the replies, -// the tool calls, and their results as they happen. It ends at the end of the -// input or when the daemon closes the connection. +// starts with /whatsapp is a WhatsApp command. /image PATH [caption] sends an +// image from the client working directory. The client shows the replies, the +// tool calls, and their results as they happen. It ends at the end of the input +// or when the daemon closes the connection. func terminal(ctx context.Context, in io.Reader, out io.Writer) error { dir, err := dataDir() if err != nil { @@ -99,6 +100,7 @@ func terminalSendLines(ctx context.Context, conn net.Conn, in io.Reader) error { name, stop := stopSubagentName(line) args, whatsapp := whatsappCommandArgs(line) + image, isImage := terminalImageRequest(line) switch { case strings.TrimSpace(line) == "": continue @@ -111,6 +113,16 @@ func terminalSendLines(ctx context.Context, conn net.Conn, in io.Reader) error { Type: "stop", Text: name, } + case isImage: + request = image + if request.Path != "" { + var err error + + request.Path, err = filepath.Abs(request.Path) + if err != nil { + return fmt.Errorf("find terminal image path: %w", err) + } + } case whatsapp: request = SocketRequest{ Type: "whatsapp", diff --git a/terminal_image.go b/terminal_image.go new file mode 100644 index 0000000..f6f4101 --- /dev/null +++ b/terminal_image.go @@ -0,0 +1,46 @@ +package main + +import "strings" + +// terminalImageRequest reads /image with one path and an optional caption. +// Single and double quotes permit spaces in a path, with no shell expansion. An +// invalid path argument becomes an empty path for the daemon to refuse. +func terminalImageRequest(line string) (SocketRequest, bool) { + rest, found := strings.CutPrefix(strings.TrimLeft(line, " \t"), "/image") + if !found || (rest != "" && rest[0] != ' ' && rest[0] != '\t') { + return SocketRequest{}, false + } + + request := SocketRequest{Type: "image"} + rest = strings.TrimLeft(rest, " \t") + if rest == "" { + return request, true + } + + if rest[0] == '\'' || rest[0] == '"' { + end := strings.IndexByte(rest[1:], rest[0]) + if end < 0 { + return request, true + } + + request.Path = rest[1 : 1+end] + rest = rest[end+2:] + if rest != "" && rest[0] != ' ' && rest[0] != '\t' { + request.Path = "" + return request, true + } + } else { + end := strings.IndexAny(rest, " \t") + if end < 0 { + request.Path = rest + return request, true + } + + request.Path = rest[:end] + rest = rest[end:] + } + + request.Text = strings.TrimLeft(rest, " \t") + + return request, true +} diff --git a/terminal_image_test.go b/terminal_image_test.go new file mode 100644 index 0000000..46a7cfd --- /dev/null +++ b/terminal_image_test.go @@ -0,0 +1,157 @@ +package main + +import ( + "bytes" + "encoding/json" + "net" + "os" + "path/filepath" + "strings" + "testing" + + "github.com/stretchr/testify/require" +) + +func TestTerminalImageLineSendsAnAbsolutePathAndCaption(t *testing.T) { + client, daemon := net.Pipe() + defer func() { + require.NoError(t, client.Close()) + require.NoError(t, daemon.Close()) + }() + + done := make(chan error, 1) + go func() { + done <- terminalSendLines(t.Context(), client, strings.NewReader("/image 'image sample.png' Read the label.\n")) + }() + + var request SocketRequest + + require.NoError(t, json.NewDecoder(daemon).Decode(&request)) + require.NoError(t, <-done) + + expected, err := filepath.Abs("image sample.png") + require.NoError(t, err) + require.Equal(t, SocketRequest{ + Type: "image", + Path: expected, + Text: "Read the label.", + }, request) +} + +func TestTerminalImagePathsAndCaptionsHaveNoShellExpansion(t *testing.T) { + testCases := []struct { + name string + line string + path string + caption string + command bool + }{ + { + name: "accepts paths without captions", + line: "/image sample.png", + path: "sample.png", + command: true, + }, + { + name: "keeps spaces in double quoted paths", + line: `/image "a b.png" caption`, + path: "a b.png", + caption: "caption", + command: true, + }, + { + name: "caption stays literal", + line: "/image sample.png $(not-a-command) ", + path: "sample.png", + caption: "$(not-a-command) ", + command: true, + }, + { + name: "missing path refused", + line: "/image", + command: true, + }, + { + name: "unclosed quote refused", + line: `/image "sample.png`, + command: true, + }, + { + name: "joined caption refused", + line: `/image "sample.png"caption`, + command: true, + }, + { + name: "other command stays text", + line: "/images sample.png", + command: false, + }, + } + for _, tc := range testCases { + t.Run(tc.name, func(t *testing.T) { + request, command := terminalImageRequest(tc.line) + require.Equal(t, tc.command, command) + require.Equal(t, tc.path, request.Path) + require.Equal(t, tc.caption, request.Text) + }) + } +} + +func TestTerminalImageIsStoredBeforeItReachesTheModel(t *testing.T) { + server := newModelFake(t) + socket := daemonStartTest(t, server) + terminal := terminalStartTestClient(t, socket) + terminal.WaitFor(t, "[connected to OneChat]") + + path := filepath.Join(t.TempDir(), "sample.png") + data := imageTestPNG(t, 2, 2) + require.NoError(t, os.WriteFile(path, data, 0o600)) + terminal.Type("/image " + path + " What is this?") + terminal.WaitFor(t, "[done]") + require.Equal(t, 1, server.Calls()) + require.Contains(t, string(server.Call(0).raw), "input_image") + require.Contains(t, string(server.Call(0).raw), "What is this?") + require.NotContains(t, string(server.Call(0).raw), path) + + files, err := os.ReadDir(filepath.Join(filepath.Dir(socket), imageDirName)) + require.NoError(t, err) + require.Len(t, files, 1) + + stored, err := os.ReadFile(filepath.Join(filepath.Dir(socket), imageDirName, files[0].Name())) + require.NoError(t, err) + require.Equal(t, data, stored) + require.NotContains(t, terminal.Output(), "base64") +} + +func TestTerminalRefusesLargeAndUnsupportedImagesWithoutAModelCall(t *testing.T) { + server := newModelFake(t) + test := newTestTurnLoop(t, server) + path := filepath.Join(t.TempDir(), "sample.png") + require.NoError(t, os.WriteFile(path, bytes.Repeat([]byte{0}, imageMaxBytes+1), 0o600)) + + request := SocketRequest{ + Type: "image", + Path: path, + } + require.ErrorContains(t, daemonRunRequest(test.loop, nil, request, func(string) { + }), "exceeds 20 MB") + require.NoError(t, os.WriteFile(path, []byte("not an image"), 0o600)) + require.ErrorContains(t, daemonRunRequest(test.loop, nil, request, func(string) { + }), "JPEG, PNG, or WebP") + require.Equal(t, 0, test.memory.Len()) + require.Equal(t, 0, server.Calls()) + + _, err := os.Stat(test.agent.Images().dir) + require.ErrorIs(t, err, os.ErrNotExist) +} + +func TestImageCommandErrorDoesNotEndTheTerminalClient(t *testing.T) { + socket := daemonStartTest(t, newModelFake(t)) + terminal := terminalStartTestClient(t, socket) + terminal.WaitFor(t, "[connected to OneChat]") + terminal.Type("/image") + terminal.WaitFor(t, "use /image PATH [caption]") + terminal.Type("Continue.") + terminal.WaitFor(t, "[done]") + require.NoError(t, terminal.Close()) +} -- 2.53.0 From 0da50dc187c8b4aa23ac1ddd0418c0da779389d9 Mon Sep 17 00:00:00 2001 From: "Anderson A. Giacomolli" Date: Tue, 10 Nov 2026 14:14:06 -0300 Subject: [PATCH 06/16] Receive images in WhatsApp batches WhatsApp stores supported images before it acknowledges their messages, so a stop cannot lose them. A download has its own limit of two minutes, apart from the ten seconds of the name lookups. Batch text keeps the image markers and captions on their own lines, so the agent and a routed child can read the pixels. Failed images and other visual media keep short text markers instead of stopping the batch. View-once messages stay unread. A failed download logs only a short chat hash and a safe cause, such as an HTTP status, because the library error text can hold signed media URLs. Tests use made-up images and a fake download client to check storage, the limits, the markers, routed delivery, and restarts. Issue: 441b3c7 --- README.rst | 2 +- whatsapp.go | 9 +- whatsapp_batch.go | 18 +- whatsapp_fake_helper_test.go | 34 ++ whatsapp_image.go | 65 ++++ whatsapp_image_test.go | 586 +++++++++++++++++++++++++++++++++++ whatsapp_inbox.go | 96 +++++- whatsapp_inbox_test.go | 12 +- whatsapp_route_test.go | 2 +- 9 files changed, 784 insertions(+), 40 deletions(-) create mode 100644 whatsapp_image.go create mode 100644 whatsapp_image_test.go diff --git a/README.rst b/README.rst index 1c344f4..a02b870 100644 --- a/README.rst +++ b/README.rst @@ -194,7 +194,7 @@ WhatsApp child of that area, and sets the route after you agree. Images - Send a photo in Telegram or with ``/image``, and the agent sees it. + Send a photo in Telegram, WhatsApp, or with ``/image`` to the agent. Its memory keeps what the image shows, and it can look at an older image again. diff --git a/whatsapp.go b/whatsapp.go index 851cfd1..f225756 100644 --- a/whatsapp.go +++ b/whatsapp.go @@ -95,6 +95,7 @@ type WhatsAppClient interface { PairPhone(ctx context.Context, phone string, showPushNotification bool, clientType whatsmeow.PairClientType, clientDisplayName string) (string, error) Logout(ctx context.Context) error GetJoinedGroups(ctx context.Context) ([]*types.GroupInfo, error) + Download(ctx context.Context, message whatsmeow.DownloadableMessage) ([]byte, error) GenerateMessageID() types.MessageID SendMessage(ctx context.Context, to types.JID, message *waE2E.Message, extra ...whatsmeow.SendRequestExtra) (whatsmeow.SendResponse, error) } @@ -123,7 +124,7 @@ func openWhatsApp(ctx context.Context, dir string, clock func() time.Time, logge return nil, err } - w.inbox = newWhatsAppInbox(w.state, clock, w.groups) + w.inbox = newWhatsAppInbox(w.state, clock, w.groups, &ImageStore{dir: filepath.Join(dir, imageDirName)}, logger) info, err := os.Stat(w.sessionPath()) if errors.Is(err, os.ErrNotExist) { @@ -168,9 +169,9 @@ func openWhatsApp(ctx context.Context, dir string, clock func() time.Time, logge // creation, close, removal, and snapshots. It is held before mutex, and stays // held when removal releases mutex for client callbacks. // -// The inbox puts the text messages into batches and logs them, and groups -// holds the names of the groups. Both get a WhatsAppLink from link for each -// call, so they use the current device and client. +// The inbox puts messages into batches and logs them, and groups holds the +// names of the groups. Both get a WhatsAppLink from link for each call, so they +// use the current device and client. // // The library logs nothing, because its log lines hold the ID of the own // account, which is the phone number of the user. The own number and the own diff --git a/whatsapp_batch.go b/whatsapp_batch.go index 08dfb6a..7f43a3c 100644 --- a/whatsapp_batch.go +++ b/whatsapp_batch.go @@ -40,7 +40,8 @@ func (b WhatsAppBatch) Turn() bool { // The first line starts with [WhatsApp] and names the chat, its ID, and the day // of the first message. Each message has a line with its time and its sender, // and the messages of the user have the sender You. A batch that starts a turn -// ends with the line about the empty reply. +// ends with the line about the empty reply. Image markers start on their own +// lines so that the shared image reader can find them. func (b WhatsAppBatch) Text(location *time.Location) string { kind := "private chat" if b.Chat.Group { @@ -59,7 +60,12 @@ func (b WhatsAppBatch) Text(location *time.Location) string { sender += " (replies to you)" } - lines = append(lines, message.Time.In(location).Format("15:04")+" "+sender+": "+message.Text) + prefix := message.Time.In(location).Format("15:04") + " " + sender + ":" + if len(imageFiles(message.Text)) > 0 { + lines = append(lines, prefix, message.Text) + } else { + lines = append(lines, prefix+" "+message.Text) + } } if b.Turn() { @@ -76,10 +82,10 @@ type WhatsAppChat struct { Group bool `json:"group,omitempty"` } -// WhatsAppMessage is one text message of a batch: its ID, the name of its -// sender, its WhatsApp time, and its text. Mine marks a message of the user. -// Mention and Reply mark a message that mentions the user or replies to a -// message of the user. +// WhatsAppMessage is one message of a batch: its ID, the name of its sender, +// its WhatsApp time, and its text. Mine marks a message of the user. Mention +// and Reply mark a message that mentions the user or replies to a message of +// the user. type WhatsAppMessage struct { ID string `json:"id"` Sender string `json:"sender"` diff --git a/whatsapp_fake_helper_test.go b/whatsapp_fake_helper_test.go index d8bb9bb..e936f45 100644 --- a/whatsapp_fake_helper_test.go +++ b/whatsapp_fake_helper_test.go @@ -83,6 +83,10 @@ type WhatsAppFake struct { logoutErr error groups []*types.GroupInfo sendErr error + downloadData []byte + downloadErr error + downloadWait time.Duration + downloads []whatsmeow.DownloadableMessage phones []string logouts int sent []WhatsAppFakeSend @@ -168,6 +172,36 @@ func (f *WhatsAppFake) GetJoinedGroups(_ context.Context) ([]*types.GroupInfo, e return f.groups, nil } +// Download waits for downloadWait, records the request, and returns the made-up +// image or downloadErr. A canceled context ends the wait. +func (f *WhatsAppFake) Download(ctx context.Context, message whatsmeow.DownloadableMessage) ([]byte, error) { + if f.downloadWait > 0 { + timer := time.NewTimer(f.downloadWait) + defer timer.Stop() + + select { + case <-ctx.Done(): + return nil, ctx.Err() + case <-timer.C: + } + } + + f.mutex.Lock() + defer f.mutex.Unlock() + + f.downloads = append(f.downloads, message) + + return slices.Clone(f.downloadData), f.downloadErr +} + +// Downloads returns the media requested from the fake. +func (f *WhatsAppFake) Downloads() []whatsmeow.DownloadableMessage { + f.mutex.Lock() + defer f.mutex.Unlock() + + return slices.Clone(f.downloads) +} + // GenerateMessageID returns a new message ID. func (f *WhatsAppFake) GenerateMessageID() types.MessageID { f.mutex.Lock() diff --git a/whatsapp_image.go b/whatsapp_image.go new file mode 100644 index 0000000..4277b0a --- /dev/null +++ b/whatsapp_image.go @@ -0,0 +1,65 @@ +package main + +import ( + "context" + "errors" + "fmt" + "time" + + "go.mau.fi/whatsmeow" + "go.mau.fi/whatsmeow/proto/waE2E" +) + +// whatsappImageTime is the longest time of an image download. +const whatsappImageTime = 2 * time.Minute + +// whatsappImage downloads only supported images and stores them before the +// event can be acknowledged. Both the declared size and the downloaded bytes +// must fit the image store. Decoded bytes determine the stored format. +func whatsappImage(ctx context.Context, client WhatsAppClient, store *ImageStore, message *waE2E.ImageMessage) (string, error) { + if message.GetFileLength() > imageMaxBytes { + return "", errImageSize + } + + switch message.GetMimetype() { + case "image/jpeg", "image/png", "image/webp": + default: + return "", errImageFormat + } + + if client == nil { + return "", errors.New("WhatsApp client is not available") + } + + data, err := client.Download(ctx, message) + if err != nil { + return "", fmt.Errorf("download image: %w", whatsappImageDownloadError(err)) + } + + name, err := store.Save(data) + if err != nil { + return "", fmt.Errorf("store image: %w", err) + } + + return name, nil +} + +// whatsappImageDownloadError keeps only known safe causes. Library errors can +// contain signed URLs or direct paths even when request preparation fails, so +// arbitrary error text cannot go to the daemon log. +func whatsappImageDownloadError(err error) error { + switch { + case errors.Is(err, context.Canceled): + return context.Canceled + case errors.Is(err, context.DeadlineExceeded): + return context.DeadlineExceeded + } + + var response whatsmeow.DownloadHTTPError + + if errors.As(err, &response) && response.Response != nil { + return fmt.Errorf("media server returned HTTP %d", response.StatusCode) + } + + return errors.New("media download failed") +} diff --git a/whatsapp_image_test.go b/whatsapp_image_test.go new file mode 100644 index 0000000..f00bc3a --- /dev/null +++ b/whatsapp_image_test.go @@ -0,0 +1,586 @@ +package main + +import ( + "bytes" + "context" + "encoding/base64" + "errors" + "fmt" + "image" + "image/gif" + "image/jpeg" + "net/http" + "net/url" + "os" + "path/filepath" + "strings" + "testing" + "time" + + "github.com/stretchr/testify/require" + "go.mau.fi/whatsmeow" + "go.mau.fi/whatsmeow/proto/waE2E" + "go.mau.fi/whatsmeow/types" + "go.mau.fi/whatsmeow/types/events" +) + +func TestWhatsAppImagesAreDurableBeforeAcknowledgement(t *testing.T) { + testCases := []struct { + name string + sender types.JID + }{ + { + name: "a contact image keeps its caption and pixels", + sender: whatsappTestAna, + }, + { + name: "an own image keeps its caption and pixels", + sender: whatsappTestOwn, + }, + } + + for _, tc := range testCases { + t.Run(tc.name, func(t *testing.T) { + dir := t.TempDir() + whatsappTestSession(t, dir, true) + + whatsapp, fake, logs := whatsappStartTest(t, dir, newTestClock()) + whatsapp.client = fake + data := imageTestPNG(t, 3, 2) + fake.downloadData = data + event := whatsappImageTestEvent(tc.sender, "Read this label.") + require.True(t, whatsapp.handle(event)) + require.Len(t, fake.Downloads(), 1) + require.Same(t, event.Message.ImageMessage, fake.Downloads()[0]) + + restored := whatsappTestState(t, filepath.Join(whatsapp.dir, whatsappStateFileName)) + require.NoError(t, restored.CloseAll()) + require.Len(t, restored.Closed(), 1) + + batch := restored.Closed()[0] + names := imageFiles(batch.Messages[0].Text) + require.Len(t, names, 1) + require.Equal(t, tc.sender == whatsappTestOwn, batch.Messages[0].Mine) + require.Equal(t, imageText(names, "Read this label."), batch.Messages[0].Text) + require.Equal(t, names, imageFiles(batch.Text(time.UTC))) + + store := &ImageStore{dir: filepath.Join(dir, imageDirName)} + saved, err := store.Read(names[0]) + require.NoError(t, err) + require.Equal(t, data, saved) + + info, err := os.Stat(filepath.Join(store.dir, names[0])) + require.NoError(t, err) + require.Equal(t, os.FileMode(0o600), info.Mode().Perm()) + require.NotContains(t, logs.String(), "Read this label") + require.NotContains(t, logs.String(), "base64") + }) + } +} + +func TestWhatsAppImageDownloadCanOutlastNameLookup(t *testing.T) { + whatsapp, fake := whatsappStartTestLinked(t) + whatsapp.client = fake + whatsapp.inbox.lookupTime = 10 * time.Millisecond + whatsapp.inbox.imageTime = time.Second + fake.downloadWait = 50 * time.Millisecond + data := imageTestPNG(t, 3, 2) + fake.downloadData = data + + require.True(t, whatsapp.handle(whatsappImageTestEvent(whatsappTestAna, "Slow photo."))) + + restored := whatsappTestState(t, whatsapp.state.path) + require.NoError(t, restored.CloseAll()) + require.Len(t, restored.Closed(), 1) + + text := restored.Closed()[0].Messages[0].Text + names := imageFiles(text) + require.Len(t, names, 1) + require.Equal(t, imageText(names, "Slow photo."), text) + + saved, err := whatsapp.inbox.images.Read(names[0]) + require.NoError(t, err) + require.Equal(t, data, saved) +} + +func TestWhatsAppImageFailureKeepsTheBatchMoving(t *testing.T) { + dir := t.TempDir() + whatsappTestSession(t, dir, true) + + whatsapp, fake, logs := whatsappStartTest(t, dir, newTestClock()) + whatsapp.client = fake + fake.downloadErr = errors.New("download failed for " + whatsappTestNumber + " and " + whatsappTestAna.User) + event := whatsappImageTestEvent(whatsappTestAna, "Keep this caption.") + require.True(t, whatsapp.handle(event)) + require.True(t, whatsapp.handle(whatsappTestMessage(whatsappTestAna, whatsappTestAna, "next", "Next text.", event.Info.Timestamp))) + + restored := whatsappTestState(t, filepath.Join(whatsapp.dir, whatsappStateFileName)) + require.NoError(t, restored.CloseAll()) + require.Len(t, restored.Closed(), 1) + + messages := restored.Closed()[0].Messages + require.Len(t, messages, 2) + require.Equal(t, "(image not downloaded)\nKeep this caption.", messages[0].Text) + require.Equal(t, "Next text.", messages[1].Text) + require.Empty(t, imageFiles(restored.Closed()[0].Text(time.UTC))) + require.Contains(t, logs.String(), "<3>image receive failed chat=") + require.Contains(t, logs.String(), ": download image: media download failed") + require.NotContains(t, logs.String(), whatsappTestNumber) + require.NotContains(t, logs.String(), whatsappTestAna.User) + require.NotContains(t, logs.String(), "Keep this caption") + require.NotContains(t, logs.String(), "base64") +} + +func TestWhatsAppImageDownloadErrorsKeepOnlySafeCauses(t *testing.T) { + mediaURL := "https://example.com/media/secret-path?auth=secret-token&hash=secret-hash" + directPath := "media/secret-path?auth=secret-token&hash=secret-hash" + testCases := []struct { + name string + err error + cause string + }{ + { + name: "signed request URLs do not reach the log", + err: &url.Error{ + Op: "Get", + URL: mediaURL, + Err: errors.New("connection failed"), + }, + cause: "media download failed", + }, + { + name: "invalid direct paths do not reach the log", + err: fmt.Errorf("media download path does not start with slash: %s", directPath), + cause: "media download failed", + }, + { + name: "request preparation errors do not expose media paths", + err: fmt.Errorf("failed to prepare request: parse %q: invalid control character", mediaURL), + cause: "media download failed", + }, + { + name: "cancellation keeps its safe cause without the URL", + err: fmt.Errorf("request %s: %w", mediaURL, context.Canceled), + cause: "context canceled", + }, + { + name: "a deadline keeps its safe cause without the path", + err: fmt.Errorf("request %s: %w", directPath, context.DeadlineExceeded), + cause: "context deadline exceeded", + }, + { + name: "HTTP failures keep only the status code", + err: whatsmeow.DownloadHTTPError{Response: &http.Response{ + StatusCode: http.StatusForbidden, + Status: "403 " + mediaURL, + }}, + cause: "media server returned HTTP 403", + }, + } + + for _, tc := range testCases { + t.Run(tc.name, func(t *testing.T) { + dir := t.TempDir() + whatsappTestSession(t, dir, true) + + whatsapp, fake, logs := whatsappStartTest(t, dir, newTestClock()) + whatsapp.client = fake + fake.downloadErr = tc.err + + require.True(t, whatsapp.handle(whatsappImageTestEvent(whatsappTestAna, "Keep this caption."))) + require.NoError(t, whatsapp.state.CloseAll()) + require.Equal(t, "(image not downloaded)\nKeep this caption.", whatsapp.state.Closed()[0].Messages[0].Text) + require.Contains(t, logs.String(), "<3>image receive failed chat=") + require.Contains(t, logs.String(), ": download image: "+tc.cause) + + for _, secret := range []string{mediaURL, directPath, "example.com", "secret-path", "secret-token", "secret-hash", whatsappTestAna.User, "Keep this caption."} { + require.NotContains(t, logs.String(), secret) + } + }) + } +} + +func TestWhatsAppImageLimitsKeepMarkersWithoutStoredPixels(t *testing.T) { + var gifBytes bytes.Buffer + + require.NoError(t, gif.Encode(&gifBytes, image.NewRGBA(image.Rect(0, 0, 2, 2)), nil)) + + testCases := []struct { + name string + mime string + size uint64 + data []byte + downloads int + }{ + { + name: "declared large images are not downloaded", + mime: "image/png", + size: imageMaxBytes + 1, + }, + { + name: "large downloaded images become markers", + mime: "image/png", + data: bytes.Repeat([]byte{0}, imageMaxBytes+1), + downloads: 1, + }, + { + name: "unsupported image types are not downloaded", + mime: "image/gif", + }, + { + name: "GIF bytes cannot claim to be PNG", + mime: "image/png", + data: gifBytes.Bytes(), + downloads: 1, + }, + { + name: "invalid image bytes become markers", + mime: "image/png", + data: []byte("not an image"), + downloads: 1, + }, + } + + for _, tc := range testCases { + t.Run(tc.name, func(t *testing.T) { + whatsapp, fake := whatsappStartTestLinked(t) + whatsapp.client = fake + fake.downloadData = tc.data + event := whatsappImageTestEvent(whatsappTestAna, "") + event.Message.ImageMessage.Mimetype = new(tc.mime) + event.Message.ImageMessage.FileLength = new(tc.size) + require.True(t, whatsapp.handle(event)) + require.Len(t, fake.Downloads(), tc.downloads) + require.NoError(t, whatsapp.state.CloseAll()) + require.Len(t, whatsapp.state.Closed(), 1) + require.Equal(t, "(image not downloaded)", whatsapp.state.Closed()[0].Messages[0].Text) + + _, err := os.Stat(whatsapp.inbox.images.dir) + require.ErrorIs(t, err, os.ErrNotExist) + }) + } +} + +func TestWhatsAppSupportsJPEGPNGAndWebPBytes(t *testing.T) { + var jpegBytes bytes.Buffer + + require.NoError(t, jpeg.Encode(&jpegBytes, image.NewRGBA(image.Rect(0, 0, 2, 2)), nil)) + + testCases := []struct { + name string + mime string + data []byte + suffix string + }{ + { + name: "PNG bytes keep the PNG extension", + mime: "image/png", + data: imageTestPNG(t, 2, 2), + suffix: ".png", + }, + { + name: "JPEG bytes set the JPEG extension even with a PNG label", + mime: "image/png", + data: jpegBytes.Bytes(), + suffix: ".jpg", + }, + { + name: "WebP bytes keep the WebP extension", + mime: "image/webp", + data: []byte{'R', 'I', 'F', 'F', 22, 0, 0, 0, 'W', 'E', 'B', 'P', 'V', 'P', '8', 'L', 10, 0, 0, 0, 0x2f, 0, 0, 0, 0, 0x07, 0x10, 0x11, 0x11, 0x88}, + suffix: ".webp", + }, + } + + for _, tc := range testCases { + t.Run(tc.name, func(t *testing.T) { + whatsapp, fake := whatsappStartTestLinked(t) + whatsapp.client = fake + fake.downloadData = tc.data + event := whatsappImageTestEvent(whatsappTestAna, "") + event.Message.ImageMessage.Mimetype = new(tc.mime) + require.True(t, whatsapp.handle(event)) + require.NoError(t, whatsapp.state.CloseAll()) + + names := imageFiles(whatsapp.state.Closed()[0].Text(time.UTC)) + require.Len(t, names, 1) + require.True(t, strings.HasSuffix(names[0], tc.suffix)) + + data, err := whatsapp.inbox.images.Read(names[0]) + require.NoError(t, err) + require.Equal(t, tc.data, data) + }) + } +} + +func TestWhatsAppMediaMarkersNeverDownload(t *testing.T) { + testCases := []struct { + name string + message *waE2E.Message + text string + }{ + { + name: "stickers keep a text marker", + message: &waE2E.Message{ + StickerMessage: &waE2E.StickerMessage{}, + }, + text: "(sticker)", + }, + { + name: "videos keep a text marker and caption", + message: &waE2E.Message{ + VideoMessage: &waE2E.VideoMessage{Caption: new("Watch this.")}, + }, + text: "(video)\nWatch this.", + }, + { + name: "GIF playback keeps a text marker", + message: &waE2E.Message{ + VideoMessage: &waE2E.VideoMessage{GifPlayback: new(true)}, + }, + text: "(GIF)", + }, + } + + for _, tc := range testCases { + t.Run(tc.name, func(t *testing.T) { + whatsapp, fake := whatsappStartTestLinked(t) + whatsapp.client = fake + event := whatsappImageTestEvent(whatsappTestAna, "") + event.Message = tc.message + require.True(t, whatsapp.handle(event)) + require.NoError(t, whatsapp.state.CloseAll()) + require.Len(t, whatsapp.state.Closed(), 1) + require.Equal(t, tc.text, whatsapp.state.Closed()[0].Messages[0].Text) + require.Empty(t, fake.Downloads()) + }) + } +} + +func TestWhatsAppPrivateImagesAreNotDownloaded(t *testing.T) { + testCases := []struct { + name string + change func(event *events.Message) + }{ + { + name: "view-once events are not read", + change: func(event *events.Message) { + event.IsViewOnce = true + }, + }, + { + name: "view-once image fields are not read", + change: func(event *events.Message) { + event.Message.ImageMessage.ViewOnce = new(true) + }, + }, + { + name: "disappearing images are not read", + change: func(event *events.Message) { + event.IsEphemeral = true + }, + }, + { + name: "self-chat images are not read", + change: func(event *events.Message) { + event.Info.Chat = whatsappTestOwn + }, + }, + } + + for _, tc := range testCases { + t.Run(tc.name, func(t *testing.T) { + whatsapp, fake := whatsappStartTestLinked(t) + whatsapp.client = fake + event := whatsappImageTestEvent(whatsappTestAna, "Secret caption.") + tc.change(event) + require.True(t, whatsapp.handle(event)) + require.NoError(t, whatsapp.state.CloseAll()) + require.Empty(t, whatsapp.state.Closed()) + require.Empty(t, fake.Downloads()) + }) + } +} + +func TestWhatsAppImageStateFailureDoesNotAcknowledge(t *testing.T) { + whatsapp, fake := whatsappStartTestLinked(t) + whatsapp.client = fake + fake.downloadData = imageTestPNG(t, 2, 2) + require.NoError(t, os.Mkdir(whatsapp.state.path, 0o700)) + require.False(t, whatsapp.handle(whatsappImageTestEvent(whatsappTestAna, "Keep this."))) + require.ErrorContains(t, <-whatsapp.inbox.failures, "save WhatsApp state") +} + +func TestWhatsAppImagesReachTheirRootOrRoutedChild(t *testing.T) { + testCases := []struct { + name string + target int + group bool + }{ + { + name: "private images wake the root with pixels", + }, + { + name: "routed group images wake only their child with pixels", + target: 1, + group: true, + }, + } + + for _, tc := range testCases { + t.Run(tc.name, func(t *testing.T) { + dir := t.TempDir() + server := newModelFake(t) + server.reply = whatsappImageTestReply + test := newDaemonMessageTest(t, dir, server) + test.agents[0].agent.config.Model = "gpt-test-root" + + whatsappTestSession(t, dir, true) + + whatsapp, fake, logs := whatsappStartTest(t, dir, newTestClock()) + whatsapp.client = fake + data := imageTestPNG(t, 3, 2) + fake.downloadData = data + event := whatsappImageTestEvent(whatsappTestAna, "What do you see?") + if tc.group { + event.Info.Chat = whatsappTestFamily + event.Info.IsGroup = true + + whatsapp.groups.set(whatsappTestFamily.String(), "Garden plans") + require.NoError(t, test.messages.routes.Set("whatsapp:"+whatsappTestFamily.String(), "Garden")) + } + + test.Start(t, tc.target) + require.True(t, whatsapp.handle(event)) + require.NoError(t, whatsapp.inbox.deliver(t.Context(), test.loops[0], whatsapp.link(), true)) + test.events[tc.target].WaitFor(t, "idle", 1) + + text := test.events[tc.target].Last("work") + names := imageFiles(text) + require.Len(t, names, 1) + require.Contains(t, text, "[image "+names[0]+"]\nWhat do you see?") + require.Equal(t, 1, test.agents[tc.target].memory.Len()) + require.Empty(t, whatsapp.state.Closed()) + + if tc.target != 0 { + require.Zero(t, test.agents[0].memory.Len()) + require.False(t, test.loops[0].waiting()) + } + + model := test.agents[tc.target].agent.config.Model + turnCalls := 0 + for index := range server.Calls() { + call := server.Call(index) + if call.body["model"] == model { + turnCalls++ + + require.Contains(t, string(call.raw), "data:image/png;base64,"+base64.StdEncoding.EncodeToString(data)) + } + } + + require.Equal(t, 1, turnCalls) + require.NotContains(t, logs.String()+test.logs.String(), "What do you see?") + require.NotContains(t, logs.String()+test.logs.String(), "base64") + }) + } +} + +func TestWhatsAppImageRoutingSurvivesRestartWithoutDuplicateEntries(t *testing.T) { + dir := t.TempDir() + server := newModelFake(t) + server.reply = whatsappImageTestReply + first := newDaemonMessageTest(t, dir, server) + whatsappTestSession(t, dir, true) + + whatsapp, fake, _ := whatsappStartTest(t, dir, newTestClock()) + whatsapp.client = fake + fake.downloadData = imageTestPNG(t, 3, 2) + event := whatsappImageTestEvent(whatsappTestAna, "Keep this image.") + require.NoError(t, first.messages.routes.Set("whatsapp:"+whatsappTestAna.String(), "Garden")) + require.True(t, whatsapp.handle(event)) + require.NoError(t, whatsapp.state.CloseAll()) + + batch := whatsapp.state.Closed()[0] + text := batch.Text(time.UTC) + routed, err := first.messages.RouteWhatsApp(batch, text) + require.NoError(t, err) + require.True(t, routed) + first.Close(t) + + second := newDaemonMessageTest(t, dir, server) + restored := whatsappRouteTestInbox(t, dir) + require.NoError(t, restored.deliver(t.Context(), second.loops[0], WhatsAppLink{}, true)) + require.Empty(t, restored.state.Closed()) + require.Zero(t, second.agents[0].memory.Len()) + require.Equal(t, 1, second.agents[1].memory.Len()) + require.True(t, second.loops[1].waiting()) + + message, err := second.agents[1].memory.Message(0) + require.NoError(t, err) + require.Equal(t, text, message.Text) + require.Len(t, imageFiles(message.Text), 1) + second.Start(t, 1) + second.events[1].WaitFor(t, "idle", 1) + require.Empty(t, second.Queue(t).Pending) + + turnCalls := 0 + for index := range server.Calls() { + call := server.Call(index) + if call.body["model"] == "gpt-test-garden" { + turnCalls++ + + require.Contains(t, string(call.raw), "input_image") + } + } + + require.Equal(t, 1, turnCalls) + second.Close(t) + + third := newDaemonMessageTest(t, dir, server) + require.Equal(t, 1, third.agents[1].memory.Len()) + require.False(t, third.loops[1].waiting()) +} + +func TestWhatsAppImageMentionsAndRepliesUseImageContext(t *testing.T) { + whatsapp, fake := whatsappStartTestLinked(t) + whatsapp.client = fake + fake.downloadData = imageTestPNG(t, 2, 2) + event := whatsappImageTestEvent(whatsappTestAna, "@"+whatsappTestLID+" check "+whatsappTestNumber) + event.Info.Chat = whatsappTestFamily + event.Message.ImageMessage.ContextInfo = &waE2E.ContextInfo{ + MentionedJID: []string{whatsappTestLID + "@lid"}, + Participant: new(whatsappTestOwn.String()), + } + + require.True(t, whatsapp.handle(event)) + require.NoError(t, whatsapp.state.CloseAll()) + + batch := whatsapp.state.Closed()[0] + require.True(t, batch.Turn()) + require.True(t, batch.Messages[0].Mention) + require.True(t, batch.Messages[0].Reply) + require.Contains(t, batch.Text(time.UTC), "@you check OWN") + require.NotContains(t, batch.Text(time.UTC), whatsappTestNumber) + require.NotContains(t, batch.Text(time.UTC), whatsappTestLID) +} + +func whatsappImageTestEvent(sender types.JID, caption string) *events.Message { + event := whatsappTestMessage(whatsappTestAna, sender, "image", "", time.Date(2026, 1, 1, 12, 0, 0, 0, time.UTC)) + event.Message = &waE2E.Message{ + ImageMessage: &waE2E.ImageMessage{ + Mimetype: new("image/png"), + Caption: new(caption), + }, + } + + return event +} + +func whatsappImageTestReply(w http.ResponseWriter, call ModelFakeCall) { + text := "A red test image." + if call.body["model"] == "gpt-test-root" || call.body["model"] == "gpt-test-garden" { + text = "" + } + + modelWriteEvents(w, modelCompletedEvent([]any{modelMessageItem(text)}, 100, 0, 5)) +} diff --git a/whatsapp_inbox.go b/whatsapp_inbox.go index adff79e..5121976 100644 --- a/whatsapp_inbox.go +++ b/whatsapp_inbox.go @@ -2,7 +2,9 @@ package main import ( "context" + "crypto/sha256" "fmt" + "log/slog" "slices" "strings" "sync" @@ -23,19 +25,24 @@ const ( ) // newWhatsAppInbox returns the inbox of the batches in state, with clock for -// the times of the batch texts and groups for the names of the groups. -func newWhatsAppInbox(state *WhatsAppState, clock func() time.Time, groups *WhatsAppGroups) *WhatsAppInbox { +// the times of the batch texts and groups for the names of the groups. images +// holds shared images, and logger gets image failures. +func newWhatsAppInbox(state *WhatsAppState, clock func() time.Time, groups *WhatsAppGroups, images *ImageStore, logger *slog.Logger) *WhatsAppInbox { return &WhatsAppInbox{ state: state, clock: clock, groups: groups, + images: images, + logger: logger, + lookupTime: whatsappLookupTime, + imageTime: whatsappImageTime, catchupLimit: whatsappCatchupLimit, wake: make(chan struct{}, 1), failures: make(chan error, 1), } } -// WhatsAppInbox puts the text messages into batches, one chat at a time, as +// WhatsAppInbox puts messages into batches, one chat at a time, as // WhatsAppState says. deliver logs each closed batch as a work message that // starts with [WhatsApp], and a batch that needs the user wakes the turn loop. // @@ -61,6 +68,10 @@ type WhatsAppInbox struct { state *WhatsAppState clock func() time.Time groups *WhatsAppGroups + images *ImageStore + logger *slog.Logger + lookupTime time.Duration + imageTime time.Duration catchupLimit time.Duration wake chan struct{} failures chan error @@ -212,13 +223,13 @@ func (i *WhatsAppInbox) startCatchup() { }) } -// receive puts the text of event into its batch and wakes Run. link gives the -// device and the own IDs. It returns false when the state file cannot hold the +// receive stores an image before its batch and wakes Run. link gives the device +// and the own IDs. It returns false when the state file cannot hold the // message, so that the library does not acknowledge it, and Run stops the -// daemon. A message that is not text, or that the batches do not take, is -// handled with nothing to do. +// daemon. An image that cannot be downloaded or stored gets a text marker. A +// message that the batches do not take is handled with nothing to do. func (i *WhatsAppInbox) receive(link WhatsAppLink, event *events.Message) bool { - ctx, cancel := context.WithTimeout(context.Background(), whatsappLookupTime) + ctx, cancel := context.WithTimeout(context.Background(), i.lookupTime) defer cancel() chat, message, found := whatsappMessage(ctx, link.device, link.own, event) @@ -226,6 +237,13 @@ func (i *WhatsAppInbox) receive(link WhatsAppLink, event *events.Message) bool { return true } + if event.Message.GetImageMessage() != nil { + ctx, cancel := context.WithTimeout(context.Background(), i.imageTime) + defer cancel() + + message.Text = i.imageText(ctx, link, event, message.Text) + } + _, err := i.state.Add(chat, message, time.Now()) if err != nil { select { @@ -244,6 +262,28 @@ func (i *WhatsAppInbox) receive(link WhatsAppLink, event *events.Message) bool { return true } +// imageText stores the image of event before returning its marker and caption. +// Failures keep the caption with a text marker. The daemon log identifies the +// chat by a short hash, not its phone number. +func (i *WhatsAppInbox) imageText(ctx context.Context, link WhatsAppLink, event *events.Message, caption string) string { + name, err := whatsappImage(ctx, link.client, i.images, event.Message.GetImageMessage()) + if err != nil { + chat := event.Info.Chat.ToNonAD() + digest := sha256.Sum256([]byte(chat.String())) + redact := append(slices.Clone(link.own), chat.User) + i.logger.Error("image receive failed", "chat", fmt.Sprintf("%x", digest[:6]), "err", whatsappRedact(redact, err.Error())) + + text := "(image not downloaded)" + if caption != "" { + text += "\n" + caption + } + + return text + } + + return imageText([]string{name}, caption) +} + // caughtUpBy ends the catch-up of the connection with the number catchups, when // no later connection started a new one. func (i *WhatsAppInbox) caughtUpBy(catchups int) { @@ -303,10 +343,10 @@ func (i *WhatsAppInbox) close() { // whatsappMessage returns the chat and the batch message of event, or false // when the batches do not take it. device is the linked device, and own holds -// the own IDs. The batches take the text of a private chat or a group, from -// another person or from the user. They do not take media, reactions, edits, -// deletions, view-once and disappearing messages, old history, status -// broadcasts, channels, and the chat of the user with itself. +// the own IDs. The batches take text, images, and media markers from a private +// chat or a group, from another person or from the user. They do not take +// reactions, edits, deletions, view-once and disappearing messages, old +// history, status broadcasts, channels, and the chat of the user with itself. // // A mention of the user becomes @you, another mention @NAME when the name is // known, and the own number and LID become OWN, so that they never go to the @@ -316,7 +356,7 @@ func whatsappMessage(ctx context.Context, device *store.Device, own []string, ev chatJID := info.Chat.ToNonAD() switch { case device == nil: - case event.IsEdit, info.Edit != "", event.IsEphemeral, event.IsViewOnce, event.SourceWebMsg != nil: + case event.IsEdit, info.Edit != "", event.IsEphemeral, event.IsViewOnce, event.Message.GetImageMessage().GetViewOnce(), event.Message.GetVideoMessage().GetViewOnce(), event.SourceWebMsg != nil: case chatJID.Server != types.DefaultUserServer && chatJID.Server != types.HiddenUserServer && chatJID.Server != types.GroupServer: case slices.Contains(own, chatJID.User): default: @@ -327,7 +367,7 @@ func whatsappMessage(ctx context.Context, device *store.Device, own []string, ev } // whatsappText returns the chat and the batch message of event, a message that -// the batches take when it has text. +// the batches take. An image returns its caption until receive stores it. func whatsappText(ctx context.Context, device *store.Device, own []string, event *events.Message) (WhatsAppChat, WhatsAppMessage, bool) { info := event.Info text := event.Message.GetConversation() @@ -336,11 +376,33 @@ func whatsappText(ctx context.Context, device *store.Device, own []string, event text = extended.GetText() } - if strings.TrimSpace(text) == "" { - return WhatsAppChat{}, WhatsAppMessage{}, false + contextInfo := extended.GetContextInfo() + switch { + case event.Message.GetImageMessage() != nil: + image := event.Message.GetImageMessage() + text = image.GetCaption() + contextInfo = image.GetContextInfo() + case event.Message.GetStickerMessage() != nil: + text = "(sticker)" + contextInfo = event.Message.GetStickerMessage().GetContextInfo() + case event.Message.GetVideoMessage() != nil: + video := event.Message.GetVideoMessage() + text = "(video)" + if video.GetGifPlayback() { + text = "(GIF)" + } + + if video.GetCaption() != "" { + text += "\n" + video.GetCaption() + } + + contextInfo = video.GetContextInfo() + default: + if strings.TrimSpace(text) == "" { + return WhatsAppChat{}, WhatsAppMessage{}, false + } } - contextInfo := extended.GetContextInfo() message := WhatsAppMessage{ ID: info.ID, Time: info.Timestamp, diff --git a/whatsapp_inbox_test.go b/whatsapp_inbox_test.go index c3937df..cea65fe 100644 --- a/whatsapp_inbox_test.go +++ b/whatsapp_inbox_test.go @@ -133,7 +133,7 @@ func TestWhatsAppGroupTurns(t *testing.T) { } } -func TestWhatsAppTakesOnlyText(t *testing.T) { +func TestWhatsAppSkipsExcludedMessages(t *testing.T) { at := time.Date(2026, 1, 1, 12, 0, 0, 0, time.UTC) testCases := []struct { name string @@ -175,16 +175,6 @@ func TestWhatsAppTakesOnlyText(t *testing.T) { event.Info.Chat = types.StatusBroadcastJID }, }, - { - name: "a photo with a caption", - change: func(event *events.Message) { - event.Message = &waE2E.Message{ - ImageMessage: &waE2E.ImageMessage{ - Caption: new("look"), - }, - } - }, - }, { name: "a text of spaces", change: func(event *events.Message) { diff --git a/whatsapp_route_test.go b/whatsapp_route_test.go index 92c9cf3..9c855c9 100644 --- a/whatsapp_route_test.go +++ b/whatsapp_route_test.go @@ -393,7 +393,7 @@ func whatsappRouteTestInbox(t *testing.T, dir string) *WhatsAppInbox { state, err := openWhatsAppState(filepath.Join(dir, whatsappDirName, whatsappStateFileName), whatsappBatchQuiet, whatsappBatchWindow) require.NoError(t, err) - return newWhatsAppInbox(state, newTestClock().Now, newWhatsAppGroups(discardLog())) + return newWhatsAppInbox(state, newTestClock().Now, newWhatsAppGroups(discardLog()), &ImageStore{dir: filepath.Join(dir, imageDirName)}, discardLog()) } func whatsappRouteTestAdd(t *testing.T, inbox *WhatsAppInbox, chat WhatsAppChat, text string) { -- 2.53.0 From 6915dbdfeb246551bf1132823d820beb321fd58c Mon Sep 17 00:00:00 2001 From: "Anderson A. Giacomolli" Date: Tue, 10 Nov 2026 14:37:54 -0300 Subject: [PATCH 07/16] Derive contexts from the caller and name timeouts The go-code skill asks for contexts derived from the caller, with context.WithoutCancel for work that must finish after a cancel and context.Background only where no caller context exists. It also names each operation limit after its operation with the Timeout suffix, in the file that applies it. Handlers with a background context could not be stopped by the daemon, and limits named only Time did not say what they limit. --- .agents/skills/go-code/SKILL.md | 9 +++++++++ 1 file changed, 9 insertions(+) diff --git a/.agents/skills/go-code/SKILL.md b/.agents/skills/go-code/SKILL.md index f9ba075..213a7d5 100644 --- a/.agents/skills/go-code/SKILL.md +++ b/.agents/skills/go-code/SKILL.md @@ -40,6 +40,15 @@ description: Go style rules. Use before you write or review Go code. Call `defer cancel()` immediately after you create the context. - Reuse `ctx` when a child context replaces its parent. Use a different name only when both contexts stay in use. Return `ctx.Err()` directly. +- Derive each context from the context of the caller. When work must + finish even after the caller is cancelled, such as a final write, use + `context.WithoutCancel` of the caller's context. Use + `context.Background()` only where no caller context exists: in `run`, + and in methods that follow `io.Closer`. +- Name a duration that limits one operation after that operation, and + end the name with `Timeout`, so that the name says what it limits + without its comment. Define it in the file that applies it, not in the + file of another type. ## Errors -- 2.53.0 From 3b2bf881d484ba1a33ff77a342564b8195a75c80 Mon Sep 17 00:00:00 2001 From: "Anderson A. Giacomolli" Date: Tue, 10 Nov 2026 14:43:29 -0300 Subject: [PATCH 08/16] Pass caller contexts through WhatsApp callbacks Library callbacks use the context of WhatsApp.Run, so daemon cancellation ends name lookups and image downloads. Each image keeps its own deadline without the shorter name lookup deadline. Tests use their caller context. --- daemon_log_test.go | 3 +-- whatsapp.go | 15 +++++++++++++-- whatsapp_image_test.go | 31 +++++++++++++++++++++++++++++++ whatsapp_inbox.go | 8 ++++---- 4 files changed, 49 insertions(+), 8 deletions(-) diff --git a/daemon_log_test.go b/daemon_log_test.go index 31a3278..b3a0fa4 100644 --- a/daemon_log_test.go +++ b/daemon_log_test.go @@ -1,7 +1,6 @@ package main import ( - "context" "errors" "io" "log/slog" @@ -124,7 +123,7 @@ func TestDaemonLogReturnsWriteFailure(t *testing.T) { mutex: &sync.Mutex{}, } - err := handler.Handle(context.Background(), slog.NewRecord(time.Now(), slog.LevelInfo, "ready", 0)) + err := handler.Handle(t.Context(), slog.NewRecord(time.Now(), slog.LevelInfo, "ready", 0)) require.ErrorIs(t, err, io.ErrClosedPipe) require.ErrorContains(t, err, "write daemon log") } diff --git a/whatsapp.go b/whatsapp.go index f225756..be1acc6 100644 --- a/whatsapp.go +++ b/whatsapp.go @@ -107,6 +107,7 @@ type WhatsAppClient interface { // the client of a device with its event handler. func openWhatsApp(ctx context.Context, dir string, clock func() time.Time, logger *slog.Logger, newClient func(device *store.Device, handle func(event any) bool) WhatsAppClient) (*WhatsApp, error) { w := &WhatsApp{ + ctx: ctx, dir: filepath.Join(dir, whatsappDirName), clock: clock, logger: logger, @@ -169,6 +170,9 @@ func openWhatsApp(ctx context.Context, dir string, clock func() time.Time, logge // creation, close, removal, and snapshots. It is held before mutex, and stays // held when removal releases mutex for client callbacks. // +// ctx is the caller context until Run sets its context for library callbacks. +// The mutex protects it, so callbacks follow daemon cancellation. +// // The inbox puts messages into batches and logs them, and groups holds the // names of the groups. Both get a WhatsAppLink from link for each call, so they // use the current device and client. @@ -181,6 +185,7 @@ func openWhatsApp(ctx context.Context, dir string, clock func() time.Time, logge // log, and those of private chats hold phone numbers, because the agent needs // them to send. type WhatsApp struct { + ctx context.Context dir string clock func() time.Time logger *slog.Logger @@ -224,6 +229,7 @@ type WhatsApp struct { func (w *WhatsApp) Run(ctx context.Context, l *TurnLoop, notify func(Event)) error { w.mutex.Lock() + w.ctx = ctx w.notify = notify linked := w.linked w.retrying = linked @@ -900,9 +906,14 @@ func (w *WhatsApp) linkedBy(client WhatsAppClient) { // connected again with the length of the outage. The retries of the library and // Run between them log nothing. func (w *WhatsApp) handle(event any) bool { + w.mutex.Lock() + + ctx := w.ctx + w.mutex.Unlock() + switch event := event.(type) { case *events.Message: - return w.inbox.receive(w.link(), event) + return w.inbox.receive(ctx, w.link(), event) case *events.GroupInfo: if event.Name != nil { w.groups.set(event.JID.String(), event.Name.Name) @@ -987,7 +998,7 @@ func (w *WhatsApp) handle(event any) bool { w.mutex.Unlock() if line != "" { - w.logger.Log(context.Background(), level, line, attrs...) + w.logger.Log(ctx, level, line, attrs...) } if notice != "" { diff --git a/whatsapp_image_test.go b/whatsapp_image_test.go index f00bc3a..919d656 100644 --- a/whatsapp_image_test.go +++ b/whatsapp_image_test.go @@ -15,6 +15,7 @@ import ( "path/filepath" "strings" "testing" + "testing/synctest" "time" "github.com/stretchr/testify/require" @@ -103,6 +104,36 @@ func TestWhatsAppImageDownloadCanOutlastNameLookup(t *testing.T) { require.Equal(t, data, saved) } +func TestWhatsAppRunCancellationEndsAnImageDownload(t *testing.T) { + synctest.Test(t, func(t *testing.T) { + test := whatsappRetryTest(t) + test.fake.downloadWait = time.Hour + test.start(t) + synctest.Wait() + + acknowledged := make(chan bool, 1) + go func() { + acknowledged <- test.fake.Emit(whatsappImageTestEvent(whatsappTestAna, "Keep this caption.")) + }() + + synctest.Wait() + + select { + case <-acknowledged: + t.Fatal("download ended before daemon cancellation") + default: + } + + test.cancel() + require.True(t, <-acknowledged) + require.NoError(t, <-test.done) + require.Empty(t, test.fake.Downloads()) + require.Contains(t, test.logs.String(), "context canceled") + require.NoError(t, test.whatsapp.state.CloseAll()) + require.Equal(t, "(image not downloaded)\nKeep this caption.", test.whatsapp.state.Closed()[0].Messages[0].Text) + }) +} + func TestWhatsAppImageFailureKeepsTheBatchMoving(t *testing.T) { dir := t.TempDir() whatsappTestSession(t, dir, true) diff --git a/whatsapp_inbox.go b/whatsapp_inbox.go index 5121976..06f1c55 100644 --- a/whatsapp_inbox.go +++ b/whatsapp_inbox.go @@ -228,17 +228,17 @@ func (i *WhatsAppInbox) startCatchup() { // message, so that the library does not acknowledge it, and Run stops the // daemon. An image that cannot be downloaded or stored gets a text marker. A // message that the batches do not take is handled with nothing to do. -func (i *WhatsAppInbox) receive(link WhatsAppLink, event *events.Message) bool { - ctx, cancel := context.WithTimeout(context.Background(), i.lookupTime) +func (i *WhatsAppInbox) receive(ctx context.Context, link WhatsAppLink, event *events.Message) bool { + lookupCtx, cancel := context.WithTimeout(ctx, i.lookupTime) defer cancel() - chat, message, found := whatsappMessage(ctx, link.device, link.own, event) + chat, message, found := whatsappMessage(lookupCtx, link.device, link.own, event) if !found { return true } if event.Message.GetImageMessage() != nil { - ctx, cancel := context.WithTimeout(context.Background(), i.imageTime) + ctx, cancel := context.WithTimeout(ctx, i.imageTime) defer cancel() message.Text = i.imageText(ctx, link, event, message.Text) -- 2.53.0 From 48d7bce28b18ecef4c52c98821675605464cecb9 Mon Sep 17 00:00:00 2001 From: "Anderson A. Giacomolli" Date: Tue, 10 Nov 2026 15:01:07 -0300 Subject: [PATCH 09/16] Keep time limits with their operations Operation files own their duration constants without struct fields that only supply shorter test waits. Fake time checks the full send, lookup, batch, catch-up, notice, and schedule waits. Retry delays and batch policy values stay the same. --- schedule.go | 10 ++- schedule_test.go | 42 +++++++++++- telegram.go | 37 ++++++----- telegram_api.go | 34 +++++----- telegram_api_test.go | 48 ++++++++------ telegram_fake_helper_test.go | 30 +++++++++ telegram_image_test.go | 2 - telegram_test.go | 125 ++++++++++++++++++++--------------- turn_loop_helper_test.go | 31 +++++++++ whatsapp.go | 57 ++++++++-------- whatsapp_fake_helper_test.go | 51 +++++++++++++- whatsapp_groups.go | 6 +- whatsapp_image.go | 4 -- whatsapp_image_test.go | 44 ++++++------ whatsapp_inbox.go | 69 +++++++++---------- whatsapp_inbox_test.go | 48 +++++++++++--- whatsapp_route_test.go | 2 +- whatsapp_send_test.go | 2 +- whatsapp_state.go | 31 ++++----- whatsapp_state_test.go | 4 +- whatsapp_test.go | 109 +++++++----------------------- 21 files changed, 454 insertions(+), 332 deletions(-) create mode 100644 turn_loop_helper_test.go diff --git a/schedule.go b/schedule.go index b3a65ea..a1eba1c 100644 --- a/schedule.go +++ b/schedule.go @@ -58,7 +58,6 @@ func openSchedules(path, owner string, clock func() time.Time, logger *slog.Logg owner: owner, clock: clock, logger: logger, - check: scheduleCheck, changed: make(chan struct{}, 1), } @@ -118,7 +117,6 @@ type Schedules struct { owner string clock func() time.Time logger *slog.Logger - check time.Duration changed chan struct{} mutex sync.Mutex @@ -126,9 +124,9 @@ type Schedules struct { } // Run starts the runs of the due schedules in l until ctx ends. It waits until -// the first due time, at most check at a time, and also wakes at a set or a -// remove. So a schedule that came due while the daemon was stopped runs at the -// start. It returns the error of a write of the log or of the file. +// the first due time, at most scheduleCheck at a time, and also wakes at a set +// or a remove. So a schedule that came due while the daemon was stopped runs at +// the start. It returns the error of a write of the log or of the file. func (s *Schedules) Run(ctx context.Context, l *TurnLoop) error { for { err := s.deliver(ctx, l) @@ -143,7 +141,7 @@ func (s *Schedules) Run(ctx context.Context, l *TurnLoop) error { wait, found := s.Wait(s.clock()) if found { - timer = time.NewTimer(min(wait, s.check)) + timer = time.NewTimer(min(wait, scheduleCheck)) timeout = timer.C } diff --git a/schedule_test.go b/schedule_test.go index 6610c26..5743a22 100644 --- a/schedule_test.go +++ b/schedule_test.go @@ -7,6 +7,7 @@ import ( "strings" "sync" "testing" + "testing/synctest" "time" "github.com/stretchr/testify/require" @@ -251,6 +252,9 @@ func TestSchedulesRunStartsATurnWhenDue(t *testing.T) { require.NoError(t, err) clock.Set(clock.Now().Add(5 * time.Minute)) + + schedules.changed <- struct{}{} + test.events.WaitFor(t, "idle", 1) require.Equal(t, 1, server.Calls()) @@ -295,7 +299,6 @@ func TestDaemonRunsADueSchedule(t *testing.T) { schedules, err := openSchedules(filepath.Join(t.TempDir(), scheduleFileName), "OneChat", clock.Now, discardLog()) require.NoError(t, err) - schedules.check = time.Millisecond whatsapp, _, _ := whatsappStartTest(t, t.TempDir(), newTestClock()) socket, _ := daemonStartTestWithSchedules(t, newModelFake(t), nil, whatsapp, schedules, clock.Now) client := terminalStartTestClient(t, socket) @@ -305,9 +308,43 @@ func TestDaemonRunsADueSchedule(t *testing.T) { require.NoError(t, err) clock.Set(clock.Now().Add(5 * time.Minute)) + + schedules.changed <- struct{}{} + client.WaitFor(t, "[Schedule 1] Sunday 2026-11-08 20:05 -03:00, once\nRemind the user to drink water.\n"+scheduleEmptyReply+"\n[working]\na line\n[done]") } +func TestSchedulesCheckTheClockAfterItsWaitLimit(t *testing.T) { + synctest.Test(t, func(t *testing.T) { + whatsapp, _, _ := whatsappStartTest(t, t.TempDir(), newTestClock()) + loop, events := newTestIdleTurnLoop(t, whatsapp) + clock := &ScheduleTestClock{now: time.Now()} + schedules, err := openSchedules(filepath.Join(t.TempDir(), scheduleFileName), "OneChat", clock.Now, discardLog()) + require.NoError(t, err) + + _, err = schedules.Set("Check the clock.", new(clock.Now().Add(5*time.Minute).Format(scheduleAtLayout)), nil) + require.NoError(t, err) + + ctx, cancel := context.WithCancel(t.Context()) + defer cancel() + + done := make(chan error, 1) + go func() { + done <- schedules.Run(ctx, loop) + }() + + synctest.Wait() + require.Zero(t, loop.memory.Len()) + clock.Set(clock.Now().Add(5 * time.Minute)) + time.Sleep(scheduleCheck) + synctest.Wait() + require.Equal(t, 1, loop.memory.Len()) + require.Contains(t, events.Last("work"), "Check the clock.") + cancel() + require.NoError(t, <-done) + }) +} + func TestToolboxRunsTheScheduleTools(t *testing.T) { schedules, _ := scheduleTestOpen(t) box := &Toolbox{ @@ -396,11 +433,10 @@ func scheduleTestOpen(t *testing.T) (*Schedules, string) { } // scheduleTestRun runs schedules in a new turn loop that calls server, until -// the test ends. Run looks at the times every millisecond. +// the test ends. func scheduleTestRun(t *testing.T, schedules *Schedules, server *ModelFake) *TurnTestLoop { t.Helper() - schedules.check = time.Millisecond test := newTestTurnLoop(t, server) ctx, cancel := context.WithCancel(t.Context()) done := make(chan error) diff --git a/telegram.go b/telegram.go index a595b68..f3b3bb2 100644 --- a/telegram.go +++ b/telegram.go @@ -18,6 +18,14 @@ import ( ) const ( + // telegramRetryDelay is the wait before the first retry of a failed + // call. + telegramRetryDelay = time.Second + + // telegramRetryLimit is the longest wait between retries of a failed + // call. + telegramRetryLimit = time.Minute + // telegramAPIURL is the Telegram Bot API service. telegramAPIURL = "https://api.telegram.org" @@ -103,8 +111,6 @@ func newTelegramBot(api *TelegramAPI, owner int64, offsetPath string, logger *sl owner: owner, offsetPath: offsetPath, logger: logger, - minWait: time.Second, - maxWait: time.Minute, ready: make(chan struct{}, 1), } } @@ -146,17 +152,15 @@ func (c TelegramConfig) validate() error { // does not lose a message. A crash between the flush and the offset write // makes Telegram send the update again, and the message repeats once. // -// A failed Bot API call waits and tries again, from minWait up to maxWait, or -// as long as the Bot API asks. It never stops the agent. A TelegramOutage logs -// the first failure and the end of each run of failures, and the retries -// between them log nothing. +// A failed Bot API call waits and tries again, from telegramRetryDelay up to +// telegramRetryLimit, or as long as the Bot API asks. It never stops the agent. +// A TelegramOutage logs the first failure and the end of each run of failures, +// and the retries between them log nothing. type TelegramBot struct { api *TelegramAPI owner int64 offsetPath string logger *slog.Logger - minWait time.Duration - maxWait time.Duration mutex sync.Mutex queue []Event @@ -279,8 +283,8 @@ func (b *TelegramBot) poll(ctx context.Context, l *TurnLoop, whatsapp *WhatsApp) back: "image receive back", } - imageWait := b.minWait - wait := b.minWait + imageWait := telegramRetryDelay + wait := telegramRetryDelay poll: for ctx.Err() == nil { @@ -298,7 +302,7 @@ poll: outage.End() - wait = b.minWait + wait = telegramRetryDelay for _, update := range updates { if ctx.Err() != nil { @@ -318,7 +322,7 @@ poll: imageOutage.End() - imageWait = b.minWait + imageWait = telegramRetryDelay offset = update.UpdateID + 1 err = telegramSaveOffset(b.offsetPath, offset) @@ -373,7 +377,8 @@ func (b *TelegramBot) post(ctx context.Context, event Event) { } // pause gives err to outage and waits the longer of wait and asked, or until -// ctx ends. It returns the wait of the next failure: twice wait, up to maxWait. +// ctx ends. It returns the wait of the next failure: twice wait, up to +// telegramRetryLimit. func (b *TelegramBot) pause(ctx context.Context, outage *TelegramOutage, err error, wait, asked time.Duration) time.Duration { delay := max(wait, asked) outage.Fail(err, delay) @@ -386,7 +391,7 @@ func (b *TelegramBot) pause(ctx context.Context, outage *TelegramOutage, err err case <-ctx.Done(): } - return min(2*wait, b.maxWait) + return min(2*wait, telegramRetryLimit) } // handle runs one update. A text message of the owner in the private chat goes @@ -480,7 +485,7 @@ func (b *TelegramBot) handle(ctx context.Context, l *TurnLoop, whatsapp *WhatsAp // later try can fix, until ctx ends. It drops the part after any other failure. func (b *TelegramBot) deliver(ctx context.Context, part TelegramPart) { outage := b.sendOutage() - wait := b.minWait + wait := telegramRetryDelay for { err := b.api.SendMessage(ctx, b.owner, part.text, part.entities) retry, asked := telegramRetryable(err) @@ -521,7 +526,7 @@ func (b *TelegramBot) deliver(ctx context.Context, part TelegramPart) { // fix, until ctx ends. It drops the message after any other failure. func (b *TelegramBot) deliverRich(ctx context.Context, markdown string) { outage := b.sendOutage() - wait := b.minWait + wait := telegramRetryDelay for { err := b.api.SendRichMessage(ctx, b.owner, markdown) retry, asked := telegramRetryable(err) diff --git a/telegram_api.go b/telegram_api.go index 32ad9a9..a04b2fa 100644 --- a/telegram_api.go +++ b/telegram_api.go @@ -41,10 +41,9 @@ const ( // newTelegramAPI returns the client of the Bot API at base with token. func newTelegramAPI(base, token string) *TelegramAPI { return &TelegramAPI{ - base: base, - token: token, - http: &http.Client{}, - sendTimeout: telegramSendTimeout, + base: base, + token: token, + http: &http.Client{}, } } @@ -78,14 +77,11 @@ func telegramRejected(err error) bool { } // TelegramAPI calls the Telegram Bot API. The bot token is a part of each URL, -// so no error of TelegramAPI holds a URL, a net/http error, or the token. Its -// sendTimeout field limits each sendMessage, sendRichMessage, and -// sendChatAction call. +// so no error of TelegramAPI holds a URL, a net/http error, or the token. type TelegramAPI struct { - base string - token string - http *http.Client - sendTimeout time.Duration + base string + token string + http *http.Client } // GetUpdates waits for the updates from offset on, up to telegramPollSeconds, @@ -111,7 +107,7 @@ func (b *TelegramAPI) GetImage(ctx context.Context, id string) (data []byte, err Size int64 `json:"file_size"` } - err = b.call(ctx, b.sendTimeout, "getFile", map[string]any{"file_id": id}, &file) + err = b.call(ctx, telegramSendTimeout, "getFile", map[string]any{"file_id": id}, &file) if err != nil { return nil, fmt.Errorf("get image file: %w", err) } @@ -124,7 +120,7 @@ func (b *TelegramAPI) GetImage(ctx context.Context, id string) (data []byte, err return nil, errors.New("invalid image file path") } - ctx, cancel := context.WithTimeout(ctx, b.sendTimeout) + ctx, cancel := context.WithTimeout(ctx, telegramSendTimeout) defer cancel() request, err := http.NewRequestWithContext(ctx, http.MethodGet, b.base+"/file/bot"+b.token+"/"+file.Path, nil) @@ -134,13 +130,13 @@ func (b *TelegramAPI) GetImage(ctx context.Context, id string) (data []byte, err response, err := b.http.Do(request) if err != nil { - return nil, b.networkError("image download failed", err, b.sendTimeout) + return nil, b.networkError("image download failed", err, telegramSendTimeout) } defer func() { closeErr := response.Body.Close() if closeErr != nil { - err = errors.Join(err, b.networkError("close image download", closeErr, b.sendTimeout)) + err = errors.Join(err, b.networkError("close image download", closeErr, telegramSendTimeout)) } }() @@ -153,7 +149,7 @@ func (b *TelegramAPI) GetImage(ctx context.Context, id string) (data []byte, err data, err = io.ReadAll(io.LimitReader(response.Body, imageMaxBytes+1)) if err != nil { - return nil, b.networkError("image cut off", err, b.sendTimeout) + return nil, b.networkError("image cut off", err, telegramSendTimeout) } if len(data) > imageMaxBytes { @@ -177,14 +173,14 @@ func (b *TelegramAPI) SendMessage(ctx context.Context, chat int64, text string, params["entities"] = entities } - return b.call(ctx, b.sendTimeout, "sendMessage", params, nil) + return b.call(ctx, telegramSendTimeout, "sendMessage", params, nil) } // SendRichMessage sends markdown to chat as a rich message, which Telegram // renders itself, tables included. A call that takes longer than // telegramSendTimeout fails with a TelegramNetworkError. func (b *TelegramAPI) SendRichMessage(ctx context.Context, chat int64, markdown string) error { - return b.call(ctx, b.sendTimeout, "sendRichMessage", map[string]any{ + return b.call(ctx, telegramSendTimeout, "sendRichMessage", map[string]any{ "chat_id": chat, "rich_message": map[string]any{ "markdown": markdown, @@ -195,7 +191,7 @@ func (b *TelegramAPI) SendRichMessage(ctx context.Context, chat int64, markdown // SendTyping shows the typing action in chat for about five seconds. A call // that takes longer than telegramSendTimeout fails with a TelegramNetworkError. func (b *TelegramAPI) SendTyping(ctx context.Context, chat int64) error { - return b.call(ctx, b.sendTimeout, "sendChatAction", map[string]any{ + return b.call(ctx, telegramSendTimeout, "sendChatAction", map[string]any{ "chat_id": chat, "action": "typing", }, nil) diff --git a/telegram_api_test.go b/telegram_api_test.go index 98aa232..6fa4cba 100644 --- a/telegram_api_test.go +++ b/telegram_api_test.go @@ -6,6 +6,7 @@ import ( "net/http" "net/http/httptest" "testing" + "testing/synctest" "time" "github.com/stretchr/testify/require" @@ -103,31 +104,34 @@ func TestSendCallsTimeOutOnAStalledConnection(t *testing.T) { for _, tc := range testCases { t.Run(tc.method, func(t *testing.T) { - server := newTelegramFake(t) - server.stall = func(call TelegramFakeCall) bool { - return call.method == tc.method - } - - api := newTelegramAPI(server.URL, telegramTestToken) - api.sendTimeout = 50 * time.Millisecond - start := time.Now() - err := tc.send(t.Context(), api) - require.EqualError(t, err, "no answer in 50ms") - require.Less(t, time.Since(start), 5*time.Second) - require.NotContains(t, err.Error(), telegramTestToken) - require.NotContains(t, err.Error(), server.URL) + synctest.Test(t, func(t *testing.T) { + server := newTelegramLocalFake(t) + server.stall = func(call TelegramFakeCall) bool { + return call.method == tc.method + } + + api := newTelegramAPI(server.URL, telegramTestToken) + api.http.Transport = server + start := time.Now() + err := tc.send(t.Context(), api) + require.EqualError(t, err, "no answer in 30s") + require.Equal(t, telegramSendTimeout, time.Since(start)) + require.NotContains(t, err.Error(), telegramTestToken) + require.NotContains(t, err.Error(), server.URL) + + retry, _ := telegramRetryable(err) + require.True(t, retry) - retry, _ := telegramRetryable(err) - require.True(t, retry) + ctx, cancel := context.WithCancel(t.Context()) + defer cancel() - api.sendTimeout = time.Hour - ctx, cancel := context.WithCancel(t.Context()) - time.AfterFunc(50*time.Millisecond, cancel) + time.AfterFunc(time.Second, cancel) - start = time.Now() - err = tc.send(ctx, api) - require.Error(t, err) - require.Less(t, time.Since(start), 5*time.Second) + start = time.Now() + err = tc.send(ctx, api) + require.Error(t, err) + require.Equal(t, time.Second, time.Since(start)) + }) }) } } diff --git a/telegram_fake_helper_test.go b/telegram_fake_helper_test.go index 6a89054..f6ccd84 100644 --- a/telegram_fake_helper_test.go +++ b/telegram_fake_helper_test.go @@ -98,6 +98,19 @@ func (f *TelegramFake) Image(id string, data []byte) { f.images[id] = data } +// RoundTrip handles a request in memory, so synctest can control its timers. +func (f *TelegramFake) RoundTrip(request *http.Request) (*http.Response, error) { + response := httptest.NewRecorder() + f.serve(response, request) + + ctx := request.Context() + if ctx.Err() != nil { + return nil, ctx.Err() + } + + return response.Result(), nil +} + func (f *TelegramFake) serve(w http.ResponseWriter, r *http.Request) { id, file := strings.CutPrefix(r.URL.Path, "/file/bot"+telegramTestToken+"/") if file { @@ -256,6 +269,23 @@ func newTelegramFake(t *testing.T) *TelegramFake { return f } +// newTelegramLocalFake returns the same fake API without network goroutines. +// Its RoundTrip handles requests directly for tests with fake time. +func newTelegramLocalFake(t *testing.T) *TelegramFake { + t.Helper() + + f := &TelegramFake{ + Server: &httptest.Server{URL: "https://example.com"}, + added: make(chan struct{}, 1), + stopped: make(chan struct{}), + } + t.Cleanup(func() { + close(f.stopped) + }) + + return f +} + // telegramOwnerMessage returns a text message of the owner in the private chat. func telegramOwnerMessage(text string) *TelegramMessage { return &TelegramMessage{ diff --git a/telegram_image_test.go b/telegram_image_test.go index 837a64e..7f7b7c0 100644 --- a/telegram_image_test.go +++ b/telegram_image_test.go @@ -194,8 +194,6 @@ func TestTelegramRetriesImageDownloadBeforeAdvancingOffset(t *testing.T) { return 0, "" } bot, logs := telegramStartTestBot(t, telegram) - bot.minWait = time.Millisecond - bot.maxWait = 5 * time.Millisecond server := newModelFake(t) socket := daemonStartTestWithBot(t, server, bot) terminal := terminalStartTestClient(t, socket) diff --git a/telegram_test.go b/telegram_test.go index 60e9e86..84f41c8 100644 --- a/telegram_test.go +++ b/telegram_test.go @@ -1,6 +1,7 @@ package main import ( + "context" "net/http" "os" "path/filepath" @@ -8,6 +9,7 @@ import ( "strconv" "strings" "testing" + "testing/synctest" "time" "github.com/stretchr/testify/require" @@ -265,7 +267,7 @@ func TestBotTriesAgainAfterFailures(t *testing.T) { }) telegramWaitForSent(t, telegram, "ok") - require.Contains(t, logs.String(), "<4>receive failed retry=1ms: 502") + require.Contains(t, logs.String(), "<4>receive failed retry=1s: 502") require.Contains(t, logs.String(), "<6>receive back after=") require.Contains(t, logs.String(), "<4>send failed retry=1s: 429") @@ -279,30 +281,32 @@ func TestBotTriesAgainAfterFailures(t *testing.T) { } func TestDeliverTriesAgainAfterATimeout(t *testing.T) { - telegram := newTelegramFake(t) - stalls := 1 - telegram.stall = func(call TelegramFakeCall) bool { - if call.method != "sendMessage" || stalls == 0 { - return false - } + synctest.Test(t, func(t *testing.T) { + telegram := newTelegramLocalFake(t) + stalls := 1 + telegram.stall = func(call TelegramFakeCall) bool { + if call.method != "sendMessage" || stalls == 0 { + return false + } - stalls-- + stalls-- - return true - } + return true + } - bot, logs := telegramStartTestBot(t, telegram) - bot.api.sendTimeout = 50 * time.Millisecond - bot.deliver(t.Context(), TelegramPart{ - text: "hello", - }) + bot, logs := telegramStartTestBot(t, telegram) + bot.api.http.Transport = telegram + bot.deliver(t.Context(), TelegramPart{ + text: "hello", + }) - require.Equal(t, []string{"hello"}, telegram.Sent()) - require.Len(t, telegram.Calls(), 2) - require.Contains(t, logs.String(), "<4>send failed retry=1ms: no answer in 50ms\n") - require.Contains(t, logs.String(), "<6>send back after=") - require.Equal(t, 2, strings.Count(logs.String(), "\n")) - require.NotContains(t, logs.String(), telegramTestToken) + require.Equal(t, []string{"hello"}, telegram.Sent()) + require.Len(t, telegram.Calls(), 2) + require.Contains(t, logs.String(), "<4>send failed retry=1s: no answer in 30s\n") + require.Contains(t, logs.String(), "<6>send back after=1s\n") + require.Equal(t, 2, strings.Count(logs.String(), "\n")) + require.NotContains(t, logs.String(), telegramTestToken) + }) } func TestRepliesGoAsRichMessages(t *testing.T) { @@ -408,7 +412,7 @@ func TestRichMessageFailures(t *testing.T) { methods: []string{"sendRichMessage", "sendRichMessage"}, sent: []string{"a **secret** word"}, lines: 2, - logText: "<4>send failed retry=1ms: 500", + logText: "<4>send failed retry=1s: 500", }, { name: "other refusal drops the reply", @@ -639,45 +643,61 @@ func TestBotLogsNoTokenWhenTheNetworkFails(t *testing.T) { daemonStartTestWithBot(t, newModelFake(t), bot) require.Eventually(t, func() bool { - return strings.Contains(logs.String(), "<4>receive failed retry=1ms: no answer: ") + return strings.Contains(logs.String(), "<4>receive failed retry=1s: no answer: ") }, 10*time.Second, time.Millisecond) require.NotContains(t, logs.String(), telegramTestToken) } func TestBotLogsAnOutageOnlyAtItsStartAndEnd(t *testing.T) { - server := newModelFake(t) - server.reply = func(w http.ResponseWriter, _ ModelFakeCall) { - modelWriteEvents(w, modelCompletedEvent([]any{modelMessageItem("ok")}, 100, 0, 5)) - } + synctest.Test(t, func(t *testing.T) { + telegram := newTelegramLocalFake(t) + failures := 5 + telegram.fail = func(call TelegramFakeCall) (int, string) { + if call.method != "getUpdates" || failures == 0 { + return 0, "" + } - telegram := newTelegramFake(t) - failures := 5 - telegram.fail = func(call TelegramFakeCall) (int, string) { - if call.method != "getUpdates" || failures == 0 { - return 0, "" - } + failures-- - failures-- + return http.StatusBadGateway, "Bad Gateway" + } - return http.StatusBadGateway, "Bad Gateway" - } + bot, logs := telegramStartTestBot(t, telegram) + bot.api.http.Transport = telegram + whatsapp, _, _ := whatsappStartTest(t, t.TempDir(), newTestClock()) + loop, _ := newTestIdleTurnLoop(t, whatsapp) - bot, logs := telegramStartTestBot(t, telegram) - daemonStartTestWithBot(t, server, bot) + telegram.Add(TelegramUpdate{ + UpdateID: 1, + Message: telegramOwnerMessage("a secret word"), + }) - telegram.Add(TelegramUpdate{ - UpdateID: 1, - Message: telegramOwnerMessage("a secret word"), + ctx, cancel := context.WithCancel(t.Context()) + defer cancel() + + done := make(chan error, 1) + go func() { + done <- bot.poll(ctx, loop, whatsapp) + }() + + time.Sleep(31 * time.Second) + synctest.Wait() + require.Equal(t, 1, loop.memory.Len()) + + message, err := loop.memory.Message(0) + require.NoError(t, err) + require.Equal(t, "a secret word", message.Text) + + lines := strings.Split(strings.TrimSuffix(logs.String(), "\n"), "\n") + require.Equal(t, []string{ + "<4>receive failed retry=1s: 502 Bad Gateway", + "<6>receive back after=31s", + }, lines) + require.NotContains(t, logs.String(), "secret word") + require.NotContains(t, logs.String(), strconv.FormatInt(telegramTestOwner, 10)) + cancel() + require.NoError(t, <-done) }) - - telegramWaitForSent(t, telegram, "ok") - - lines := strings.Split(strings.TrimSuffix(logs.String(), "\n"), "\n") - require.Len(t, lines, 2) - require.Equal(t, "<4>receive failed retry=1ms: 502 Bad Gateway", lines[0]) - require.Regexp(t, `^<6>receive back after=[0-9]+m?s$`, lines[1]) - require.NotContains(t, logs.String(), "secret word") - require.NotContains(t, logs.String(), strconv.FormatInt(telegramTestOwner, 10)) } func TestSplitMessage(t *testing.T) { @@ -906,15 +926,12 @@ func TestLoadBotConfig(t *testing.T) { } // telegramStartTestBot returns a bot of telegramTestOwner on server, with its -// offset file in a temporary directory and short waits, and the buffer of its -// log. +// offset file in a temporary directory and the buffer of its log. func telegramStartTestBot(t *testing.T, server *TelegramFake) (*TelegramBot, *SyncBuffer) { t.Helper() logs := &SyncBuffer{} bot := newTelegramBot(newTelegramAPI(server.URL, telegramTestToken), telegramTestOwner, filepath.Join(t.TempDir(), telegramOffsetFileName), daemonTestLogger(logs)) - bot.minWait = time.Millisecond - bot.maxWait = 10 * time.Millisecond return bot, logs } diff --git a/turn_loop_helper_test.go b/turn_loop_helper_test.go new file mode 100644 index 0000000..06a6f93 --- /dev/null +++ b/turn_loop_helper_test.go @@ -0,0 +1,31 @@ +package main + +import ( + "path/filepath" + "testing" + "time" + + "github.com/stretchr/testify/require" +) + +// newTestIdleTurnLoop opens a loop with no model calls or turn goroutine. +// Callers can test queued messages and events with synctest fake time. +func newTestIdleTurnLoop(t *testing.T, whatsapp *WhatsApp) (*TurnLoop, *TurnEventRecorder) { + t.Helper() + + agent, err := openAgent(t.TempDir()) + require.NoError(t, err) + + memory, err := openMemory(t.Context(), agent.dir, time.Now, newFakeSummarizer(), discardLog()) + require.NoError(t, err) + t.Cleanup(func() { + require.NoError(t, memory.Close()) + }) + + events := &TurnEventRecorder{} + client := newModelClient("https://example.com", ModelAPIKey(modelTestKey), agent.CacheKey(), filepath.Join(agent.dir, "usage.jsonl")) + loop, err := newTurnLoop(agent, memory, newTestSubagents(t), whatsapp, newTestSchedules(t, time.Now), client, time.Now, discardLog(), events.Add) + require.NoError(t, err) + + return loop, events +} diff --git a/whatsapp.go b/whatsapp.go index be1acc6..9365173 100644 --- a/whatsapp.go +++ b/whatsapp.go @@ -77,10 +77,6 @@ const ( // wait for Run. whatsappRequests = 8 - // whatsappLookupTime is the longest time of a read of the names in the - // session database or of the groups from WhatsApp. - whatsappLookupTime = 10 * time.Second - // whatsappNotLinked is the answer of a WhatsApp tool without a link. whatsappNotLinked = "WhatsApp is not linked. Tell the user to run /whatsapp pair PHONE in the terminal or in Telegram." ) @@ -107,20 +103,19 @@ type WhatsAppClient interface { // the client of a device with its event handler. func openWhatsApp(ctx context.Context, dir string, clock func() time.Time, logger *slog.Logger, newClient func(device *store.Device, handle func(event any) bool) WhatsAppClient) (*WhatsApp, error) { w := &WhatsApp{ - ctx: ctx, - dir: filepath.Join(dir, whatsappDirName), - clock: clock, - logger: logger, - newClient: newClient, - noticeDelay: whatsappNoticeDelay, - requests: make(chan WhatsAppRequest, whatsappRequests), - logouts: make(chan struct{}, 1), - groups: newWhatsAppGroups(logger), + ctx: ctx, + dir: filepath.Join(dir, whatsappDirName), + clock: clock, + logger: logger, + newClient: newClient, + requests: make(chan WhatsAppRequest, whatsappRequests), + logouts: make(chan struct{}, 1), + groups: newWhatsAppGroups(logger), } var err error - w.state, err = openWhatsAppState(filepath.Join(w.dir, whatsappStateFileName), whatsappBatchQuiet, whatsappBatchWindow) + w.state, err = openWhatsAppState(filepath.Join(w.dir, whatsappStateFileName)) if err != nil { return nil, err } @@ -185,18 +180,17 @@ func openWhatsApp(ctx context.Context, dir string, clock func() time.Time, logge // log, and those of private chats hold phone numbers, because the agent needs // them to send. type WhatsApp struct { - ctx context.Context - dir string - clock func() time.Time - logger *slog.Logger - newClient func(device *store.Device, handle func(event any) bool) WhatsAppClient - noticeDelay time.Duration - requests chan WhatsAppRequest - logouts chan struct{} - state *WhatsAppState - groups *WhatsAppGroups - inbox *WhatsAppInbox - routes *Routes + ctx context.Context + dir string + clock func() time.Time + logger *slog.Logger + newClient func(device *store.Device, handle func(event any) bool) WhatsAppClient + requests chan WhatsAppRequest + logouts chan struct{} + state *WhatsAppState + groups *WhatsAppGroups + inbox *WhatsAppInbox + routes *Routes sessionMutex sync.Mutex mutex sync.Mutex @@ -575,7 +569,7 @@ func (w *WhatsApp) startOutage() bool { w.disconnectedAt = w.clock() w.disconnections++ disconnections := w.disconnections - w.timer = time.AfterFunc(w.noticeDelay, func() { + w.timer = time.AfterFunc(whatsappNoticeDelay, func() { w.disconnectedLong(disconnections) }) } @@ -896,10 +890,11 @@ func (w *WhatsApp) linkedBy(client WhatsAppClient) { // handle takes one event of the library and tells if it was handled. A message // goes to the inbox. The names of the groups follow the events of the groups. A // connection starts a catch-up of the inbox, which ends at OfflineSyncCompleted -// or after its catchupLimit. A disconnection keeps the time of its start while -// Run or the library tries again. After noticeDelay of a disconnection, all -// clients get one notice, and they get one more when the connection comes back. -// A disconnection that the library does not try again gives a notice at once. +// or after whatsappCatchupLimit. A disconnection keeps the time of its start +// while Run or the library tries again. After whatsappNoticeDelay of a +// disconnection, all clients get one notice, and they get one more when the +// connection comes back. A disconnection that the library does not try again +// gives a notice at once. // // The log of the daemon gets one line for each change of the connection of the // linked device: connected, a failed first connection, a lost connection, and diff --git a/whatsapp_fake_helper_test.go b/whatsapp_fake_helper_test.go index e936f45..9f80b38 100644 --- a/whatsapp_fake_helper_test.go +++ b/whatsapp_fake_helper_test.go @@ -9,6 +9,7 @@ import ( "slices" "sync" "testing" + "testing/synctest" "time" "github.com/stretchr/testify/require" @@ -282,6 +283,34 @@ type WhatsAppFakeSend struct { id string } +// WhatsAppRetryTest runs WhatsApp without a model or network client call. All +// timers and channels belong to its synctest bubble, so backoff uses fake time. +type WhatsAppRetryTest struct { + whatsapp *WhatsApp + fake *WhatsAppFake + logs *SyncBuffer + loop *TurnLoop + events *TurnEventRecorder + cancel context.CancelFunc + done chan error +} + +func (r *WhatsAppRetryTest) start(t *testing.T) { + t.Helper() + + ctx, cancel := context.WithCancel(t.Context()) + r.cancel = cancel + r.done = make(chan error, 1) + go func() { + r.done <- r.whatsapp.Run(ctx, r.loop, r.events.Add) + }() + + t.Cleanup(func() { + cancel() + synctest.Wait() + }) +} + // whatsappStartTest opens the WhatsApp of the data directory dir with a fake // client and the test clock, and closes it at the end of the test. The daemon // log goes to the returned buffer. @@ -385,7 +414,7 @@ func whatsappTestLogTexts(t *testing.T, memory *Memory) []string { } // whatsappStartTestLinked opens a WhatsApp with a linked device in a temporary -// directory, with a quiet time of 50 ms for its batches. +// directory. func whatsappStartTestLinked(t *testing.T) (*WhatsApp, *WhatsAppFake) { t.Helper() @@ -393,7 +422,6 @@ func whatsappStartTestLinked(t *testing.T) (*WhatsApp, *WhatsAppFake) { whatsappTestSession(t, dir, true) whatsapp, fake, _ := whatsappStartTest(t, dir, newTestClock()) - whatsapp.state.quiet = 50 * time.Millisecond return whatsapp, fake } @@ -421,3 +449,22 @@ func whatsappTestContacts(t *testing.T, whatsapp *WhatsApp) { require.NoError(t, err) require.NoError(t, contacts.PutContactName(t.Context(), whatsappTestOwn, "Me", "Me Souza")) } + +func whatsappRetryTest(t *testing.T) *WhatsAppRetryTest { + t.Helper() + + dir := t.TempDir() + whatsappTestSession(t, dir, true) + + whatsapp, fake, logs := whatsappStartTest(t, dir, newTestClock()) + whatsapp.clock = time.Now + loop, events := newTestIdleTurnLoop(t, whatsapp) + + return &WhatsAppRetryTest{ + whatsapp: whatsapp, + fake: fake, + logs: logs, + loop: loop, + events: events, + } +} diff --git a/whatsapp_groups.go b/whatsapp_groups.go index ec5c892..4d6c7e5 100644 --- a/whatsapp_groups.go +++ b/whatsapp_groups.go @@ -9,6 +9,10 @@ import ( ) const ( + // whatsappGroupsLookupTime is the longest time of a read of group + // names. + whatsappGroupsLookupTime = 10 * time.Second + // whatsappGroupsRefresh is the shortest time between two reads of the // groups from WhatsApp for a group with no name. whatsappGroupsRefresh = time.Minute @@ -77,7 +81,7 @@ func (g *WhatsAppGroups) read(ctx context.Context, link WhatsAppLink) bool { return false } - ctx, cancel := context.WithTimeout(ctx, whatsappLookupTime) + ctx, cancel := context.WithTimeout(ctx, whatsappGroupsLookupTime) defer cancel() groups, err := link.client.GetJoinedGroups(ctx) diff --git a/whatsapp_image.go b/whatsapp_image.go index 4277b0a..d4b88d8 100644 --- a/whatsapp_image.go +++ b/whatsapp_image.go @@ -4,15 +4,11 @@ import ( "context" "errors" "fmt" - "time" "go.mau.fi/whatsmeow" "go.mau.fi/whatsmeow/proto/waE2E" ) -// whatsappImageTime is the longest time of an image download. -const whatsappImageTime = 2 * time.Minute - // whatsappImage downloads only supported images and stores them before the // event can be acknowledged. Both the declared size and the downloaded bytes // must fit the image store. Decoded bytes determine the stored format. diff --git a/whatsapp_image_test.go b/whatsapp_image_test.go index 919d656..91af70a 100644 --- a/whatsapp_image_test.go +++ b/whatsapp_image_test.go @@ -80,28 +80,28 @@ func TestWhatsAppImagesAreDurableBeforeAcknowledgement(t *testing.T) { } func TestWhatsAppImageDownloadCanOutlastNameLookup(t *testing.T) { - whatsapp, fake := whatsappStartTestLinked(t) - whatsapp.client = fake - whatsapp.inbox.lookupTime = 10 * time.Millisecond - whatsapp.inbox.imageTime = time.Second - fake.downloadWait = 50 * time.Millisecond - data := imageTestPNG(t, 3, 2) - fake.downloadData = data - - require.True(t, whatsapp.handle(whatsappImageTestEvent(whatsappTestAna, "Slow photo."))) - - restored := whatsappTestState(t, whatsapp.state.path) - require.NoError(t, restored.CloseAll()) - require.Len(t, restored.Closed(), 1) - - text := restored.Closed()[0].Messages[0].Text - names := imageFiles(text) - require.Len(t, names, 1) - require.Equal(t, imageText(names, "Slow photo."), text) - - saved, err := whatsapp.inbox.images.Read(names[0]) - require.NoError(t, err) - require.Equal(t, data, saved) + synctest.Test(t, func(t *testing.T) { + whatsapp, fake := whatsappStartTestLinked(t) + whatsapp.client = fake + fake.downloadWait = whatsappLookupTime + time.Second + data := imageTestPNG(t, 3, 2) + fake.downloadData = data + + require.True(t, whatsapp.handle(whatsappImageTestEvent(whatsappTestAna, "Slow photo."))) + + restored := whatsappTestState(t, whatsapp.state.path) + require.NoError(t, restored.CloseAll()) + require.Len(t, restored.Closed(), 1) + + text := restored.Closed()[0].Messages[0].Text + names := imageFiles(text) + require.Len(t, names, 1) + require.Equal(t, imageText(names, "Slow photo."), text) + + saved, err := whatsapp.inbox.images.Read(names[0]) + require.NoError(t, err) + require.Equal(t, data, saved) + }) } func TestWhatsAppRunCancellationEndsAnImageDownload(t *testing.T) { diff --git a/whatsapp_inbox.go b/whatsapp_inbox.go index 06f1c55..392593f 100644 --- a/whatsapp_inbox.go +++ b/whatsapp_inbox.go @@ -16,6 +16,13 @@ import ( ) const ( + // whatsappLookupTime is the longest time of a read of names in the + // session. + whatsappLookupTime = 10 * time.Second + + // whatsappImageTime is the longest time of an image download. + whatsappImageTime = 2 * time.Minute + // whatsappSource is the source of the work events of the batches. whatsappSource = "whatsapp" @@ -29,16 +36,13 @@ const ( // holds shared images, and logger gets image failures. func newWhatsAppInbox(state *WhatsAppState, clock func() time.Time, groups *WhatsAppGroups, images *ImageStore, logger *slog.Logger) *WhatsAppInbox { return &WhatsAppInbox{ - state: state, - clock: clock, - groups: groups, - images: images, - logger: logger, - lookupTime: whatsappLookupTime, - imageTime: whatsappImageTime, - catchupLimit: whatsappCatchupLimit, - wake: make(chan struct{}, 1), - failures: make(chan error, 1), + state: state, + clock: clock, + groups: groups, + images: images, + logger: logger, + wake: make(chan struct{}, 1), + failures: make(chan error, 1), } } @@ -48,16 +52,16 @@ func newWhatsAppInbox(state *WhatsAppState, clock func() time.Time, groups *What // // After each connection, the servers send the messages that came while the // daemon was away, and then OfflineSyncCompleted. Until then, or until -// catchupLimit passes, the closed batches wait in the state file. At the end of -// the catch-up, deliver closes all batches and logs them. It wakes the root -// once for its batches, so their catch-up gives one turn, not one for each -// chat. The batches of the state file at the start wait for the first catch-up -// in the same way. The event handler acknowledges a message only after the -// state file holds it, so the servers send again a message that a crash lost. A -// batch leaves the state file only after its work message is durable in the log -// or agent queue. A crash between the two can repeat an unrouted batch once. A -// routed batch enters the durable agent queue, which prevents a second log -// entry and wakes only the child. A missing child uses the root. +// whatsappCatchupLimit passes, the closed batches wait in the state file. At +// the end of the catch-up, deliver closes all batches and logs them. It wakes +// the root once for its batches, so their catch-up gives one turn, not one for +// each chat. The batches of the state file at the start wait for the first +// catch-up in the same way. The event handler acknowledges a message only after +// the state file holds it, so the servers send again a message that a crash +// lost. A batch leaves the state file only after its work message is durable in +// the log or agent queue. A crash between the two can repeat an unrouted batch +// once. A routed batch enters the durable agent queue, which prevents a second +// log entry and wakes only the child. A missing child uses the root. // // The state file is the one of WhatsApp, which keeps the send records in it // too. groups gives the names of the groups. The mutex protects the state of @@ -65,16 +69,13 @@ func newWhatsAppInbox(state *WhatsAppState, clock func() time.Time, groups *What // goroutines. wake tells Run that a batch can be due or that a catch-up ended, // and failures gives Run the error of a write of the state file. type WhatsAppInbox struct { - state *WhatsAppState - clock func() time.Time - groups *WhatsAppGroups - images *ImageStore - logger *slog.Logger - lookupTime time.Duration - imageTime time.Duration - catchupLimit time.Duration - wake chan struct{} - failures chan error + state *WhatsAppState + clock func() time.Time + groups *WhatsAppGroups + images *ImageStore + logger *slog.Logger + wake chan struct{} + failures chan error mutex sync.Mutex catchingUp bool @@ -206,7 +207,7 @@ func (i *WhatsAppInbox) waitForCatchup() { } // startCatchup starts the catch-up of a new connection, which ends at -// endCatchup or after catchupLimit. +// endCatchup or after whatsappCatchupLimit. func (i *WhatsAppInbox) startCatchup() { i.mutex.Lock() defer i.mutex.Unlock() @@ -218,7 +219,7 @@ func (i *WhatsAppInbox) startCatchup() { i.catchupTimer.Stop() } - i.catchupTimer = time.AfterFunc(i.catchupLimit, func() { + i.catchupTimer = time.AfterFunc(whatsappCatchupLimit, func() { i.caughtUpBy(catchups) }) } @@ -229,7 +230,7 @@ func (i *WhatsAppInbox) startCatchup() { // daemon. An image that cannot be downloaded or stored gets a text marker. A // message that the batches do not take is handled with nothing to do. func (i *WhatsAppInbox) receive(ctx context.Context, link WhatsAppLink, event *events.Message) bool { - lookupCtx, cancel := context.WithTimeout(ctx, i.lookupTime) + lookupCtx, cancel := context.WithTimeout(ctx, whatsappLookupTime) defer cancel() chat, message, found := whatsappMessage(lookupCtx, link.device, link.own, event) @@ -238,7 +239,7 @@ func (i *WhatsAppInbox) receive(ctx context.Context, link WhatsAppLink, event *e } if event.Message.GetImageMessage() != nil { - ctx, cancel := context.WithTimeout(ctx, i.imageTime) + ctx, cancel := context.WithTimeout(ctx, whatsappImageTime) defer cancel() message.Text = i.imageText(ctx, link, event, message.Text) diff --git a/whatsapp_inbox_test.go b/whatsapp_inbox_test.go index cea65fe..a0a0f9d 100644 --- a/whatsapp_inbox_test.go +++ b/whatsapp_inbox_test.go @@ -3,6 +3,7 @@ package main import ( "net/http" "testing" + "testing/synctest" "time" "github.com/stretchr/testify/require" @@ -115,6 +116,9 @@ func TestWhatsAppGroupTurns(t *testing.T) { event.Info.PushName = "Bruno Lima" tc.message(event) require.True(t, fake.Emit(event)) + require.NoError(t, whatsapp.state.CloseAll()) + + whatsapp.inbox.wake <- struct{}{} text := "[WhatsApp] Family, group 120363000000000001@g.us, Thursday 2026-01-01\n" + tc.line if tc.turn { @@ -133,6 +137,27 @@ func TestWhatsAppGroupTurns(t *testing.T) { } } +func TestWhatsAppRunClosesABatchAfterQuietTime(t *testing.T) { + synctest.Test(t, func(t *testing.T) { + test := whatsappRetryTest(t) + test.start(t) + synctest.Wait() + require.True(t, test.fake.Emit(&events.Connected{})) + require.True(t, test.fake.Emit(&events.OfflineSyncCompleted{})) + synctest.Wait() + require.True(t, test.fake.Emit(whatsappTestMessage(whatsappTestAna, whatsappTestOwn, "m1", "On my way.", time.Now()))) + synctest.Wait() + time.Sleep(whatsappBatchQuiet - time.Nanosecond) + synctest.Wait() + require.Zero(t, test.loop.memory.Len()) + + time.Sleep(time.Nanosecond) + synctest.Wait() + require.Equal(t, 1, test.loop.memory.Len()) + require.Contains(t, test.events.Last("work"), "You: On my way.") + }) +} + func TestWhatsAppSkipsExcludedMessages(t *testing.T) { at := time.Date(2026, 1, 1, 12, 0, 0, 0, time.UTC) testCases := []struct { @@ -235,17 +260,20 @@ func TestWhatsAppCatchupGivesOneTurn(t *testing.T) { } func TestWhatsAppCatchupEndsAfterItsLimit(t *testing.T) { - whatsapp, fake := whatsappStartTestLinked(t) - whatsapp.inbox.catchupLimit = 100 * time.Millisecond - socket, memory := daemonStartTestWithWhatsApp(t, newModelFake(t), nil, whatsapp) - client := terminalStartTestClient(t, socket) - client.WaitFor(t, "[connected to OneChat]") - <-fake.connects + synctest.Test(t, func(t *testing.T) { + test := whatsappRetryTest(t) + test.whatsapp.client = test.fake + require.True(t, test.whatsapp.handle(&events.Connected{})) + require.True(t, test.whatsapp.handle(whatsappTestMessage(whatsappTestAna, whatsappTestOwn, "m1", "On my way.", time.Date(2026, 1, 1, 12, 0, 0, 0, time.UTC)))) + require.NoError(t, test.whatsapp.inbox.deliver(t.Context(), test.loop, test.whatsapp.link(), false)) + require.Zero(t, test.loop.memory.Len()) - require.True(t, fake.Emit(&events.Connected{})) - require.True(t, fake.Emit(whatsappTestMessage(whatsappTestAna, whatsappTestOwn, "m1", "On my way.", time.Date(2026, 1, 1, 12, 0, 0, 0, time.UTC)))) - client.WaitFor(t, "12:00 You: On my way.") - require.Len(t, whatsappTestLogTexts(t, memory), 1) + time.Sleep(whatsappCatchupLimit) + synctest.Wait() + require.NoError(t, test.whatsapp.inbox.deliver(t.Context(), test.loop, test.whatsapp.link(), false)) + require.Equal(t, 1, test.loop.memory.Len()) + require.Contains(t, test.events.Last("work"), "12:00 You: On my way.") + }) } func TestWhatsAppStateOfTheStateFileWaitForTheCatchup(t *testing.T) { diff --git a/whatsapp_route_test.go b/whatsapp_route_test.go index 9c855c9..c6dbf44 100644 --- a/whatsapp_route_test.go +++ b/whatsapp_route_test.go @@ -390,7 +390,7 @@ func TestRoutedWhatsAppQueueFailureKeepsTheInboxBatch(t *testing.T) { func whatsappRouteTestInbox(t *testing.T, dir string) *WhatsAppInbox { t.Helper() - state, err := openWhatsAppState(filepath.Join(dir, whatsappDirName, whatsappStateFileName), whatsappBatchQuiet, whatsappBatchWindow) + state, err := openWhatsAppState(filepath.Join(dir, whatsappDirName, whatsappStateFileName)) require.NoError(t, err) return newWhatsAppInbox(state, newTestClock().Now, newWhatsAppGroups(discardLog()), &ImageStore{dir: filepath.Join(dir, imageDirName)}, discardLog()) diff --git a/whatsapp_send_test.go b/whatsapp_send_test.go index 469912b..480e7db 100644 --- a/whatsapp_send_test.go +++ b/whatsapp_send_test.go @@ -39,7 +39,7 @@ func TestWhatsAppSendSendsOnceForEachID(t *testing.T) { }, }, fake.Sent()) - again, err := openWhatsAppState(filepath.Join(whatsapp.dir, whatsappStateFileName), whatsappBatchQuiet, whatsappBatchWindow) + again, err := openWhatsAppState(filepath.Join(whatsapp.dir, whatsappStateFileName)) require.NoError(t, err) record, found := again.Send("keys") diff --git a/whatsapp_state.go b/whatsapp_state.go index c09423d..41bd2f0 100644 --- a/whatsapp_state.go +++ b/whatsapp_state.go @@ -32,13 +32,11 @@ const ( ) // openWhatsAppState reads the batches and the send records of the state file at -// path. A missing file holds none. A batch closes after quiet with no message, -// or after window at the latest. -func openWhatsAppState(path string, quiet, window time.Duration) (*WhatsAppState, error) { +// path. A missing file holds none. A batch closes after whatsappBatchQuiet with +// no message, or after whatsappBatchWindow at the latest. +func openWhatsAppState(path string) (*WhatsAppState, error) { s := &WhatsAppState{ - path: path, - quiet: quiet, - window: window, + path: path, } data, err := os.ReadFile(path) @@ -61,11 +59,12 @@ func openWhatsAppState(path string, quiet, window time.Duration) (*WhatsAppState // WhatsAppState holds the state file: the send records, and the batches of the // WhatsApp messages that are not in the log yet, open and closed, in the order // of their first message. The messages of one chat form one batch while the -// times of the messages are at most quiet apart, at most window after the -// first, and the batch has fewer than whatsappBatchMessages messages. The times -// are the WhatsApp times, so a catch-up after a stop gives the same batches as -// live messages. A batch also closes when quiet passes after the arrival of its -// last message, or window after the arrival of its first message, in real time. +// times of the messages are at most whatsappBatchQuiet apart, at most +// whatsappBatchWindow after the first, and the batch has fewer than +// whatsappBatchMessages messages. The times are the WhatsApp times, so a +// catch-up after a stop gives the same batches as live messages. A batch also +// closes when whatsappBatchQuiet passes after the arrival of its last message, +// or whatsappBatchWindow after the arrival of its first message, in real time. // // A send record keeps the result of a send by its ID, so that a repeated call // of the send tool does not send again. The state file keeps the last @@ -76,9 +75,7 @@ func openWhatsAppState(path string, quiet, window time.Duration) (*WhatsAppState // library adds messages while Run closes and removes batches, and the turns // send. type WhatsAppState struct { - path string - quiet time.Duration - window time.Duration + path string mutex sync.Mutex file WhatsAppStateFile @@ -102,7 +99,7 @@ func (s *WhatsAppState) Add(chat WhatsAppChat, message WhatsAppMessage, arrival return false, nil } - if message.Time.Sub(batch.Messages[len(batch.Messages)-1].Time) > s.quiet || message.Time.Sub(batch.Messages[0].Time) > s.window { + if message.Time.Sub(batch.Messages[len(batch.Messages)-1].Time) > whatsappBatchQuiet || message.Time.Sub(batch.Messages[0].Time) > whatsappBatchWindow { batch.Closed = true closed = true index = -1 @@ -261,8 +258,8 @@ func (s *WhatsAppState) open(chat string) int { // deadline returns when batch closes in real time. func (s *WhatsAppState) deadline(batch WhatsAppBatch) time.Time { - quiet := batch.Last.Add(s.quiet) - window := batch.First.Add(s.window) + quiet := batch.Last.Add(whatsappBatchQuiet) + window := batch.First.Add(whatsappBatchWindow) if window.Before(quiet) { return window } diff --git a/whatsapp_state_test.go b/whatsapp_state_test.go index 2631b4d..1cfedee 100644 --- a/whatsapp_state_test.go +++ b/whatsapp_state_test.go @@ -134,7 +134,7 @@ func TestWhatsAppStateRefusesABrokenFile(t *testing.T) { path := filepath.Join(t.TempDir(), whatsappStateFileName) require.NoError(t, os.WriteFile(path, []byte("{"), 0o600)) - _, err := openWhatsAppState(path, whatsappBatchQuiet, whatsappBatchWindow) + _, err := openWhatsAppState(path) require.ErrorContains(t, err, path) } @@ -159,7 +159,7 @@ func TestWhatsAppStateKeepsTheLastSendRecords(t *testing.T) { func whatsappTestState(t *testing.T, path string) *WhatsAppState { t.Helper() - batches, err := openWhatsAppState(path, whatsappBatchQuiet, whatsappBatchWindow) + batches, err := openWhatsAppState(path) require.NoError(t, err) return batches diff --git a/whatsapp_test.go b/whatsapp_test.go index 96ec2df..f098c90 100644 --- a/whatsapp_test.go +++ b/whatsapp_test.go @@ -1,7 +1,6 @@ package main import ( - "context" "errors" "os" "path/filepath" @@ -606,34 +605,33 @@ func TestWhatsAppLogoutFromThePhone(t *testing.T) { } func TestWhatsAppLongDisconnectionGivesOneNotice(t *testing.T) { - dir := t.TempDir() - whatsappTestSession(t, dir, true) - - clock := newTestClock() - whatsapp, fake, _ := whatsappStartTest(t, dir, clock) - whatsapp.noticeDelay = 300 * time.Millisecond - socket, _ := daemonStartTestWithWhatsApp(t, newModelFake(t), nil, whatsapp) - client := terminalStartTestClient(t, socket) - client.WaitFor(t, "[connected to OneChat]") - <-fake.connects - - require.True(t, fake.Emit(&events.Connected{})) - require.True(t, fake.Emit(&events.Disconnected{})) - require.True(t, fake.Emit(&events.Connected{})) - time.Sleep(400 * time.Millisecond) - require.Equal(t, "[connected to OneChat]\n", client.Output()) + synctest.Test(t, func(t *testing.T) { + test := whatsappRetryTest(t) + test.start(t) + synctest.Wait() - clock.now = clock.now.Add(time.Hour) + require.True(t, test.fake.Emit(&events.Connected{})) + require.True(t, test.fake.Emit(&events.Disconnected{})) + require.True(t, test.fake.Emit(&events.Connected{})) + time.Sleep(whatsappNoticeDelay + time.Second) + synctest.Wait() + require.Zero(t, test.events.Count("notice")) - require.True(t, fake.Emit(&events.Disconnected{})) - require.True(t, fake.Emit(&events.Disconnected{})) - client.WaitFor(t, "WhatsApp is disconnected since 2026-01-01 13:00. OneChat tries again.\n") - require.True(t, fake.Emit(&events.Disconnected{})) - time.Sleep(400 * time.Millisecond) + require.True(t, test.fake.Emit(&events.Disconnected{})) + require.True(t, test.fake.Emit(&events.Disconnected{})) + time.Sleep(whatsappNoticeDelay) + synctest.Wait() + require.Equal(t, 1, test.events.Count("notice")) + require.Contains(t, test.events.Last("notice"), "OneChat tries again.") + require.True(t, test.fake.Emit(&events.Disconnected{})) + time.Sleep(whatsappNoticeDelay) + synctest.Wait() + require.Equal(t, 1, test.events.Count("notice")) - require.True(t, fake.Emit(&events.Connected{})) - client.WaitFor(t, "WhatsApp is connected again.\n") - require.Equal(t, 1, strings.Count(client.Output(), "WhatsApp is disconnected since")) + require.True(t, test.fake.Emit(&events.Connected{})) + require.Equal(t, 2, test.events.Count("notice")) + require.Equal(t, "WhatsApp is connected again.", test.events.Last("notice")) + }) } func TestWhatsAppDisconnectionWithNoRetryGivesANoticeAtOnce(t *testing.T) { @@ -755,62 +753,3 @@ func TestSubagentHasNoWhatsAppTools(t *testing.T) { require.Equal(t, `error: unknown tool "whatsapp_find"`, box.run(t.Context(), "whatsapp_find", `{"name":"ana"}`)) require.Equal(t, `error: unknown tool "whatsapp_send"`, box.run(t.Context(), "whatsapp_send", `{"id":"a","chat":"15550002222@s.whatsapp.net","text":"hi"}`)) } - -// WhatsAppRetryTest runs WhatsApp without a model or network client call. All -// timers and channels belong to its synctest bubble, so backoff uses fake time. -type WhatsAppRetryTest struct { - whatsapp *WhatsApp - fake *WhatsAppFake - logs *SyncBuffer - loop *TurnLoop - events *TurnEventRecorder - cancel context.CancelFunc - done chan error -} - -func (r *WhatsAppRetryTest) start(t *testing.T) { - t.Helper() - - ctx, cancel := context.WithCancel(t.Context()) - r.cancel = cancel - r.done = make(chan error, 1) - go func() { - r.done <- r.whatsapp.Run(ctx, r.loop, r.events.Add) - }() - - t.Cleanup(func() { - cancel() - synctest.Wait() - }) -} - -func whatsappRetryTest(t *testing.T) *WhatsAppRetryTest { - t.Helper() - - dir := t.TempDir() - whatsappTestSession(t, dir, true) - - whatsapp, fake, logs := whatsappStartTest(t, dir, newTestClock()) - whatsapp.clock = time.Now - agent, err := openAgent(t.TempDir()) - require.NoError(t, err) - - memory, err := openMemory(t.Context(), agent.dir, time.Now, newFakeSummarizer(), discardLog()) - require.NoError(t, err) - t.Cleanup(func() { - require.NoError(t, memory.Close()) - }) - - events := &TurnEventRecorder{} - client := newModelClient("https://example.com", ModelAPIKey(modelTestKey), agent.CacheKey(), filepath.Join(agent.dir, "usage.jsonl")) - loop, err := newTurnLoop(agent, memory, newTestSubagents(t), whatsapp, newTestSchedules(t, time.Now), client, time.Now, discardLog(), events.Add) - require.NoError(t, err) - - return &WhatsAppRetryTest{ - whatsapp: whatsapp, - fake: fake, - logs: logs, - loop: loop, - events: events, - } -} -- 2.53.0 From add3cf6416e33c24331fd1e90e99284f44d2aae2 Mon Sep 17 00:00:00 2001 From: "Anderson A. Giacomolli" Date: Tue, 10 Nov 2026 15:09:23 -0300 Subject: [PATCH 10/16] Name timeouts for the operations they limit Each operation limit ends in Timeout and names its operation. Telegram file lookup, image download, and typing have separate names from message sends. All wait values stay the same, including the server wait for updates and the memory build wait. --- build_run.go | 5 +++- schedule.go | 17 +++++++------ schedule_test.go | 2 +- telegram_api.go | 55 ++++++++++++++++++++++++------------------ telegram_api_test.go | 2 +- tools.go | 6 ++--- whatsapp.go | 26 ++++++++++---------- whatsapp_groups.go | 6 ++--- whatsapp_image_test.go | 2 +- whatsapp_inbox.go | 27 +++++++++++---------- whatsapp_inbox_test.go | 4 +-- whatsapp_send.go | 6 ++--- whatsapp_state.go | 28 +++++++++++---------- whatsapp_test.go | 10 ++++---- 14 files changed, 106 insertions(+), 90 deletions(-) diff --git a/build_run.go b/build_run.go index 9c27ff2..cc12cf2 100644 --- a/build_run.go +++ b/build_run.go @@ -19,6 +19,9 @@ import ( // buildLineBytes is the most bytes of one line in a build input or usage log. const buildLineBytes = 1 << 20 +// buildWaitTimeout is the longest wait for pending memory builds. +const buildWaitTimeout = time.Second + // buildQuotaLimit is the largest increase in used quota points of a run. const buildQuotaLimit = 15.0 @@ -113,7 +116,7 @@ func (r *BuildRun) check() string { // waitBuilds waits up to one second without leaving a child context open. func (r *BuildRun) waitBuilds(ctx context.Context) error { - ctx, cancel := context.WithTimeout(ctx, time.Second) + ctx, cancel := context.WithTimeout(ctx, buildWaitTimeout) defer cancel() return r.memory.Wait(ctx) diff --git a/schedule.go b/schedule.go index a1eba1c..28a5a76 100644 --- a/schedule.go +++ b/schedule.go @@ -33,10 +33,10 @@ const ( // scheduleSource is the source of the work events of the runs. scheduleSource = "schedule" - // scheduleCheck is the longest wait of Run before it looks at the times - // again, so that it sees a suspend of the machine or a change of the - // clock. - scheduleCheck = time.Minute + // scheduleClockWaitTimeout is the longest wait of Run before it looks + // at the times again, so that it sees a suspend of the machine or a + // change of the clock. + scheduleClockWaitTimeout = time.Minute // scheduleLate is how long after its time a run is late. scheduleLate = time.Minute @@ -124,9 +124,10 @@ type Schedules struct { } // Run starts the runs of the due schedules in l until ctx ends. It waits until -// the first due time, at most scheduleCheck at a time, and also wakes at a set -// or a remove. So a schedule that came due while the daemon was stopped runs at -// the start. It returns the error of a write of the log or of the file. +// the first due time, at most scheduleClockWaitTimeout at a time, and also +// wakes at a set or a remove. So a schedule that came due while the daemon was +// stopped runs at the start. It returns the error of a write of the log or of +// the file. func (s *Schedules) Run(ctx context.Context, l *TurnLoop) error { for { err := s.deliver(ctx, l) @@ -141,7 +142,7 @@ func (s *Schedules) Run(ctx context.Context, l *TurnLoop) error { wait, found := s.Wait(s.clock()) if found { - timer = time.NewTimer(min(wait, scheduleCheck)) + timer = time.NewTimer(min(wait, scheduleClockWaitTimeout)) timeout = timer.C } diff --git a/schedule_test.go b/schedule_test.go index 5743a22..cb839d5 100644 --- a/schedule_test.go +++ b/schedule_test.go @@ -336,7 +336,7 @@ func TestSchedulesCheckTheClockAfterItsWaitLimit(t *testing.T) { synctest.Wait() require.Zero(t, loop.memory.Len()) clock.Set(clock.Now().Add(5 * time.Minute)) - time.Sleep(scheduleCheck) + time.Sleep(scheduleClockWaitTimeout) synctest.Wait() require.Equal(t, 1, loop.memory.Len()) require.Contains(t, events.Last("work"), "Check the clock.") diff --git a/telegram_api.go b/telegram_api.go index a04b2fa..8f78387 100644 --- a/telegram_api.go +++ b/telegram_api.go @@ -15,25 +15,33 @@ import ( ) const ( - // telegramPollSeconds is how long one getUpdates call waits for an - // update. - telegramPollSeconds = 50 + // telegramUpdateWaitTimeout is how long one getUpdates call waits for + // an update. + telegramUpdateWaitTimeout = 50 * time.Second - // telegramPollSlack is how much longer than telegramPollSeconds a + // telegramPollSlack is how much longer than telegramUpdateWaitTimeout a // getUpdates call can take before the bot gives it up. telegramPollSlack = 20 * time.Second // telegramPollTimeout is how long one getUpdates call can take. - telegramPollTimeout = telegramPollSeconds*time.Second + telegramPollSlack - - // telegramSendTimeout is how long a sendMessage, sendRichMessage, or - // sendChatAction call can take. A connection that stops without a - // close, such as after a network change, then fails the call, and the - // sender tries again, instead of a wait until TCP gives up. The client - // has no timeout of its own, because the long poll of getUpdates needs - // a longer limit. + telegramPollTimeout = telegramUpdateWaitTimeout + telegramPollSlack + + // telegramSendTimeout is how long a sendMessage or sendRichMessage call + // can take. A connection that stops without a close, such as after a + // network change, then fails the call, and the sender tries again, + // instead of a wait until TCP gives up. The client has no timeout of + // its own, because the long poll of getUpdates needs a longer limit. telegramSendTimeout = 30 * time.Second + // telegramFileLookupTimeout is the longest wait for image file details. + telegramFileLookupTimeout = 30 * time.Second + + // telegramImageDownloadTimeout is the longest wait for image bytes. + telegramImageDownloadTimeout = 30 * time.Second + + // telegramTypingTimeout is the longest wait to show the typing action. + telegramTypingTimeout = 30 * time.Second + // telegramAnswerBytes is the most bytes of one answer of the Bot API. telegramAnswerBytes = 16 << 20 ) @@ -84,15 +92,15 @@ type TelegramAPI struct { http *http.Client } -// GetUpdates waits for the updates from offset on, up to telegramPollSeconds, -// and returns them. Telegram forgets each update before offset. Only new -// messages are asked for. +// GetUpdates waits for the updates from offset on, up to +// telegramUpdateWaitTimeout, and returns them. Telegram forgets each update +// before offset. Only new messages are asked for. func (b *TelegramAPI) GetUpdates(ctx context.Context, offset int64) ([]TelegramUpdate, error) { var updates []TelegramUpdate err := b.call(ctx, telegramPollTimeout, "getUpdates", map[string]any{ "offset": offset, - "timeout": telegramPollSeconds, + "timeout": int(telegramUpdateWaitTimeout / time.Second), "allowed_updates": []string{"message"}, }, &updates) @@ -107,7 +115,7 @@ func (b *TelegramAPI) GetImage(ctx context.Context, id string) (data []byte, err Size int64 `json:"file_size"` } - err = b.call(ctx, telegramSendTimeout, "getFile", map[string]any{"file_id": id}, &file) + err = b.call(ctx, telegramFileLookupTimeout, "getFile", map[string]any{"file_id": id}, &file) if err != nil { return nil, fmt.Errorf("get image file: %w", err) } @@ -120,7 +128,7 @@ func (b *TelegramAPI) GetImage(ctx context.Context, id string) (data []byte, err return nil, errors.New("invalid image file path") } - ctx, cancel := context.WithTimeout(ctx, telegramSendTimeout) + ctx, cancel := context.WithTimeout(ctx, telegramImageDownloadTimeout) defer cancel() request, err := http.NewRequestWithContext(ctx, http.MethodGet, b.base+"/file/bot"+b.token+"/"+file.Path, nil) @@ -130,13 +138,13 @@ func (b *TelegramAPI) GetImage(ctx context.Context, id string) (data []byte, err response, err := b.http.Do(request) if err != nil { - return nil, b.networkError("image download failed", err, telegramSendTimeout) + return nil, b.networkError("image download failed", err, telegramImageDownloadTimeout) } defer func() { closeErr := response.Body.Close() if closeErr != nil { - err = errors.Join(err, b.networkError("close image download", closeErr, telegramSendTimeout)) + err = errors.Join(err, b.networkError("close image download", closeErr, telegramImageDownloadTimeout)) } }() @@ -149,7 +157,7 @@ func (b *TelegramAPI) GetImage(ctx context.Context, id string) (data []byte, err data, err = io.ReadAll(io.LimitReader(response.Body, imageMaxBytes+1)) if err != nil { - return nil, b.networkError("image cut off", err, telegramSendTimeout) + return nil, b.networkError("image cut off", err, telegramImageDownloadTimeout) } if len(data) > imageMaxBytes { @@ -189,9 +197,10 @@ func (b *TelegramAPI) SendRichMessage(ctx context.Context, chat int64, markdown } // SendTyping shows the typing action in chat for about five seconds. A call -// that takes longer than telegramSendTimeout fails with a TelegramNetworkError. +// that takes longer than telegramTypingTimeout fails with a +// TelegramNetworkError. func (b *TelegramAPI) SendTyping(ctx context.Context, chat int64) error { - return b.call(ctx, telegramSendTimeout, "sendChatAction", map[string]any{ + return b.call(ctx, telegramTypingTimeout, "sendChatAction", map[string]any{ "chat_id": chat, "action": "typing", }, nil) diff --git a/telegram_api_test.go b/telegram_api_test.go index 6fa4cba..ca11274 100644 --- a/telegram_api_test.go +++ b/telegram_api_test.go @@ -29,7 +29,7 @@ func TestGetUpdatesAsksForMessagesFromTheOffset(t *testing.T) { require.Equal(t, "getUpdates", call.method) require.Equal(t, map[string]any{ "offset": float64(7), - "timeout": float64(telegramPollSeconds), + "timeout": float64(telegramUpdateWaitTimeout / time.Second), "allowed_updates": []any{"message"}, }, call.params) } diff --git a/tools.go b/tools.go index 96cbb97..bbd34cf 100644 --- a/tools.go +++ b/tools.go @@ -27,9 +27,9 @@ const ( // output, enough for toolClipCharacters/2 characters of 4 bytes. toolClipBytes = 4 * toolClipCharacters / 2 - // toolShellWaitDelay is how long a command can keep its output open + // toolShellOutputTimeout is how long a command can keep its output open // after it ends or after its kill. - toolShellWaitDelay = 2 * time.Second + toolShellOutputTimeout = 2 * time.Second ) // ToolLog is the log that the tools write their calls and results to. @@ -379,7 +379,7 @@ func (b *Toolbox) shell(ctx context.Context, command string) string { cmd.Cancel = func() error { return syscall.Kill(-cmd.Process.Pid, syscall.SIGKILL) } - cmd.WaitDelay = toolShellWaitDelay + cmd.WaitDelay = toolShellOutputTimeout output := &ToolHeadTail{} cmd.Stdout = output diff --git a/whatsapp.go b/whatsapp.go index 9365173..97b1e85 100644 --- a/whatsapp.go +++ b/whatsapp.go @@ -54,17 +54,17 @@ const ( // whatsappTimeLayout is the layout of the times in the answers. whatsappTimeLayout = "2006-01-02 15:04" - // whatsappPairTime is the longest time of a pairing. The library closes - // the pairing connection after about 160 seconds. - whatsappPairTime = 3 * time.Minute + // whatsappPairTimeout is the longest time of a pairing. The library + // closes the pairing connection after about 160 seconds. + whatsappPairTimeout = 3 * time.Minute - // whatsappLogoutTime is the longest time of the logout call of an + // whatsappLogoutTimeout is the longest time of the logout call of an // unlink. - whatsappLogoutTime = 30 * time.Second + whatsappLogoutTimeout = 30 * time.Second - // whatsappNoticeDelay is how long a disconnection lasts before the user - // gets a notice. - whatsappNoticeDelay = 10 * time.Minute + // whatsappOutageNoticeDelay is the delay from a disconnection to its + // notice to the user. + whatsappOutageNoticeDelay = 10 * time.Minute // whatsappRetryDelay is the first wait after a failed starting // connection. @@ -309,7 +309,7 @@ func (w *WhatsApp) Run(ctx context.Context, l *TurnLoop, notify func(Event)) err case w.isLinked(): request.answer("WhatsApp is linked already. Run /whatsapp unlink first.") default: - pairCtx, cancelPair := context.WithTimeout(ctx, whatsappPairTime) + pairCtx, cancelPair := context.WithTimeout(ctx, whatsappPairTimeout) cancel = cancelPair pairing.Go(func() { defer cancelPair() @@ -569,7 +569,7 @@ func (w *WhatsApp) startOutage() bool { w.disconnectedAt = w.clock() w.disconnections++ disconnections := w.disconnections - w.timer = time.AfterFunc(whatsappNoticeDelay, func() { + w.timer = time.AfterFunc(whatsappOutageNoticeDelay, func() { w.disconnectedLong(disconnections) }) } @@ -654,7 +654,7 @@ func (w *WhatsApp) unlink(ctx context.Context, l *TurnLoop, answer func(text str var logoutErr error if client != nil { - ctx, cancel := context.WithTimeout(ctx, whatsappLogoutTime) + ctx, cancel := context.WithTimeout(ctx, whatsappLogoutTimeout) defer cancel() logoutErr = client.Logout(ctx) @@ -890,8 +890,8 @@ func (w *WhatsApp) linkedBy(client WhatsAppClient) { // handle takes one event of the library and tells if it was handled. A message // goes to the inbox. The names of the groups follow the events of the groups. A // connection starts a catch-up of the inbox, which ends at OfflineSyncCompleted -// or after whatsappCatchupLimit. A disconnection keeps the time of its start -// while Run or the library tries again. After whatsappNoticeDelay of a +// or after whatsappCatchupTimeout. A disconnection keeps the time of its start +// while Run or the library tries again. After whatsappOutageNoticeDelay of a // disconnection, all clients get one notice, and they get one more when the // connection comes back. A disconnection that the library does not try again // gives a notice at once. diff --git a/whatsapp_groups.go b/whatsapp_groups.go index 4d6c7e5..6e9bed3 100644 --- a/whatsapp_groups.go +++ b/whatsapp_groups.go @@ -9,9 +9,9 @@ import ( ) const ( - // whatsappGroupsLookupTime is the longest time of a read of group + // whatsappGroupLookupTimeout is the longest time of a read of group // names. - whatsappGroupsLookupTime = 10 * time.Second + whatsappGroupLookupTimeout = 10 * time.Second // whatsappGroupsRefresh is the shortest time between two reads of the // groups from WhatsApp for a group with no name. @@ -81,7 +81,7 @@ func (g *WhatsAppGroups) read(ctx context.Context, link WhatsAppLink) bool { return false } - ctx, cancel := context.WithTimeout(ctx, whatsappGroupsLookupTime) + ctx, cancel := context.WithTimeout(ctx, whatsappGroupLookupTimeout) defer cancel() groups, err := link.client.GetJoinedGroups(ctx) diff --git a/whatsapp_image_test.go b/whatsapp_image_test.go index 91af70a..75a95dc 100644 --- a/whatsapp_image_test.go +++ b/whatsapp_image_test.go @@ -83,7 +83,7 @@ func TestWhatsAppImageDownloadCanOutlastNameLookup(t *testing.T) { synctest.Test(t, func(t *testing.T) { whatsapp, fake := whatsappStartTestLinked(t) whatsapp.client = fake - fake.downloadWait = whatsappLookupTime + time.Second + fake.downloadWait = whatsappContactLookupTimeout + time.Second data := imageTestPNG(t, 3, 2) fake.downloadData = data diff --git a/whatsapp_inbox.go b/whatsapp_inbox.go index 392593f..9d2183e 100644 --- a/whatsapp_inbox.go +++ b/whatsapp_inbox.go @@ -16,19 +16,20 @@ import ( ) const ( - // whatsappLookupTime is the longest time of a read of names in the - // session. - whatsappLookupTime = 10 * time.Second + // whatsappContactLookupTimeout is the longest time of a read of names + // in the session. + whatsappContactLookupTimeout = 10 * time.Second - // whatsappImageTime is the longest time of an image download. - whatsappImageTime = 2 * time.Minute + // whatsappImageDownloadTimeout is the longest time of an image + // download. + whatsappImageDownloadTimeout = 2 * time.Minute // whatsappSource is the source of the work events of the batches. whatsappSource = "whatsapp" - // whatsappCatchupLimit is the longest wait for the end of the catch-up - // after a connection. - whatsappCatchupLimit = time.Minute + // whatsappCatchupTimeout is the longest wait for the end of the + // catch-up after a connection. + whatsappCatchupTimeout = time.Minute ) // newWhatsAppInbox returns the inbox of the batches in state, with clock for @@ -52,7 +53,7 @@ func newWhatsAppInbox(state *WhatsAppState, clock func() time.Time, groups *What // // After each connection, the servers send the messages that came while the // daemon was away, and then OfflineSyncCompleted. Until then, or until -// whatsappCatchupLimit passes, the closed batches wait in the state file. At +// whatsappCatchupTimeout passes, the closed batches wait in the state file. At // the end of the catch-up, deliver closes all batches and logs them. It wakes // the root once for its batches, so their catch-up gives one turn, not one for // each chat. The batches of the state file at the start wait for the first @@ -207,7 +208,7 @@ func (i *WhatsAppInbox) waitForCatchup() { } // startCatchup starts the catch-up of a new connection, which ends at -// endCatchup or after whatsappCatchupLimit. +// endCatchup or after whatsappCatchupTimeout. func (i *WhatsAppInbox) startCatchup() { i.mutex.Lock() defer i.mutex.Unlock() @@ -219,7 +220,7 @@ func (i *WhatsAppInbox) startCatchup() { i.catchupTimer.Stop() } - i.catchupTimer = time.AfterFunc(whatsappCatchupLimit, func() { + i.catchupTimer = time.AfterFunc(whatsappCatchupTimeout, func() { i.caughtUpBy(catchups) }) } @@ -230,7 +231,7 @@ func (i *WhatsAppInbox) startCatchup() { // daemon. An image that cannot be downloaded or stored gets a text marker. A // message that the batches do not take is handled with nothing to do. func (i *WhatsAppInbox) receive(ctx context.Context, link WhatsAppLink, event *events.Message) bool { - lookupCtx, cancel := context.WithTimeout(ctx, whatsappLookupTime) + lookupCtx, cancel := context.WithTimeout(ctx, whatsappContactLookupTimeout) defer cancel() chat, message, found := whatsappMessage(lookupCtx, link.device, link.own, event) @@ -239,7 +240,7 @@ func (i *WhatsAppInbox) receive(ctx context.Context, link WhatsAppLink, event *e } if event.Message.GetImageMessage() != nil { - ctx, cancel := context.WithTimeout(ctx, whatsappImageTime) + ctx, cancel := context.WithTimeout(ctx, whatsappImageDownloadTimeout) defer cancel() message.Text = i.imageText(ctx, link, event, message.Text) diff --git a/whatsapp_inbox_test.go b/whatsapp_inbox_test.go index a0a0f9d..1ce9a72 100644 --- a/whatsapp_inbox_test.go +++ b/whatsapp_inbox_test.go @@ -147,7 +147,7 @@ func TestWhatsAppRunClosesABatchAfterQuietTime(t *testing.T) { synctest.Wait() require.True(t, test.fake.Emit(whatsappTestMessage(whatsappTestAna, whatsappTestOwn, "m1", "On my way.", time.Now()))) synctest.Wait() - time.Sleep(whatsappBatchQuiet - time.Nanosecond) + time.Sleep(whatsappBatchIdleTimeout - time.Nanosecond) synctest.Wait() require.Zero(t, test.loop.memory.Len()) @@ -268,7 +268,7 @@ func TestWhatsAppCatchupEndsAfterItsLimit(t *testing.T) { require.NoError(t, test.whatsapp.inbox.deliver(t.Context(), test.loop, test.whatsapp.link(), false)) require.Zero(t, test.loop.memory.Len()) - time.Sleep(whatsappCatchupLimit) + time.Sleep(whatsappCatchupTimeout) synctest.Wait() require.NoError(t, test.whatsapp.inbox.deliver(t.Context(), test.loop, test.whatsapp.link(), false)) require.Equal(t, 1, test.loop.memory.Len()) diff --git a/whatsapp_send.go b/whatsapp_send.go index ce3f70b..d5995df 100644 --- a/whatsapp_send.go +++ b/whatsapp_send.go @@ -16,8 +16,8 @@ import ( ) const ( - // whatsappSendTime is the longest time of a send. - whatsappSendTime = 30 * time.Second + // whatsappSendTimeout is the longest time of a send. + whatsappSendTimeout = 30 * time.Second // whatsappSendBytes is the most bytes of the text of a send. whatsappSendBytes = 65536 @@ -77,7 +77,7 @@ func whatsappSend(ctx context.Context, link WhatsAppLink, groups *WhatsAppGroups return "", err } - sendCtx, cancel := context.WithTimeout(ctx, whatsappSendTime) + sendCtx, cancel := context.WithTimeout(ctx, whatsappSendTimeout) defer cancel() response, err := link.client.SendMessage(sendCtx, jid, &waE2E.Message{ diff --git a/whatsapp_state.go b/whatsapp_state.go index 41bd2f0..b1b5935 100644 --- a/whatsapp_state.go +++ b/whatsapp_state.go @@ -16,12 +16,12 @@ const ( // the batches that are not in the log yet and the records of the sends. whatsappStateFileName = "state.json" - // whatsappBatchQuiet is how long a chat is quiet before its batch + // whatsappBatchIdleTimeout is how long a chat is quiet before its batch // closes. - whatsappBatchQuiet = 20 * time.Second + whatsappBatchIdleTimeout = 20 * time.Second - // whatsappBatchWindow is the longest time of a batch. - whatsappBatchWindow = 2 * time.Minute + // whatsappBatchTimeout is the longest time of a batch. + whatsappBatchTimeout = 2 * time.Minute // whatsappBatchMessages is the most messages of a batch. whatsappBatchMessages = 50 @@ -32,8 +32,9 @@ const ( ) // openWhatsAppState reads the batches and the send records of the state file at -// path. A missing file holds none. A batch closes after whatsappBatchQuiet with -// no message, or after whatsappBatchWindow at the latest. +// path. A missing file holds none. A batch closes after +// whatsappBatchIdleTimeout with no message, or after whatsappBatchTimeout at +// the latest. func openWhatsAppState(path string) (*WhatsAppState, error) { s := &WhatsAppState{ path: path, @@ -59,12 +60,13 @@ func openWhatsAppState(path string) (*WhatsAppState, error) { // WhatsAppState holds the state file: the send records, and the batches of the // WhatsApp messages that are not in the log yet, open and closed, in the order // of their first message. The messages of one chat form one batch while the -// times of the messages are at most whatsappBatchQuiet apart, at most -// whatsappBatchWindow after the first, and the batch has fewer than +// times of the messages are at most whatsappBatchIdleTimeout apart, at most +// whatsappBatchTimeout after the first, and the batch has fewer than // whatsappBatchMessages messages. The times are the WhatsApp times, so a // catch-up after a stop gives the same batches as live messages. A batch also -// closes when whatsappBatchQuiet passes after the arrival of its last message, -// or whatsappBatchWindow after the arrival of its first message, in real time. +// closes when whatsappBatchIdleTimeout passes after the arrival of its last +// message, or whatsappBatchTimeout after the arrival of its first message, in +// real time. // // A send record keeps the result of a send by its ID, so that a repeated call // of the send tool does not send again. The state file keeps the last @@ -99,7 +101,7 @@ func (s *WhatsAppState) Add(chat WhatsAppChat, message WhatsAppMessage, arrival return false, nil } - if message.Time.Sub(batch.Messages[len(batch.Messages)-1].Time) > whatsappBatchQuiet || message.Time.Sub(batch.Messages[0].Time) > whatsappBatchWindow { + if message.Time.Sub(batch.Messages[len(batch.Messages)-1].Time) > whatsappBatchIdleTimeout || message.Time.Sub(batch.Messages[0].Time) > whatsappBatchTimeout { batch.Closed = true closed = true index = -1 @@ -258,8 +260,8 @@ func (s *WhatsAppState) open(chat string) int { // deadline returns when batch closes in real time. func (s *WhatsAppState) deadline(batch WhatsAppBatch) time.Time { - quiet := batch.Last.Add(whatsappBatchQuiet) - window := batch.First.Add(whatsappBatchWindow) + quiet := batch.Last.Add(whatsappBatchIdleTimeout) + window := batch.First.Add(whatsappBatchTimeout) if window.Before(quiet) { return window } diff --git a/whatsapp_test.go b/whatsapp_test.go index f098c90..c1471e9 100644 --- a/whatsapp_test.go +++ b/whatsapp_test.go @@ -249,12 +249,12 @@ func TestWhatsAppLongFirstConnectionOutageGivesOneNotice(t *testing.T) { since := time.Now().Format(whatsappTimeLayout) - time.Sleep(whatsappNoticeDelay) + time.Sleep(whatsappOutageNoticeDelay) synctest.Wait() require.Equal(t, 1, test.events.Count("notice")) require.Equal(t, "WhatsApp is disconnected since "+since+". OneChat tries again.", test.events.Last("notice")) - time.Sleep(whatsappNoticeDelay) + time.Sleep(whatsappOutageNoticeDelay) synctest.Wait() require.Equal(t, 1, test.events.Count("notice")) test.fake.mutex.Lock() @@ -613,18 +613,18 @@ func TestWhatsAppLongDisconnectionGivesOneNotice(t *testing.T) { require.True(t, test.fake.Emit(&events.Connected{})) require.True(t, test.fake.Emit(&events.Disconnected{})) require.True(t, test.fake.Emit(&events.Connected{})) - time.Sleep(whatsappNoticeDelay + time.Second) + time.Sleep(whatsappOutageNoticeDelay + time.Second) synctest.Wait() require.Zero(t, test.events.Count("notice")) require.True(t, test.fake.Emit(&events.Disconnected{})) require.True(t, test.fake.Emit(&events.Disconnected{})) - time.Sleep(whatsappNoticeDelay) + time.Sleep(whatsappOutageNoticeDelay) synctest.Wait() require.Equal(t, 1, test.events.Count("notice")) require.Contains(t, test.events.Last("notice"), "OneChat tries again.") require.True(t, test.fake.Emit(&events.Disconnected{})) - time.Sleep(whatsappNoticeDelay) + time.Sleep(whatsappOutageNoticeDelay) synctest.Wait() require.Equal(t, 1, test.events.Count("notice")) -- 2.53.0 From 3cdbd017da9ae7a7045f4272f16d4b262fd01c79 Mon Sep 17 00:00:00 2001 From: "Anderson A. Giacomolli" Date: Tue, 10 Nov 2026 15:59:13 -0300 Subject: [PATCH 11/16] Show stored images to the user The root can show a stored image as a Telegram photo and return its name and path to terminal clients. A refused photo goes as a document, while an uncertain send does not cause a second send. Children and subagents have no image output tool. --- agent.go | 4 +- daemon.go | 4 ++ daemon_children_test.go | 2 + image_show.go | 37 ++++++++++ image_show_test.go | 136 +++++++++++++++++++++++++++++++++++ prompt.go | 9 +++ prompt_test.go | 2 +- telegram_api.go | 65 +++++++++++++++++ telegram_fake_helper_test.go | 35 ++++++++- tools.go | 21 +++++- turn.go | 4 ++ 11 files changed, 313 insertions(+), 6 deletions(-) create mode 100644 image_show.go create mode 100644 image_show_test.go diff --git a/agent.go b/agent.go index 97d6685..b86880e 100644 --- a/agent.go +++ b/agent.go @@ -117,12 +117,12 @@ func (a *Agent) Prompt() (string, error) { } // Tools returns the tools shared by the turns and compactions of this agent. A -// child has no WhatsApp, child creation, or route tools. +// child has no image output, WhatsApp, child creation, or route tools. func (a *Agent) Tools() []ModelTool { tools := promptTools() if a.config.Parent != "" { tools = slices.DeleteFunc(tools, func(tool ModelTool) bool { - return strings.HasPrefix(tool.Name, "whatsapp_") || tool.Name == "agent_create" || tool.Name == "route" || tool.Name == "unroute" || tool.Name == "routes" + return strings.HasPrefix(tool.Name, "whatsapp_") || tool.Name == "image_show" || tool.Name == "agent_create" || tool.Name == "route" || tool.Name == "unroute" || tool.Name == "routes" }) } diff --git a/daemon.go b/daemon.go index 137cc6f..5c94a4f 100644 --- a/daemon.go +++ b/daemon.go @@ -353,6 +353,10 @@ func daemonServe(ctx context.Context, listener net.Listener, bot *TelegramBot, w return errors.Join(err, listener.Close()) } + if bot != nil { + l.tools.imageShow.bot = bot + } + loops := []*TurnLoop{l} for _, child := range children { loop, err := child.Loop(nil, func(Event) { diff --git a/daemon_children_test.go b/daemon_children_test.go index 16f326f..b9e212f 100644 --- a/daemon_children_test.go +++ b/daemon_children_test.go @@ -335,6 +335,8 @@ func TestChildToolsExcludeCreationAndWhatsApp(t *testing.T) { names := promptToolNames(child.agent.Tools()) require.NotContains(t, names, "agent_create") + require.NotContains(t, names, "image_show") + require.Equal(t, `error: unknown tool "image_show"`, child.tools.run(t.Context(), "image_show", `{"image":"unknown","caption":""}`)) require.NotContains(t, names, "whatsapp_send") require.NotContains(t, names, "whatsapp_find") require.Contains(t, names, "spawn") diff --git a/image_show.go b/image_show.go new file mode 100644 index 0000000..80204c1 --- /dev/null +++ b/image_show.go @@ -0,0 +1,37 @@ +package main + +import ( + "context" + "fmt" + "path/filepath" +) + +// ImageShow shows a stored image to the root's user. A configured bot sends it +// to its owner, and the tool result gives terminal clients its name and path. +type ImageShow struct { + store *ImageStore + bot *TelegramBot +} + +// Show checks the stored name and bytes before any output. A refused Telegram +// photo goes as a document. An uncertain send is not repeated. +func (s *ImageShow) Show(ctx context.Context, name, caption string) (string, error) { + data, err := s.store.Read(name) + if err != nil { + return "", fmt.Errorf("read image to show: %w", err) + } + + path, err := filepath.Abs(filepath.Join(s.store.dir, name)) + if err != nil { + return "", fmt.Errorf("find image path: %w", err) + } + + if s.bot != nil { + err = s.bot.api.SendImage(ctx, s.bot.owner, name, data, caption) + if err != nil { + return "", fmt.Errorf("show Telegram image: %w", err) + } + } + + return imageText([]string{name}, caption) + "\n" + path, nil +} diff --git a/image_show_test.go b/image_show_test.go new file mode 100644 index 0000000..ed475fe --- /dev/null +++ b/image_show_test.go @@ -0,0 +1,136 @@ +package main + +import ( + "encoding/json" + "net/http" + "path/filepath" + "testing" + + "github.com/stretchr/testify/require" +) + +func TestImageShowUsesStoredImageForTelegramAndTerminal(t *testing.T) { + picture := imageTestPNG(t, 5, 4) + store := &ImageStore{dir: filepath.Join(t.TempDir(), imageDirName)} + name, err := store.Save(picture) + require.NoError(t, err) + + telegram := newTelegramFake(t) + bot, logs := telegramStartTestBot(t, telegram) + box := &Toolbox{imageShow: &ImageShow{ + store: store, + bot: bot, + }} + args, err := json.Marshal(ToolImageShowArguments{ + Image: name, + Caption: "A red point.", + }) + require.NoError(t, err) + + output := box.run(t.Context(), "image_show", string(args)) + require.Equal(t, imageText([]string{name}, "A red point.")+"\n"+filepath.Join(store.dir, name), output) + + calls := telegram.Calls() + require.Len(t, calls, 1) + require.Equal(t, "sendPhoto", calls[0].method) + require.Equal(t, "4242", calls[0].params["chat_id"]) + require.Equal(t, "A red point.", calls[0].params["caption"]) + require.Equal(t, name, calls[0].params["photo_name"]) + require.Equal(t, picture, calls[0].params["photo"]) + require.Empty(t, logs.String()) +} + +func TestImageShowRefusesUnknownNamesBeforeSending(t *testing.T) { + telegram := newTelegramFake(t) + bot, _ := telegramStartTestBot(t, telegram) + show := &ImageShow{ + store: &ImageStore{dir: t.TempDir()}, + bot: bot, + } + + for _, name := range []string{"../image.png", "unknown", "aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa.png"} { + _, err := show.Show(t.Context(), name, "Do not send this.") + require.Error(t, err) + } + + require.Empty(t, telegram.Calls()) + + box := &Toolbox{} + require.Equal(t, `error: unknown tool "image_show"`, box.run(t.Context(), "image_show", `{"image":"unknown","caption":""}`)) +} + +func TestImageShowFallsBackOnlyWhenTelegramRefusesPhoto(t *testing.T) { + testCases := []struct { + name string + status int + methods []string + failure bool + }{ + { + name: "refused photo sends document", + status: http.StatusBadRequest, + methods: []string{"sendPhoto", "sendDocument"}, + }, + { + name: "uncertain server error does not send twice", + status: http.StatusBadGateway, + methods: []string{"sendPhoto"}, + failure: true, + }, + } + for _, tc := range testCases { + t.Run(tc.name, func(t *testing.T) { + telegram := newTelegramFake(t) + telegram.fail = func(call TelegramFakeCall) (int, string) { + if call.method == "sendPhoto" { + return tc.status, "photo refused" + } + + return 0, "" + } + store := &ImageStore{dir: t.TempDir()} + picture := imageTestPNG(t, 3, 3) + name, err := store.Save(picture) + require.NoError(t, err) + + bot, _ := telegramStartTestBot(t, telegram) + show := &ImageShow{ + store: store, + bot: bot, + } + _, err = show.Show(t.Context(), name, "Caption") + if tc.failure { + require.Error(t, err) + } else { + require.NoError(t, err) + } + + var methods []string + + for _, call := range telegram.Calls() { + methods = append(methods, call.method) + if call.method == "sendDocument" { + require.Equal(t, picture, call.params["document"]) + require.Equal(t, name, call.params["document_name"]) + require.Equal(t, "Caption", call.params["caption"]) + } + } + + require.Equal(t, tc.methods, methods) + }) + } +} + +func TestAgentShowsImageInTerminalWithoutTelegram(t *testing.T) { + server := newModelFake(t) + test := newTestTurnLoop(t, server) + name, err := test.agent.Images().Save(imageTestPNG(t, 2, 2)) + require.NoError(t, err) + + output := test.loop.tools.run(t.Context(), "image_show", `{"image":"`+name+`","caption":"Here it is."}`) + require.Equal(t, imageText([]string{name}, "Here it is.")+"\n"+filepath.Join(test.agent.Images().dir, name), output) + require.Contains(t, terminalEventText(Event{ + Kind: "echo", + Text: output, + }), filepath.Join(test.agent.Images().dir, name)) +} diff --git a/prompt.go b/prompt.go index df6f17d..5847ee8 100644 --- a/prompt.go +++ b/prompt.go @@ -274,6 +274,15 @@ func promptTools() []ModelTool { "message": promptStringSchema("The message."), }), Strict: true, + }, ModelTool{ + Type: "function", + Name: "image_show", + Description: "Show a stored image to the user, only when the user asks to see it. Give its stored filename, not a path.", + Parameters: promptObjectSchema(map[string]any{ + "image": promptStringSchema("The stored image filename from an [image FILE] line."), + "caption": promptStringSchema("The caption, or an empty string."), + }), + Strict: true, }, ModelTool{ Type: "function", Name: "whatsapp_find", diff --git a/prompt_test.go b/prompt_test.go index e9e87f3..06fea49 100644 --- a/prompt_test.go +++ b/prompt_test.go @@ -129,7 +129,7 @@ func TestProbePromptKeepsTheInstructionsPathStable(t *testing.T) { func TestAgentToolsAreStrictAndStable(t *testing.T) { tools := promptTools() - require.Equal(t, []string{"zoom", "date", "shell", "read", "write", "edit", "spawn", "agent_create", "tell", "whatsapp_find", "whatsapp_send", "route", "unroute", "routes", "schedule_set", "schedule_list", "schedule_remove"}, promptToolNames(tools)) + require.Equal(t, []string{"zoom", "date", "shell", "read", "write", "edit", "spawn", "agent_create", "tell", "image_show", "whatsapp_find", "whatsapp_send", "route", "unroute", "routes", "schedule_set", "schedule_list", "schedule_remove"}, promptToolNames(tools)) require.Equal(t, []string{"zoom", "date", "shell", "read", "write", "edit"}, promptToolNames(promptSubagentTools())) for _, tool := range tools { diff --git a/telegram_api.go b/telegram_api.go index 8f78387..508010e 100644 --- a/telegram_api.go +++ b/telegram_api.go @@ -7,9 +7,11 @@ import ( "errors" "fmt" "io" + "mime/multipart" "net/http" "net/url" "path" + "strconv" "strings" "time" ) @@ -196,6 +198,64 @@ func (b *TelegramAPI) SendRichMessage(ctx context.Context, chat int64, markdown }, nil) } +// SendImage uploads an image to the owner as a photo, or as a document when the +// Bot API refuses the photo. Network failures do not trigger a second send. +func (b *TelegramAPI) SendImage(ctx context.Context, chat int64, name string, data []byte, caption string) error { + err := b.sendImage(ctx, "sendPhoto", "photo", chat, name, data, caption) + if telegramRejected(err) { + err = b.sendImage(ctx, "sendDocument", "document", chat, name, data, caption) + } + + return err +} + +func (b *TelegramAPI) sendImage(ctx context.Context, method, field string, chat int64, name string, data []byte, caption string) error { + ctx, cancel := context.WithTimeout(ctx, telegramSendTimeout) + defer cancel() + + var body bytes.Buffer + + writer := multipart.NewWriter(&body) + err := writer.WriteField("chat_id", strconv.FormatInt(chat, 10)) + if err != nil { + return fmt.Errorf("write image chat: %w", err) + } + + err = writer.WriteField("caption", caption) + if err != nil { + return fmt.Errorf("write image caption: %w", err) + } + + part, err := writer.CreateFormFile(field, name) + if err != nil { + return fmt.Errorf("make image part: %w", err) + } + + _, err = part.Write(data) + if err != nil { + return fmt.Errorf("write image part: %w", err) + } + + err = writer.Close() + if err != nil { + return fmt.Errorf("close image form: %w", err) + } + + request, err := http.NewRequestWithContext(ctx, http.MethodPost, b.base+"/bot"+b.token+"/"+method, &body) + if err != nil { + return fmt.Errorf("make image upload: %s", b.redact(err)) + } + + request.Header.Set("Content-Type", writer.FormDataContentType()) + + err = b.answer(request, telegramSendTimeout, nil) + if err != nil { + return fmt.Errorf("upload image: %w", err) + } + + return nil +} + // SendTyping shows the typing action in chat for about five seconds. A call // that takes longer than telegramTypingTimeout fails with a // TelegramNetworkError. @@ -226,6 +286,11 @@ func (b *TelegramAPI) call(ctx context.Context, timeout time.Duration, method st request.Header.Set("Content-Type", "application/json") + return b.answer(request, timeout, result) +} + +// answer reads the bounded Bot API response without keeping its request URL. +func (b *TelegramAPI) answer(request *http.Request, timeout time.Duration, result any) error { response, err := b.http.Do(request) if err != nil { return b.networkError("no answer", err, timeout) diff --git a/telegram_fake_helper_test.go b/telegram_fake_helper_test.go index f6ccd84..b71eee0 100644 --- a/telegram_fake_helper_test.go +++ b/telegram_fake_helper_test.go @@ -2,8 +2,10 @@ package main import ( "encoding/json" + "errors" "fmt" "io" + "mime/multipart" "net/http" "net/http/httptest" "strings" @@ -139,7 +141,38 @@ func (f *TelegramFake) serve(w http.ResponseWriter, r *http.Request) { var params map[string]any - err := json.NewDecoder(r.Body).Decode(¶ms) + var err error + + if strings.HasPrefix(r.Header.Get("Content-Type"), "multipart/form-data;") { + err = r.ParseMultipartForm(imageMaxBytes + (1 << 20)) + if err == nil { + params = make(map[string]any) + for key, values := range r.MultipartForm.Value { + params[key] = values[0] + } + + for key, files := range r.MultipartForm.File { + var file multipart.File + + file, err = files[0].Open() + if err != nil { + break + } + + params[key], err = io.ReadAll(file) + err = errors.Join(err, file.Close()) + params[key+"_name"] = files[0].Filename + if err != nil { + break + } + } + + err = errors.Join(err, r.MultipartForm.RemoveAll()) + } + } else { + err = json.NewDecoder(r.Body).Decode(¶ms) + } + if err != nil { http.Error(w, err.Error(), http.StatusBadRequest) diff --git a/tools.go b/tools.go index bbd34cf..9f2905a 100644 --- a/tools.go +++ b/tools.go @@ -54,8 +54,8 @@ type ToolChat interface { // spawn and tell, the WhatsApp link for the WhatsApp tools, and the schedules // for the schedule tools. Each permanent agent has the parent-child queue. Only // the root has child creation and source routes. A child has neither of them -// nor a WhatsApp link. The toolbox of a subagent has none of them, so these are -// unknown tools there. +// nor a WhatsApp link. Only the root can show images to the user. The toolbox +// of a subagent has none of them, so these are unknown tools there. type Toolbox struct { log ToolLog chat ToolChat @@ -69,6 +69,7 @@ type Toolbox struct { children *DaemonChildren routes *Routes images *ImageStore + imageShow *ImageShow } // Call logs the call of the tool name with its JSON arguments, makes the log @@ -231,6 +232,16 @@ func (b *Toolbox) run(ctx context.Context, name, arguments string) string { case err == nil: output, err = b.runs.Tell(ctx, args.Name, args.Message) } + case "image_show": + var args ToolImageShowArguments + + err = toolDecodeArguments(arguments, &args) + switch { + case b.imageShow == nil: + err = fmt.Errorf("unknown tool %q", name) + case err == nil: + output, err = b.imageShow.Show(ctx, args.Image, args.Caption) + } case "whatsapp_find": var args ToolWhatsAppFindArguments @@ -549,6 +560,12 @@ type ToolTellArguments struct { Message string `json:"message"` } +// ToolImageShowArguments are the arguments of image_show. +type ToolImageShowArguments struct { + Image string `json:"image"` + Caption string `json:"caption"` +} + // ToolWhatsAppFindArguments are the arguments of whatsapp_find. type ToolWhatsAppFindArguments struct { Name string `json:"name"` diff --git a/turn.go b/turn.go index 2c1d9c2..9e6a1a7 100644 --- a/turn.go +++ b/turn.go @@ -63,6 +63,10 @@ func newTurnLoop(agent *Agent, memory *Memory, subagents *Subagents, whatsapp *W images: agent.Images(), } + if agent.config.Parent == "" { + l.tools.imageShow = &ImageShow{store: agent.Images()} + } + return l, nil } -- 2.53.0 From b1be0ee8530df5db8973bfe355f2f6d3149383a0 Mon Sep 17 00:00:00 2001 From: "Anderson A. Giacomolli" Date: Tue, 10 Nov 2026 16:11:27 -0300 Subject: [PATCH 12/16] Send stored images to WhatsApp chats WhatsApp sends accept a stored image and use the text as its caption. The send record includes the image name and precedes the upload, so a repeated ID cannot upload or send again. Media errors keep safe causes without URLs or keys. The root asks for consent as for text sends, and image markers in tell give children the shared pixels. The user can also ask to see a received Telegram photo without a separate import. --- README.rst | 2 +- image_show_test.go | 95 +++++++++++++ prompt.go | 13 +- prompt_test.go | 11 ++ tools.go | 9 +- whatsapp.go | 9 +- whatsapp_fake_helper_test.go | 47 +++++++ whatsapp_image.go | 41 ++++++ whatsapp_send.go | 65 ++++++--- whatsapp_send_test.go | 253 +++++++++++++++++++++++++++++++++-- whatsapp_state.go | 6 +- 11 files changed, 504 insertions(+), 47 deletions(-) diff --git a/README.rst b/README.rst index a02b870..bbb4475 100644 --- a/README.rst +++ b/README.rst @@ -196,7 +196,7 @@ WhatsApp Images Send a photo in Telegram, WhatsApp, or with ``/image`` to the agent. Its memory keeps what the image shows, and it can look at an older - image again. + image again. Ask it to show an image or send one to a WhatsApp chat. Schedules Ask for reminders and timed tasks, such as "every weekday at 8, tell diff --git a/image_show_test.go b/image_show_test.go index ed475fe..65e422c 100644 --- a/image_show_test.go +++ b/image_show_test.go @@ -1,6 +1,7 @@ package main import ( + "encoding/base64" "encoding/json" "net/http" "path/filepath" @@ -134,3 +135,97 @@ func TestAgentShowsImageInTerminalWithoutTelegram(t *testing.T) { Text: output, }), filepath.Join(test.agent.Images().dir, name)) } + +func TestTellImageMarkerGivesTheChildPixels(t *testing.T) { + server := newModelFake(t) + test := newDaemonMessageTest(t, t.TempDir(), server) + picture := imageTestPNG(t, 4, 2) + name, err := test.agents[0].agent.Images().Save(picture) + require.NoError(t, err) + test.Start(t, 0, 1) + + arguments, err := json.Marshal(ToolTellArguments{ + Name: "Garden", + Message: imageText([]string{name}, "Review this plan."), + }) + require.NoError(t, err) + require.Contains(t, test.loops[0].tools.run(t.Context(), "tell", string(arguments)), "Sent.") + test.events[1].WaitFor(t, "idle", 1) + + var childCall *ModelFakeCall + + for i := range server.Calls() { + call := server.Call(i) + if call.body["model"] == "gpt-test-garden" { + childCall = &call + break + } + } + + require.NotNil(t, childCall) + require.Contains(t, string(childCall.raw), "input_image") + require.Contains(t, string(childCall.raw), base64.StdEncoding.EncodeToString(picture)) + require.Contains(t, test.events[1].Last("user"), "[image "+name+"]") + require.NotContains(t, promptToolNames(test.agents[1].agent.Tools()), "image_show") +} + +func TestAgentShowsTelegramReceivedImageToItsOwner(t *testing.T) { + picture := imageTestPNG(t, 6, 4) + store := &ImageStore{dir: t.TempDir()} + name, err := store.Save(picture) + require.NoError(t, err) + + telegram := newTelegramFake(t) + telegram.Image("owner-photo", picture) + + bot, logs := telegramStartTestBot(t, telegram) + server := newModelFake(t) + server.reply = func(w http.ResponseWriter, call ModelFakeCall) { + if call.number == 0 { + arguments, err := json.Marshal(ToolImageShowArguments{ + Image: name, + Caption: "Here is your photo.", + }) + require.NoError(t, err) + modelWriteEvents(w, modelCompletedEvent([]any{modelFunctionCallItem("fc_1", "call_1", "image_show", string(arguments))}, 100, 0, 5)) + + return + } + + modelWriteEvents(w, modelCompletedEvent([]any{modelMessageItem("Shown.")}, 100, 0, 5)) + } + socket := daemonStartTestWithBot(t, server, bot) + terminal := terminalStartTestClient(t, socket) + terminal.WaitFor(t, "[connected to OneChat]") + + message := telegramOwnerMessage("") + message.Caption = "Show this photo." + message.Photo = []TelegramPhoto{{ + FileID: "owner-photo", + Width: 6, + Height: 4, + }} + telegram.Add(TelegramUpdate{ + UpdateID: 1, + Message: message, + }) + terminal.WaitFor(t, "Shown.\n[done]") + require.Contains(t, terminal.Output(), "[result] [image "+name+"]") + require.Contains(t, terminal.Output(), filepath.Join(imageDirName, name)) + + var photos []TelegramFakeCall + + for _, call := range telegram.Calls() { + if call.method == "sendPhoto" { + photos = append(photos, call) + } + } + + require.Len(t, photos, 1) + require.Equal(t, picture, photos[0].params["photo"]) + require.Equal(t, "4242", photos[0].params["chat_id"]) + require.Equal(t, "Here is your photo.", photos[0].params["caption"]) + require.NotContains(t, logs.String(), "Here is your photo.") + require.NotContains(t, logs.String(), "Show this photo.") + require.NotContains(t, logs.String(), telegramTestToken) +} diff --git a/prompt.go b/prompt.go index 5847ee8..0852bec 100644 --- a/prompt.go +++ b/prompt.go @@ -154,7 +154,9 @@ Pass area messages from mixed chats to the right child with sender, date, and words. Never send a WhatsApp reply drafted by a child without the user's agreement. Propose routing a source that talks only about one area to its child. -Set or remove a route only after the user agrees.` +Set or remove a route only after the user agrees. +Use image_show or whatsapp_send with an image only on the user's request. +Pass an image to a child with an [image FILE] line in tell.` // promptSubagentText is the system prompt of a subagent, from the subagent // prompt of the reference design: its head, then the doc of the view, without @@ -294,11 +296,12 @@ func promptTools() []ModelTool { }, ModelTool{ Type: "function", Name: "whatsapp_send", - Description: "Send a text message to a WhatsApp chat. Send only a message that the user asked you to send, to the chat that the user named. Give a new id for each message: a call with an id that was used gives the answer of the first call and sends nothing.", + Description: "Send text or a stored image with text as its caption to a WhatsApp chat. Send only a message that the user asked you to send, to the chat that the user named. Give a new id for each message: a call with an id that was used gives the answer of the first call and sends nothing.", Parameters: promptObjectSchema(map[string]any{ - "id": promptStringSchema("A new id for this message, such as a short random word."), - "chat": promptStringSchema("The ID of the chat, from whatsapp_find or from a [WhatsApp] message."), - "text": promptStringSchema("The text of the message."), + "id": promptStringSchema("A new id for this message, such as a short random word."), + "chat": promptStringSchema("The ID of the chat, from whatsapp_find or from a [WhatsApp] message."), + "text": promptStringSchema("The text, or the caption when sending an image. Can be empty with an image."), + "image": promptNullableStringSchema("The stored image filename from an [image FILE] line, or null for text only."), }), Strict: true, }, ModelTool{ diff --git a/prompt_test.go b/prompt_test.go index 06fea49..888057f 100644 --- a/prompt_test.go +++ b/prompt_test.go @@ -166,3 +166,14 @@ func TestRootPromptDefinesChildRoutingAndConsent(t *testing.T) { require.NotContains(t, child, "Use agent_create") require.Contains(t, child, "Send nothing to WhatsApp. Draft replies in your report instead") } + +func TestRootPromptKeepsImageConsentAndChildForwardingRules(t *testing.T) { + text := prompt("Ada", "/AGENT/instructions.md", "") + require.Contains(t, text, "Use image_show or whatsapp_send with an image only on the user's request.") + require.Contains(t, text, "Pass an image to a child with an [image FILE] line in tell.") + + child := promptChild("House", "Ada", "Keep house plans.", "/CHILD/instructions.md", "") + require.Contains(t, child, "Send nothing to WhatsApp. Draft replies in your report instead") + require.NotContains(t, promptToolNames(promptSubagentTools()), "image_show") + require.NotContains(t, promptToolNames(promptSubagentTools()), "whatsapp_send") +} diff --git a/tools.go b/tools.go index 9f2905a..621cd84 100644 --- a/tools.go +++ b/tools.go @@ -260,7 +260,7 @@ func (b *Toolbox) run(ctx context.Context, name, arguments string) string { case b.whatsapp == nil: err = fmt.Errorf("unknown tool %q", name) case err == nil: - output, err = b.whatsapp.Send(ctx, args.ID, args.Chat, args.Text) + output, err = b.whatsapp.Send(ctx, args.ID, args.Chat, args.Text, args.Image) } case "schedule_set": var args ToolScheduleSetArguments @@ -573,9 +573,10 @@ type ToolWhatsAppFindArguments struct { // ToolWhatsAppSendArguments are the arguments of whatsapp_send. type ToolWhatsAppSendArguments struct { - ID string `json:"id"` - Chat string `json:"chat"` - Text string `json:"text"` + ID string `json:"id"` + Chat string `json:"chat"` + Text string `json:"text"` + Image string `json:"image"` } // ToolScheduleSetArguments are the arguments of schedule_set. Exactly one of At diff --git a/whatsapp.go b/whatsapp.go index 97b1e85..57bfab0 100644 --- a/whatsapp.go +++ b/whatsapp.go @@ -92,6 +92,7 @@ type WhatsAppClient interface { Logout(ctx context.Context) error GetJoinedGroups(ctx context.Context) ([]*types.GroupInfo, error) Download(ctx context.Context, message whatsmeow.DownloadableMessage) ([]byte, error) + Upload(ctx context.Context, data []byte, mediaType whatsmeow.MediaType) (whatsmeow.UploadResponse, error) GenerateMessageID() types.MessageID SendMessage(ctx context.Context, to types.JID, message *waE2E.Message, extra ...whatsmeow.SendRequestExtra) (whatsmeow.SendResponse, error) } @@ -462,10 +463,10 @@ func (w *WhatsApp) Find(ctx context.Context, name string) (string, error) { return whatsappFind(ctx, w.link(), w.groups, name) } -// Send returns the answer of whatsapp_send for the send id of text to chat, as -// whatsappSend says. -func (w *WhatsApp) Send(ctx context.Context, id, chat, text string) (string, error) { - return whatsappSend(ctx, w.link(), w.groups, w.state, w.clock, id, chat, text) +// Send returns the answer of whatsapp_send for text and an optional stored +// image to chat, with the durable send id that whatsappSend describes. +func (w *WhatsApp) Send(ctx context.Context, id, chat, text, image string) (string, error) { + return whatsappSend(ctx, w.link(), w.groups, w.state, w.inbox.images, w.clock, id, chat, text, image) } func (w *WhatsApp) sessionPath() string { diff --git a/whatsapp_fake_helper_test.go b/whatsapp_fake_helper_test.go index 9f80b38..8e0f237 100644 --- a/whatsapp_fake_helper_test.go +++ b/whatsapp_fake_helper_test.go @@ -84,6 +84,9 @@ type WhatsAppFake struct { logoutErr error groups []*types.GroupInfo sendErr error + uploadErr error + uploads [][]byte + imageSends []*waE2E.ImageMessage downloadData []byte downloadErr error downloadWait time.Duration @@ -203,6 +206,46 @@ func (f *WhatsAppFake) Downloads() []whatsmeow.DownloadableMessage { return slices.Clone(f.downloads) } +// Upload records the pixels and returns made-up media metadata. +func (f *WhatsAppFake) Upload(_ context.Context, data []byte, mediaType whatsmeow.MediaType) (whatsmeow.UploadResponse, error) { + f.mutex.Lock() + defer f.mutex.Unlock() + + if mediaType != whatsmeow.MediaImage { + return whatsmeow.UploadResponse{}, fmt.Errorf("unexpected media type %q", mediaType) + } + + f.uploads = append(f.uploads, slices.Clone(data)) + if f.uploadErr != nil { + return whatsmeow.UploadResponse{}, f.uploadErr + } + + return whatsmeow.UploadResponse{ + URL: "https://example.com/image", + DirectPath: "/image", + MediaKey: []byte("made-up key"), + FileSHA256: []byte("made-up hash"), + FileEncSHA256: []byte("made-up encrypted hash"), + FileLength: uint64(len(data)), + }, nil +} + +// Uploads returns the pixels passed to Upload. +func (f *WhatsAppFake) Uploads() [][]byte { + f.mutex.Lock() + defer f.mutex.Unlock() + + return slices.Clone(f.uploads) +} + +// ImageSends returns the outgoing image messages. +func (f *WhatsAppFake) ImageSends() []*waE2E.ImageMessage { + f.mutex.Lock() + defer f.mutex.Unlock() + + return slices.Clone(f.imageSends) +} + // GenerateMessageID returns a new message ID. func (f *WhatsAppFake) GenerateMessageID() types.MessageID { f.mutex.Lock() @@ -216,6 +259,10 @@ func (f *WhatsAppFake) SendMessage(_ context.Context, to types.JID, message *waE f.mutex.Lock() defer f.mutex.Unlock() + if message.GetImageMessage() != nil { + f.imageSends = append(f.imageSends, message.GetImageMessage()) + } + f.sent = append(f.sent, WhatsAppFakeSend{ to: to.String(), text: message.GetConversation(), diff --git a/whatsapp_image.go b/whatsapp_image.go index d4b88d8..5ce13a8 100644 --- a/whatsapp_image.go +++ b/whatsapp_image.go @@ -4,11 +4,16 @@ import ( "context" "errors" "fmt" + "net/http" + "time" "go.mau.fi/whatsmeow" "go.mau.fi/whatsmeow/proto/waE2E" ) +// whatsappImageUploadTimeout limits one image upload to WhatsApp. +const whatsappImageUploadTimeout = 2 * time.Minute + // whatsappImage downloads only supported images and stores them before the // event can be acknowledged. Both the declared size and the downloaded bytes // must fit the image store. Decoded bytes determine the stored format. @@ -59,3 +64,39 @@ func whatsappImageDownloadError(err error) error { return errors.New("media download failed") } + +// whatsappImageMessage uploads stored pixels and uses the upload keys only in +// the outgoing message. The text is the caption, not a separate message. +func whatsappImageMessage(ctx context.Context, client WhatsAppClient, data []byte, text string) (*waE2E.Message, error) { + ctx, cancel := context.WithTimeout(ctx, whatsappImageUploadTimeout) + defer cancel() + + upload, err := client.Upload(ctx, data, whatsmeow.MediaImage) + if err != nil { + return nil, fmt.Errorf("upload image: %w", whatsappImageSendError(err)) + } + + return &waE2E.Message{ImageMessage: &waE2E.ImageMessage{ + Caption: new(text), + Mimetype: new(http.DetectContentType(data)), + URL: &upload.URL, + DirectPath: &upload.DirectPath, + MediaKey: upload.MediaKey, + FileEncSHA256: upload.FileEncSHA256, + FileSHA256: upload.FileSHA256, + FileLength: &upload.FileLength, + }}, nil +} + +// whatsappImageSendError drops arbitrary upload and send errors, which can hold +// media URLs and keys. Cancellation and deadline causes are safe to keep. +func whatsappImageSendError(err error) error { + switch { + case errors.Is(err, context.Canceled): + return context.Canceled + case errors.Is(err, context.DeadlineExceeded): + return context.DeadlineExceeded + default: + return errors.New("media send failed") + } +} diff --git a/whatsapp_send.go b/whatsapp_send.go index d5995df..b95a82e 100644 --- a/whatsapp_send.go +++ b/whatsapp_send.go @@ -31,21 +31,24 @@ const ( whatsappFailed = "failed" ) -// whatsappSend sends text to chat, the ID of a private chat or a group, and -// returns the answer of the tool. id names the send: a call with an id that has -// a record gives the answer of the record and sends nothing, and the same id -// with another chat or text is an error. The record "sending" is durable before -// the send, so a stop of the daemon during the send gives an answer that the -// result is unknown, and not a second send. A disconnected WhatsApp sends -// nothing and makes no record. link gives the device, the client, and the own -// IDs, groups the names of the groups, and state the send records. The clock -// gives the times of the records and the time zone of the answer. -func whatsappSend(ctx context.Context, link WhatsAppLink, groups *WhatsAppGroups, state *WhatsAppState, clock func() time.Time, id, chat, text string) (string, error) { +// whatsappSend sends text and an optional stored image to chat, the ID of a +// private chat or a group, and returns the answer of the tool. id names the +// send: a call with an id that has a record gives the answer of the record and +// sends nothing, and the same id with another chat, text, or image is an error. +// The record "sending" is durable before the send, so a stop of the daemon +// during the send gives an answer that the result is unknown, and not a second +// send. A disconnected WhatsApp sends nothing and makes no record. link gives +// the device, the client, and the own IDs, groups the names of the groups, and +// state the send records. The clock gives the times of the records and the time +// zone of the answer. Image names are checked before the record is written, and +// uploads start only after the record is durable. Text-only records keep their +// text digest. +func whatsappSend(ctx context.Context, link WhatsAppLink, groups *WhatsAppGroups, state *WhatsAppState, images *ImageStore, clock func() time.Time, id, chat, text, image string) (string, error) { if !link.linked { return whatsappNotLinked, nil } - jid, err := whatsappSendCheck(id, chat, text, link.own) + jid, err := whatsappSendCheck(id, chat, text, image, link.own) if err != nil { return "", err } @@ -53,6 +56,7 @@ func whatsappSend(ctx context.Context, link WhatsAppLink, groups *WhatsAppGroups digest := sha256.Sum256([]byte(text)) record := WhatsAppSend{ ID: id, + Image: image, Chat: chat, Digest: hex.EncodeToString(digest[:]), State: whatsappSending, @@ -61,7 +65,7 @@ func whatsappSend(ctx context.Context, link WhatsAppLink, groups *WhatsAppGroups held, found := state.Send(id) switch { - case found && (held.Chat != record.Chat || held.Digest != record.Digest): + case found && (held.Chat != record.Chat || held.Digest != record.Digest || held.Image != record.Image): return "", fmt.Errorf("the id %s belongs to another message: give a new id for each message", id) case found && held.State == whatsappSending: return "The send with this id started, but its result is unknown, because the daemon stopped during it. Ask the user to look at the chat in WhatsApp before you send the message again with a new id.", nil @@ -71,20 +75,39 @@ func whatsappSend(ctx context.Context, link WhatsAppLink, groups *WhatsAppGroups return "", errors.New("WhatsApp is disconnected, so nothing was sent: try again later") } + var data []byte + + if image != "" { + data, err = images.Read(image) + if err != nil { + return "", fmt.Errorf("read image to send: %w", err) + } + } + record.MessageID = link.client.GenerateMessageID() err = state.PutSend(record) if err != nil { return "", err } + message := &waE2E.Message{Conversation: new(text)} + if image != "" { + message, err = whatsappImageMessage(ctx, link.client, data, text) + } + sendCtx, cancel := context.WithTimeout(ctx, whatsappSendTimeout) defer cancel() - response, err := link.client.SendMessage(sendCtx, jid, &waE2E.Message{ - Conversation: new(text), - }, whatsmeow.SendRequestExtra{ - ID: record.MessageID, - }) + var response whatsmeow.SendResponse + + if err == nil { + response, err = link.client.SendMessage(sendCtx, jid, message, whatsmeow.SendRequestExtra{ + ID: record.MessageID, + }) + if err != nil && image != "" { + err = fmt.Errorf("send image: %w", whatsappImageSendError(err)) + } + } record.State = whatsappSent if err != nil { @@ -108,14 +131,14 @@ func whatsappSend(ctx context.Context, link WhatsAppLink, groups *WhatsAppGroups } // whatsappSendCheck returns the JID of chat, and an error when id, chat, or -// text cannot go to a send. The chat is a private chat or a group, and not the -// chat of the user with itself. -func whatsappSendCheck(id, chat, text string, own []string) (types.JID, error) { +// text cannot go to a send. Text can be empty only with an image. The chat is a +// private chat or a group, and not the chat of the user with itself. +func whatsappSendCheck(id, chat, text, image string, own []string) (types.JID, error) { jid, err := types.ParseJID(chat) switch { case strings.TrimSpace(id) == "" || len(id) > whatsappSendIDBytes: return types.EmptyJID, fmt.Errorf("the id must have 1 to %d bytes", whatsappSendIDBytes) - case strings.TrimSpace(text) == "" || len(text) > whatsappSendBytes: + case (strings.TrimSpace(text) == "" && image == "") || len(text) > whatsappSendBytes: return types.EmptyJID, fmt.Errorf("the text must have 1 to %d bytes", whatsappSendBytes) case err != nil || jid.User == "" || jid != jid.ToNonAD() || jid.String() != chat: return types.EmptyJID, fmt.Errorf("%q is not the ID of a WhatsApp chat: find the ID with whatsapp_find", chat) diff --git a/whatsapp_send_test.go b/whatsapp_send_test.go index 480e7db..65c2fc6 100644 --- a/whatsapp_send_test.go +++ b/whatsapp_send_test.go @@ -1,6 +1,7 @@ package main import ( + "context" "crypto/sha256" "encoding/hex" "errors" @@ -8,6 +9,10 @@ import ( "path/filepath" "strings" "testing" + "testing/synctest" + "time" + + "go.mau.fi/whatsmeow" "github.com/stretchr/testify/require" "go.mau.fi/whatsmeow/types" @@ -20,15 +25,15 @@ func TestWhatsAppSendSendsOnceForEachID(t *testing.T) { daemonStartTestWithWhatsApp(t, newModelFake(t), nil, whatsapp) whatsappTestConnect(t, fake) - answer, err := whatsapp.Send(t.Context(), "keys", whatsappTestAna.String(), "The keys are under the mat.") + answer, err := whatsapp.Send(t.Context(), "keys", whatsappTestAna.String(), "The keys are under the mat.", "") require.NoError(t, err) require.Equal(t, "Sent to Ana Souza at 12:05.", answer) - answer, err = whatsapp.Send(t.Context(), "keys", whatsappTestAna.String(), "The keys are under the mat.") + answer, err = whatsapp.Send(t.Context(), "keys", whatsappTestAna.String(), "The keys are under the mat.", "") require.NoError(t, err) require.Equal(t, "Sent to Ana Souza at 12:05. This is the answer of the first call with this id: nothing was sent again.", answer) - _, err = whatsapp.Send(t.Context(), "keys", whatsappTestAna.String(), "Another text.") + _, err = whatsapp.Send(t.Context(), "keys", whatsappTestAna.String(), "Another text.", "") require.EqualError(t, err, "the id keys belongs to another message: give a new id for each message") require.Equal(t, []WhatsAppFakeSend{ @@ -62,7 +67,7 @@ func TestWhatsAppSendToAGroup(t *testing.T) { daemonStartTestWithWhatsApp(t, newModelFake(t), nil, whatsapp) whatsappTestConnect(t, fake) - answer, err := whatsapp.Send(t.Context(), "cake", whatsappTestFamily.String(), "I bring the cake.") + answer, err := whatsapp.Send(t.Context(), "cake", whatsappTestFamily.String(), "I bring the cake.", "") require.NoError(t, err) require.Equal(t, "Sent to Family at 12:05.", answer) } @@ -75,7 +80,7 @@ func TestWhatsAppSendThatFails(t *testing.T) { whatsappTestConnect(t, fake) for range 2 { - answer, err := whatsapp.Send(t.Context(), "keys", whatsappTestAna.String(), "hi") + answer, err := whatsapp.Send(t.Context(), "keys", whatsappTestAna.String(), "hi", "") require.NoError(t, err) require.True(t, strings.HasPrefix(answer, "The send failed: server error for OWN."), answer) } @@ -96,7 +101,7 @@ func TestWhatsAppSendCutByAStop(t *testing.T) { State: whatsappSending, })) - answer, err := whatsapp.Send(t.Context(), "keys", whatsappTestAna.String(), "hi") + answer, err := whatsapp.Send(t.Context(), "keys", whatsappTestAna.String(), "hi", "") require.NoError(t, err) require.Contains(t, answer, "its result is unknown") require.Empty(t, fake.Sent()) @@ -153,7 +158,7 @@ func TestWhatsAppSendRefuses(t *testing.T) { for _, tc := range testCases { t.Run(tc.name, func(t *testing.T) { - _, err := whatsapp.Send(t.Context(), tc.id, tc.chat, tc.text) + _, err := whatsapp.Send(t.Context(), tc.id, tc.chat, tc.text, "") require.EqualError(t, err, tc.err) }) } @@ -167,12 +172,12 @@ func TestWhatsAppSendWhileDisconnected(t *testing.T) { whatsappTestConnect(t, fake) require.True(t, fake.Emit(&events.Disconnected{})) - _, err := whatsapp.Send(t.Context(), "keys", whatsappTestAna.String(), "hi") + _, err := whatsapp.Send(t.Context(), "keys", whatsappTestAna.String(), "hi", "") require.EqualError(t, err, "WhatsApp is disconnected, so nothing was sent: try again later") require.True(t, fake.Emit(&events.Connected{})) - answer, err := whatsapp.Send(t.Context(), "keys", whatsappTestAna.String(), "hi") + answer, err := whatsapp.Send(t.Context(), "keys", whatsappTestAna.String(), "hi", "") require.NoError(t, err) require.Equal(t, "Sent to +15550002222 at 12:05.", answer) } @@ -180,7 +185,7 @@ func TestWhatsAppSendWhileDisconnected(t *testing.T) { func TestWhatsAppSendWithoutALink(t *testing.T) { whatsapp, _, _ := whatsappStartTest(t, t.TempDir(), newTestClock()) - answer, err := whatsapp.Send(t.Context(), "keys", whatsappTestAna.String(), "hi") + answer, err := whatsapp.Send(t.Context(), "keys", whatsappTestAna.String(), "hi", "") require.NoError(t, err) require.Equal(t, whatsappNotLinked, answer) } @@ -208,3 +213,231 @@ func TestAgentSendsAWhatsAppMessage(t *testing.T) { client.WaitFor(t, "[result] Sent to Ana Souza at 12:05.\nSent.\n[done]") require.Len(t, fake.Sent(), 1) } + +func TestWhatsAppImageSendUploadsOnceAndKeepsCaption(t *testing.T) { + whatsapp, fake := whatsappStartTestLinked(t) + daemonStartTestWithWhatsApp(t, newModelFake(t), nil, whatsapp) + whatsappTestConnect(t, fake) + + picture := imageTestPNG(t, 4, 3) + name, err := whatsapp.inbox.images.Save(picture) + require.NoError(t, err) + + box := &Toolbox{whatsapp: whatsapp} + arguments := `{"id":"photo","chat":"15550002222@s.whatsapp.net","text":"The plan.","image":"` + name + `"}` + answer := box.run(t.Context(), "whatsapp_send", arguments) + require.Contains(t, answer, "Sent to") + require.Contains(t, box.run(t.Context(), "whatsapp_send", arguments), "nothing was sent again") + require.Equal(t, [][]byte{picture}, fake.Uploads()) + require.Len(t, fake.Sent(), 1) + require.Equal(t, "MSG1", fake.Sent()[0].id) + require.Empty(t, fake.Sent()[0].text) + + messages := fake.ImageSends() + require.Len(t, messages, 1) + + message := messages[0] + require.Equal(t, "The plan.", message.GetCaption()) + require.Equal(t, "image/png", message.GetMimetype()) + require.Equal(t, "https://example.com/image", message.GetURL()) + require.Equal(t, "/image", message.GetDirectPath()) + require.Equal(t, []byte("made-up key"), message.GetMediaKey()) + require.Equal(t, []byte("made-up hash"), message.GetFileSHA256()) + require.Equal(t, []byte("made-up encrypted hash"), message.GetFileEncSHA256()) + require.Equal(t, uint64(len(picture)), message.GetFileLength()) + + state, err := openWhatsAppState(whatsapp.state.path) + require.NoError(t, err) + + record, found := state.Send("photo") + require.True(t, found) + require.Equal(t, name, record.Image) + require.Equal(t, whatsappSent, record.State) + + again, err := whatsappSend(t.Context(), whatsapp.link(), whatsapp.groups, state, whatsapp.inbox.images, whatsapp.clock, "photo", whatsappTestAna.String(), "The plan.", name) + require.NoError(t, err) + require.Contains(t, again, "nothing was sent again") + require.Len(t, fake.Uploads(), 1) + require.Len(t, fake.Sent(), 1) + + other, err := whatsapp.inbox.images.Save(imageTestPNG(t, 1, 1)) + require.NoError(t, err) + + _, err = whatsapp.Send(t.Context(), "photo", whatsappTestAna.String(), "The plan.", other) + require.ErrorContains(t, err, "belongs to another message") + + _, err = whatsapp.Send(t.Context(), "photo", whatsappTestAna.String(), "The plan.", "") + require.ErrorContains(t, err, "belongs to another message") + require.Len(t, fake.Uploads(), 1) +} + +func TestWhatsAppImageSendAllowsEmptyCaptionAndRefusesUnknownImage(t *testing.T) { + whatsapp, fake := whatsappStartTestLinked(t) + daemonStartTestWithWhatsApp(t, newModelFake(t), nil, whatsapp) + whatsappTestConnect(t, fake) + + _, err := whatsapp.Send(t.Context(), "unknown", whatsappTestAna.String(), "A caption.", "../image.png") + require.ErrorContains(t, err, "invalid image name") + + _, found := whatsapp.state.Send("unknown") + require.False(t, found) + require.Empty(t, fake.Uploads()) + require.Empty(t, fake.Sent()) + + name, err := whatsapp.inbox.images.Save(imageTestPNG(t, 3, 2)) + require.NoError(t, err) + + _, err = whatsapp.Send(t.Context(), "no-caption", whatsappTestAna.String(), "", name) + require.NoError(t, err) + require.Len(t, fake.ImageSends(), 1) + require.Empty(t, fake.ImageSends()[0].GetCaption()) +} + +func TestWhatsAppImageFailureKeepsNoSecretsAndDoesNotRetry(t *testing.T) { + testCases := []struct { + name string + upload bool + }{ + { + name: "upload failure stores a safe cause", + upload: true, + }, + {name: "send failure stores a safe cause"}, + } + for _, tc := range testCases { + t.Run(tc.name, func(t *testing.T) { + whatsapp, fake := whatsappStartTestLinked(t) + failure := errors.New("https://example.com/media?token=secret /private-media/key phone=" + whatsappTestNumber) + if tc.upload { + fake.uploadErr = failure + } else { + fake.sendErr = failure + } + + daemonStartTestWithWhatsApp(t, newModelFake(t), nil, whatsapp) + whatsappTestConnect(t, fake) + + name, err := whatsapp.inbox.images.Save(imageTestPNG(t, 2, 1)) + require.NoError(t, err) + + for range 2 { + answer, err := whatsapp.Send(t.Context(), "failed-photo", whatsappTestAna.String(), "Private caption.", name) + require.NoError(t, err) + require.Contains(t, answer, "media send failed") + require.NotContains(t, answer, "secret") + require.NotContains(t, answer, "/private-media") + require.NotContains(t, answer, whatsappTestNumber) + require.NotContains(t, answer, "Private caption") + } + + require.Len(t, fake.Uploads(), 1) + + if tc.upload { + require.Empty(t, fake.Sent()) + } else { + require.Len(t, fake.Sent(), 1) + } + + record, found := whatsapp.state.Send("failed-photo") + require.True(t, found) + require.Equal(t, whatsappFailed, record.State) + require.NotContains(t, record.Answer, "secret") + }) + } +} + +func TestWhatsAppImageInProgressRecordDoesNotUploadAgain(t *testing.T) { + whatsapp, fake := whatsappStartTestLinked(t) + daemonStartTestWithWhatsApp(t, newModelFake(t), nil, whatsapp) + whatsappTestConnect(t, fake) + + name, err := whatsapp.inbox.images.Save(imageTestPNG(t, 2, 2)) + require.NoError(t, err) + + digest := sha256.Sum256([]byte("The plan.")) + require.NoError(t, whatsapp.state.PutSend(WhatsAppSend{ + ID: "interrupted-image", + Chat: whatsappTestAna.String(), + Digest: hex.EncodeToString(digest[:]), + Image: name, + State: whatsappSending, + })) + + answer, err := whatsapp.Send(t.Context(), "interrupted-image", whatsappTestAna.String(), "The plan.", name) + require.NoError(t, err) + require.Contains(t, answer, "its result is unknown") + require.Empty(t, fake.Uploads()) + require.Empty(t, fake.Sent()) +} + +func TestTelegramReceivedPhotoCanBeShownAndSentToWhatsApp(t *testing.T) { + whatsapp, fake := whatsappStartTestLinked(t) + daemonStartTestWithWhatsApp(t, newModelFake(t), nil, whatsapp) + whatsappTestConnect(t, fake) + + telegram := newTelegramFake(t) + picture := imageTestPNG(t, 5, 5) + telegram.Image("received", picture) + + api := newTelegramAPI(telegram.URL, telegramTestToken) + text, err := telegramImageText(t.Context(), api, whatsapp.inbox.images, &TelegramMessage{ + Caption: "The new plan.", + Photo: []TelegramPhoto{{ + FileID: "received", + Width: 5, + Height: 5, + }}, + }) + require.NoError(t, err) + + names := imageFiles(text) + require.Len(t, names, 1) + + bot, logs := telegramStartTestBot(t, telegram) + box := &Toolbox{ + whatsapp: whatsapp, + imageShow: &ImageShow{ + store: whatsapp.inbox.images, + bot: bot, + }, + } + require.NotContains(t, box.run(t.Context(), "image_show", `{"image":"`+names[0]+`","caption":"The plan."}`), "error:") + require.Contains(t, box.run(t.Context(), "whatsapp_send", `{"id":"forward","chat":"15550002222@s.whatsapp.net","text":"The plan.","image":"`+names[0]+`"}`), "Sent to") + require.Equal(t, [][]byte{picture}, fake.Uploads()) + require.Equal(t, "The plan.", fake.ImageSends()[0].GetCaption()) + + calls := telegram.Calls() + require.Equal(t, "sendPhoto", calls[len(calls)-1].method) + require.Equal(t, picture, calls[len(calls)-1].params["photo"]) + require.Empty(t, logs.String()) +} + +func TestWhatsAppImageUploadEndsWithItsCallerOrTimeout(t *testing.T) { + synctest.Test(t, func(t *testing.T) { + client := &WhatsAppUploadWaitFake{WhatsAppFake: &WhatsAppFake{}} + start := time.Now() + _, err := whatsappImageMessage(t.Context(), client, imageTestPNG(t, 1, 1), "") + require.ErrorIs(t, err, context.DeadlineExceeded) + require.Equal(t, 2*time.Minute, time.Since(start)) + + ctx, cancel := context.WithCancel(t.Context()) + defer cancel() + + cancel() + + start = time.Now() + _, err = whatsappImageMessage(ctx, client, imageTestPNG(t, 1, 1), "") + require.ErrorIs(t, err, context.Canceled) + require.Zero(t, time.Since(start)) + }) +} + +type WhatsAppUploadWaitFake struct { + *WhatsAppFake +} + +func (f *WhatsAppUploadWaitFake) Upload(ctx context.Context, _ []byte, _ whatsmeow.MediaType) (whatsmeow.UploadResponse, error) { + <-ctx.Done() + + return whatsmeow.UploadResponse{}, ctx.Err() +} diff --git a/whatsapp_state.go b/whatsapp_state.go index b1b5935..3d21b16 100644 --- a/whatsapp_state.go +++ b/whatsapp_state.go @@ -298,12 +298,14 @@ type WhatsAppStateFile struct { } // WhatsAppSend is the record of one send: the ID that the agent gave, the chat, -// the SHA-256 of the text, the WhatsApp message ID, the state, which is -// sending, sent, or failed, the answer of the tool, and the time of its start. +// the SHA-256 of the text, an optional stored image name, the WhatsApp message +// ID, the state, which is sending, sent, or failed, the answer of the tool, and +// the time of its start. type WhatsAppSend struct { ID string `json:"id"` Chat string `json:"chat"` Digest string `json:"digest"` + Image string `json:"image,omitempty"` MessageID string `json:"message_id"` State string `json:"state"` Answer string `json:"answer,omitempty"` -- 2.53.0 From 195c3fbb075383ac47bba310eb701b02774e8d10 Mon Sep 17 00:00:00 2001 From: "Anderson A. Giacomolli" Date: Tue, 10 Nov 2026 16:27:23 -0300 Subject: [PATCH 13/16] Keep test daemon sockets under the path limit The test daemon keeps its data, and so its socket, in a short directory under /tmp. t.TempDir holds the test name and the long TMPDIR of the development shell, so a test with a long name failed to listen on a socket path over 107 bytes, and bin/check failed in the shell unless TMPDIR was set short by hand. --- daemon_start_helper_test.go | 2 +- socket_helper_test.go | 24 ++++++++++++++++++++++++ 2 files changed, 25 insertions(+), 1 deletion(-) create mode 100644 socket_helper_test.go diff --git a/daemon_start_helper_test.go b/daemon_start_helper_test.go index 81399ce..c43ff9c 100644 --- a/daemon_start_helper_test.go +++ b/daemon_start_helper_test.go @@ -35,7 +35,7 @@ func daemonStartTestWithWhatsApp(t *testing.T, server *ModelFake, bot *TelegramB func daemonStartTestWithSchedules(t *testing.T, server *ModelFake, bot *TelegramBot, whatsapp *WhatsApp, schedules *Schedules, clock func() time.Time) (string, *Memory) { t.Helper() - dir := t.TempDir() + dir := socketTestDir(t) agent, err := openAgent(filepath.Join(dir, daemonAgentDirName)) require.NoError(t, err) diff --git a/socket_helper_test.go b/socket_helper_test.go new file mode 100644 index 0000000..44177bd --- /dev/null +++ b/socket_helper_test.go @@ -0,0 +1,24 @@ +package main + +import ( + "os" + "testing" + + "github.com/stretchr/testify/require" +) + +// socketTestDir returns a new short directory under /tmp for data that holds a +// daemon socket. t.TempDir holds the test name and the TMPDIR of the shell, so +// its paths can pass the 107-byte limit of a Unix socket path. +func socketTestDir(t *testing.T) string { + t.Helper() + + dir, err := os.MkdirTemp("/tmp", "onechat-") + require.NoError(t, err) + + t.Cleanup(func() { + require.NoError(t, os.RemoveAll(dir)) + }) + + return dir +} -- 2.53.0 From 7b8d8e8f4b57bd51fd6d0f0d926b821f06b0e1f0 Mon Sep 17 00:00:00 2001 From: "Anderson A. Giacomolli" Date: Tue, 10 Nov 2026 16:45:36 -0300 Subject: [PATCH 14/16] Log image and message sends Each WhatsApp send logs one line, message sent at info or message send failed at warning with its safe cause, and an image shown in Telegram logs image sent. A chat shows as a short hash, the same for received images and sends, so the log tells chats apart with no phone number, text, or caption. Before, a successful send left no trace in the journal. Issue: 441b3c7 --- image_show.go | 2 ++ image_show_test.go | 2 +- whatsapp.go | 11 ++++++++++- whatsapp_inbox.go | 4 +--- whatsapp_send.go | 8 ++++++-- whatsapp_send_test.go | 35 +++++++++++++++++++++++++++++++++-- 6 files changed, 53 insertions(+), 9 deletions(-) diff --git a/image_show.go b/image_show.go index 80204c1..0047029 100644 --- a/image_show.go +++ b/image_show.go @@ -31,6 +31,8 @@ func (s *ImageShow) Show(ctx context.Context, name, caption string) (string, err if err != nil { return "", fmt.Errorf("show Telegram image: %w", err) } + + s.bot.logger.Info("image sent") } return imageText([]string{name}, caption) + "\n" + path, nil diff --git a/image_show_test.go b/image_show_test.go index 65e422c..8df642a 100644 --- a/image_show_test.go +++ b/image_show_test.go @@ -38,7 +38,7 @@ func TestImageShowUsesStoredImageForTelegramAndTerminal(t *testing.T) { require.Equal(t, "A red point.", calls[0].params["caption"]) require.Equal(t, name, calls[0].params["photo_name"]) require.Equal(t, picture, calls[0].params["photo"]) - require.Empty(t, logs.String()) + require.Equal(t, "<6>image sent\n", logs.String()) } func TestImageShowRefusesUnknownNamesBeforeSending(t *testing.T) { diff --git a/whatsapp.go b/whatsapp.go index 57bfab0..4950d28 100644 --- a/whatsapp.go +++ b/whatsapp.go @@ -2,6 +2,7 @@ package main import ( "context" + "crypto/sha256" "database/sql" "errors" "fmt" @@ -466,7 +467,7 @@ func (w *WhatsApp) Find(ctx context.Context, name string) (string, error) { // Send returns the answer of whatsapp_send for text and an optional stored // image to chat, with the durable send id that whatsappSend describes. func (w *WhatsApp) Send(ctx context.Context, id, chat, text, image string) (string, error) { - return whatsappSend(ctx, w.link(), w.groups, w.state, w.inbox.images, w.clock, id, chat, text, image) + return whatsappSend(ctx, w.link(), w.groups, w.state, w.inbox.images, w.logger, w.clock, id, chat, text, image) } func (w *WhatsApp) sessionPath() string { @@ -1144,6 +1145,14 @@ func whatsappOwnIDs(ids ...string) []string { return own } +// whatsappChatHash returns a short hash of chat for the daemon log, so that a +// log line can tell chats apart without a phone number. +func whatsappChatHash(chat types.JID) string { + digest := sha256.Sum256([]byte(chat.ToNonAD().String())) + + return fmt.Sprintf("%x", digest[:6]) +} + // whatsappRedact replaces each of the own IDs own in text with OWN. func whatsappRedact(own []string, text string) string { for _, id := range own { diff --git a/whatsapp_inbox.go b/whatsapp_inbox.go index 9d2183e..cff43c7 100644 --- a/whatsapp_inbox.go +++ b/whatsapp_inbox.go @@ -2,7 +2,6 @@ package main import ( "context" - "crypto/sha256" "fmt" "log/slog" "slices" @@ -271,9 +270,8 @@ func (i *WhatsAppInbox) imageText(ctx context.Context, link WhatsAppLink, event name, err := whatsappImage(ctx, link.client, i.images, event.Message.GetImageMessage()) if err != nil { chat := event.Info.Chat.ToNonAD() - digest := sha256.Sum256([]byte(chat.String())) redact := append(slices.Clone(link.own), chat.User) - i.logger.Error("image receive failed", "chat", fmt.Sprintf("%x", digest[:6]), "err", whatsappRedact(redact, err.Error())) + i.logger.Error("image receive failed", "chat", whatsappChatHash(chat), "err", whatsappRedact(redact, err.Error())) text := "(image not downloaded)" if caption != "" { diff --git a/whatsapp_send.go b/whatsapp_send.go index b95a82e..d3d3cbc 100644 --- a/whatsapp_send.go +++ b/whatsapp_send.go @@ -6,6 +6,7 @@ import ( "encoding/hex" "errors" "fmt" + "log/slog" "slices" "strings" "time" @@ -39,11 +40,11 @@ const ( // during the send gives an answer that the result is unknown, and not a second // send. A disconnected WhatsApp sends nothing and makes no record. link gives // the device, the client, and the own IDs, groups the names of the groups, and -// state the send records. The clock gives the times of the records and the time +// state the send records, and logger gets one line for each send. The clock gives the times of the records and the time // zone of the answer. Image names are checked before the record is written, and // uploads start only after the record is durable. Text-only records keep their // text digest. -func whatsappSend(ctx context.Context, link WhatsAppLink, groups *WhatsAppGroups, state *WhatsAppState, images *ImageStore, clock func() time.Time, id, chat, text, image string) (string, error) { +func whatsappSend(ctx context.Context, link WhatsAppLink, groups *WhatsAppGroups, state *WhatsAppState, images *ImageStore, logger *slog.Logger, clock func() time.Time, id, chat, text, image string) (string, error) { if !link.linked { return whatsappNotLinked, nil } @@ -113,7 +114,10 @@ func whatsappSend(ctx context.Context, link WhatsAppLink, groups *WhatsAppGroups if err != nil { record.State = whatsappFailed record.Answer = "The send failed: " + whatsappRedact(link.own, err.Error()) + "." + logger.Warn("message send failed", "chat", whatsappChatHash(jid), "image", image != "", "err", whatsappRedact(append(slices.Clone(link.own), jid.User), err.Error())) } else { + logger.Info("message sent", "chat", whatsappChatHash(jid), "image", image != "") + name := groups.name(ctx, link, jid.String()) if jid.Server != types.GroupServer { name = whatsappContactName(ctx, link.device, jid, "") diff --git a/whatsapp_send_test.go b/whatsapp_send_test.go index 65c2fc6..85b7d15 100644 --- a/whatsapp_send_test.go +++ b/whatsapp_send_test.go @@ -88,6 +88,37 @@ func TestWhatsAppSendThatFails(t *testing.T) { require.Len(t, fake.Sent(), 1) } +func TestWhatsAppSendLogsEachSendWithoutTextOrNumbers(t *testing.T) { + dir := t.TempDir() + whatsappTestSession(t, dir, true) + + whatsapp, fake, logs := whatsappStartTest(t, dir, newTestClock()) + daemonStartTestWithWhatsApp(t, newModelFake(t), nil, whatsapp) + whatsappTestConnect(t, fake) + + name, err := whatsapp.inbox.images.Save(imageTestPNG(t, 4, 3)) + require.NoError(t, err) + + _, err = whatsapp.Send(t.Context(), "keys", whatsappTestAna.String(), "The keys are under the mat.", "") + require.NoError(t, err) + + _, err = whatsapp.Send(t.Context(), "plan", whatsappTestAna.String(), "The plan.", name) + require.NoError(t, err) + + fake.sendErr = errors.New("server error for " + whatsappTestNumber) + _, err = whatsapp.Send(t.Context(), "late", whatsappTestAna.String(), "Late.", "") + require.NoError(t, err) + + chat := whatsappChatHash(whatsappTestAna) + require.Contains(t, logs.String(), "<6>message sent chat="+chat+" image=false\n") + require.Contains(t, logs.String(), "<6>message sent chat="+chat+" image=true\n") + require.Contains(t, logs.String(), "<4>message send failed chat="+chat+" image=false: server error for OWN\n") + + for _, secret := range []string{"keys", "plan", "Late", whatsappTestNumber, whatsappTestAna.User} { + require.NotContains(t, logs.String(), secret) + } +} + func TestWhatsAppSendCutByAStop(t *testing.T) { whatsapp, fake := whatsappStartTestLinked(t) daemonStartTestWithWhatsApp(t, newModelFake(t), nil, whatsapp) @@ -254,7 +285,7 @@ func TestWhatsAppImageSendUploadsOnceAndKeepsCaption(t *testing.T) { require.Equal(t, name, record.Image) require.Equal(t, whatsappSent, record.State) - again, err := whatsappSend(t.Context(), whatsapp.link(), whatsapp.groups, state, whatsapp.inbox.images, whatsapp.clock, "photo", whatsappTestAna.String(), "The plan.", name) + again, err := whatsappSend(t.Context(), whatsapp.link(), whatsapp.groups, state, whatsapp.inbox.images, discardLog(), whatsapp.clock, "photo", whatsappTestAna.String(), "The plan.", name) require.NoError(t, err) require.Contains(t, again, "nothing was sent again") require.Len(t, fake.Uploads(), 1) @@ -409,7 +440,7 @@ func TestTelegramReceivedPhotoCanBeShownAndSentToWhatsApp(t *testing.T) { calls := telegram.Calls() require.Equal(t, "sendPhoto", calls[len(calls)-1].method) require.Equal(t, picture, calls[len(calls)-1].params["photo"]) - require.Empty(t, logs.String()) + require.Equal(t, "<6>image sent\n", logs.String()) } func TestWhatsAppImageUploadEndsWithItsCallerOrTimeout(t *testing.T) { -- 2.53.0 From 59948b44c0fe2165b90e3aa0d18646f6c3d74af3 Mon Sep 17 00:00:00 2001 From: "Anderson A. Giacomolli" Date: Tue, 10 Nov 2026 17:47:30 -0300 Subject: [PATCH 15/16] Reflow comments over the reflow width bin/reflow wraps each changed paragraph again, so that no comment line passes 80 columns with tabs as 8 columns, as the docs skill asks. Ten lines were over, from edits that skipped the reflow. Only comments change. --- memory.go | 6 +++--- model.go | 6 +++--- subagent_run.go | 9 +++++---- subagent_run_test.go | 3 ++- telegram_markdown.go | 4 ++-- telegram_test.go | 7 ++++--- whatsapp_find.go | 4 ++-- whatsapp_send.go | 8 ++++---- 8 files changed, 25 insertions(+), 22 deletions(-) diff --git a/memory.go b/memory.go index 602b86c..1b734c0 100644 --- a/memory.go +++ b/memory.go @@ -485,9 +485,9 @@ func (m *Memory) dispatch() { } // addMessage builds the leaf of message when it needs no model call, or else -// adds the leaf to the queue. Then it appends the leaf to the views. A crash can -// stop an append after the leaf build and before the view save, so a leaf that -// is built already stays as it is, and restoreQueue settles its parent. +// adds the leaf to the queue. Then it appends the leaf to the views. A crash +// can stop an append after the leaf build and before the view save, so a leaf +// that is built already stays as it is, and restoreQueue settles its parent. func (m *Memory) addMessage(message Message) error { built, err := m.tree.Built(0, message.ID) if err != nil { diff --git a/model.go b/model.go index 6ddd300..cc77f9a 100644 --- a/model.go +++ b/model.go @@ -41,9 +41,9 @@ const ( var errModelCallLimit = errors.New("call limit reached") // modelEndpoint returns the URL and the authorizer of the endpoint in config. -// The ChatGPT backend uses the token file in dir. The OpenAI API uses the key in -// OPENAI_API_KEY, and there is no fallback from one to the other. All clients -// share the authorizer, so one refresh runs at a time. +// The ChatGPT backend uses the token file in dir. The OpenAI API uses the key +// in OPENAI_API_KEY, and there is no fallback from one to the other. All +// clients share the authorizer, so one refresh runs at a time. func modelEndpoint(config Config, dir string) (string, ModelAuthorizer, error) { if config.Endpoint == configEndpointAPI { key := os.Getenv("OPENAI_API_KEY") diff --git a/subagent_run.go b/subagent_run.go index 891766c..faa6207 100644 --- a/subagent_run.go +++ b/subagent_run.go @@ -424,8 +424,9 @@ func (r *SubagentRuns) workPath() string { } // run runs the chains of run and reports and logs its end with the length of -// the run, unless Close ended it. A chain follows while messages wait after the final answer of the chain before -// it, and no chain failed. A chain that Stop ends gives the reply "stopped". +// the run, unless Close ended it. A chain follows while messages wait after the +// final answer of the chain before it, and no chain failed. A chain that Stop +// ends gives the reply "stopped". func (r *SubagentRuns) run(ctx context.Context, run *SubagentRun) { defer r.workers.Done() defer run.cancel() @@ -552,8 +553,8 @@ func (r *SubagentRuns) take(run *SubagentRun) []Message { // SubagentRun is one run of a subagent: the view that its requests get, its // effort, the end of its context, the messages of Tell that wait for a request, -// whether Stop ended it, and the time it started. The mutex of SubagentRuns protects the messages -// and the stop. +// whether Stop ended it, and the time it started. The mutex of SubagentRuns +// protects the messages and the stop. type SubagentRun struct { subagent *Subagent view []string diff --git a/subagent_run_test.go b/subagent_run_test.go index c5ad299..71c96ea 100644 --- a/subagent_run_test.go +++ b/subagent_run_test.go @@ -29,7 +29,8 @@ func TestSpawnRunsASubagentInTheBackground(t *testing.T) { last := subagentTestLastItem(call) switch { case subagentTestCall(call) && last["type"] == "function_call_output": - // The report comes after the reply of the turn of the spawn. + // The report comes after the reply of the turn of the + // spawn. test.events.WaitFor(t, "reply", 1) modelWriteEvents(w, modelCompletedEvent([]any{modelMessageItem("There are 3 files.")}, 100, 0, 5)) case subagentTestCall(call): diff --git a/telegram_markdown.go b/telegram_markdown.go index a978036..fc68634 100644 --- a/telegram_markdown.go +++ b/telegram_markdown.go @@ -6,8 +6,8 @@ import ( ) var ( - // telegramTagEscaper writes the characters that rich Markdown reads as HTML - // as HTML entities. + // telegramTagEscaper writes the characters that rich Markdown reads as + // HTML as HTML entities. telegramTagEscaper = strings.NewReplacer("&", "&", "<", "<") // telegramTagUnescaper reverses telegramTagEscaper. diff --git a/telegram_test.go b/telegram_test.go index 84f41c8..dee692d 100644 --- a/telegram_test.go +++ b/telegram_test.go @@ -145,7 +145,8 @@ func TestBotShowsOnlyTheStartAndTheEndOfASubagent(t *testing.T) { last := subagentTestLastItem(call) switch { case subagentTestCall(call): - // The report comes after the reply of the turn of the spawn. + // The report comes after the reply of the turn of the + // spawn. require.Eventually(t, func() bool { return slices.Contains(telegram.Sent(), "Started.") }, 10*time.Second, time.Millisecond) @@ -271,8 +272,8 @@ func TestBotTriesAgainAfterFailures(t *testing.T) { require.Contains(t, logs.String(), "<6>receive back after=") require.Contains(t, logs.String(), "<4>send failed retry=1s: 429") - // The bot logs the end of the outage after the send returns, so the line - // can come after the fake server has the message. + // The bot logs the end of the outage after the send returns, so the + // line can come after the fake server has the message. require.Eventually(t, func() bool { return strings.Contains(logs.String(), "<6>send back after=") }, 10*time.Second, time.Millisecond) diff --git a/whatsapp_find.go b/whatsapp_find.go index b825a3e..9e3cbf8 100644 --- a/whatsapp_find.go +++ b/whatsapp_find.go @@ -112,8 +112,8 @@ func whatsappFold(text string) string { return strings.ToLower(strings.TrimSpace(folded)) } -// whatsappFindAnswer returns the answer of whatsappFind for name with matches. Without -// groups, the answer says that it has no groups. +// whatsappFindAnswer returns the answer of whatsappFind for name with matches. +// Without groups, the answer says that it has no groups. func whatsappFindAnswer(name string, matches []WhatsAppMatch, groups bool) string { slices.SortFunc(matches, func(a, b WhatsAppMatch) int { return cmp.Or(strings.Compare(whatsappFold(a.name), whatsappFold(b.name)), strings.Compare(a.id, b.id)) diff --git a/whatsapp_send.go b/whatsapp_send.go index d3d3cbc..b3bbe95 100644 --- a/whatsapp_send.go +++ b/whatsapp_send.go @@ -40,10 +40,10 @@ const ( // during the send gives an answer that the result is unknown, and not a second // send. A disconnected WhatsApp sends nothing and makes no record. link gives // the device, the client, and the own IDs, groups the names of the groups, and -// state the send records, and logger gets one line for each send. The clock gives the times of the records and the time -// zone of the answer. Image names are checked before the record is written, and -// uploads start only after the record is durable. Text-only records keep their -// text digest. +// state the send records, and logger gets one line for each send. The clock +// gives the times of the records and the time zone of the answer. Image names +// are checked before the record is written, and uploads start only after the +// record is durable. Text-only records keep their text digest. func whatsappSend(ctx context.Context, link WhatsAppLink, groups *WhatsAppGroups, state *WhatsAppState, images *ImageStore, logger *slog.Logger, clock func() time.Time, id, chat, text, image string) (string, error) { if !link.linked { return whatsappNotLinked, nil -- 2.53.0 From 12bd9e96f3b3e67290484e3829461e054110db27 Mon Sep 17 00:00:00 2001 From: "Anderson A. Giacomolli" Date: Tue, 10 Nov 2026 17:48:48 -0300 Subject: [PATCH 16/16] Check the width of comment lines bin/check fails on a Go comment line over 80 columns, counted as bin/reflow counts them, with tabs as 8 columns. gofumpt does not wrap comments, and the line length linter would also count code lines, which the go-code skill keeps on one line, so ten long comment lines had passed every check. --- bin/check | 9 +++++++++ 1 file changed, 9 insertions(+) diff --git a/bin/check b/bin/check index 73256cb..e8e6e97 100755 --- a/bin/check +++ b/bin/check @@ -13,6 +13,15 @@ if [ -n "$unformatted" ]; then exit 1 fi +# Comment lines stay within the width of bin/reflow, with tabs as 8 columns. +long=$(files '*.go' | while read -r file; do + expand -t8 "$file" | awk -v file="$file" 'length > 80 && /^[[:space:]]*\/\//{print file ":" FNR}' +done) +if [ -n "$long" ]; then + printf 'comment lines over 80 columns, run bin/reflow:\n%s\n' "$long" >&2 + exit 1 +fi + go vet ./... golangci-lint run go test -race -short ./... -- 2.53.0