From: Anderson A. Giacomolli Date: Tue, 10 Nov 2026 15:03:38 +0000 (-0300) Subject: Send terminal images by local path X-Git-Url: http://git.giacomolli.uk/onechat/commitdiff_plain/refs/heads/master?ds=sidebyside;hp=855a45e6448537c3eed1ad6ec7f1ca9d6caf91fc Send terminal images by local path Accept /image PATH with an optional caption and quoted paths with spaces. Resolve relative paths in the client and let the daemon store the image before it submits the marker and caption. The socket carries only the path, so image bytes never enter its events or the chat log. --- diff --git a/.agents/skills/docs/SKILL.md b/.agents/skills/docs/SKILL.md index b802f77..2f7f3dc 100644 --- a/.agents/skills/docs/SKILL.md +++ b/.agents/skills/docs/SKILL.md @@ -9,6 +9,12 @@ description: Docs and comment rules. Use before you write any prose. - `README.rst` is the entry point. It says what OneChat is, its status, and how to build, run, and check it. Keep it short. +- Write the README for the user: what to set up, which commands to run, + and what the agent does, in the user's words. A new feature adds at + most a few lines to it. +- Keep internals out of the README: internal files and their formats, + locks, queues, recovery after a stop, limits, and edge cases. Put them + in the Go doc comments of the code that owns them. - Do not add other documentation files or a `docs/` folder. Document commands and flags in the command help, and code contracts and design reasons in Go doc comments. diff --git a/README.rst b/README.rst index 49963a1..1c344f4 100644 --- a/README.rst +++ b/README.rst @@ -3,31 +3,23 @@ ######### OneChat is one AI agent in a single chat that never ends. A daemon owns -the agent and its files, and a terminal client and a Telegram bot talk -to it. It is in early development. Linux is supported. +the agent and its memory, and you talk to it from a terminal client, +Telegram, and through your WhatsApp chats. It is in early development. +Linux is supported. -**************** - Data directory -**************** - -OneChat keeps its data and its token file in one directory. It uses the -first of these: - -#. ``ONECHAT_DIR``, when it is set. -#. ``$XDG_DATA_HOME/onechat``, when ``XDG_DATA_HOME`` is an absolute - path. A relative value is ignored. -#. ``~/.local/share/onechat``. With no home directory, OneChat exits - with an error. +******* + Setup +******* -OneChat creates the directory with mode 0700. +Data directory +============== -*************** - Configuration -*************** +OneChat keeps all its data in one directory, with mode 0700. It uses +``ONECHAT_DIR`` when it is set, else ``$XDG_DATA_HOME/onechat``, else +``~/.local/share/onechat``. The development shell sets ``ONECHAT_DIR`` +to ``.devenv/onechat`` in the checkout. -``config.json`` in the data directory holds the settings. OneChat does -not create it. A missing file or field takes the default, and an unknown -field is an error. +Optional settings go in ``config.json`` in the data directory: .. code:: json @@ -36,155 +28,60 @@ field is an error. "compactor": {"model": "gpt-6.1-sol", "effort": "high"} } -``endpoint`` - ``chatgpt`` (the default) sends model calls to the ChatGPT backend - with the subscription login. ``api`` sends them to the OpenAI API - with the key in ``OPENAI_API_KEY``, which must then be set. With - ``chatgpt``, OneChat ignores ``OPENAI_API_KEY``. +``endpoint`` is ``chatgpt`` (the ChatGPT subscription, the default) or +``api`` (the OpenAI API, with ``OPENAI_API_KEY`` set). ``compactor`` is +the model and effort of the memory summaries. -``compactor`` - The model and the reasoning effort of the summaries. The defaults are - ``gpt-6.1-sol`` and ``high``. - -******* - Login -******* - -The ``chatgpt`` endpoint needs a login with a ChatGPT account: +Login +===== .. code:: console onechat login -The command shows a link and a code. Open the link on any device, enter -the code, and approve the login. Device code login must be allowed in -the ChatGPT security settings. OneChat writes the tokens to -``auth.json`` in the data directory, with mode 0600, and refreshes them -before they expire. The file belongs to OneChat only. Do not share it -with the Codex CLI, because each program rotates the refresh token and -logs the other out. +Open the link on any device and enter the code. Device code login must +be allowed in the ChatGPT security settings. Do not share the login with +the Codex CLI: each program logs the other out. -******* - Agent -******* +Telegram +======== -Start the daemon, and then the client in another terminal: +#. Make a bot with BotFather and keep its token. +#. Find your numeric Telegram user ID. +#. Write both to ``telegram.json`` in the data directory, with mode + 0600: -.. code:: console +.. code:: json - onechat daemon - onechat + {"token": "123456:ABC-token-from-BotFather", "owner": 123456789} -``onechat`` with no command is the same as ``onechat chat``. ``onechat -help`` lists the commands. - -The daemon owns the root agent in ``agent/`` in the data directory and -listens on the Unix socket ``onechat.sock`` there. In the client, each -line is a message to the agent, the line ``/stop`` interrupts the -running turn, the line ``/stop NAME`` stops the subagent NAME, and -Ctrl-D ends the client. On terminal output, the client shows ``> `` -after each event while idle and clears it before the next event. An -event can interrupt a line while the user types. The client shows the -replies, the tool calls, and the first lines of each tool result as they -happen. Replies and subagent reports render Markdown with styles and -wrapped tables only when the output is a terminal, and stay unchanged in -a pipe or file. It also shows each message that the user sends from -another client, with the client as a mark, such as ``[telegram] text``. -When a subagent starts, the client shows ``[NAME] started``, and it -shows the whole report of the subagent, which starts with ``[NAME]``, -but not the tool calls of the subagent. The daemon is the only process -that uses the token file while it runs, so ``onechat login`` and -``onechat probe`` refuse to start until it stops. - -The agent runs commands with bash as the user in its working directory, -with no sandbox. It has the rights of the user, also on the token file. - -When the user asks for it, the agent starts subagents for long tasks. -Each subagent has a name, such as ``GreenOtter``, and runs in the -background while the agent answers other messages. It gets the view of -the agent and its task, and it runs its tools as the agent does, in -``agent/subagents/NAME/work``. A task in a repository gives the path of -the repository. The steps of a subagent go to its own log in -``agent/subagents/NAME``, not to the chat, and its final reply comes -back to the agent as one message that starts with ``[NAME]``. The agent -can send a subagent more messages: a running subagent reads them between -its tool calls, and a finished one goes on from its log, and its reply -comes back in the same way. When the user stops a subagent, the chat -gets a note, and the report of the subagent is logged, but it starts no -turn. Each model call of a subagent has the name in the ``agent`` field -of ``agent/usage.jsonl``. ``agent/work.json`` holds the names of the -running subagents. A subagent that runs when the daemon stops goes on at -the next start: the chat gets the note that the daemon restarted, and -the subagent gets the same note and goes on from its log. - -``agent/config.json`` holds the settings of the agent. The first start -writes it with the defaults and a random ID. Do not change the ID: the -prompt cache key comes from it. +Start the daemon and write to the bot in a private chat. It answers only +you. -.. code:: json +WhatsApp +======== - { - "name": "OneChat", - "id": "0123456789abcdef0123456789abcdef", - "model": "gpt-6.1-sol", - "effort": "medium", - "workdir": "/home/user" - } +Link your WhatsApp account as a linked device from the terminal client +or Telegram: + +``/whatsapp pair PHONE`` + Gives a pairing code. On the phone, open Linked devices, Link a + device, Link with phone number instead, and enter the code. -``workdir`` is an absolute path, and its default is the home directory. -``agent/instructions.md`` holds the user's instructions: who they are, -how their files are organized, and how they want work done. The prompt -ends with them, and a change applies at the next call. The agent can -propose a change to the file, and it makes the change only after the -user agrees. - -A permanent child agent lives in ``agent/children/NAME/`` with the same -files as the root. The root's ``agent_create(name, responsibility)`` -tool creates and starts it with the root's model, effort, and working -directory, and a separate random ID. A name describes the area, such as -``House`` or ``Calendar``. It is one word of 2 to 32 ASCII letters, -starts with a capital letter, and cannot name the root, an existing -child, or a subagent. The daemon opens all children at startup. A -missing or invalid child config stops startup. Only one level of -children is supported. Remove a child by hand while the daemon is -stopped. The root's turn state lists each child and its responsibility. -The root passes area messages from mixed chats with sender, date, and -words, and asks the user before it sends a child's draft to WhatsApp. - -Each child has its own memory, model client, usage file, subagents, -schedules, and turn loop. Its prompt limits work to its area, asks for -only reports that need the parent or the user, and permits an empty -reply. Children have no WhatsApp tools and draft replies instead of -sending them. Clients address only the root. A child's saved schedules -and subagents can resume at startup. The ready log line lists the child -names. - -A message from a parent to its own child enters the child's log as -``user`` and wakes it. A child's final reply reaches the parent as -``work`` with the prefix ``[NAME] `` and wakes it. An empty reply sends -nothing. A busy recipient takes the message between tool calls or in its -next turn. ``agent/messages.json``, with mode 0600, keeps messages until -their recipient finishes its turn. A daemon stop preserves queued -messages and reports for the next start without a second log entry. An -interrupted turn can repeat its model calls and tools after restart. -Backups lock this queue with the agent files, including children made -during the daemon run. The root uses ``tell`` for both children and -subagents. ``zoom("Name")`` gives a child's view or a subagent's whole -log. ``spawn`` starts short-lived subagents, not permanent children. - -************** - User service -************** - -The Nix package includes ``share/systemd/user/onechat.service``. It runs -``onechat daemon`` from that package, starts at login when enabled, and -restarts 5 seconds after a failure. A stop sends SIGTERM to the daemon -and allows up to 120 seconds for cleanup. Its output goes to the user -journal. The service has the rights of the user, as the terminal daemon -does. - -Build a package with a separate result link. Keep this link so that Nix -does not collect the package while the service uses it: +``/whatsapp status`` + Shows the link and the routes of chats to child agents. + +``/whatsapp unlink`` + Ends the link. + +The session file gives full access to the account. If it leaks, remove +the device in Linked devices on the phone. + +User service +============ + +Run the daemon as a systemd user service. Keep the ``result-service`` +link, so that Nix keeps the package: .. code:: console @@ -192,112 +89,41 @@ does not collect the package while the service uses it: systemctl --user link --force "$PWD/result-service/share/systemd/user/onechat.service" systemctl --user edit onechat.service -In the editor, set the data directory with this drop-in. Replace -``DATA_DIR`` with an absolute path to the data directory. Set the same -``ONECHAT_DIR`` for each terminal client. Without a drop-in, the data -directory rules above apply to the environment of the user manager, not -to the development shell. +In the editor, set the data directory, and the tools that the agent may +use in its shell: .. code:: ini [Service] Environment="ONECHAT_DIR=DATA_DIR" + Environment="PATH=TOOLS_PATH" -The service does not read the environment of an open terminal. If the -agent needs tools from a shell, set ``Environment="PATH=TOOLS_PATH"`` in -the same drop-in, with ``TOOLS_PATH`` replaced by the complete search -path, including bash. For the API endpoint, use -``EnvironmentFile=ENV_FILE``, with ``ENV_FILE`` replaced by an absolute -path to a mode 0600 file that sets ``OPENAI_API_KEY``. Keep secrets out -of the unit and Nix sources. - -Use the existing data directory, or copy the development data before the -first service start. For a copy, first stop the terminal daemon with -Ctrl-C and wait for it to exit. Replace ``DATA_DIR`` below with a new, -empty destination. If the development shell uses a custom -``ONECHAT_DIR``, replace ``.devenv/onechat`` with that source. Copy the -whole directory so that ``auth.json``, ``telegram.json``, -``config.json``, ``agent/``, and ``whatsapp/`` stay together, including -the WhatsApp session and its database files. The copy preserves the file -modes and the source as a backup. Do not copy or use the same data from -two running daemons. - -.. code:: console - - install -d -m 0700 DATA_DIR - cp -a .devenv/onechat/. DATA_DIR/ - -After the terminal daemon exits and the data is ready, enable and start -the service: +Then start it, and let it run while you are logged out: .. code:: console systemctl --user daemon-reload systemctl --user enable --now onechat.service - systemctl --user status onechat.service - ONECHAT_DIR=DATA_DIR ./result-service/bin/onechat - journalctl --user -u onechat.service -f + loginctl enable-linger "$USER" -After a pull, update the package and unit with: +After a ``git pull``, ``bin/service-update`` builds and restarts it. +Never run two daemons on the same data. -.. code:: console - - git pull - bin/service-update - -The script builds with the same ``result-service`` link, reloads the -user manager, and restarts the service only if it is active. The unit -link follows ``result-service``, so no new link is needed. - -The service starts with the user manager at login, not at boot without a -login. To stop it, or also remove its start at login: +Backups +======= .. code:: console - systemctl --user stop onechat.service - systemctl --user disable onechat.service - -******************** - Backup and restore -******************** - -Back up the running daemon to one mode 0600 ``.tar.gz`` archive. The -command prints its path and size in bytes. With no daemon, it fails. + onechat backup BACKUP_DIR/onechat.tar.gz + ONECHAT_DIR=EMPTY_DIR onechat restore BACKUP_DIR/onechat.tar.gz -.. code:: console +A backup runs while the daemon runs. Restore needs a stopped daemon and +an empty directory. **An archive holds your logins, the bot token, and +the WhatsApp link**, so keep it private. - onechat backup BACKUP_FILE.tar.gz - ONECHAT_DIR=EMPTY_DATA_DIR onechat restore BACKUP_FILE.tar.gz - -The daemon flushes memory and holds its write locks during the agent -copy. It also flushes and locks the child agents and all subagent logs. -It copies the WhatsApp session with SQLite ``VACUUM INTO``, not as raw -database files. It then compresses the copy and installs the archive -through a temporary file and a rename. Use a backup path outside the -data directory. - -The archive includes configuration, instructions, the log, tree, -indexes, views, layout, schedules, subagents, children, Telegram offset, -tokens, and WhatsApp state and session. It excludes sockets, lock files, -``probe/``, and the ``agent-backup-*`` import copies. Links and special -files fail, rather than enter the archive. **The archive holds -secrets**, including login tokens, the Telegram bot token, and the -WhatsApp link. Keep it private. There is no encryption or remote copy. - -Restore needs a stopped daemon and an empty destination. It checks the -token lock and validates the archive before it installs files, with -their modes. It rejects unsafe paths, links, and special files. A failed -restore can leave ``auth.lock`` in the destination. Use another empty -directory for a retry. Do not run the source and restored WhatsApp links -at the same time. Check the restored log and view before you use it in -place of the source. - -For a daily local archive with the last 14 kept, save this script as -``BACKUP_SCRIPT``. Replace ``BACKUP_DIR`` with a private archive -directory outside the data directory. Replace ``ONECHAT_BINARY`` with -the absolute path through the stable ``result-service/bin/onechat`` -package link. Make the archive directory with mode 0700 before you -enable the timer. +For a daily backup with the last 14 kept, save this script, with +``BACKUP_DIR`` replaced by a private directory outside the data +directory: .. code:: sh @@ -305,385 +131,118 @@ enable the timer. set -eu umask 077 cd BACKUP_DIR - ONECHAT_BINARY backup "onechat-$(date +%F).tar.gz" - find . -maxdepth 1 -type f -name 'onechat-????-??-??.tar.gz' -printf '%f\n' | - sort -r | tail -n +15 | - while IFS= read -r file; do - rm -- "$file" - done - -Save these units as ``onechat-backup.service`` and -``onechat-backup.timer`` in the systemd user unit directory. Replace -``DATA_DIR`` and ``BACKUP_SCRIPT`` with absolute paths. The service uses -the daemon unit from User service above. - -.. code:: ini + onechat backup "onechat-$(date +%F).tar.gz" + ls -1 onechat-????-??-??.tar.gz | sort -r | tail -n +15 | xargs -r rm -- - [Unit] - Description=Back up OneChat data - Requires=onechat.service - After=onechat.service +and run it from a ``Type=oneshot`` user service with the same +``ONECHAT_DIR`` and an ``OnCalendar=daily`` timer. - [Service] - Type=oneshot - Environment="ONECHAT_DIR=DATA_DIR" - UMask=0077 - ExecStart=/bin/sh BACKUP_SCRIPT - -.. code:: ini - - [Unit] - Description=Back up OneChat each day - - [Timer] - OnCalendar=daily - Persistent=true - - [Install] - WantedBy=timers.target - -Enable the timer with the user manager. A failed backup leaves the -existing archives in place and skips removal of old archives. +********* + Talking +********* .. code:: console - systemctl --user daemon-reload - systemctl --user enable --now onechat-backup.timer - systemctl --user list-timers onechat-backup.timer - -************ - Daemon log -************ - -The daemon writes short events with the part prefix and ordered -``key=value`` attributes. An error cause comes last. Normal events use -info, failures that can be retried or lose nothing use warning, and -failures that lose data or stop work use error. - -With ``JOURNAL_STREAM`` set, each line carries a syslog priority without -a time stamp or level word. The user journal can show only warnings and -errors: - -.. code:: console - - journalctl --user -u onechat -p warning + onechat -The summary warning names its node. A schedule run has ``late`` only -when it ran late: +``onechat`` connects to the running daemon. Each line is a message, and +replies show with Markdown styles and tables. These lines are commands: -.. code:: text +``/stop`` + Interrupts the running turn. - <4>onechat: summary failed level=0 index=0: fake failure - <6>onechat: schedule ran id=1 - <6>onechat: schedule ran id=2 late=1m0s +``/stop NAME`` + Stops the subagent NAME. -Outside the journal, each line starts with the time stamp and level. +``/image PATH [caption]`` + Sends a JPEG, PNG, or WebP image. Quote a path with spaces. -********** - Telegram -********** +``/whatsapp ...`` + The WhatsApp commands above. -The daemon also connects the agent to a Telegram bot when -``telegram.json`` exists in the data directory. It polls the Bot API for -new messages, so it needs no open port. +In Telegram, write as in the client, also with photos and image files. +``onechat help`` lists all commands. -#. Make a bot with BotFather and keep its token. +********************* + What the agent does +********************* -#. Find your numeric Telegram user ID. +Memory + The agent remembers the whole chat as summaries that it can open + again, so it never starts over. ``agent/instructions.md`` holds your + lasting preferences. The agent can propose a change to it and makes + the change only after you agree. -#. Write the token and the ID to ``telegram.json``, and give the file - mode 0600, because the token is a secret. The daemon refuses the file - with any other mode. +Tools + The agent runs shell commands as you, in its working directory, with + no sandbox. -#. Start the daemon, open a private chat with the bot, and write. +Subagents + For long tasks it starts subagents in the background, such as + ``GreenOtter``, and gets their reports as messages. -.. code:: json +Child agents + A child is a permanent agent for one area of your life, such as + ``House``, with its own memory. The root creates it, talks to it, and + hears from it only what needs you. Children never send WhatsApp + messages; they draft replies for you to approve. - {"token": "123456:ABC-token-from-BotFather", "owner": 123456789} +WhatsApp + The agent reads your chats in batches and wakes for a private message + or a mention of you. It finds chats and sends messages only when you + ask. It proposes to route a chat that is only about one area to the + child of that area, and sets the route after you agree. -The bot reads only the private chat of the owner and ignores all other -updates. Each text message goes to the agent as a user message, the line -``/stop`` interrupts the running turn, and the line ``/stop NAME`` stops -the subagent NAME. The bot sends the replies in parts of at most 4096 -characters, with their Markdown as Telegram formats: bold, italic, -strikethrough, code, code blocks, links, and headings as bold text. When -Telegram refuses the formats of a part, the bot sends the part again as -plain text. Each error goes as a short message in plain text. It shows -the typing action while a turn runs. It does not send the tool calls. -For a subagent it sends only ``[NAME] started`` and ``[NAME] finished``, -``[NAME] stopped``, or ``[NAME] failed``, because the reply of the agent -carries the content of the report. It sends each message that the user -types in the terminal client, with the mark ``[terminal]``. - -``agent/telegram-offset.json`` holds the ID of the next update. The bot -moves it past an update only after the message is durable in the log, so -a restart of the daemon does not lose a message. A crash just after the -write of a message can make the message repeat once. A network failure -or a rate limit makes the bot wait and try again, and does not stop the -daemon. - -********** - WhatsApp -********** - -The daemon connects to the WhatsApp account of the user as a linked -device. The terminal client and Telegram take three commands: +Images + Send a photo in Telegram or with ``/image``, and the agent sees it. + Its memory keeps what the image shows, and it can look at an older + image again. -``/whatsapp pair PHONE`` - Links the account of PHONE, the number with its country code, such as - ``+15550001111``. The daemon answers with the steps and then with a - pairing code in its own message: in WhatsApp on the phone, open - Linked devices, tap Link a device, then tap Link with phone number - instead, and enter the code. The code expires in about 2 minutes, and - the same command gives a new code. When the link succeeds, the daemon - answers with the linked number. Linked devices on the phone shows the - device as OneChat. The name applies at the pairing, so an existing - link keeps its old name until ``/whatsapp unlink`` and a new - ``/whatsapp pair PHONE``. +Schedules + Ask for reminders and timed tasks, such as "every weekday at 8, tell + me what my groups talked about". Each run uses the quota, so runs are + at least 5 minutes apart. -``/whatsapp status`` - Shows the link: not linked, linked and connected, or linked and - disconnected, with the times of the connection and the WhatsApp - routes. - -``/whatsapp unlink`` - Ends the link and deletes the session. When WhatsApp is disconnected, - the phone does not get the logout, so remove the device in Linked - devices on the phone. - -The answers go only to the client that asked, and the commands do not go -to the agent, so the agent never gets the phone number or the code. The -chat gets only the notes that WhatsApp is linked or unlinked. - -The session is ``whatsapp/session.db`` in the data directory, with mode -0600 in a directory with mode 0700. It gives full access to the WhatsApp -account, so the daemon refuses the file when its mode is not 0600. The -shell of the agent runs as the user, so it can read the session. If the -session leaks, remove the linked device in WhatsApp on the phone, and -the session stops working. Without the file, WhatsApp is not linked. - -The daemon reads the text messages of the private chats and the groups, -from other people and from the user. It does not read media, reactions, -edits, deletions, view-once and disappearing messages, or the history -from before the link. The messages of one chat form one batch until the -chat is quiet for 20 seconds, for 2 minutes or 50 messages at the most. -Each batch goes to the chat of the agent as one work message that starts -with ``[WhatsApp]``, with the name and the ID of the chat, and the time, -the sender, and the text of each message. The messages of the user show -as ``You``. ``whatsapp/state.json``, with mode 0600, keeps the batches -that are not in the chat yet, so they survive a restart, and the records -of the sends. A crash just after a batch goes to the chat can make the -batch repeat once. - -A batch of a private chat with a message of another person starts a -turn, and so does a batch of a group with a message that mentions the -user or replies to a message of the user. The other batches go to the -chat with no turn, and the agent sees them at its next turn. A batch -that starts a turn ends with a line that tells the agent to end its turn -with an empty reply when nothing in the batch needs the user. No client -shows an empty reply, so such a batch gives no message in Telegram. The -terminal client shows each batch, and Telegram shows none. - -The phone number of the user never goes to the chat: a mention of the -user shows as ``@you``, the number shows as ``OWN``, and the chat of the -user with itself is not read. But the IDs of the other chats go to the -chat and to the model, because the agent needs them to send. The ID of a -private chat holds the phone number of the contact, such as -``15550002222@s.whatsapp.net``, and the ID of a group can hold the -number of the person who made it. - -While the daemon is stopped or disconnected, the WhatsApp servers keep -the messages for the linked device and send them at the next connection. -They keep them only for a limited time, so a long stop can lose -messages. These messages form batches by their original times, as live -messages do. The daemon waits for the end of this catch-up, or one -minute at the most, and then gives all its unrouted batches to the root -in one turn, not one turn for each chat. Routed batches wake their -child. The batches of ``whatsapp/state.json`` at the start wait for the -first catch-up in the same way. - -The agent finds a chat with the tool ``whatsapp_find``: it gives the -contacts and the groups whose name holds a text, with no regard to case -and accents, with their IDs. The names of the contacts come from the -session, and the names of the groups from WhatsApp, kept in memory for -one hour. The names arrive from the phone some minutes after the link, -and until then the tool says that they are not ready. The agent sends a -text message with the tool ``whatsapp_send``, only when the user asks -for it. Each send has an ID that the agent gives: a repeated call with -the same ID gives the answer of the first call and sends nothing. The -state file keeps the last 1000 send records. When the daemon stops -during a send, the result is not known, and the tool tells the agent to -ask the user to look at the chat in WhatsApp. Without a link, both tools -tell the agent to ask the user to run ``/whatsapp pair PHONE``. Only the -agent has the WhatsApp tools, not its subagents. - -Each batch of a group goes to the chat, also with no turn, and the -memory tree summarizes it with model calls, which use the quota. - -The root manages source routes with ``route(source, agent)``, -``unroute(source)``, and ``routes()``, only after the user agrees. A -source is ``whatsapp:CHAT-ID``, with the ID from ``whatsapp_find`` or a -``[WhatsApp]`` message, and the agent must be its child. The root keeps -the map in ``agent/routes.json`` with mode 0600. Children and subagents -have no route tools. There is no command to change routes. Each routed -batch enters only its child's log and wakes it, including a group batch -with no mention. The durable agent queue prevents a second log entry -after a restart. An unrouted source or a missing child uses the root. A -queued batch whose child is removed while stopped goes to the root. - -When the session holds a linked device, the daemon connects at its -start. If the first connection fails, it tries again after 2 seconds, -then doubles the wait up to 5 minutes. The inbox and commands stay -active during each wait. An unlink or a logout stops these retries. -After a network failure of an open connection, the library connects -again by itself. When a disconnection lasts 10 minutes, all clients get -one notice, and one more when the connection comes back. When WhatsApp -ends the connection for good, for example because another program uses -the same session, all clients get a notice at once. When the user -removes the device in the phone, the daemon deletes the session and -tells all clients one time. - -************************* - Import WhatsApp history -************************* - -Stop the daemon and finish all subagents before an import. Export each -chat from WhatsApp as a ZIP with ``_chat.txt``. Keep the chat name in -the ZIP name, such as ``WhatsApp Chat - Example.zip``. Read the plan -first: +Import + Old WhatsApp chats can join the memory in date order. Stop the + daemon, export the chat as a ZIP, and run the plan before the import. + The import rebuilds the target's memory with model calls: .. code:: console - onechat import -n CHAT.zip OTHER.zip - onechat import CHAT.zip OTHER.zip - -``-n`` makes no model call and prints the chats, chat-day batches, local -dates, cutoff, text bytes, merged message count, and estimated calls. -The import keeps multiline text, system lines, and omitted-media -markers. It maps the session's profile name to ``You``. Without that -name, it keeps the senders as written and says so in the plan. An exact, -unambiguous local contact or batch name supplies the chat ID when known. -No network lookup is made. - -Each chat and local day becomes one work message marked ``(imported)``. -Only messages before the first message of the earliest live WhatsApp -batch for the same chat enter the import. Live times have minute -precision, so the cutoff excludes that whole minute. A chat-day already -imported is skipped. Overlapping chat-days in the supplied ZIPs are -refused. Review the exports before import: their text goes to the model -as chat history. - -The import merges the full log in date order and builds its tree with -real compactor calls, the agent prompt, and the agent cache key. It uses -a limit of 1,500 calls and the probe's 15-quota-point limit, and prints -the same usage report. Its estimate is about 1.3 calls per merged -message, not a cost guarantee. A stopped or failed run leaves the agent -unchanged. A new run starts over. Success keeps the other agent files -and renames the old directory to ``agent-backup-TIMESTAMP`` before it -installs the new one. Keep the backup until the new memory is checked, -then start the daemon. Message IDs change to their positions in the -merged log. - -*********** - Schedules -*********** - -The agent sets reminders and timed tasks when the user asks for them, -such as "remind me tomorrow at 9" or "every weekday at 8, tell me what -my WhatsApp groups talked about". A schedule has a text, the task for -the agent, and either one time or a cron expression. The agent also -lists the schedules with their next times, and removes one when the user -asks. The root and its children have the schedule tools, not their -subagents. Each schedule records its owner and wakes only that agent. - -A cron expression has five fields: minute, hour, day of month, month, -and day of week, where 0 and 7 are Sunday. A field is a comma list of -``*``, ``N``, and ``N-M``, and ``*`` and ``N-M`` can have a step ``/S``, -as in ``0 8 * * 1-5`` or ``*/15 9-17 * * *``. When both day fields start -with something other than ``*``, a day matches when either field -matches. Names such as ``MON``, macros such as ``@daily``, the -characters ``? L W #``, and ``N/S`` are refused. The runs must be at -least 5 minutes apart, because each run is a full turn that uses the -quota, so ``*/5 * * * *`` is the shortest schedule. - -The times use the time zone of the machine. A time that the clock skips -at the start of daylight saving time runs later by the length of the -skip. A time that the clock repeats at its end runs once, at its first -occurrence. - -At its time, a schedule goes to the chat as a work message that starts -with ``[Schedule ID]``, with its time and its text, and it starts a -turn. The message ends with a line that tells the agent to end its turn -with an empty reply when nothing needs the user. The terminal client -shows the message, and Telegram shows only the reply. After its run, a -one-time schedule goes, and a repeating schedule gets its next time. - -``agent/schedules.json``, with mode 0600, holds the schedules. Each -change writes it before the tool answers, so the schedules survive a -restart. A schedule that came due while the daemon was stopped runs once -at the next start, with its original time and a line that says that the -run is late. A repeating schedule that missed several times runs once. A -crash just after a run goes to the chat can make the run repeat once. - -******* - Probe -******* + onechat import -n [--agent NAME] CHAT.zip + onechat import [--agent NAME] CHAT.zip -``onechat probe LOG COUNT [BUILDS]`` measures the compactor on a log. It -appends the first COUNT messages of LOG, one JSON object ``{"kind", -"text"}`` on each line, to a new chat in ``probe/`` in the data -directory, and builds its tree with real model calls. The calls use the -quota of the account. The run stops at the first failed call, after 660 -calls, or when the used percent of a quota window grows by more than 15 -points. At the end it reports the calls per message, the cached share of -the input tokens, the retries, the line sizes, and the quota at the -first and the last call. BUILDS, from 1 to 8, limits the model calls at -once in the run. +***** + Log +***** -A test writes a generated log of readable made-up messages: +The daemon writes short events with ``key=value`` attributes to the user +journal, at three levels: info, warning for a failure that is retried or +loses nothing, and error for a failure that loses data or stops work. .. code:: console - ONECHAT_PROBE_LOG=probe.jsonl go test -run TestProbeLogHasTheSyntheticMix + journalctl --user -u onechat -f + journalctl --user -u onechat -p warning ************* Development ************* Nix supplies the Go toolchain and all tools. Enter the development shell -with direnv: - -.. code:: console - - direnv allow - -Or enter it directly: - -.. code:: console - - nix develop - -The shell sets ``ONECHAT_DIR`` to ``.devenv/onechat`` in the checkout, -unless it is already set. Development data goes there, and Git ignores -``.devenv``. - -Build and run: +with ``direnv allow`` or ``nix develop``. Then: .. code:: console nix build - ./result/bin/onechat - -``onechat version`` prints ``yyyy-mm-dd-gitshort`` for a Nix build, adds -``-dirty`` for a dirty tree, and prints ``dev`` for a direct Go build. - -Format and check all files: - -.. code:: console - + ./result/bin/onechat version bin/format bin/check nix flake check -``bin/check`` and ``nix build`` run a short replay of the synthetic log. -``nix flake check`` also runs the full replay of 100,000 messages. +``onechat version`` prints ``yyyy-mm-dd-gitshort`` for a Nix build and +``dev`` for a direct Go build. ``bin/check`` runs the linters and a +short replay of a synthetic log, and ``nix flake check`` also runs the +full replay of 100,000 messages. ``onechat probe LOG COUNT`` measures +the memory summaries on a log with real model calls. diff --git a/agent.go b/agent.go index dbcabd2..97d6685 100644 --- a/agent.go +++ b/agent.go @@ -80,6 +80,16 @@ type Agent struct { config AgentConfig } +// Images returns the shared data-directory image store, also for a child. +func (a *Agent) Images() *ImageStore { + root := a.dir + if a.config.Parent != "" { + root = filepath.Dir(filepath.Dir(root)) + } + + return &ImageStore{dir: filepath.Join(filepath.Dir(root), imageDirName)} +} + // CacheKey returns the prompt cache key of every model call of the agent. func (a *Agent) CacheKey() string { return "onechat-" + a.config.ID diff --git a/backup_archive.go b/backup_archive.go index 9560cd1..2ec9352 100644 --- a/backup_archive.go +++ b/backup_archive.go @@ -270,7 +270,7 @@ func backupExcluded(name string) bool { return true } - if filepath.Base(name) == "lock" { + if filepath.Base(name) == "lock" || (first == imageDirName && strings.HasPrefix(filepath.Base(name), ".image-")) { return true } diff --git a/build_run_test.go b/build_run_test.go index b45745e..ef9c3c0 100644 --- a/build_run_test.go +++ b/build_run_test.go @@ -215,7 +215,7 @@ func newTestBuildRun(t *testing.T, server *ModelFake) *BuildRun { root := t.TempDir() usagePath := filepath.Join(root, "usage.jsonl") client := newModelClient(server.URL, ModelAPIKey(modelTestKey), "onechat-"+probeName, usagePath) - memory, err := openMemory(t.Context(), root, newTestClock().Now, newCompactor(client, probePrompt, defaultConfig().Compactor, promptTools()), discardLog()) + memory, err := openMemory(t.Context(), root, newTestClock().Now, newCompactor(client, probePrompt, defaultConfig().Compactor, promptTools(), &ImageStore{dir: t.TempDir()}), discardLog()) require.NoError(t, err) return &BuildRun{ diff --git a/chain.go b/chain.go index c4a4302..23c7dec 100644 --- a/chain.go +++ b/chain.go @@ -32,6 +32,7 @@ type Chain struct { tools *Toolbox request ModelRequest take func() []Message + images *ImageStore } // Run runs the chain from input, the input of its first request, and returns @@ -122,7 +123,12 @@ func (c *Chain) handle(ctx context.Context, input []any, response ModelResponse) input = append(input, outputs...) for _, message := range c.take() { - input = append(input, modelUserMessage(message.Text)) + item, err := imageUserMessage(c.images, message.Text) + if err != nil { + return nil, "", fmt.Errorf("read queued images: %w", err) + } + + input = append(input, item) } return input, "", nil @@ -141,7 +147,7 @@ type ChainFunctionCall struct { type ChainFunctionCallOutput struct { Type string `json:"type"` CallID string `json:"call_id"` - Output string `json:"output"` + Output any `json:"output"` } // chainInputItem returns an output item as an input item for the next request. diff --git a/command_test.go b/command_test.go index 3d89327..602424d 100644 --- a/command_test.go +++ b/command_test.go @@ -22,6 +22,12 @@ func TestHelpListsVersionCommand(t *testing.T) { require.Contains(t, commandOutput(t, "help"), " version show the build version\n") } +func TestHelpListsTheImportTargetFlag(t *testing.T) { + help := commandOutput(t, "help") + require.Contains(t, help, "import [-n] [--agent NAME] ZIP...") + require.Contains(t, help, "--agent NAME import into an existing direct child (default: root)") +} + func commandOutput(t *testing.T, args ...string) string { t.Helper() diff --git a/compactor.go b/compactor.go index 8a16c6e..2b23901 100644 --- a/compactor.go +++ b/compactor.go @@ -4,7 +4,6 @@ import ( "context" "errors" "fmt" - "slices" "strings" "unicode/utf8" ) @@ -20,12 +19,13 @@ var compactorRuler = strings.Repeat("-", nodeBytes) // client with model. instructions gives the system prompt of the agent at each // build. tools are the tools of that agent, so a build sends the same prompt // and tools as its turns. -func newCompactor(client *ModelClient, instructions func() (string, error), model ModelConfig, tools []ModelTool) *Compactor { +func newCompactor(client *ModelClient, instructions func() (string, error), model ModelConfig, tools []ModelTool, images *ImageStore) *Compactor { return &Compactor{ client: client, model: model, instructions: instructions, tools: tools, + images: images, } } @@ -37,17 +37,19 @@ func newCompactor(client *ModelClient, instructions func() (string, error), mode // // When the line is over nodeBytes, the next try sends the same message, plus // one more user message with a note with the size of the newest line and its -// cut at the limit. A try never sends the model's earlier reply or reasoning -// back, so every try begins with the same view and task and has about the same -// size. After at most compactorTries tries, the shortest line is the node. A -// few bytes over the limit is fine, because the view measures the real sizes. A -// call that fails before any line fails the build, and the core tries it again -// at the next message. +// cut at the limit. Only the first call of an image leaf sends its pixels. A +// try never sends the model's earlier reply or reasoning back, so every try +// begins with the same view and task and has about the same size. After at most +// compactorTries tries, the shortest line is the node. A few bytes over the +// limit is fine, because the view measures the real sizes. A call that fails +// before any line fails the build, and the core tries it again at the next +// message. type Compactor struct { client *ModelClient model ModelConfig instructions func() (string, error) tools []ModelTool + images *ImageStore } // Summarize returns the line of the node of task. @@ -62,7 +64,13 @@ func (c *Compactor) Summarize(ctx context.Context, task Task) (string, error) { return "", err } - first := []any{modelUserMessage(task.Context + "\n" + text)} + source := task.Context + "\n" + text + parts, err := imageParts(c.images, source, task.Images) + if err != nil { + return "", fmt.Errorf("read compaction images: %w", err) + } + + first := []any{modelUserParts(parts)} var lines []string @@ -70,7 +78,7 @@ func (c *Compactor) Summarize(ctx context.Context, task Task) (string, error) { for len(lines) < compactorTries { input := first if note != "" { - input = append(slices.Clip(first), modelUserMessage(note)) + input = []any{modelUserMessage(source), modelUserMessage(note)} } var response ModelResponse @@ -111,12 +119,17 @@ func (c *Compactor) Summarize(ctx context.Context, task Task) (string, error) { // compactorTask returns the task of a build, with its input. func compactorTask(task Task) (string, error) { if task.Level == 0 && len(task.Input) == 1 { - return fmt.Sprintf(`Compaction: compress message %d into one line of at most %d bytes + text := fmt.Sprintf(`Compaction: compress message %d into one line of at most %d bytes (about 70 words), the length of this ruler: %s %s -`, task.Index, nodeBytes, compactorRuler, task.Input[0]), nil +`, task.Index, nodeBytes, compactorRuler, task.Input[0]) + if len(task.Images) > 0 { + text += "\nKeep each [image FILE] name and describe its kind, visible text, amounts, names, and context." + } + + return text, nil } if task.Level > 0 && len(task.Input) == 2 { diff --git a/compactor_test.go b/compactor_test.go index 8540515..baa7f57 100644 --- a/compactor_test.go +++ b/compactor_test.go @@ -208,7 +208,7 @@ func newTestCompactor(t *testing.T, server *ModelFake) *Compactor { return newCompactor(client, compactorAdaPrompt, ModelConfig{ Model: "gpt-test", Effort: "high", - }, promptTools()) + }, promptTools(), &ImageStore{dir: t.TempDir()}) } // compactorInputTexts returns the text of each user message of call. Each diff --git a/daemon.go b/daemon.go index 2a93f1d..137cc6f 100644 --- a/daemon.go +++ b/daemon.go @@ -493,6 +493,27 @@ func daemonRunRequest(l *TurnLoop, whatsapp *WhatsApp, request SocketRequest, an _, err := l.Submit(terminalSource, request.Text) return err + case "image": + if !filepath.IsAbs(request.Path) { + return errors.New("use /image PATH [caption]") + } + + data, err := imageReadFile(request.Path) + if err != nil { + return fmt.Errorf("read terminal image: %w", err) + } + + name, err := l.agent.Images().Save(data) + if err != nil { + return fmt.Errorf("store terminal image: %w", err) + } + + _, err = l.Submit(terminalSource, imageText([]string{name}, request.Text)) + if err != nil { + return fmt.Errorf("submit terminal image: %w", err) + } + + return nil case "interrupt": if !l.Interrupt() { return errors.New("no turn runs") diff --git a/daemon_agent.go b/daemon_agent.go index 662291c..227d23b 100644 --- a/daemon_agent.go +++ b/daemon_agent.go @@ -55,7 +55,7 @@ func openDaemonAgents(ctx context.Context, dir, address string, auth ModelAuthor // one agent. All model calls use its cache key and usage file. func openDaemonAgent(ctx context.Context, agent *Agent, address string, auth ModelAuthorizer, compactor ModelConfig, clock func() time.Time, logger *slog.Logger) (*DaemonAgent, error) { client := newModelClient(address, auth, agent.CacheKey(), filepath.Join(agent.dir, "usage.jsonl")) - memory, err := openMemory(ctx, agent.dir, clock, newCompactor(client, agent.Prompt, compactor, agent.Tools()), logger) + memory, err := openMemory(ctx, agent.dir, clock, newCompactor(client, agent.Prompt, compactor, agent.Tools(), agent.Images()), logger) if err != nil { return nil, fmt.Errorf("open memory: %w", err) } diff --git a/go.mod b/go.mod index fcf4681..4643a53 100644 --- a/go.mod +++ b/go.mod @@ -5,6 +5,7 @@ go 1.26.5 require ( github.com/stretchr/testify v1.12.0 go.mau.fi/whatsmeow v0.0.0-20261107160454-79dc2a8c33df + golang.org/x/image v0.47.0 golang.org/x/sys v0.49.0 golang.org/x/text v0.43.0 modernc.org/sqlite v1.62.0 diff --git a/go.sum b/go.sum index 13d902b..16298eb 100644 --- a/go.sum +++ b/go.sum @@ -52,6 +52,8 @@ golang.org/x/crypto v0.58.0 h1:COkYLr4k7nDI4r6QtMML2AfzeIswWHAJthLS6K/M/54= golang.org/x/crypto v0.58.0/go.mod h1:Xh+kl5A+M0gMiWDytH2Hhr5Z7WbNCpv/eB/EFgfSk+w= golang.org/x/exp v0.0.0-20261014143641-f0251eb4aa3e h1:HgI6W1OolKK6tcV74is+rl9SMT0njsHlp0TbQiGXI/A= golang.org/x/exp v0.0.0-20261014143641-f0251eb4aa3e/go.mod h1:DBoxwWhjaW8DJJw/YeM4zpeOKMC1FMXSblsYy+1mFSw= +golang.org/x/image v0.47.0 h1:za/8gVz++PPZqmZJBNSISICN5Rrj5zixP/8K++fgmqo= +golang.org/x/image v0.47.0/go.mod h1:IkN5o4A7vq55vXN8FWjjnhg9Rs2GOtffqsSi9mWAKYE= golang.org/x/mod v0.42.0 h1:ICeL3t2L30ljSPHHxDnR4fqVcBt/zR2zJAb9juavmQg= golang.org/x/mod v0.42.0/go.mod h1:u5fzIrRInkL8MaO1cbRkUHT64v303YIT5KEqpNr7jRs= golang.org/x/net v0.61.0 h1:RR+6j/BTBCrggOLugxa+66R8FvoN6MHlzDlZIbN9YFw= diff --git a/image.go b/image.go new file mode 100644 index 0000000..4b23f7c --- /dev/null +++ b/image.go @@ -0,0 +1,180 @@ +package main + +import ( + "bytes" + "crypto/sha256" + "encoding/hex" + "errors" + "fmt" + "image" + _ "image/jpeg" + _ "image/png" + "io" + "os" + "path/filepath" + "strings" + + _ "golang.org/x/image/webp" +) + +const ( + imageDirName = "images" + imageMaxBytes = 20 << 20 +) + +var ( + errImageSize = errors.New("image exceeds 20 MB") + errImageFormat = errors.New("use a JPEG, PNG, or WebP image") +) + +// ImageStore holds immutable images shared by the root, children, and +// subagents. Names are the SHA-256 of the bytes with the decoded format as +// extension. Publication is atomic and does not replace an existing image. +type ImageStore struct { + dir string +} + +// Save validates and stores data, and returns its filename. Only JPEG, PNG, and +// WebP images up to 20 MB are accepted. +func (s *ImageStore) Save(data []byte) (name string, err error) { + format, err := imageFormat(data) + if err != nil { + return "", fmt.Errorf("check image: %w", err) + } + + sum := sha256.Sum256(data) + name = hex.EncodeToString(sum[:]) + "." + format + err = os.MkdirAll(s.dir, 0o700) + if err != nil { + return "", fmt.Errorf("create image directory: %w", err) + } + + file, err := os.CreateTemp(s.dir, ".image-*") + if err != nil { + return "", fmt.Errorf("create image file: %w", err) + } + + defer func() { + err = errors.Join(err, os.Remove(file.Name())) + }() + + _, err = file.Write(data) + if err == nil { + err = file.Sync() + } + + err = errors.Join(err, file.Close()) + if err != nil { + return "", fmt.Errorf("write image: %w", err) + } + + err = os.Link(file.Name(), filepath.Join(s.dir, name)) + if err != nil && !errors.Is(err, os.ErrExist) { + return "", fmt.Errorf("publish image: %w", err) + } + + return name, nil +} + +// Read returns the bytes of a stored image after checking its name and content. +func (s *ImageStore) Read(name string) ([]byte, error) { + if !imageName(name) { + return nil, errors.New("invalid image name") + } + + data, err := imageReadFile(filepath.Join(s.dir, name)) + if err != nil { + return nil, fmt.Errorf("read stored image: %w", err) + } + + format, err := imageFormat(data) + if err != nil { + return nil, fmt.Errorf("check stored image: %w", err) + } + + sum := sha256.Sum256(data) + if name != hex.EncodeToString(sum[:])+"."+format { + return nil, errors.New("image content does not match its name") + } + + return data, nil +} + +func imageFormat(data []byte) (string, error) { + if len(data) > imageMaxBytes { + return "", errImageSize + } + + config, format, err := image.DecodeConfig(bytes.NewReader(data)) + if err != nil || config.Width <= 0 || config.Height <= 0 { + return "", errImageFormat + } + + switch format { + case "jpeg": + return "jpg", nil + case "png", "webp": + return format, nil + default: + return "", errImageFormat + } +} + +func imageName(name string) bool { + hash, ext, found := strings.Cut(name, ".") + return found && len(hash) == 64 && strings.Trim(hash, "0123456789abcdef") == "" && (ext == "jpg" || ext == "png" || ext == "webp") +} + +func imageReadFile(path string) (data []byte, err error) { + file, err := os.Open(path) + if err != nil { + return nil, fmt.Errorf("open image: %w", err) + } + + defer func() { + err = errors.Join(err, file.Close()) + }() + + data, err = io.ReadAll(io.LimitReader(file, imageMaxBytes+1)) + if err != nil { + return nil, fmt.Errorf("read image: %w", err) + } + + if len(data) > imageMaxBytes { + return nil, errImageSize + } + + return data, nil +} + +// imageFiles reads only complete marker lines with valid store filenames. +func imageFiles(text string) []string { + var names []string + + for line := range strings.SplitSeq(text, "\n") { + name, found := strings.CutPrefix(line, "[image ") + if !found || !strings.HasSuffix(name, "]") { + continue + } + + name = strings.TrimSuffix(name, "]") + if imageName(name) { + names = append(names, name) + } + } + + return names +} + +func imageText(names []string, caption string) string { + lines := make([]string, 0, len(names)+1) + for _, name := range names { + lines = append(lines, "[image "+name+"]") + } + + if caption != "" { + lines = append(lines, caption) + } + + return strings.Join(lines, "\n") +} diff --git a/image_helper_test.go b/image_helper_test.go new file mode 100644 index 0000000..94788a2 --- /dev/null +++ b/image_helper_test.go @@ -0,0 +1,27 @@ +package main + +import ( + "bytes" + "image" + "image/color" + "image/png" + "testing" + + "github.com/stretchr/testify/require" +) + +func imageTestPNG(t *testing.T, width, height int) []byte { + t.Helper() + + picture := image.NewRGBA(image.Rect(0, 0, width, height)) + picture.Set(0, 0, color.RGBA{ + R: 255, + A: 255, + }) + + var data bytes.Buffer + + require.NoError(t, png.Encode(&data, picture)) + + return data.Bytes() +} diff --git a/image_model.go b/image_model.go new file mode 100644 index 0000000..6965142 --- /dev/null +++ b/image_model.go @@ -0,0 +1,90 @@ +package main + +import ( + "encoding/base64" + "errors" + "fmt" + "strings" +) + +// imageUserMessage attaches pixels only to messages selected as new by a turn. +func imageUserMessage(store *ImageStore, text string) (any, error) { + names := imageFiles(text) + if len(names) == 0 { + return modelUserMessage(text), nil + } + + parts, err := imageParts(store, text, names) + if err != nil { + return nil, fmt.Errorf("make image message: %w", err) + } + + return modelUserParts(parts), nil +} + +// imageParts is the shared wire format for image messages and zoom results. +// Tool results use this list directly as function_call_output.output. +func imageParts(store *ImageStore, text string, names []string) ([]ModelInputPart, error) { + parts := []ModelInputPart{{ + Type: "input_text", + Text: text, + }} + for _, name := range names { + if store == nil { + return nil, errors.New("image store is not available") + } + + data, err := store.Read(name) + if err != nil { + return nil, fmt.Errorf("read image part: %w", err) + } + + mime := "image/" + strings.TrimPrefix(name[64:], ".") + if strings.HasSuffix(name, ".jpg") { + mime = "image/jpeg" + } + + parts = append(parts, ModelInputPart{ + Type: "input_image", + ImageURL: "data:" + mime + ";base64," + base64.StdEncoding.EncodeToString(data), + }) + } + + return parts, nil +} + +// imageToolOutput leaves all results as text except zoom of one image message. +// Only the text goes to the log, never the data URLs. +func imageToolOutput(box *Toolbox, name, arguments, text string) (any, error) { + if name != "zoom" || strings.HasPrefix(text, "error:") { + return text, nil + } + + var args ToolZoomArguments + + err := toolDecodeArguments(arguments, &args) + if err != nil { + return nil, fmt.Errorf("decode image zoom: %w", err) + } + + if args.Name != nil || args.N != 1 { + return text, nil + } + + message, err := box.chat.Message(args.ID) + if err != nil { + return nil, fmt.Errorf("read image zoom: %w", err) + } + + names := imageFiles(message.Text) + if len(names) == 0 { + return text, nil + } + + parts, err := imageParts(box.images, text, names) + if err != nil { + return nil, fmt.Errorf("make image zoom: %w", err) + } + + return parts, nil +} diff --git a/image_model_test.go b/image_model_test.go new file mode 100644 index 0000000..53b0f31 --- /dev/null +++ b/image_model_test.go @@ -0,0 +1,199 @@ +package main + +import ( + "encoding/base64" + "encoding/json" + "net/http" + "path/filepath" + "strings" + "testing" + + "github.com/stretchr/testify/require" +) + +func TestNewTurnKeepsImagesInEveryChainRequestButNotLaterTurns(t *testing.T) { + server := newModelFake(t) + server.reply = func(w http.ResponseWriter, call ModelFakeCall) { + if call.number == 0 { + modelWriteEvents(w, modelCompletedEvent([]any{modelFunctionCallItem("fc_1", "call_1", "date", `{"id":0}`)}, 100, 0, 5)) + + return + } + + modelWriteEvents(w, modelCompletedEvent([]any{modelMessageItem("A red square.")}, 100, 0, 5)) + } + + test := newTestTurnLoop(t, server) + data := imageTestPNG(t, 3, 3) + name, err := test.agent.Images().Save(data) + require.NoError(t, err) + + marker := imageText([]string{name}, "What is this?") + _, err = test.loop.Submit(terminalSource, marker) + require.NoError(t, err) + test.events.WaitFor(t, "idle", 1) + require.Equal(t, 2, server.Calls()) + + first := server.Call(0).body["input"].([]any) + second := server.Call(1).body["input"].([]any) + require.Equal(t, first, second[:len(first)]) + + expected := "data:image/png;base64," + base64.StdEncoding.EncodeToString(data) + require.Contains(t, string(server.Call(0).raw), expected) + require.Contains(t, string(server.Call(1).raw), expected) + + _, err = test.loop.Submit(terminalSource, "Continue with text.") + require.NoError(t, err) + test.events.WaitFor(t, "idle", 2) + require.NotContains(t, string(server.Call(2).raw), "input_image") + require.NotContains(t, strings.Join(test.Log(t), "\n"), "base64") +} + +func TestImageArrivingDuringAChainJoinsItsNextRequest(t *testing.T) { + server := newModelFake(t) + arrived := make(chan struct{}) + release := make(chan struct{}) + server.reply = func(w http.ResponseWriter, call ModelFakeCall) { + if call.number == 0 { + close(arrived) + <-release + modelWriteEvents(w, modelCompletedEvent([]any{modelFunctionCallItem("fc_1", "call_1", "date", `{"id":0}`)}, 100, 0, 5)) + + return + } + + modelWriteEvents(w, modelCompletedEvent([]any{modelMessageItem("done")}, 100, 0, 5)) + } + + test := newTestTurnLoop(t, server) + _, err := test.loop.Submit(terminalSource, "Start.") + require.NoError(t, err) + <-arrived + + name, err := test.agent.Images().Save(imageTestPNG(t, 2, 2)) + require.NoError(t, err) + + _, err = test.loop.Submit(terminalSource, imageText([]string{name}, "Read this.")) + require.NoError(t, err) + close(release) + test.events.WaitFor(t, "idle", 1) + require.NotContains(t, string(server.Call(0).raw), "input_image") + require.Contains(t, string(server.Call(1).raw), "input_image") +} + +func TestLeafCompactionDescribesImagesOnceAndMergesOnlyText(t *testing.T) { + server := newModelFake(t) + store := &ImageStore{dir: filepath.Join(t.TempDir(), imageDirName)} + name, err := store.Save(imageTestPNG(t, 2, 2)) + require.NoError(t, err) + + server.reply = func(w http.ResponseWriter, call ModelFakeCall) { + text := "receipt, Ada, $12, red square" + if call.number == 0 { + text = strings.Repeat("detail ", 100) + } + + modelWriteEvents(w, modelCompletedEvent([]any{modelMessageItem(text)}, 100, 0, 5)) + } + + compactor := newTestCompactor(t, server) + compactor.images = store + task := Task{ + Level: 0, + Index: 0, + Context: "\n", + Input: []string{"user: " + imageText([]string{name}, "Read the receipt.")}, + Images: []string{name}, + } + text, err := compactor.Summarize(t.Context(), task) + require.NoError(t, err) + require.Equal(t, "receipt, Ada, $12, red square", text) + require.Contains(t, string(server.Call(0).raw), "input_image") + require.Contains(t, string(server.Call(0).raw), "kind, visible text, amounts, names, and context") + require.NotContains(t, string(server.Call(1).raw), "input_image") + + _, err = compactor.Summarize(t.Context(), Task{ + Level: 1, + Index: 0, + Context: "\n", + Input: []string{text, "user: next"}, + }) + require.NoError(t, err) + require.NotContains(t, string(server.Call(2).raw), "input_image") + + memory, err := openMemory(t.Context(), t.TempDir(), newTestClock().Now, compactor, discardLog()) + require.NoError(t, err) + t.Cleanup(func() { + require.NoError(t, memory.Close()) + }) + + _, err = memory.Append("user", imageText([]string{name}, "A short caption.")) + require.NoError(t, err) + require.NoError(t, memory.Settle(t.Context())) + require.Equal(t, 4, server.Calls()) + require.Contains(t, string(server.Call(3).raw), "input_image") + require.Contains(t, memory.View(), "receipt, Ada, $12") +} + +func TestZoomReturnsImagePartsButLogsOnlyText(t *testing.T) { + server := newModelFake(t) + server.reply = func(w http.ResponseWriter, call ModelFakeCall) { + if call.number == 0 { + modelWriteEvents(w, modelCompletedEvent([]any{modelFunctionCallItem("fc_1", "call_1", "zoom", `{"id":0,"n":1,"name":null}`)}, 100, 0, 5)) + + return + } + + modelWriteEvents(w, modelCompletedEvent([]any{modelMessageItem("A red square.")}, 100, 0, 5)) + } + + test := newTestTurnLoop(t, server) + name, err := test.agent.Images().Save(imageTestPNG(t, 2, 2)) + require.NoError(t, err) + + _, err = test.memory.Append("user", imageText([]string{name}, "Old image.")) + require.NoError(t, err) + + _, err = test.loop.Submit(terminalSource, "Zoom the old image.") + require.NoError(t, err) + test.events.WaitFor(t, "idle", 1) + require.NotContains(t, string(server.Call(0).raw), "input_image") + + input := server.Call(1).body["input"].([]any) + result := input[len(input)-1].(map[string]any) + require.Equal(t, "function_call_output", result["type"]) + + parts := result["output"].([]any) + require.Len(t, parts, 2) + require.Equal(t, "input_text", parts[0].(map[string]any)["type"]) + require.Equal(t, "input_image", parts[1].(map[string]any)["type"]) + require.Contains(t, parts[0].(map[string]any)["text"], "Old image.") + require.NotContains(t, strings.Join(test.Log(t), "\n"), "base64") +} + +func TestImageUserMessageKeepsTextPartsAndMarkerOrder(t *testing.T) { + store := &ImageStore{dir: t.TempDir()} + first, err := store.Save(imageTestPNG(t, 1, 2)) + require.NoError(t, err) + + second, err := store.Save(imageTestPNG(t, 2, 1)) + require.NoError(t, err) + + text := imageText([]string{first, second}, "Compare.") + item, err := imageUserMessage(store, text) + require.NoError(t, err) + + data, err := json.Marshal(item) + require.NoError(t, err) + + var message struct { + Role string `json:"role"` + Content []ModelInputPart `json:"content"` + } + + require.NoError(t, json.Unmarshal(data, &message)) + require.Equal(t, "user", message.Role) + require.Len(t, message.Content, 3) + require.Equal(t, text, message.Content[0].Text) + require.NotEqual(t, message.Content[1].ImageURL, message.Content[2].ImageURL) +} diff --git a/image_test.go b/image_test.go new file mode 100644 index 0000000..24007d7 --- /dev/null +++ b/image_test.go @@ -0,0 +1,112 @@ +package main + +import ( + "bytes" + "crypto/sha256" + "encoding/hex" + "os" + "path/filepath" + "sync" + "testing" + + "github.com/stretchr/testify/require" +) + +func TestImagesArePrivateSharedAndWrittenOnce(t *testing.T) { + root, err := openAgent(filepath.Join(t.TempDir(), "agent")) + require.NoError(t, err) + + child := &Agent{ + dir: filepath.Join(root.dir, "children", "House"), + config: AgentConfig{Parent: root.config.Name}, + } + require.Equal(t, root.Images().dir, child.Images().dir) + + data := imageTestPNG(t, 3, 2) + sum := sha256.Sum256(data) + expected := hex.EncodeToString(sum[:]) + ".png" + store := root.Images() + name, err := store.Save(data) + require.NoError(t, err) + require.Equal(t, expected, name) + + info, err := os.Stat(filepath.Join(store.dir, name)) + require.NoError(t, err) + require.Equal(t, os.FileMode(0o600), info.Mode().Perm()) + + var workers sync.WaitGroup + + for range 8 { + workers.Go(func() { + saved, saveErr := child.Images().Save(data) + require.NoError(t, saveErr) + require.Equal(t, name, saved) + }) + } + + workers.Wait() + + again, err := os.Stat(filepath.Join(store.dir, name)) + require.NoError(t, err) + require.True(t, os.SameFile(info, again)) + + files, err := os.ReadDir(store.dir) + require.NoError(t, err) + require.Len(t, files, 1) + + actual, err := store.Read(name) + require.NoError(t, err) + require.Equal(t, data, actual) +} + +func TestImageStoreRejectsLargeUnsupportedAndUnsafeImages(t *testing.T) { + store := &ImageStore{dir: filepath.Join(t.TempDir(), imageDirName)} + _, err := store.Save(bytes.Repeat([]byte{0}, imageMaxBytes+1)) + require.ErrorContains(t, err, "exceeds 20 MB") + + _, err = store.Save([]byte("not an image")) + require.ErrorContains(t, err, "JPEG, PNG, or WebP") + + _, err = store.Read("../secret.png") + require.ErrorContains(t, err, "invalid image name") + + _, err = os.Stat(store.dir) + require.ErrorIs(t, err, os.ErrNotExist) +} + +func TestBackupKeepsImagesAndTheirModes(t *testing.T) { + dir := t.TempDir() + store := &ImageStore{dir: filepath.Join(dir, imageDirName)} + data := imageTestPNG(t, 2, 2) + name, err := store.Save(data) + require.NoError(t, err) + require.NoError(t, os.WriteFile(filepath.Join(store.dir, ".image-incomplete"), []byte("partial"), 0o600)) + + stage := t.TempDir() + archive := filepath.Join(t.TempDir(), "images.tar.gz") + require.NoError(t, backupCopy(t.Context(), dir, stage, archive)) + + _, err = backupArchive(t.Context(), stage, archive) + require.NoError(t, err) + + restored := t.TempDir() + require.NoError(t, backupRestore(t.Context(), archive, restored)) + + actual, err := (&ImageStore{dir: filepath.Join(restored, imageDirName)}).Read(name) + require.NoError(t, err) + require.Equal(t, data, actual) + + info, err := os.Stat(filepath.Join(restored, imageDirName, name)) + require.NoError(t, err) + require.Equal(t, os.FileMode(0o600), info.Mode().Perm()) + + _, err = os.Stat(filepath.Join(restored, imageDirName, ".image-incomplete")) + require.ErrorIs(t, err, os.ErrNotExist) +} + +func TestImageMarkersKeepCaptionAndOnlySafeNames(t *testing.T) { + name := "aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa.png" + text := imageText([]string{name}, "Read the label.") + require.Equal(t, "[image "+name+"]\nRead the label.", text) + require.Equal(t, []string{name}, imageFiles(text+"\n[image ../secret.png]\ntext [image "+name+"]")) +} diff --git a/main.go b/main.go index efecde3..10ac995 100644 --- a/main.go +++ b/main.go @@ -18,14 +18,18 @@ Commands: restore FILE restore into an empty data directory chat talk to the agent in this terminal (default) daemon run the agent, Telegram, and WhatsApp - import [-n] ZIP... import WhatsApp history (stop the daemon first) + import [-n] [--agent NAME] ZIP... + import WhatsApp history (stop the daemon first) -n print the plan without model calls + --agent NAME import into an existing direct child (default: root) login log in to ChatGPT with a device code probe LOG COUNT [BUILDS] measure the compactor on a log, with real calls version show the build version help show this help Chat needs a running daemon. ONECHAT_DIR sets the data directory. +In chat, /image PATH [caption] sends a JPEG, PNG, or WebP image up to 20 MB. +Quote PATH with single or double quotes when it contains spaces. ` // version is set by the package build and stays dev for direct Go builds. diff --git a/memory.go b/memory.go index a240b6c..602b86c 100644 --- a/memory.go +++ b/memory.go @@ -607,6 +607,7 @@ func (m *Memory) task(key NodeKey) (Task, error) { } task.Input = []string{leafSource(message)} + task.Images = imageFiles(message.Text) return task, nil } @@ -781,12 +782,14 @@ func (m *Memory) buildBuiltByModel(key NodeKey) (bool, error) { // Task is the build of the node (Level, Index). Context is the compaction view // for the build. Input holds the source "kind: text" of a leaf, or the two -// child lines of a merge with their newlines as spaces. +// child lines of a merge with their newlines as spaces. Images holds the leaf +// image filenames and is empty for merges. type Task struct { Level int Index int Context string Input []string + Images []string } // MemoryViewFile is the content of view.json. diff --git a/model.go b/model.go index 507fbb1..6ddd300 100644 --- a/model.go +++ b/model.go @@ -73,22 +73,22 @@ func newModelClient(url string, auth ModelAuthorizer, cacheKey, usagePath string // modelUserMessage returns a user input item with one text part for each text. func modelUserMessage(texts ...string) any { - type part struct { - Type string `json:"type"` - Text string `json:"text"` - } - - content := make([]part, len(texts)) + content := make([]ModelInputPart, len(texts)) for n, text := range texts { - content[n] = part{ + content[n] = ModelInputPart{ Type: "input_text", Text: text, } } + return modelUserParts(content) +} + +// modelUserParts keeps text and image parts in one user message. +func modelUserParts(content []ModelInputPart) any { return struct { - Role string `json:"role"` - Content []part `json:"content"` + Role string `json:"role"` + Content []ModelInputPart `json:"content"` }{ Role: "user", Content: content, @@ -321,6 +321,13 @@ type ModelOutputItem struct { } `json:"content"` } +// ModelInputPart is a text or image part in a user message or tool result. +type ModelInputPart struct { + Type string `json:"type"` + Text string `json:"text,omitempty"` + ImageURL string `json:"image_url,omitempty"` +} + // ModelRequest is one model call. Input holds the input items in order, such as // modelUserMessage values or output items of an earlier response. Node names // the node that a compaction builds, or is nil. Turn names the request of a diff --git a/nix/package.nix b/nix/package.nix index bbd5fb6..41cd24f 100644 --- a/nix/package.nix +++ b/nix/package.nix @@ -10,7 +10,7 @@ buildGoModule { subPackages = [ "." ]; ldflags = [ "-X main.version=${version}" ]; - vendorHash = "sha256-HEYbx27w3s+x29/gDLUOCrkBJu9Ke1ue+Ms++LdHpyQ="; + vendorHash = "sha256-nRDUAGSXAK5/L2xyfDh3NbMOZmrGGEIm4vtgfWsxHYU="; # The flake check runs the full replay. checkFlags = [ "-short" ]; diff --git a/probe.go b/probe.go index b004829..384d0f8 100644 --- a/probe.go +++ b/probe.go @@ -103,7 +103,7 @@ func probe(ctx context.Context, args []string, out io.Writer) (err error) { client.LimitCalls(probeCallLimit) summarizer := ProbeSlotSummarizer{ - inner: newCompactor(client, probePrompt, config.Compactor, promptTools()), + inner: newCompactor(client, probePrompt, config.Compactor, promptTools(), &ImageStore{dir: filepath.Join(dir, imageDirName)}), slots: make(chan struct{}, builds), } diff --git a/prompt.go b/prompt.go index 36fda32..df6f17d 100644 --- a/prompt.go +++ b/prompt.go @@ -41,10 +41,10 @@ var promptEfforts = []string{"low", "medium", "high", "xhigh"} // one added paragraph on the user's instructions file. The reference names its // own agent, so promptText has the name of the agent and the kind reply in its // place. It describes only what OneChat can do, so it leaves out the spec text -// on images, computers, and computer tasks. Its paragraph on persistent mode -// tells the model that a final answer ends the turn. prompt adds the clause on -// the instructions of the user only when there are instructions. It holds no -// date and no state, so turns and compactions share its prompt cache. +// on computers and computer tasks. Its paragraph on persistent mode tells the +// model that a final answer ends the turn. prompt adds the clause on the +// instructions of the user only when there are instructions. It holds no date +// and no state, so turns and compactions share its prompt cache. const promptText = `You are {name}, an AI agent that works for one user in a single chat that never ends. Each call to you is a turn or a compaction: the view below is followed by the user's new message, or by a task starting "Compaction:". @@ -76,6 +76,9 @@ Tools: - zoom("Name") gives an agent's whole chat - date(id) gives the date and time of message id +An [image FILE] line names a stored image. New images include pixels for this +turn only. Use zoom(id, 1) to see an older image again. + # Turns Do the user's tasks yourself, with your tools. @@ -155,10 +158,10 @@ Set or remove a route only after the user agrees.` // promptSubagentText is the system prompt of a subagent, from the subagent // prompt of the reference design: its head, then the doc of the view, without -// the parts on devices and images. A subagent gets the view of the agent, then -// its whole log, so it goes on from its log after a tell or a restart. -// promptSubagent adds the clause on the instructions of the user only when -// there are instructions. +// the parts on devices. A subagent gets the view of the agent, then its whole +// log, so it goes on from its log after a tell or a restart. promptSubagent +// adds the clause on the instructions of the user only when there are +// instructions. const promptSubagentText = `You are a subagent of {name}, an AI agent that works for one user in a single chat that never ends. {name} gave you a task. Do it yourself, with your tools. @@ -189,7 +192,8 @@ was made from; zoom(id, 1) gives message id in full. Zoom whenever a summary only mentions something you need, such as what your last reply said, a decision, a past attempt or where a file is, before you act, guess or ask. date(id) gives the date and time of message id. zoom("Name") gives an agent's -whole chat.` +whole chat. An [image FILE] line names a stored image. Use zoom(id, 1) +to see its pixels.` // prompt returns the system prompt of the reference with one added paragraph on // the user's instructions file at instructionsPath. It uses the agent name and diff --git a/prompt_test.go b/prompt_test.go index c9cadc2..e9e87f3 100644 --- a/prompt_test.go +++ b/prompt_test.go @@ -18,6 +18,7 @@ func TestPromptNamesTheAgent(t *testing.T) { require.Contains(t, text, `- work: an agent's report, starting "[Name]"`) require.Contains(t, text, "- zoom(id, 1) gives message id whole\n- zoom(\"Name\") gives an agent's whole chat\n") require.NotContains(t, text, promptName) + require.Contains(t, text, "[image FILE]") // The old agent name is built from parts, so the source does not spell // it. @@ -26,7 +27,7 @@ func TestPromptNamesTheAgent(t *testing.T) { func TestPromptLeavesOutAbsentFeatures(t *testing.T) { text := prompt("Ada", "/AGENT/instructions.md", "") - for _, absent := range []string{"image", "computer", "device", "instructions at the end"} { + for _, absent := range []string{"computer", "device", "instructions at the end"} { require.NotContains(t, text, absent) } } @@ -50,7 +51,7 @@ func TestSubagentPromptNamesTheAgentAndEndsWithTheInstructions(t *testing.T) { require.True(t, strings.HasSuffix(text, "\n\n# The user's instructions\n\nI am Lena.")) require.NotContains(t, text, promptName) - for _, absent := range []string{"image", "computer", "device", "Compaction"} { + for _, absent := range []string{"computer", "device", "Compaction"} { require.NotContains(t, text, absent) } diff --git a/rebuild.go b/rebuild.go index 11c6bfc..598371c 100644 --- a/rebuild.go +++ b/rebuild.go @@ -26,7 +26,9 @@ const rebuildCallLimit = 1500 // removes the new directory and leaves the original directory unchanged. // Success copies the other agent files, adds this run's usage, and swaps the // directories, with the original kept as a timestamped backup. Message IDs are -// new positions in the merged log, but kind, text, and date stay exact. +// new positions in the merged log, but kind, text, and date stay exact. A child +// keeps its staging directory and backup outside children so that neither is +// read as a configured child. func rebuild(ctx context.Context, agent *Agent, added []DatedMessage, client *ModelClient, summarizer Summarizer, out io.Writer) (backup string, err error) { err = rebuildIdle(agent.dir) if err != nil { @@ -43,7 +45,14 @@ func rebuild(ctx context.Context, agent *Agent, added []DatedMessage, client *Mo return "", fmt.Errorf("merge rebuild messages: %w", err) } - root, err := os.MkdirTemp(filepath.Dir(agent.dir), ".agent-rebuild-") + parent := filepath.Dir(agent.dir) + backupDir := agent.dir + if agent.config.Parent != "" { + parent = filepath.Dir(parent) + backupDir = filepath.Join(parent, agent.config.Name) + } + + root, err := os.MkdirTemp(parent, ".agent-rebuild-") if err != nil { return "", fmt.Errorf("create rebuild directory: %w", err) } @@ -110,7 +119,7 @@ func rebuild(ctx context.Context, agent *Agent, added []DatedMessage, client *Mo return "", ctx.Err() } - backup = agent.dir + "-backup-" + time.Now().Format("20060102-150405.000000000") + backup = backupDir + "-backup-" + time.Now().Format("20060102-150405.000000000") err = os.Rename(agent.dir, backup) if err != nil { return "", fmt.Errorf("back up agent: %w", err) diff --git a/rebuild_test.go b/rebuild_test.go index fc4ff7e..104ec88 100644 --- a/rebuild_test.go +++ b/rebuild_test.go @@ -63,7 +63,7 @@ func TestRebuildMergesDatesAndKeepsAgentFiles(t *testing.T) { }, } - backup, err := rebuild(t.Context(), agent, added, client, newCompactor(client, agent.Prompt, defaultConfig().Compactor, agent.Tools()), &report) + backup, err := rebuild(t.Context(), agent, added, client, newCompactor(client, agent.Prompt, defaultConfig().Compactor, agent.Tools(), agent.Images()), &report) require.NoError(t, err) require.Equal(t, before, rebuildTestSnapshot(t, backup)) @@ -143,7 +143,7 @@ func TestRebuildFailureAndCancellationKeepOriginal(t *testing.T) { Kind: "work", Text: strings.Repeat("long ", 200), Date: "2025-01-01T00:00:00Z", - }}, client, newCompactor(client, agent.Prompt, defaultConfig().Compactor, agent.Tools()), &strings.Builder{}) + }}, client, newCompactor(client, agent.Prompt, defaultConfig().Compactor, agent.Tools(), agent.Images()), &strings.Builder{}) require.Error(t, err) require.Empty(t, backup) require.Equal(t, before, rebuildTestSnapshot(t, agent.dir)) @@ -179,7 +179,7 @@ func TestRebuildKeepsOriginalWhenAnInvisibleMergeFails(t *testing.T) { } client := newModelClient(server.URL, ModelAPIKey(modelTestKey), agent.CacheKey(), filepath.Join(t.TempDir(), "usage.jsonl")) - _, err := rebuild(t.Context(), agent, nil, client, newCompactor(client, agent.Prompt, defaultConfig().Compactor, agent.Tools()), &strings.Builder{}) + _, err := rebuild(t.Context(), agent, nil, client, newCompactor(client, agent.Prompt, defaultConfig().Compactor, agent.Tools(), agent.Images()), &strings.Builder{}) require.ErrorContains(t, err, "summary tree has unfinished nodes") require.Equal(t, before, rebuildTestSnapshot(t, agent.dir)) } diff --git a/socket.go b/socket.go index 7b2da2b..0dffeb0 100644 --- a/socket.go +++ b/socket.go @@ -33,8 +33,10 @@ func checkSocketPath(path string) error { // message, "interrupt", "stop" with the name of a subagent in Text, or // "whatsapp" with the text after /whatsapp of a WhatsApp command, or "backup" // with the absolute output path in Text. The backup answer is a "backup" event -// with the path and size, or an "error" event. +// with the path and size, or an "error" event. An "image" request has an +// absolute Path and the caption in Text. type SocketRequest struct { Type string `json:"type"` Text string `json:"text,omitempty"` + Path string `json:"path,omitempty"` } diff --git a/subagent_run.go b/subagent_run.go index 8084c39..891766c 100644 --- a/subagent_run.go +++ b/subagent_run.go @@ -515,11 +515,13 @@ func (r *SubagentRuns) chain(ctx context.Context, run *SubagentRun) (string, err input = append(input, modelUserMessage(r.state()), modelUserMessage(log)) chain := &Chain{ client: r.client, + images: r.agent.Images(), tools: &Toolbox{ log: run.subagent, chat: r.memory, subagents: r.subagents, workdir: run.subagent.WorkDir(), + images: r.agent.Images(), }, request: ModelRequest{ Model: r.agent.config.Model, diff --git a/telegram.go b/telegram.go index f2c687a..a595b68 100644 --- a/telegram.go +++ b/telegram.go @@ -133,13 +133,13 @@ func (c TelegramConfig) validate() error { } // TelegramBot connects the private chat of the owner with the agent. It polls -// the Bot API for new messages and sends each text message of the owner to the -// agent. It ignores every other update. Its sender sends the replies, the -// errors, the notices of the bot, and the user messages of the other clients to -// the chat, and shows the typing action while a turn runs. Only the replies -// have a format: their Markdown goes as rich messages, which Telegram renders -// itself, and as message entities only when Telegram refuses a rich message. -// The rest is plain text. Tool calls do not go to the chat. +// the Bot API for new messages and sends owner text and images to the agent. It +// ignores every other update. Its sender sends the replies, the errors, the +// notices of the bot, and the user messages of the other clients to the chat, +// and shows the typing action while a turn runs. Only the replies have a +// format: their Markdown goes as rich messages, which Telegram renders itself, +// and as message entities only when Telegram refuses a rich message. The rest +// is plain text. Tool calls do not go to the chat. // // The offset file holds the ID of the next update. It moves past an update // only after the message of the update is durable in the log, so a restart @@ -193,8 +193,8 @@ func (b *TelegramBot) Notify(event Event) { } // Run polls for the messages of the owner and sends the queued events to the -// chat until ctx ends. It returns an error only when a message or the offset -// cannot be written, because then the bot would lose or repeat messages. +// chat until ctx ends. It returns an error only when a message or the offset or +// image cannot be written, because then the bot would lose or repeat messages. func (b *TelegramBot) Run(ctx context.Context, l *TurnLoop, whatsapp *WhatsApp) error { ctx, cancel := context.WithCancel(ctx) defer cancel() @@ -258,9 +258,9 @@ func (b *TelegramBot) send(ctx context.Context) { } } -// poll gets the updates from the stored offset on and handles each one, until -// ctx ends. Each update is handled to its end also when ctx ends during it, so -// a stop does not repeat its message. +// poll gets updates from the stored offset and handles them until ctx ends. A +// failed image download retries from the same offset. A submitted message is +// flushed even if ctx ends, and only then can the offset advance. func (b *TelegramBot) poll(ctx context.Context, l *TurnLoop, whatsapp *WhatsApp) error { offset, err := telegramLoadOffset(b.offsetPath) if err != nil { @@ -273,7 +273,16 @@ func (b *TelegramBot) poll(ctx context.Context, l *TurnLoop, whatsapp *WhatsApp) back: "receive back", } + imageOutage := &TelegramOutage{ + logger: b.logger, + loss: "image receive failed", + back: "image receive back", + } + + imageWait := b.minWait wait := b.minWait + +poll: for ctx.Err() == nil { updates, err := b.api.GetUpdates(ctx, offset) if ctx.Err() != nil { @@ -290,16 +299,27 @@ func (b *TelegramBot) poll(ctx context.Context, l *TurnLoop, whatsapp *WhatsApp) outage.End() wait = b.minWait + for _, update := range updates { if ctx.Err() != nil { return nil } - err = b.handle(context.WithoutCancel(ctx), l, whatsapp, update) + err = b.handle(ctx, l, whatsapp, update) if err != nil { + retry, asked := telegramRetryable(err) + if retry { + imageWait = b.pause(ctx, imageOutage, err, imageWait, asked) + continue poll + } + return err } + imageOutage.End() + + imageWait = b.minWait + offset = update.UpdateID + 1 err = telegramSaveOffset(b.offsetPath, offset) if err != nil { @@ -373,9 +393,9 @@ func (b *TelegramBot) pause(ctx context.Context, outage *TelegramOutage, err err // to the agent, and handle returns when it is durable in the log. The line // /stop interrupts the turn, the line /stop NAME stops the subagent NAME, and a // line that starts with /whatsapp runs a WhatsApp command on whatsapp, whose -// answers go to the chat only. Any other message of the owner gets the answer -// that only text is supported. Every other update is ignored, and its content -// is not logged. +// answers go to the chat only. Photos and image documents enter the shared +// image store with their caption. Other media keep the text-only answer. Every +// other update is ignored, and its content is not logged. func (b *TelegramBot) handle(ctx context.Context, l *TurnLoop, whatsapp *WhatsApp, update TelegramUpdate) error { message := update.Message if message == nil || message.Chat.Type != "private" || message.Chat.ID != b.owner || message.From == nil || message.From.ID != b.owner { @@ -387,6 +407,41 @@ func (b *TelegramBot) handle(ctx context.Context, l *TurnLoop, whatsapp *WhatsAp name, stop := stopSubagentName(message.Text) args, isWhatsApp := whatsappCommandArgs(message.Text) switch { + case len(message.Photo) > 0 || message.Document != nil: + text, err := telegramImageText(ctx, b.api, l.agent.Images(), message) + if err != nil { + retry, _ := telegramRetryable(err) + if retry { + return fmt.Errorf("receive image: %w", err) + } + + switch { + case errors.Is(err, errTelegramOtherMedia): + b.notice(telegramTextOnly) + case errors.Is(err, errImageSize), errors.Is(err, errImageFormat), errors.Is(err, errTelegramPhotoSize): + b.notice(err.Error()) + default: + var answer *TelegramError + + if !errors.As(err, &answer) { + return fmt.Errorf("receive Telegram image: %w", err) + } + + b.notice("Cannot download image.") + } + + return nil + } + + _, err = l.Submit(telegramSource, text) + if err != nil { + return fmt.Errorf("submit Telegram image: %w", err) + } + + err = l.Flush(context.WithoutCancel(ctx)) + if err != nil { + return fmt.Errorf("flush Telegram image: %w", err) + } case message.Text == "": b.notice(telegramTextOnly) case strings.TrimSpace(message.Text) == stopCommand: @@ -410,7 +465,7 @@ func (b *TelegramBot) handle(ctx context.Context, l *TurnLoop, whatsapp *WhatsAp return err } - err = l.Flush(ctx) + err = l.Flush(context.WithoutCancel(ctx)) if err != nil { return fmt.Errorf("flush the message from Telegram: %w", err) } diff --git a/telegram_api.go b/telegram_api.go index 07d2bf1..32ad9a9 100644 --- a/telegram_api.go +++ b/telegram_api.go @@ -9,6 +9,7 @@ import ( "io" "net/http" "net/url" + "path" "strings" "time" ) @@ -102,6 +103,66 @@ func (b *TelegramAPI) GetUpdates(ctx context.Context, offset int64) ([]TelegramU return updates, err } +// GetImage downloads a file through the authenticated file endpoint. Errors +// never include its token or URL, and the body is limited before storage. +func (b *TelegramAPI) GetImage(ctx context.Context, id string) (data []byte, err error) { + var file struct { + Path string `json:"file_path"` + Size int64 `json:"file_size"` + } + + err = b.call(ctx, b.sendTimeout, "getFile", map[string]any{"file_id": id}, &file) + if err != nil { + return nil, fmt.Errorf("get image file: %w", err) + } + + if file.Size > imageMaxBytes { + return nil, errImageSize + } + + if file.Path == "" || path.Clean(file.Path) != file.Path || strings.HasPrefix(file.Path, "/") || strings.HasPrefix(file.Path, "../") || strings.ContainsAny(file.Path, "?%#\\") { + return nil, errors.New("invalid image file path") + } + + ctx, cancel := context.WithTimeout(ctx, b.sendTimeout) + defer cancel() + + request, err := http.NewRequestWithContext(ctx, http.MethodGet, b.base+"/file/bot"+b.token+"/"+file.Path, nil) + if err != nil { + return nil, fmt.Errorf("make image request: %s", b.redact(err)) + } + + response, err := b.http.Do(request) + if err != nil { + return nil, b.networkError("image download failed", err, b.sendTimeout) + } + + defer func() { + closeErr := response.Body.Close() + if closeErr != nil { + err = errors.Join(err, b.networkError("close image download", closeErr, b.sendTimeout)) + } + }() + + if response.StatusCode != http.StatusOK { + return nil, &TelegramError{ + Status: response.StatusCode, + Description: http.StatusText(response.StatusCode), + } + } + + data, err = io.ReadAll(io.LimitReader(response.Body, imageMaxBytes+1)) + if err != nil { + return nil, b.networkError("image cut off", err, b.sendTimeout) + } + + if len(data) > imageMaxBytes { + return nil, errImageSize + } + + return data, nil +} + // SendMessage sends text to chat with the format of entities, with no parse // mode, so the text needs no escaping. With no entities, the text is plain. A // call that takes longer than telegramSendTimeout fails with a @@ -291,12 +352,29 @@ type TelegramAnswer struct { } `json:"parameters"` } -// TelegramMessage is a message to the bot. Text is empty for a message that is -// not text, such as a photo. +// TelegramMessage holds the text, caption, and image sizes of a bot message. type TelegramMessage struct { - Chat TelegramChat `json:"chat"` - From *TelegramUser `json:"from"` - Text string `json:"text"` + Chat TelegramChat `json:"chat"` + From *TelegramUser `json:"from"` + Text string `json:"text"` + Caption string `json:"caption"` + Photo []TelegramPhoto `json:"photo"` + Document *TelegramDocument `json:"document"` +} + +// TelegramPhoto is one supplied size of a photo. +type TelegramPhoto struct { + FileID string `json:"file_id"` + Width int `json:"width"` + Height int `json:"height"` + FileSize int64 `json:"file_size"` +} + +// TelegramDocument is a file sent without photo compression. +type TelegramDocument struct { + FileID string `json:"file_id"` + MimeType string `json:"mime_type"` + FileSize int64 `json:"file_size"` } // TelegramChat is the chat of a message. Type is private for a chat with one diff --git a/telegram_fake_helper_test.go b/telegram_fake_helper_test.go index 3056aa1..6a89054 100644 --- a/telegram_fake_helper_test.go +++ b/telegram_fake_helper_test.go @@ -45,6 +45,7 @@ type TelegramFake struct { stall func(call TelegramFakeCall) bool added chan struct{} stopped chan struct{} + images map[string][]byte } // Add makes update available to getUpdates. @@ -86,7 +87,36 @@ func (f *TelegramFake) Sent() []string { return texts } +func (f *TelegramFake) Image(id string, data []byte) { + f.mutex.Lock() + defer f.mutex.Unlock() + + if f.images == nil { + f.images = make(map[string][]byte) + } + + f.images[id] = data +} + func (f *TelegramFake) serve(w http.ResponseWriter, r *http.Request) { + id, file := strings.CutPrefix(r.URL.Path, "/file/bot"+telegramTestToken+"/") + if file { + f.mutex.Lock() + + data, exists := f.images[id] + f.mutex.Unlock() + + if !exists { + http.NotFound(w, r) + + return + } + + _, _ = w.Write(data) + + return + } + method, found := strings.CutPrefix(r.URL.Path, "/bot"+telegramTestToken+"/") if !found { http.Error(w, `{"ok":false,"error_code":404,"description":"Not Found"}`, http.StatusNotFound) @@ -146,6 +176,15 @@ func (f *TelegramFake) serve(w http.ResponseWriter, r *http.Request) { call.ok = true f.record(call) + if method == "getFile" { + _ = json.NewEncoder(w).Encode(map[string]any{ + "ok": true, + "result": map[string]any{"file_path": params["file_id"]}, + }) + + return + } + if method != "getUpdates" { _, _ = io.WriteString(w, `{"ok":true,"result":true}`) return diff --git a/telegram_image.go b/telegram_image.go new file mode 100644 index 0000000..9b73341 --- /dev/null +++ b/telegram_image.go @@ -0,0 +1,71 @@ +package main + +import ( + "context" + "errors" + "fmt" + "strings" +) + +var ( + errTelegramOtherMedia = errors.New("only text messages are supported") + errTelegramPhotoSize = errors.New("no photo size fits 1568 pixels") +) + +// telegramImageText selects a supplied photo size, or an image document, and +// stores it before returning marker text with the unchanged caption. +func telegramImageText(ctx context.Context, api *TelegramAPI, store *ImageStore, message *TelegramMessage) (string, error) { + var ( + id string + size int64 + ) + + if len(message.Photo) > 0 { + var area int + + for _, photo := range message.Photo { + if photo.Width <= 0 || photo.Height <= 0 || max(photo.Width, photo.Height) > 1568 { + continue + } + + if photo.Width*photo.Height > area { + id = photo.FileID + size = photo.FileSize + area = photo.Width * photo.Height + } + } + + if id == "" { + return "", errTelegramPhotoSize + } + } else if message.Document != nil { + document := message.Document + switch document.MimeType { + case "image/jpeg", "image/png", "image/webp": + id = document.FileID + size = document.FileSize + default: + if strings.HasPrefix(document.MimeType, "image/") { + return "", errImageFormat + } + + return "", errTelegramOtherMedia + } + } + + if size > imageMaxBytes { + return "", errImageSize + } + + data, err := api.GetImage(ctx, id) + if err != nil { + return "", fmt.Errorf("download image: %w", err) + } + + name, err := store.Save(data) + if err != nil { + return "", fmt.Errorf("store image: %w", err) + } + + return imageText([]string{name}, message.Caption), nil +} diff --git a/telegram_image_test.go b/telegram_image_test.go new file mode 100644 index 0000000..837a64e --- /dev/null +++ b/telegram_image_test.go @@ -0,0 +1,259 @@ +package main + +import ( + "bytes" + "image" + "image/jpeg" + "net/http" + "os" + "path/filepath" + "strings" + "testing" + "time" + + "github.com/stretchr/testify/require" +) + +func TestTelegramSelectsLargestPhotoAtMost1568Pixels(t *testing.T) { + server := newModelFake(t) + test := newTestTurnLoop(t, server) + telegram := newTelegramFake(t) + bot, logs := telegramStartTestBot(t, telegram) + picture := imageTestPNG(t, 8, 6) + telegram.Image("selected", picture) + + message := telegramOwnerMessage("") + message.Caption = "Read this label.\n" + message.Photo = []TelegramPhoto{ + { + FileID: "small", + Width: 100, + Height: 200, + }, + { + FileID: "too-large", + Width: 1600, + Height: 1200, + }, + { + FileID: "selected", + Width: 1568, + Height: 1176, + }, + { + FileID: "medium", + Width: 800, + Height: 600, + }, + } + require.NoError(t, bot.handle(t.Context(), test.loop, nil, TelegramUpdate{ + UpdateID: 1, + Message: message, + })) + test.events.WaitFor(t, "idle", 1) + + files, err := os.ReadDir(test.agent.Images().dir) + require.NoError(t, err) + require.Len(t, files, 1) + + logged, err := test.memory.Message(0) + require.NoError(t, err) + require.Equal(t, imageText([]string{files[0].Name()}, message.Caption), logged.Text) + require.True(t, strings.HasSuffix(logged.Text, "\n")) + require.Contains(t, string(server.Call(0).raw), "input_image") + require.NotContains(t, logs.String(), "base64") + require.NotContains(t, logs.String(), "Read this label") + + for _, call := range telegram.Calls() { + if call.method == "getFile" { + require.Equal(t, "selected", call.params["file_id"]) + } + } +} + +func TestTelegramImageDocumentsUseContentFormatAndRejectOtherMedia(t *testing.T) { + telegram := newTelegramFake(t) + png := imageTestPNG(t, 2, 2) + + var jpegBytes bytes.Buffer + + require.NoError(t, jpeg.Encode(&jpegBytes, image.NewRGBA(image.Rect(0, 0, 2, 2)), nil)) + + // A made-up one-pixel lossless WebP image. + webp := []byte{'R', 'I', 'F', 'F', 22, 0, 0, 0, 'W', 'E', 'B', 'P', 'V', 'P', '8', 'L', 10, 0, 0, 0, 0x2f, 0, 0, 0, 0, 0x07, 0x10, 0x11, 0x11, 0x88} + + telegram.Image("png", png) + telegram.Image("jpeg", jpegBytes.Bytes()) + telegram.Image("webp", webp) + telegram.Image("invalid", []byte("not an image")) + telegram.Image("large", bytes.Repeat([]byte{0}, imageMaxBytes+1)) + + api := newTelegramAPI(telegram.URL, telegramTestToken) + + store := &ImageStore{dir: filepath.Join(t.TempDir(), imageDirName)} + testCases := []struct { + name string + document TelegramDocument + suffix string + failure string + }{ + { + name: "accepts PNG documents", + document: TelegramDocument{ + FileID: "png", + MimeType: "image/png", + }, + suffix: ".png]", + }, + { + name: "accepts JPEG documents", + document: TelegramDocument{ + FileID: "jpeg", + MimeType: "image/jpeg", + }, + suffix: ".jpg]", + }, + { + name: "accepts WebP documents", + document: TelegramDocument{ + FileID: "webp", + MimeType: "image/webp", + }, + suffix: ".webp]", + }, + { + name: "other documents keep text answer", + document: TelegramDocument{ + FileID: "pdf", + MimeType: "application/pdf", + }, + failure: "only text messages are supported", + }, + { + name: "other images refused", + document: TelegramDocument{ + FileID: "gif", + MimeType: "image/gif", + }, + failure: "JPEG, PNG, or WebP", + }, + { + name: "invalid image refused", + document: TelegramDocument{ + FileID: "invalid", + MimeType: "image/png", + }, + failure: "JPEG, PNG, or WebP", + }, + { + name: "large download refused", + document: TelegramDocument{ + FileID: "large", + MimeType: "image/png", + }, + failure: "exceeds 20 MB", + }, + { + name: "large metadata refused", + document: TelegramDocument{ + FileID: "not-downloaded", + MimeType: "image/png", + FileSize: imageMaxBytes + 1, + }, + failure: "exceeds 20 MB", + }, + } + for _, tc := range testCases { + t.Run(tc.name, func(t *testing.T) { + text, err := telegramImageText(t.Context(), api, store, &TelegramMessage{Document: &tc.document}) + if tc.failure != "" { + require.ErrorContains(t, err, tc.failure) + + return + } + + require.NoError(t, err) + require.True(t, strings.HasSuffix(text, tc.suffix)) + }) + } +} + +func TestTelegramRetriesImageDownloadBeforeAdvancingOffset(t *testing.T) { + telegram := newTelegramFake(t) + telegram.Image("photo", imageTestPNG(t, 2, 2)) + + tries := 0 + telegram.fail = func(call TelegramFakeCall) (int, string) { + if call.method == "getFile" { + tries++ + if tries == 1 { + return http.StatusServiceUnavailable, "try again" + } + } + + return 0, "" + } + bot, logs := telegramStartTestBot(t, telegram) + bot.minWait = time.Millisecond + bot.maxWait = 5 * time.Millisecond + server := newModelFake(t) + socket := daemonStartTestWithBot(t, server, bot) + terminal := terminalStartTestClient(t, socket) + terminal.WaitFor(t, "[connected to OneChat]") + + message := telegramOwnerMessage("") + message.Photo = []TelegramPhoto{{ + FileID: "photo", + Width: 2, + Height: 2, + }} + telegram.Add(TelegramUpdate{ + UpdateID: 7, + Message: message, + }) + terminal.WaitFor(t, "[done]") + require.Eventually(t, func() bool { + offset, err := telegramLoadOffset(bot.offsetPath) + return err == nil && offset == 8 + }, 10*time.Second, time.Millisecond) + require.Contains(t, logs.String(), "receive failed") + require.Contains(t, logs.String(), "receive back") + require.NotContains(t, logs.String(), telegramTestToken) + require.NotContains(t, logs.String(), "base64") +} + +func TestTelegramRefusesPhotoWhenEverySizeIsTooLarge(t *testing.T) { + telegram := newTelegramFake(t) + _, err := telegramImageText(t.Context(), newTelegramAPI(telegram.URL, telegramTestToken), &ImageStore{dir: t.TempDir()}, &TelegramMessage{Photo: []TelegramPhoto{{ + FileID: "too-big", + Width: 1600, + Height: 1000, + }}}) + require.ErrorContains(t, err, "1568 pixels") + require.Empty(t, telegram.Calls()) +} + +func TestTelegramStorageFailureDoesNotAcceptTheImage(t *testing.T) { + telegram := newTelegramFake(t) + telegram.Image("photo", imageTestPNG(t, 2, 2)) + + test := newTestTurnLoop(t, newModelFake(t)) + require.NoError(t, os.WriteFile(test.agent.Images().dir, []byte("blocked"), 0o600)) + + bot, _ := telegramStartTestBot(t, telegram) + message := telegramOwnerMessage("") + message.Photo = []TelegramPhoto{{ + FileID: "photo", + Width: 2, + Height: 2, + }} + err := bot.handle(t.Context(), test.loop, nil, TelegramUpdate{ + UpdateID: 1, + Message: message, + }) + require.ErrorContains(t, err, "create image directory") + require.Equal(t, 0, test.memory.Len()) + + _, found := bot.next() + require.False(t, found) +} diff --git a/terminal.go b/terminal.go index 9945620..da0d78a 100644 --- a/terminal.go +++ b/terminal.go @@ -23,9 +23,10 @@ var errTerminalDaemonClosed = errors.New("the daemon closed the connection") // terminal runs "onechat" and "onechat chat": the terminal client of the // daemon. Each line of input is a user message, the line /stop interrupts the // running turn, the line /stop NAME stops the subagent NAME, and a line that -// starts with /whatsapp is a WhatsApp command. The client shows the replies, -// the tool calls, and their results as they happen. It ends at the end of the -// input or when the daemon closes the connection. +// starts with /whatsapp is a WhatsApp command. /image PATH [caption] sends an +// image from the client working directory. The client shows the replies, the +// tool calls, and their results as they happen. It ends at the end of the input +// or when the daemon closes the connection. func terminal(ctx context.Context, in io.Reader, out io.Writer) error { dir, err := dataDir() if err != nil { @@ -99,6 +100,7 @@ func terminalSendLines(ctx context.Context, conn net.Conn, in io.Reader) error { name, stop := stopSubagentName(line) args, whatsapp := whatsappCommandArgs(line) + image, isImage := terminalImageRequest(line) switch { case strings.TrimSpace(line) == "": continue @@ -111,6 +113,16 @@ func terminalSendLines(ctx context.Context, conn net.Conn, in io.Reader) error { Type: "stop", Text: name, } + case isImage: + request = image + if request.Path != "" { + var err error + + request.Path, err = filepath.Abs(request.Path) + if err != nil { + return fmt.Errorf("find terminal image path: %w", err) + } + } case whatsapp: request = SocketRequest{ Type: "whatsapp", diff --git a/terminal_image.go b/terminal_image.go new file mode 100644 index 0000000..f6f4101 --- /dev/null +++ b/terminal_image.go @@ -0,0 +1,46 @@ +package main + +import "strings" + +// terminalImageRequest reads /image with one path and an optional caption. +// Single and double quotes permit spaces in a path, with no shell expansion. An +// invalid path argument becomes an empty path for the daemon to refuse. +func terminalImageRequest(line string) (SocketRequest, bool) { + rest, found := strings.CutPrefix(strings.TrimLeft(line, " \t"), "/image") + if !found || (rest != "" && rest[0] != ' ' && rest[0] != '\t') { + return SocketRequest{}, false + } + + request := SocketRequest{Type: "image"} + rest = strings.TrimLeft(rest, " \t") + if rest == "" { + return request, true + } + + if rest[0] == '\'' || rest[0] == '"' { + end := strings.IndexByte(rest[1:], rest[0]) + if end < 0 { + return request, true + } + + request.Path = rest[1 : 1+end] + rest = rest[end+2:] + if rest != "" && rest[0] != ' ' && rest[0] != '\t' { + request.Path = "" + return request, true + } + } else { + end := strings.IndexAny(rest, " \t") + if end < 0 { + request.Path = rest + return request, true + } + + request.Path = rest[:end] + rest = rest[end:] + } + + request.Text = strings.TrimLeft(rest, " \t") + + return request, true +} diff --git a/terminal_image_test.go b/terminal_image_test.go new file mode 100644 index 0000000..46a7cfd --- /dev/null +++ b/terminal_image_test.go @@ -0,0 +1,157 @@ +package main + +import ( + "bytes" + "encoding/json" + "net" + "os" + "path/filepath" + "strings" + "testing" + + "github.com/stretchr/testify/require" +) + +func TestTerminalImageLineSendsAnAbsolutePathAndCaption(t *testing.T) { + client, daemon := net.Pipe() + defer func() { + require.NoError(t, client.Close()) + require.NoError(t, daemon.Close()) + }() + + done := make(chan error, 1) + go func() { + done <- terminalSendLines(t.Context(), client, strings.NewReader("/image 'image sample.png' Read the label.\n")) + }() + + var request SocketRequest + + require.NoError(t, json.NewDecoder(daemon).Decode(&request)) + require.NoError(t, <-done) + + expected, err := filepath.Abs("image sample.png") + require.NoError(t, err) + require.Equal(t, SocketRequest{ + Type: "image", + Path: expected, + Text: "Read the label.", + }, request) +} + +func TestTerminalImagePathsAndCaptionsHaveNoShellExpansion(t *testing.T) { + testCases := []struct { + name string + line string + path string + caption string + command bool + }{ + { + name: "accepts paths without captions", + line: "/image sample.png", + path: "sample.png", + command: true, + }, + { + name: "keeps spaces in double quoted paths", + line: `/image "a b.png" caption`, + path: "a b.png", + caption: "caption", + command: true, + }, + { + name: "caption stays literal", + line: "/image sample.png $(not-a-command) ", + path: "sample.png", + caption: "$(not-a-command) ", + command: true, + }, + { + name: "missing path refused", + line: "/image", + command: true, + }, + { + name: "unclosed quote refused", + line: `/image "sample.png`, + command: true, + }, + { + name: "joined caption refused", + line: `/image "sample.png"caption`, + command: true, + }, + { + name: "other command stays text", + line: "/images sample.png", + command: false, + }, + } + for _, tc := range testCases { + t.Run(tc.name, func(t *testing.T) { + request, command := terminalImageRequest(tc.line) + require.Equal(t, tc.command, command) + require.Equal(t, tc.path, request.Path) + require.Equal(t, tc.caption, request.Text) + }) + } +} + +func TestTerminalImageIsStoredBeforeItReachesTheModel(t *testing.T) { + server := newModelFake(t) + socket := daemonStartTest(t, server) + terminal := terminalStartTestClient(t, socket) + terminal.WaitFor(t, "[connected to OneChat]") + + path := filepath.Join(t.TempDir(), "sample.png") + data := imageTestPNG(t, 2, 2) + require.NoError(t, os.WriteFile(path, data, 0o600)) + terminal.Type("/image " + path + " What is this?") + terminal.WaitFor(t, "[done]") + require.Equal(t, 1, server.Calls()) + require.Contains(t, string(server.Call(0).raw), "input_image") + require.Contains(t, string(server.Call(0).raw), "What is this?") + require.NotContains(t, string(server.Call(0).raw), path) + + files, err := os.ReadDir(filepath.Join(filepath.Dir(socket), imageDirName)) + require.NoError(t, err) + require.Len(t, files, 1) + + stored, err := os.ReadFile(filepath.Join(filepath.Dir(socket), imageDirName, files[0].Name())) + require.NoError(t, err) + require.Equal(t, data, stored) + require.NotContains(t, terminal.Output(), "base64") +} + +func TestTerminalRefusesLargeAndUnsupportedImagesWithoutAModelCall(t *testing.T) { + server := newModelFake(t) + test := newTestTurnLoop(t, server) + path := filepath.Join(t.TempDir(), "sample.png") + require.NoError(t, os.WriteFile(path, bytes.Repeat([]byte{0}, imageMaxBytes+1), 0o600)) + + request := SocketRequest{ + Type: "image", + Path: path, + } + require.ErrorContains(t, daemonRunRequest(test.loop, nil, request, func(string) { + }), "exceeds 20 MB") + require.NoError(t, os.WriteFile(path, []byte("not an image"), 0o600)) + require.ErrorContains(t, daemonRunRequest(test.loop, nil, request, func(string) { + }), "JPEG, PNG, or WebP") + require.Equal(t, 0, test.memory.Len()) + require.Equal(t, 0, server.Calls()) + + _, err := os.Stat(test.agent.Images().dir) + require.ErrorIs(t, err, os.ErrNotExist) +} + +func TestImageCommandErrorDoesNotEndTheTerminalClient(t *testing.T) { + socket := daemonStartTest(t, newModelFake(t)) + terminal := terminalStartTestClient(t, socket) + terminal.WaitFor(t, "[connected to OneChat]") + terminal.Type("/image") + terminal.WaitFor(t, "use /image PATH [caption]") + terminal.Type("Continue.") + terminal.WaitFor(t, "[done]") + require.NoError(t, terminal.Close()) +} diff --git a/tools.go b/tools.go index a75514f..96cbb97 100644 --- a/tools.go +++ b/tools.go @@ -68,13 +68,15 @@ type Toolbox struct { messages *DaemonMessages children *DaemonChildren routes *Routes + images *ImageStore } // Call logs the call of the tool name with its JSON arguments, makes the log -// durable, runs the tool, and logs its clipped output, which it returns. A -// failed tool gives its error as the output. An error of Call itself is an -// error of the log. When ctx ends before the flush, the tool does not run. -func (b *Toolbox) Call(ctx context.Context, name, arguments string) (string, error) { +// durable, runs the tool, and logs its clipped text. It returns that text or +// the text and image parts of a single-message zoom. Pixels never enter the +// log. A failed tool gives its error as the output. An error of Call itself is +// an error of the log. When ctx ends before the flush, the tool does not run. +func (b *Toolbox) Call(ctx context.Context, name, arguments string) (any, error) { _, err := b.log.Append("tool", toolCallText(name, arguments)) if err != nil { return "", fmt.Errorf("append tool call: %w", err) @@ -90,12 +92,26 @@ func (b *Toolbox) Call(ctx context.Context, name, arguments string) (string, err output = toolClip(b.run(ctx, name, arguments)) } + var ( + result any = output + resultErr error + ) + + if err == nil { + result, resultErr = imageToolOutput(b, name, arguments, output) + } + + if resultErr != nil { + output = "error: " + resultErr.Error() + result = output + } + _, err = b.log.Append("echo", output) if err != nil { return "", fmt.Errorf("append tool result: %w", err) } - return output, nil + return result, nil } // run runs one tool and returns its output, or its error as text. diff --git a/tree.go b/tree.go index 69558c5..079969c 100644 --- a/tree.go +++ b/tree.go @@ -55,11 +55,11 @@ func leafSource(message Message) string { return message.Kind + ": " + message.Text } -// freeLeaf returns the leaf of message when its source fits in a node. Such a -// leaf holds the source word for word and needs no model call. +// freeLeaf returns a text-only leaf when its source fits in a node. Images need +// a model description even when their text fits. func freeLeaf(message Message) (Node, bool) { text := leafSource(message) - if len(text) > nodeBytes { + if len(text) > nodeBytes || len(imageFiles(message.Text)) > 0 { return Node{}, false } diff --git a/turn.go b/turn.go index e0bd489..2c1d9c2 100644 --- a/turn.go +++ b/turn.go @@ -60,6 +60,7 @@ func newTurnLoop(agent *Agent, memory *Memory, subagents *Subagents, whatsapp *W schedules: schedules, workdir: agent.config.Workdir, agentName: agent.config.Name, + images: agent.Images(), } return l, nil @@ -423,10 +424,19 @@ func (l *TurnLoop) chain(ctx context.Context) (string, error) { Instructions: instructions, Tools: l.agent.Tools(), }, - take: l.take, + take: l.take, + images: l.agent.Images(), } - reply, err := chain.Run(ctx, turnInput(view, l.state(), messages)) + input := turnInput(view, l.state(), messages) + for n, message := range messages { + input[len(input)-len(messages)+n], err = imageUserMessage(l.agent.Images(), message.Text) + if err != nil { + return "", fmt.Errorf("read turn images: %w", err) + } + } + + reply, err := chain.Run(ctx, input) if err != nil { return "", fmt.Errorf("run turn chain: %w", err) } diff --git a/turn_test.go b/turn_test.go index 949c0b8..3b86f0c 100644 --- a/turn_test.go +++ b/turn_test.go @@ -388,7 +388,7 @@ func turnInputTexts(t *testing.T, call ModelFakeCall) []string { func newTestTurnLoopWith(t *testing.T, server *ModelFake, summarizer Summarizer) *TurnTestLoop { t.Helper() - agent, err := openAgent(t.TempDir()) + agent, err := openAgent(filepath.Join(t.TempDir(), "agent")) require.NoError(t, err) agent.config.Workdir = t.TempDir() diff --git a/whatsapp_import.go b/whatsapp_import.go index 2361915..1092b49 100644 --- a/whatsapp_import.go +++ b/whatsapp_import.go @@ -37,21 +37,12 @@ var ( // whatsappImport runs the offline import command. It holds the same token lock // as the daemon and probe, even for a plan. The plan reads no credentials for a // model and makes no network call. Only a completed rebuild replaces the agent -// directory, and unfinished subagents prevent either mode. +// directory. Unfinished subagents and pending target deliveries prevent either +// mode. func whatsappImport(ctx context.Context, args []string, out io.Writer) (err error) { - planOnly := len(args) > 0 && args[0] == "-n" - if planOnly { - args = args[1:] - } - - if len(args) == 0 { - return errors.New("usage: onechat import [-n] ZIP [ZIP...]") - } - - for _, arg := range args { - if strings.HasPrefix(arg, "-") { - return errors.New("usage: onechat import [-n] ZIP [ZIP...]") - } + options, err := whatsappImportParse(args) + if err != nil { + return fmt.Errorf("parse import arguments: %w", err) } dir, err := dataDir() @@ -72,47 +63,57 @@ func whatsappImport(ctx context.Context, args []string, out io.Writer) (err erro }() root := filepath.Join(dir, "agent") - err = rebuildIdle(root) + agent, err := whatsappImportAgent(root, options.agent) + if err != nil { + return fmt.Errorf("select import agent: %w", err) + } + + err = whatsappImportIdle(root, agent) if err != nil { return fmt.Errorf("check import work: %w", err) } - old, err := rebuildReadLog(ctx, root) + old, err := rebuildReadLog(ctx, agent.dir) if err != nil { return fmt.Errorf("read import log: %w", err) } + var rootLog []DatedMessage + + if agent.dir != root { + rootLog, err = rebuildReadLog(ctx, root) + if err != nil { + return fmt.Errorf("read root import history: %w", err) + } + } + names, owner, err := whatsappImportNames(ctx, dir) if err != nil { return fmt.Errorf("read import names: %w", err) } - plan, err := whatsappImportPlan(ctx, args, old, nil, names, owner, time.Local) + plan, err := whatsappImportPlan(ctx, options.paths, old, rootLog, names, owner, time.Local) if err != nil { return fmt.Errorf("plan import: %w", err) } + plan.agent = agent.config.Name err = plan.Write(out) if err != nil { return fmt.Errorf("write import plan: %w", err) } - if planOnly || len(plan.added) == 0 { + if options.planOnly || len(plan.added) == 0 { return nil } // An existing agent is required so that a failed import never creates // or changes the original agent's configuration. - _, err = os.Stat(filepath.Join(root, "config.json")) + _, err = os.Stat(filepath.Join(agent.dir, "config.json")) if err != nil { return fmt.Errorf("read agent config: %w", err) } - agent, err := openAgent(root) - if err != nil { - return fmt.Errorf("open import agent: %w", err) - } - config, err := loadConfig(dir) if err != nil { return fmt.Errorf("load import configuration: %w", err) @@ -142,7 +143,7 @@ func whatsappImport(ctx context.Context, args []string, out io.Writer) (err erro } client := newModelClient(address, auth, agent.CacheKey(), usagePath) - backup, err := rebuild(ctx, agent, plan.added, client, newCompactor(client, agent.Prompt, config.Compactor, agent.Tools()), out) + backup, err := rebuild(ctx, agent, plan.added, client, newCompactor(client, agent.Prompt, config.Compactor, agent.Tools(), agent.Images()), out) if err != nil { return fmt.Errorf("rebuild imported memory: %w", err) } @@ -159,6 +160,7 @@ func whatsappImport(ctx context.Context, args []string, out io.Writer) (err erro // log. Its estimate uses the measured cost of 1.3 calls per message, not a // promise about a particular export or compaction retry. type WhatsAppImportPlan struct { + agent string cutoffs map[string]time.Time added []DatedMessage chats []string @@ -188,7 +190,7 @@ func (p WhatsAppImportPlan) Write(out io.Writer) error { owner = "unknown (sender names kept as written)" } - _, err := fmt.Fprintf(out, "chats: %d\n", len(p.chats)) + _, err := fmt.Fprintf(out, "agent: %s\nchats: %d\n", p.agent, len(p.chats)) if err != nil { return fmt.Errorf("write import plan: %w", err) } @@ -229,6 +231,140 @@ func (p WhatsAppImportPlan) Write(out io.Writer) error { return nil } +// WhatsAppImportOptions selects the target, exports, and plan-only mode. +type WhatsAppImportOptions struct { + planOnly bool + agent string + paths []string +} + +// whatsappImportParse accepts GNU-style flags, including --agent=NAME and -- +// before paths that start with a dash. +func whatsappImportParse(args []string) (WhatsAppImportOptions, error) { + var options WhatsAppImportOptions + + flags := true + for i := 0; i < len(args); i++ { + arg := args[i] + switch { + case flags && arg == "--": + flags = false + case flags && arg == "-n": + options.planOnly = true + case flags && (arg == "--agent" || strings.HasPrefix(arg, "--agent=")): + _, name, found := strings.Cut(arg, "=") + if !found { + i++ + if i == len(args) { + return options, errors.New("--agent needs a child name") + } + + name = args[i] + } + + if name == "" { + return options, errors.New("--agent needs a child name") + } + + options.agent = name + case flags && strings.HasPrefix(arg, "-"): + return options, fmt.Errorf("unknown import flag %q", arg) + default: + options.paths = append(options.paths, arg) + } + } + + if len(options.paths) == 0 { + return options, errors.New("usage: onechat import [-n] [--agent NAME] ZIP [ZIP...]") + } + + return options, nil +} + +// whatsappImportAgent selects the root or an existing direct child without +// creating a directory or config. A root without a config can have a plan, +// using the default name. +func whatsappImportAgent(root, name string) (*Agent, error) { + _, err := os.Stat(filepath.Join(root, "config.json")) + if errors.Is(err, os.ErrNotExist) && name == "" { + var config AgentConfig + + config, err = defaultAgentConfig() + if err != nil { + return nil, fmt.Errorf("read default import agent: %w", err) + } + + return &Agent{ + dir: root, + config: config, + }, nil + } + + if err != nil { + return nil, fmt.Errorf("read root agent config: %w", err) + } + + agent, err := openAgent(root) + if err != nil { + return nil, fmt.Errorf("open root agent: %w", err) + } + + if name == "" { + return agent, nil + } + + if agent.config.Parent != "" { + return nil, errors.New("import root cannot have a parent") + } + + children, err := agent.Children() + if err != nil { + return nil, fmt.Errorf("read import children: %w", err) + } + + for _, child := range children { + if child.config.Name == name { + return child, nil + } + } + + return nil, fmt.Errorf("child %q does not exist", name) +} + +// whatsappImportIdle refuses unfinished subagents and pending deliveries to the +// target. Rebuilding renumbers its log, so saved delivery IDs must not refer to +// it until their turns finish. The caller holds the token lock. +func whatsappImportIdle(root string, agent *Agent) error { + err := rebuildIdle(agent.dir) + if err != nil { + return fmt.Errorf("check target work: %w", err) + } + + data, err := os.ReadFile(filepath.Join(root, "messages.json")) + if errors.Is(err, os.ErrNotExist) { + return nil + } + + if err != nil { + return fmt.Errorf("read agent deliveries: %w", err) + } + + var queue DaemonMessageFile + + err = json.Unmarshal(data, &queue) + if err != nil { + return fmt.Errorf("decode agent deliveries: %w", err) + } + + for _, message := range queue.Pending { + if message.Target == agent.config.Name { + return fmt.Errorf("import refused: agent %q has pending deliveries", agent.config.Name) + } + } + + return nil +} + // whatsappImportNames reads only local session data. Group names that the // library keeps only in memory can also come from logged or pending batches. // Ambiguous names have no ID. A missing session keeps sender names unchanged. diff --git a/whatsapp_import_test.go b/whatsapp_import_test.go index 1e24039..4bd0a1f 100644 --- a/whatsapp_import_test.go +++ b/whatsapp_import_test.go @@ -2,6 +2,8 @@ package main import ( "archive/zip" + "encoding/json" + "net/http" "os" "path/filepath" "strings" @@ -368,6 +370,380 @@ func TestWhatsAppImportCutoffUsesEachChatAndBothLogs(t *testing.T) { } } +func TestWhatsAppImportSelectsOnlyAnExistingDirectChild(t *testing.T) { + root := newRebuildTestAgent(t) + dir := filepath.Dir(root.dir) + t.Setenv("ONECHAT_DIR", dir) + daemonChildConfig(t, dir, "House") + + before := rebuildTestSnapshot(t, root.dir) + for _, name := range []string{"Missing", "../House", root.config.Name} { + err := whatsappImport(t.Context(), []string{"-n", "--agent", name, whatsappImportSample(t)}, &strings.Builder{}) + require.ErrorContains(t, err, "does not exist") + require.Equal(t, before, rebuildTestSnapshot(t, root.dir)) + } +} + +func TestWhatsAppImportParsesGNUFlags(t *testing.T) { + testCases := []struct { + name string + args []string + want WhatsAppImportOptions + }{ + { + name: "default target remains root", + args: []string{"One.zip", "Two.zip"}, + want: WhatsAppImportOptions{paths: []string{"One.zip", "Two.zip"}}, + }, + { + name: "long flag takes a separate child name", + args: []string{"-n", "--agent", "House", "One.zip"}, + want: WhatsAppImportOptions{ + planOnly: true, + agent: "House", + paths: []string{"One.zip"}, + }, + }, + { + name: "flags can follow paths", + args: []string{"One.zip", "--agent=House", "-n"}, + want: WhatsAppImportOptions{ + planOnly: true, + agent: "House", + paths: []string{"One.zip"}, + }, + }, + { + name: "double dash ends flag parsing", + args: []string{"-n", "--", "-Example.zip"}, + want: WhatsAppImportOptions{ + planOnly: true, + paths: []string{"-Example.zip"}, + }, + }, + } + + for _, tc := range testCases { + t.Run(tc.name, func(t *testing.T) { + options, err := whatsappImportParse(tc.args) + require.NoError(t, err) + require.Equal(t, tc.want, options) + }) + } +} + +func TestWhatsAppImportRefusesInvalidFlags(t *testing.T) { + testCases := []struct { + name string + args []string + want string + }{ + { + name: "missing exports are refused", + args: []string{"-n", "--agent", "House"}, + want: "usage:", + }, + { + name: "missing child name is refused", + args: []string{"--agent"}, + want: "needs a child name", + }, + { + name: "empty child name is refused", + args: []string{"--agent=", "One.zip"}, + want: "needs a child name", + }, + { + name: "single dash long flag is refused", + args: []string{"-agent", "House", "One.zip"}, + want: "unknown import flag", + }, + } + + for _, tc := range testCases { + t.Run(tc.name, func(t *testing.T) { + _, err := whatsappImportParse(tc.args) + require.ErrorContains(t, err, tc.want) + }) + } +} + +func TestWhatsAppImportChildPlanReadsRootCutoffButNotRootImportedDays(t *testing.T) { + root := newRebuildTestAgent(t) + dir := filepath.Dir(root.dir) + t.Setenv("ONECHAT_DIR", dir) + + childDir := daemonChildConfig(t, dir, "House") + rebuildTestLog(t, root.dir, []DatedMessage{ + { + Kind: "work", + Text: "[WhatsApp] Plans, Wednesday 2025-01-01 (imported)\n08:00 Ada: Root history", + Date: "2025-01-01T08:00:00Z", + }, + { + Kind: "work", + Text: "[WhatsApp] Plans, group example@g.us, Friday 2025-01-03\n10:00 Ada: Root live\n10:20 Ada: Last", + Date: "2025-01-03T10:20:00Z", + }, + }) + rebuildTestLog(t, childDir, []DatedMessage{ + { + Kind: "work", + Text: "[WhatsApp] Plans, Thursday 2025-01-02 (imported)\n08:00 Ada: Child history", + Date: "2025-01-02T08:00:00Z", + }, + { + Kind: "work", + Text: "[WhatsApp] Plans, group example@g.us, Saturday 2025-01-04\n10:00 Ada: Child live", + Date: "2025-01-04T10:00:00Z", + }, + }) + + path := whatsappImportZip(t, "Plans", "[01/01/2025, 08:00:00] Ada: Root history\n[02/01/2025, 08:00:00] Ada: Child history\n[03/01/2025, 09:59:59] Ada: Before\n[03/01/2025, 10:00:00] Ada: Boundary\n[03/01/2025, 10:01:00] Ada: After\n") + before := rebuildTestSnapshot(t, root.dir) + + var report strings.Builder + + require.NoError(t, whatsappImport(t.Context(), []string{"-n", "--agent", "House", path}, &report)) + require.Contains(t, report.String(), "agent: House\n") + require.Contains(t, report.String(), "batches: 2 (1 chat-days skipped)") + require.Contains(t, report.String(), "2025-01-03T10:00:00") + require.Contains(t, report.String(), "messages: 4 (merged log)") + require.Equal(t, before, rebuildTestSnapshot(t, root.dir)) + + report.Reset() + require.NoError(t, whatsappImport(t.Context(), []string{"-n", path}, &report)) + require.Contains(t, report.String(), "agent: "+root.config.Name+"\n") + require.Contains(t, report.String(), "batches: 2 (1 chat-days skipped)") + require.Equal(t, before, rebuildTestSnapshot(t, root.dir)) +} + +func TestWhatsAppImportChildRebuildUsesOwnPromptAndCache(t *testing.T) { + root := newRebuildTestAgent(t) + dir := filepath.Dir(root.dir) + childDir := daemonChildConfig(t, dir, "House") + daemonChildConfig(t, dir, "Calendar") + require.NoError(t, os.WriteFile(filepath.Join(childDir, "instructions.md"), []byte("Keep room dimensions exact."), 0o600)) + rebuildTestLog(t, root.dir, []DatedMessage{{ + Kind: "user", + Text: "Keep the root history.", + Date: "2025-02-01T00:00:00Z", + }}) + rebuildTestLog(t, childDir, []DatedMessage{{ + Kind: "user", + Text: "Keep the child history.", + Date: "2025-02-02T00:00:00Z", + }}) + + child, err := whatsappImportAgent(root.dir, "House") + require.NoError(t, err) + + old, err := rebuildReadLog(t.Context(), child.dir) + require.NoError(t, err) + + rootLog, err := rebuildReadLog(t.Context(), root.dir) + require.NoError(t, err) + + path := whatsappImportZip(t, "Plans", "[01/01/2025, 08:00:00] Ada: "+strings.Repeat("The room is ready. ", 80)+"\n") + plan, err := whatsappImportPlan(t.Context(), []string{path}, old, rootLog, nil, "", time.UTC) + require.NoError(t, err) + + before := rebuildTestSnapshot(t, child.dir) + sibling := rebuildTestSnapshot(t, filepath.Join(root.dir, "children", "Calendar")) + server := newModelFake(t) + server.reply = func(w http.ResponseWriter, _ ModelFakeCall) { + children, err := root.Children() + if err != nil || len(children) != 2 { + w.WriteHeader(http.StatusBadRequest) + + return + } + + modelWriteEvents(w, modelCompletedEvent([]any{modelMessageItem("The room is ready.")}, 100, 0, 5)) + } + + client := newModelClient(server.URL, ModelAPIKey(modelTestKey), child.CacheKey(), filepath.Join(t.TempDir(), "usage.jsonl")) + backup, err := rebuild(t.Context(), child, plan.added, client, newCompactor(client, child.Prompt, defaultConfig().Compactor, child.Tools(), child.Images()), &strings.Builder{}) + require.NoError(t, err) + require.Equal(t, before, rebuildTestSnapshot(t, backup)) + require.Equal(t, root.dir, filepath.Dir(backup)) + require.Equal(t, sibling, rebuildTestSnapshot(t, filepath.Join(root.dir, "children", "Calendar"))) + + children, err := root.Children() + require.NoError(t, err) + require.Len(t, children, 2) + + entries, err := rebuildReadLog(t.Context(), child.dir) + require.NoError(t, err) + require.Equal(t, append(plan.added, old...), entries) + + unchanged, err := rebuildReadLog(t.Context(), root.dir) + require.NoError(t, err) + require.Equal(t, rootLog, unchanged) + + prompt, err := child.Prompt() + require.NoError(t, err) + require.NotEqual(t, root.CacheKey(), child.CacheKey()) + require.Positive(t, server.Calls()) + + for i := range server.Calls() { + call := server.Call(i) + require.Equal(t, child.CacheKey(), call.body["prompt_cache_key"]) + require.Equal(t, prompt, call.body["instructions"]) + require.Contains(t, prompt, "Keep room dimensions exact.") + require.NotContains(t, string(call.raw), `"name":"whatsapp_send"`) + } +} + +func TestWhatsAppImportChildFailureKeepsConfiguredChildren(t *testing.T) { + root := newRebuildTestAgent(t) + daemonChildConfig(t, filepath.Dir(root.dir), "House") + + child, err := whatsappImportAgent(root.dir, "House") + require.NoError(t, err) + + before := rebuildTestSnapshot(t, root.dir) + server := newModelFake(t) + server.reply = func(w http.ResponseWriter, _ ModelFakeCall) { + w.WriteHeader(http.StatusBadRequest) + } + + client := newModelClient(server.URL, ModelAPIKey(modelTestKey), child.CacheKey(), filepath.Join(t.TempDir(), "usage.jsonl")) + _, err = rebuild(t.Context(), child, []DatedMessage{{ + Kind: "work", + Text: strings.Repeat("The plan is ready. ", 80), + Date: "2025-01-01T08:00:00Z", + }}, client, newCompactor(client, child.Prompt, defaultConfig().Compactor, child.Tools(), child.Images()), &strings.Builder{}) + require.ErrorContains(t, err, "rebuild") + require.Equal(t, before, rebuildTestSnapshot(t, root.dir)) + + children, err := root.Children() + require.NoError(t, err) + require.Len(t, children, 1) +} + +func TestWhatsAppImportRefusesPendingDeliveriesAndKeepsRestartSafe(t *testing.T) { + testCases := []struct { + name string + child bool + }{ + { + name: "pending child delivery keeps its log ID", + child: true, + }, + { + name: "pending root report keeps its log ID", + }, + } + + for _, tc := range testCases { + t.Run(tc.name, func(t *testing.T) { + dir := t.TempDir() + t.Setenv("ONECHAT_DIR", dir) + + server := newModelFake(t) + server.reply = func(w http.ResponseWriter, _ ModelFakeCall) { + modelWriteEvents(w, modelCompletedEvent([]any{modelMessageItem("")}, 100, 0, 5)) + } + + test := newDaemonMessageTest(t, dir, server) + index := 0 + args := []string{whatsappImportZip(t, "Plans", "[01/01/2025, 08:00:00] Ada: The room is ready.\n")} + if tc.child { + index = 1 + args = append([]string{"--agent", "Garden"}, args...) + + require.NoError(t, test.messages.Tell("OneChat", "Garden", "Keep the plan.")) + } else { + require.NoError(t, test.messages.Finish(test.loops[1], nil, "The plan is ready.")) + } + + queue := test.Queue(t) + require.Len(t, queue.Pending, 1) + require.NotNil(t, queue.Pending[0].LogID) + + before, err := test.agents[index].memory.Message(*queue.Pending[0].LogID) + require.NoError(t, err) + test.Close(t) + + root := filepath.Join(dir, "agent") + snapshot := rebuildTestSnapshot(t, root) + require.NoError(t, os.WriteFile(filepath.Join(dir, "config.json"), []byte("invalid model settings"), 0o600)) + + for _, planOnly := range []bool{true, false} { + input := append([]string(nil), args...) + if planOnly { + input = append([]string{"-n"}, input...) + } + + err = whatsappImport(t.Context(), input, &strings.Builder{}) + require.ErrorContains(t, err, "has pending deliveries") + require.Equal(t, snapshot, rebuildTestSnapshot(t, root)) + require.Zero(t, server.Calls()) + } + + restarted := newDaemonMessageTest(t, dir, server) + require.Equal(t, queue, restarted.Queue(t)) + require.Equal(t, 1, restarted.agents[index].memory.Len()) + + after, err := restarted.agents[index].memory.Message(before.ID) + require.NoError(t, err) + require.Equal(t, before, after) + restarted.Start(t, index) + restarted.events[index].WaitFor(t, "idle", 1) + require.Empty(t, restarted.Queue(t).Pending) + require.Equal(t, 1, server.Calls()) + restarted.Close(t) + + again := newDaemonMessageTest(t, dir, server) + require.False(t, again.loops[index].waiting()) + require.Equal(t, 1, server.Calls()) + }) + } +} + +func TestWhatsAppImportPendingDeliveriesBlockOnlyTheirTarget(t *testing.T) { + dir := t.TempDir() + t.Setenv("ONECHAT_DIR", dir) + + server := newModelFake(t) + test := newDaemonMessageTest(t, dir, server) + require.NoError(t, test.messages.Tell("OneChat", "Garden", "Keep the plan.")) + + queue := test.Queue(t) + test.Close(t) + + path := whatsappImportZip(t, "Plans", "[01/01/2025, 08:00:00] Ada: The room is ready.\n") + for _, name := range []string{"", "Library"} { + args := []string{"-n", path} + if name != "" { + args = append(args, "--agent", name) + } + + require.NoError(t, whatsappImport(t.Context(), args, &strings.Builder{})) + } + + queue.Pending[0].LogID = nil + data, err := json.Marshal(queue) + require.NoError(t, err) + require.NoError(t, os.WriteFile(filepath.Join(dir, "agent", "messages.json"), data, 0o600)) + require.ErrorContains(t, whatsappImport(t.Context(), []string{"-n", "--agent", "Garden", path}, &strings.Builder{}), "has pending deliveries") + require.Zero(t, server.Calls()) +} + +func TestWhatsAppImportDefaultPlanDoesNotCreateAgent(t *testing.T) { + dir := t.TempDir() + t.Setenv("ONECHAT_DIR", dir) + + var report strings.Builder + + require.NoError(t, whatsappImport(t.Context(), []string{"-n", whatsappImportSample(t)}, &report)) + require.Contains(t, report.String(), "agent: OneChat\n") + + _, err := os.Stat(filepath.Join(dir, "agent")) + require.ErrorIs(t, err, os.ErrNotExist) +} + func whatsappImportSample(t *testing.T) string { t.Helper()