X-Git-Url: https://git.giacomolli.uk/onechat/blobdiff_plain/e1e48ba388c28105fff3b70c8ca02304f5d57f53..277a37a79b51910074a29ca6d70fae75c73619b7:/README.rst?ds=sidebyside diff --git a/README.rst b/README.rst index 3819e25..7ca3d74 100644 --- a/README.rst +++ b/README.rst @@ -73,17 +73,28 @@ Start the daemon, and then the client in another terminal: .. code:: console onechat daemon - onechat chat + onechat + +``onechat`` with no command is the same as ``onechat chat``. ``onechat +help`` lists the commands. The daemon owns the root agent in ``agent/`` in the data directory and listens on the Unix socket ``onechat.sock`` there. In the client, each line is a message to the agent, the line ``/stop`` interrupts the -running turn, and Ctrl-D ends the client. The client shows the replies, -the tool calls, and the first lines of each tool result as they happen. -It also shows each message that the user sends from another client, with -the client as a mark, such as ``[telegram] text``. The daemon is the -only process that uses the token file while it runs, so ``onechat -login`` and ``onechat probe`` refuse to start until it stops. +running turn, the line ``/stop NAME`` stops the subagent NAME, and +Ctrl-D ends the client. On terminal output, the client shows ``> `` +after each event while idle and clears it before the next event. An +event can interrupt a line while the user types. The client shows the +replies, the tool calls, and the first lines of each tool result as they +happen. Replies and subagent reports render Markdown with styles and +wrapped tables only when the output is a terminal, and stay unchanged in +a pipe or file. It also shows each message that the user sends from +another client, with the client as a mark, such as ``[telegram] text``. +When a subagent starts, the client shows ``[NAME] started``, and it +shows the whole report of the subagent, which starts with ``[NAME]``, +but not the tool calls of the subagent. The daemon is the only process +that uses the token file while it runs, so ``onechat login`` and +``onechat probe`` refuse to start until it stops. The agent runs commands with bash as the user in its working directory, with no sandbox. It has the rights of the user, also on the token file. @@ -98,8 +109,13 @@ the repository. The steps of a subagent go to its own log in back to the agent as one message that starts with ``[NAME]``. The agent can send a subagent more messages: a running subagent reads them between its tool calls, and a finished one goes on from its log, and its reply -comes back in the same way. Each model call of a subagent has the name -in the ``agent`` field of ``agent/usage.jsonl``. +comes back in the same way. When the user stops a subagent, the chat +gets a note, and the report of the subagent is logged, but it starts no +turn. Each model call of a subagent has the name in the ``agent`` field +of ``agent/usage.jsonl``. ``agent/work.json`` holds the names of the +running subagents. A subagent that runs when the daemon stops goes on at +the next start: the chat gets the note that the daemon restarted, and +the subagent gets the same note and goes on from its log. ``agent/config.json`` holds the settings of the agent. The first start writes it with the defaults and a random ID. Do not change the ID: the @@ -118,7 +134,250 @@ prompt cache key comes from it. ``workdir`` is an absolute path, and its default is the home directory. ``agent/instructions.md`` holds the user's instructions: who they are, how their files are organized, and how they want work done. The prompt -ends with them, and a change applies at the next call. +ends with them, and a change applies at the next call. The agent can +propose a change to the file, and it makes the change only after the +user agrees. + +A permanent child agent lives in ``agent/children/NAME/`` with the same +files as the root. The root's ``agent_create(name, responsibility)`` +tool creates and starts it with the root's model, effort, and working +directory, and a separate random ID. A name describes the area, such as +``House`` or ``Calendar``. It is one word of 2 to 32 ASCII letters, +starts with a capital letter, and cannot name the root, an existing +child, or a subagent. The daemon opens all children at startup. A +missing or invalid child config stops startup. Only one level of +children is supported. Remove a child by hand while the daemon is +stopped. The root's turn state lists each child and its responsibility. +The root passes area messages from mixed chats with sender, date, and +words, and asks the user before it sends a child's draft to WhatsApp. + +Each child has its own memory, model client, usage file, subagents, +schedules, and turn loop. Its prompt limits work to its area, asks for +only reports that need the parent or the user, and permits an empty +reply. Children have no WhatsApp tools and draft replies instead of +sending them. Clients address only the root. A child's saved schedules +and subagents can resume at startup. The ready log line lists the child +names. + +A message from a parent to its own child enters the child's log as +``user`` and wakes it. A child's final reply reaches the parent as +``work`` with the prefix ``[NAME] `` and wakes it. An empty reply sends +nothing. A busy recipient takes the message between tool calls or in its +next turn. ``agent/messages.json``, with mode 0600, keeps messages until +their recipient finishes its turn. A daemon stop preserves queued +messages and reports for the next start without a second log entry. An +interrupted turn can repeat its model calls and tools after restart. +Backups lock this queue with the agent files, including children made +during the daemon run. The root uses ``tell`` for both children and +subagents. ``zoom("Name")`` gives a child's view or a subagent's whole +log. ``spawn`` starts short-lived subagents, not permanent children. + +************** + User service +************** + +The Nix package includes ``share/systemd/user/onechat.service``. It runs +``onechat daemon`` from that package, starts at login when enabled, and +restarts 5 seconds after a failure. A stop sends SIGTERM to the daemon +and allows up to 120 seconds for cleanup. Its output goes to the user +journal. The service has the rights of the user, as the terminal daemon +does. + +Build a package with a separate result link. Keep this link so that Nix +does not collect the package while the service uses it: + +.. code:: console + + nix build --out-link result-service + systemctl --user link --force "$PWD/result-service/share/systemd/user/onechat.service" + systemctl --user edit onechat.service + +In the editor, set the data directory with this drop-in. Replace +``DATA_DIR`` with an absolute path to the data directory. Set the same +``ONECHAT_DIR`` for each terminal client. Without a drop-in, the data +directory rules above apply to the environment of the user manager, not +to the development shell. + +.. code:: ini + + [Service] + Environment="ONECHAT_DIR=DATA_DIR" + +The service does not read the environment of an open terminal. If the +agent needs tools from a shell, set ``Environment="PATH=TOOLS_PATH"`` in +the same drop-in, with ``TOOLS_PATH`` replaced by the complete search +path, including bash. For the API endpoint, use +``EnvironmentFile=ENV_FILE``, with ``ENV_FILE`` replaced by an absolute +path to a mode 0600 file that sets ``OPENAI_API_KEY``. Keep secrets out +of the unit and Nix sources. + +Use the existing data directory, or copy the development data before the +first service start. For a copy, first stop the terminal daemon with +Ctrl-C and wait for it to exit. Replace ``DATA_DIR`` below with a new, +empty destination. If the development shell uses a custom +``ONECHAT_DIR``, replace ``.devenv/onechat`` with that source. Copy the +whole directory so that ``auth.json``, ``telegram.json``, +``config.json``, ``agent/``, and ``whatsapp/`` stay together, including +the WhatsApp session and its database files. The copy preserves the file +modes and the source as a backup. Do not copy or use the same data from +two running daemons. + +.. code:: console + + install -d -m 0700 DATA_DIR + cp -a .devenv/onechat/. DATA_DIR/ + +After the terminal daemon exits and the data is ready, enable and start +the service: + +.. code:: console + + systemctl --user daemon-reload + systemctl --user enable --now onechat.service + systemctl --user status onechat.service + ONECHAT_DIR=DATA_DIR ./result-service/bin/onechat + journalctl --user -u onechat.service -f + +After a pull, update the package and unit with: + +.. code:: console + + git pull + bin/service-update + +The script builds with the same ``result-service`` link, reloads the +user manager, and restarts the service only if it is active. The unit +link follows ``result-service``, so no new link is needed. + +The service starts with the user manager at login, not at boot without a +login. To stop it, or also remove its start at login: + +.. code:: console + + systemctl --user stop onechat.service + systemctl --user disable onechat.service + +******************** + Backup and restore +******************** + +Back up the running daemon to one mode 0600 ``.tar.gz`` archive. The +command prints its path and size in bytes. With no daemon, it fails. + +.. code:: console + + onechat backup BACKUP_FILE.tar.gz + ONECHAT_DIR=EMPTY_DATA_DIR onechat restore BACKUP_FILE.tar.gz + +The daemon flushes memory and holds its write locks during the agent +copy. It also flushes and locks the child agents and all subagent logs. +It copies the WhatsApp session with SQLite ``VACUUM INTO``, not as raw +database files. It then compresses the copy and installs the archive +through a temporary file and a rename. Use a backup path outside the +data directory. + +The archive includes configuration, instructions, the log, tree, +indexes, views, layout, schedules, subagents, children, Telegram offset, +tokens, and WhatsApp state and session. It excludes sockets, lock files, +``probe/``, and the ``agent-backup-*`` import copies. Links and special +files fail, rather than enter the archive. **The archive holds +secrets**, including login tokens, the Telegram bot token, and the +WhatsApp link. Keep it private. There is no encryption or remote copy. + +Restore needs a stopped daemon and an empty destination. It checks the +token lock and validates the archive before it installs files, with +their modes. It rejects unsafe paths, links, and special files. A failed +restore can leave ``auth.lock`` in the destination. Use another empty +directory for a retry. Do not run the source and restored WhatsApp links +at the same time. Check the restored log and view before you use it in +place of the source. + +For a daily local archive with the last 14 kept, save this script as +``BACKUP_SCRIPT``. Replace ``BACKUP_DIR`` with a private archive +directory outside the data directory. Replace ``ONECHAT_BINARY`` with +the absolute path through the stable ``result-service/bin/onechat`` +package link. Make the archive directory with mode 0700 before you +enable the timer. + +.. code:: sh + + #!/bin/sh + set -eu + umask 077 + cd BACKUP_DIR + ONECHAT_BINARY backup "onechat-$(date +%F).tar.gz" + find . -maxdepth 1 -type f -name 'onechat-????-??-??.tar.gz' -printf '%f\n' | + sort -r | tail -n +15 | + while IFS= read -r file; do + rm -- "$file" + done + +Save these units as ``onechat-backup.service`` and +``onechat-backup.timer`` in the systemd user unit directory. Replace +``DATA_DIR`` and ``BACKUP_SCRIPT`` with absolute paths. The service uses +the daemon unit from User service above. + +.. code:: ini + + [Unit] + Description=Back up OneChat data + Requires=onechat.service + After=onechat.service + + [Service] + Type=oneshot + Environment="ONECHAT_DIR=DATA_DIR" + UMask=0077 + ExecStart=/bin/sh BACKUP_SCRIPT + +.. code:: ini + + [Unit] + Description=Back up OneChat each day + + [Timer] + OnCalendar=daily + Persistent=true + + [Install] + WantedBy=timers.target + +Enable the timer with the user manager. A failed backup leaves the +existing archives in place and skips removal of old archives. + +.. code:: console + + systemctl --user daemon-reload + systemctl --user enable --now onechat-backup.timer + systemctl --user list-timers onechat-backup.timer + +************ + Daemon log +************ + +The daemon writes short events with the part prefix and ordered +``key=value`` attributes. An error cause comes last. Normal events use +info, failures that can be retried or lose nothing use warning, and +failures that lose data or stop work use error. + +With ``JOURNAL_STREAM`` set, each line carries a syslog priority without +a time stamp or level word. The user journal can show only warnings and +errors: + +.. code:: console + + journalctl --user -u onechat -p warning + +The summary warning names its node. A schedule run has ``late`` only +when it ran late: + +.. code:: text + + <4>onechat: summary failed level=0 index=0: fake failure + <6>onechat: schedule ran id=1 + <6>onechat: schedule ran id=2 late=1m0s + +Outside the journal, each line starts with the time stamp and level. ********** Telegram @@ -143,15 +402,18 @@ new messages, so it needs no open port. {"token": "123456:ABC-token-from-BotFather", "owner": 123456789} The bot reads only the private chat of the owner and ignores all other -updates. Each text message goes to the agent as a user message, and the -line ``/stop`` interrupts the running turn. The bot sends the replies in -parts of at most 4096 characters, with their Markdown as Telegram -formats: bold, italic, strikethrough, code, code blocks, links, and -headings as bold text. When Telegram refuses the formats of a part, the -bot sends the part again as plain text. Each error goes as a short -message in plain text. It shows the typing action while a turn runs. It -does not send the tool calls. It sends each message that the user types -in the terminal client, with the mark ``[terminal]``. +updates. Each text message goes to the agent as a user message, the line +``/stop`` interrupts the running turn, and the line ``/stop NAME`` stops +the subagent NAME. The bot sends the replies in parts of at most 4096 +characters, with their Markdown as Telegram formats: bold, italic, +strikethrough, code, code blocks, links, and headings as bold text. When +Telegram refuses the formats of a part, the bot sends the part again as +plain text. Each error goes as a short message in plain text. It shows +the typing action while a turn runs. It does not send the tool calls. +For a subagent it sends only ``[NAME] started`` and ``[NAME] finished``, +``[NAME] stopped``, or ``[NAME] failed``, because the reply of the agent +carries the content of the report. It sends each message that the user +types in the terminal client, with the mark ``[terminal]``. ``agent/telegram-offset.json`` holds the ID of the next update. The bot moves it past an update only after the message is durable in the log, so @@ -160,6 +422,223 @@ write of a message can make the message repeat once. A network failure or a rate limit makes the bot wait and try again, and does not stop the daemon. +********** + WhatsApp +********** + +The daemon connects to the WhatsApp account of the user as a linked +device. The terminal client and Telegram take three commands: + +``/whatsapp pair PHONE`` + Links the account of PHONE, the number with its country code, such as + ``+15550001111``. The daemon answers with the steps and then with a + pairing code in its own message: in WhatsApp on the phone, open + Linked devices, tap Link a device, then tap Link with phone number + instead, and enter the code. The code expires in about 2 minutes, and + the same command gives a new code. When the link succeeds, the daemon + answers with the linked number. Linked devices on the phone shows the + device as OneChat. The name applies at the pairing, so an existing + link keeps its old name until ``/whatsapp unlink`` and a new + ``/whatsapp pair PHONE``. + +``/whatsapp status`` + Shows the link: not linked, linked and connected, or linked and + disconnected, with the times of the connection and the WhatsApp + routes. + +``/whatsapp unlink`` + Ends the link and deletes the session. When WhatsApp is disconnected, + the phone does not get the logout, so remove the device in Linked + devices on the phone. + +The answers go only to the client that asked, and the commands do not go +to the agent, so the agent never gets the phone number or the code. The +chat gets only the notes that WhatsApp is linked or unlinked. + +The session is ``whatsapp/session.db`` in the data directory, with mode +0600 in a directory with mode 0700. It gives full access to the WhatsApp +account, so the daemon refuses the file when its mode is not 0600. The +shell of the agent runs as the user, so it can read the session. If the +session leaks, remove the linked device in WhatsApp on the phone, and +the session stops working. Without the file, WhatsApp is not linked. + +The daemon reads the text messages of the private chats and the groups, +from other people and from the user. It does not read media, reactions, +edits, deletions, view-once and disappearing messages, or the history +from before the link. The messages of one chat form one batch until the +chat is quiet for 20 seconds, for 2 minutes or 50 messages at the most. +Each batch goes to the chat of the agent as one work message that starts +with ``[WhatsApp]``, with the name and the ID of the chat, and the time, +the sender, and the text of each message. The messages of the user show +as ``You``. ``whatsapp/state.json``, with mode 0600, keeps the batches +that are not in the chat yet, so they survive a restart, and the records +of the sends. A crash just after a batch goes to the chat can make the +batch repeat once. + +A batch of a private chat with a message of another person starts a +turn, and so does a batch of a group with a message that mentions the +user or replies to a message of the user. The other batches go to the +chat with no turn, and the agent sees them at its next turn. A batch +that starts a turn ends with a line that tells the agent to end its turn +with an empty reply when nothing in the batch needs the user. No client +shows an empty reply, so such a batch gives no message in Telegram. The +terminal client shows each batch, and Telegram shows none. + +The phone number of the user never goes to the chat: a mention of the +user shows as ``@you``, the number shows as ``OWN``, and the chat of the +user with itself is not read. But the IDs of the other chats go to the +chat and to the model, because the agent needs them to send. The ID of a +private chat holds the phone number of the contact, such as +``15550002222@s.whatsapp.net``, and the ID of a group can hold the +number of the person who made it. + +While the daemon is stopped or disconnected, the WhatsApp servers keep +the messages for the linked device and send them at the next connection. +They keep them only for a limited time, so a long stop can lose +messages. These messages form batches by their original times, as live +messages do. The daemon waits for the end of this catch-up, or one +minute at the most, and then gives all its unrouted batches to the root +in one turn, not one turn for each chat. Routed batches wake their +child. The batches of ``whatsapp/state.json`` at the start wait for the +first catch-up in the same way. + +The agent finds a chat with the tool ``whatsapp_find``: it gives the +contacts and the groups whose name holds a text, with no regard to case +and accents, with their IDs. The names of the contacts come from the +session, and the names of the groups from WhatsApp, kept in memory for +one hour. The names arrive from the phone some minutes after the link, +and until then the tool says that they are not ready. The agent sends a +text message with the tool ``whatsapp_send``, only when the user asks +for it. Each send has an ID that the agent gives: a repeated call with +the same ID gives the answer of the first call and sends nothing. The +state file keeps the last 1000 send records. When the daemon stops +during a send, the result is not known, and the tool tells the agent to +ask the user to look at the chat in WhatsApp. Without a link, both tools +tell the agent to ask the user to run ``/whatsapp pair PHONE``. Only the +agent has the WhatsApp tools, not its subagents. + +Each batch of a group goes to the chat, also with no turn, and the +memory tree summarizes it with model calls, which use the quota. + +The root manages source routes with ``route(source, agent)``, +``unroute(source)``, and ``routes()``, only after the user agrees. A +source is ``whatsapp:CHAT-ID``, with the ID from ``whatsapp_find`` or a +``[WhatsApp]`` message, and the agent must be its child. The root keeps +the map in ``agent/routes.json`` with mode 0600. Children and subagents +have no route tools. There is no command to change routes. Each routed +batch enters only its child's log and wakes it, including a group batch +with no mention. The durable agent queue prevents a second log entry +after a restart. An unrouted source or a missing child uses the root. A +queued batch whose child is removed while stopped goes to the root. + +When the session holds a linked device, the daemon connects at its +start. If the first connection fails, it tries again after 2 seconds, +then doubles the wait up to 5 minutes. The inbox and commands stay +active during each wait. An unlink or a logout stops these retries. +After a network failure of an open connection, the library connects +again by itself. When a disconnection lasts 10 minutes, all clients get +one notice, and one more when the connection comes back. When WhatsApp +ends the connection for good, for example because another program uses +the same session, all clients get a notice at once. When the user +removes the device in the phone, the daemon deletes the session and +tells all clients one time. + +************************* + Import WhatsApp history +************************* + +Stop the daemon and finish all subagents before an import. Export each +chat from WhatsApp as a ZIP with ``_chat.txt``. Keep the chat name in +the ZIP name, such as ``WhatsApp Chat - Example.zip``. Read the plan +first: + +.. code:: console + + onechat import -n CHAT.zip OTHER.zip + onechat import CHAT.zip OTHER.zip + onechat import -n --agent House CHAT.zip OTHER.zip + onechat import --agent House CHAT.zip OTHER.zip + +The target is the root unless ``--agent NAME`` selects an existing +direct child. A missing child is refused. The rebuild uses the target's +log, prompt, cache key, and usage records. The root keeps its history +when the target is a child. An import or plan is refused while the +target has pending agent deliveries. Start the daemon and let those +turns finish before you stop it and try again. This keeps their saved +log IDs valid. + +``-n`` makes no model call and prints the target name, chats, chat-day +batches, local dates, cutoffs, text bytes, merged message count, and +estimated calls. The import keeps multiline text, system lines, and +omitted-media markers. It maps the session's profile name to ``You``. +Without that name, it keeps the senders as written and says so in the +plan. An exact, unambiguous local contact or batch name supplies the +chat ID when known. No network lookup is made. + +Each chat and local day becomes one work message marked ``(imported)``. +For each chat, the cutoff is the first message time of its earliest live +WhatsApp batch across the root and target logs. Only messages before +that cutoff enter the import. Live times have minute precision, so the +cutoff excludes that whole minute. A chat-day already imported into the +target is skipped. An imported day held only by the root does not skip +that day for a child. Overlapping chat-days in the supplied ZIPs are +refused. Review the exports before import: their text goes to the model +as chat history. + +The import merges the full log in date order and builds its tree with +real compactor calls, the agent prompt, and the agent cache key. It uses +a limit of 1,500 calls and the probe's 15-quota-point limit, and prints +the same usage report. Its estimate is about 1.3 calls per merged +message, not a cost guarantee. A stopped or failed run leaves the agent +unchanged. A new run starts over. Success keeps the other agent files +and renames the old directory to ``agent-backup-TIMESTAMP`` before it +installs the new one. A child's staging directory and backup stay in the +root agent directory, outside ``children/``. Its backup is +``NAME-backup-TIMESTAMP``. Keep the backup until the new memory is +checked, then start the daemon. Message IDs change to their positions in +the merged log. + +*********** + Schedules +*********** + +The agent sets reminders and timed tasks when the user asks for them, +such as "remind me tomorrow at 9" or "every weekday at 8, tell me what +my WhatsApp groups talked about". A schedule has a text, the task for +the agent, and either one time or a cron expression. The agent also +lists the schedules with their next times, and removes one when the user +asks. The root and its children have the schedule tools, not their +subagents. Each schedule records its owner and wakes only that agent. + +A cron expression has five fields: minute, hour, day of month, month, +and day of week, where 0 and 7 are Sunday. A field is a comma list of +``*``, ``N``, and ``N-M``, and ``*`` and ``N-M`` can have a step ``/S``, +as in ``0 8 * * 1-5`` or ``*/15 9-17 * * *``. When both day fields start +with something other than ``*``, a day matches when either field +matches. Names such as ``MON``, macros such as ``@daily``, the +characters ``? L W #``, and ``N/S`` are refused. The runs must be at +least 5 minutes apart, because each run is a full turn that uses the +quota, so ``*/5 * * * *`` is the shortest schedule. + +The times use the time zone of the machine. A time that the clock skips +at the start of daylight saving time runs later by the length of the +skip. A time that the clock repeats at its end runs once, at its first +occurrence. + +At its time, a schedule goes to the chat as a work message that starts +with ``[Schedule ID]``, with its time and its text, and it starts a +turn. The message ends with a line that tells the agent to end its turn +with an empty reply when nothing needs the user. The terminal client +shows the message, and Telegram shows only the reply. After its run, a +one-time schedule goes, and a repeating schedule gets its next time. + +``agent/schedules.json``, with mode 0600, holds the schedules. Each +change writes it before the tool answers, so the schedules survive a +restart. A schedule that came due while the daemon was stopped runs once +at the next start, with its original time and a line that says that the +run is late. A repeating schedule that missed several times runs once. A +crash just after a run goes to the chat can make the run repeat once. + ******* Probe ******* @@ -209,6 +688,9 @@ Build and run: nix build ./result/bin/onechat +``onechat version`` prints ``yyyy-mm-dd-gitshort`` for a Nix build, adds +``-dirty`` for a dirty tree, and prints ``dev`` for a direct Go build. + Format and check all files: .. code:: console