X-Git-Url: https://git.giacomolli.uk/onechat/blobdiff_plain/6e7c7f3d47f086b9d59fe40b91fb071e1528684e..add3cf6416e33c24331fd1e90e99284f44d2aae2:/README.rst?ds=sidebyside diff --git a/README.rst b/README.rst index 34d0d49..a02b870 100644 --- a/README.rst +++ b/README.rst @@ -3,31 +3,23 @@ ######### OneChat is one AI agent in a single chat that never ends. A daemon owns -the agent and its files, and a terminal client and a Telegram bot talk -to it. It is in early development. Linux is supported. +the agent and its memory, and you talk to it from a terminal client, +Telegram, and through your WhatsApp chats. It is in early development. +Linux is supported. -**************** - Data directory -**************** - -OneChat keeps its data and its token file in one directory. It uses the -first of these: - -#. ``ONECHAT_DIR``, when it is set. -#. ``$XDG_DATA_HOME/onechat``, when ``XDG_DATA_HOME`` is an absolute - path. A relative value is ignored. -#. ``~/.local/share/onechat``. With no home directory, OneChat exits - with an error. +******* + Setup +******* -OneChat creates the directory with mode 0700. +Data directory +============== -*************** - Configuration -*************** +OneChat keeps all its data in one directory, with mode 0700. It uses +``ONECHAT_DIR`` when it is set, else ``$XDG_DATA_HOME/onechat``, else +``~/.local/share/onechat``. The development shell sets ``ONECHAT_DIR`` +to ``.devenv/onechat`` in the checkout. -``config.json`` in the data directory holds the settings. OneChat does -not create it. A missing file or field takes the default, and an unknown -field is an error. +Optional settings go in ``config.json`` in the data directory: .. code:: json @@ -36,299 +28,221 @@ field is an error. "compactor": {"model": "gpt-6.1-sol", "effort": "high"} } -``endpoint`` - ``chatgpt`` (the default) sends model calls to the ChatGPT backend - with the subscription login. ``api`` sends them to the OpenAI API - with the key in ``OPENAI_API_KEY``, which must then be set. With - ``chatgpt``, OneChat ignores ``OPENAI_API_KEY``. - -``compactor`` - The model and the reasoning effort of the summaries. The defaults are - ``gpt-6.1-sol`` and ``high``. - -******* - Login -******* +``endpoint`` is ``chatgpt`` (the ChatGPT subscription, the default) or +``api`` (the OpenAI API, with ``OPENAI_API_KEY`` set). ``compactor`` is +the model and effort of the memory summaries. -The ``chatgpt`` endpoint needs a login with a ChatGPT account: +Login +===== .. code:: console onechat login -The command shows a link and a code. Open the link on any device, enter -the code, and approve the login. Device code login must be allowed in -the ChatGPT security settings. OneChat writes the tokens to -``auth.json`` in the data directory, with mode 0600, and refreshes them -before they expire. The file belongs to OneChat only. Do not share it -with the Codex CLI, because each program rotates the refresh token and -logs the other out. +Open the link on any device and enter the code. Device code login must +be allowed in the ChatGPT security settings. Do not share the login with +the Codex CLI: each program logs the other out. -******* - Agent -******* +Telegram +======== -Start the daemon, and then the client in another terminal: +#. Make a bot with BotFather and keep its token. +#. Find your numeric Telegram user ID. +#. Write both to ``telegram.json`` in the data directory, with mode + 0600: -.. code:: console +.. code:: json - onechat daemon - onechat chat - -The daemon owns the root agent in ``agent/`` in the data directory and -listens on the Unix socket ``onechat.sock`` there. In the client, each -line is a message to the agent, the line ``/stop`` interrupts the -running turn, the line ``/stop NAME`` stops the subagent NAME, and -Ctrl-D ends the client. The client shows the replies, the tool calls, -and the first lines of each tool result as they happen. It also shows -each message that the user sends from another client, with the client as -a mark, such as ``[telegram] text``. When a subagent starts, the client -shows ``[NAME] started``, and it shows the whole report of the subagent, -which starts with ``[NAME]``, but not the tool calls of the subagent. -The daemon is the only process that uses the token file while it runs, -so ``onechat login`` and ``onechat probe`` refuse to start until it -stops. - -The agent runs commands with bash as the user in its working directory, -with no sandbox. It has the rights of the user, also on the token file. - -When the user asks for it, the agent starts subagents for long tasks. -Each subagent has a name, such as ``GreenOtter``, and runs in the -background while the agent answers other messages. It gets the view of -the agent and its task, and it runs its tools as the agent does, in -``agent/subagents/NAME/work``. A task in a repository gives the path of -the repository. The steps of a subagent go to its own log in -``agent/subagents/NAME``, not to the chat, and its final reply comes -back to the agent as one message that starts with ``[NAME]``. The agent -can send a subagent more messages: a running subagent reads them between -its tool calls, and a finished one goes on from its log, and its reply -comes back in the same way. When the user stops a subagent, the chat -gets a note, and the report of the subagent is logged, but it starts no -turn. Each model call of a subagent has the name in the ``agent`` field -of ``agent/usage.jsonl``. ``agent/work.json`` holds the names of the -running subagents. A subagent that runs when the daemon stops goes on at -the next start: the chat gets the note that the daemon restarted, and -the subagent gets the same note and goes on from its log. - -``agent/config.json`` holds the settings of the agent. The first start -writes it with the defaults and a random ID. Do not change the ID: the -prompt cache key comes from it. + {"token": "123456:ABC-token-from-BotFather", "owner": 123456789} -.. code:: json +Start the daemon and write to the bot in a private chat. It answers only +you. - { - "name": "OneChat", - "id": "0123456789abcdef0123456789abcdef", - "model": "gpt-6.1-sol", - "effort": "medium", - "workdir": "/home/user" - } +WhatsApp +======== -``workdir`` is an absolute path, and its default is the home directory. -``agent/instructions.md`` holds the user's instructions: who they are, -how their files are organized, and how they want work done. The prompt -ends with them, and a change applies at the next call. +Link your WhatsApp account as a linked device from the terminal client +or Telegram: -********** - Telegram -********** +``/whatsapp pair PHONE`` + Gives a pairing code. On the phone, open Linked devices, Link a + device, Link with phone number instead, and enter the code. -The daemon also connects the agent to a Telegram bot when -``telegram.json`` exists in the data directory. It polls the Bot API for -new messages, so it needs no open port. +``/whatsapp status`` + Shows the link and the routes of chats to child agents. -#. Make a bot with BotFather and keep its token. +``/whatsapp unlink`` + Ends the link. -#. Find your numeric Telegram user ID. +The session file gives full access to the account. If it leaks, remove +the device in Linked devices on the phone. -#. Write the token and the ID to ``telegram.json``, and give the file - mode 0600, because the token is a secret. The daemon refuses the file - with any other mode. +User service +============ -#. Start the daemon, open a private chat with the bot, and write. +Run the daemon as a systemd user service. Keep the ``result-service`` +link, so that Nix keeps the package: -.. code:: json +.. code:: console - {"token": "123456:ABC-token-from-BotFather", "owner": 123456789} + nix build --out-link result-service + systemctl --user link --force "$PWD/result-service/share/systemd/user/onechat.service" + systemctl --user edit onechat.service -The bot reads only the private chat of the owner and ignores all other -updates. Each text message goes to the agent as a user message, the line -``/stop`` interrupts the running turn, and the line ``/stop NAME`` stops -the subagent NAME. The bot sends the replies in parts of at most 4096 -characters, with their Markdown as Telegram formats: bold, italic, -strikethrough, code, code blocks, links, and headings as bold text. When -Telegram refuses the formats of a part, the bot sends the part again as -plain text. Each error goes as a short message in plain text. It shows -the typing action while a turn runs. It does not send the tool calls. -For a subagent it sends only ``[NAME] started`` and ``[NAME] finished``, -``[NAME] stopped``, or ``[NAME] failed``, because the reply of the agent -carries the content of the report. It sends each message that the user -types in the terminal client, with the mark ``[terminal]``. - -``agent/telegram-offset.json`` holds the ID of the next update. The bot -moves it past an update only after the message is durable in the log, so -a restart of the daemon does not lose a message. A crash just after the -write of a message can make the message repeat once. A network failure -or a rate limit makes the bot wait and try again, and does not stop the -daemon. - -********** - WhatsApp -********** - -The daemon connects to the WhatsApp account of the user as a linked -device. The terminal client and Telegram take three commands: +In the editor, set the data directory, and the tools that the agent may +use in its shell: -``/whatsapp pair PHONE`` - Links the account of PHONE, the number with its country code, such as - ``+15550001111``. The daemon answers with the steps and then with a - pairing code in its own message: in WhatsApp on the phone, open - Linked devices, tap Link a device, then tap Link with phone number - instead, and enter the code. The code expires in about 2 minutes, and - the same command gives a new code. When the link succeeds, the daemon - answers with the linked number. +.. code:: ini -``/whatsapp status`` - Shows the link: not linked, linked and connected, or linked and - disconnected, with the times of the connection. + [Service] + Environment="ONECHAT_DIR=DATA_DIR" + Environment="PATH=TOOLS_PATH" -``/whatsapp unlink`` - Ends the link and deletes the session. When WhatsApp is disconnected, - the phone does not get the logout, so remove the device in Linked - devices on the phone. - -The answers go only to the client that asked, and the commands do not go -to the agent, so the agent never gets the phone number or the code. The -chat gets only the notes that WhatsApp is linked or unlinked. - -The session is ``whatsapp/session.db`` in the data directory, with mode -0600 in a directory with mode 0700. It gives full access to the WhatsApp -account, so the daemon refuses the file when its mode is not 0600. The -shell of the agent runs as the user, so it can read the session. If the -session leaks, remove the linked device in WhatsApp on the phone, and -the session stops working. Without the file, WhatsApp is not linked. - -The daemon reads the text messages of the private chats and the groups, -from other people and from the user. It does not read media, reactions, -edits, deletions, view-once and disappearing messages, or the history -from before the link. The messages of one chat form one batch until the -chat is quiet for 20 seconds, for 2 minutes or 50 messages at the most. -Each batch goes to the chat of the agent as one work message that starts -with ``[WhatsApp]``, with the name and the ID of the chat, and the time, -the sender, and the text of each message. The messages of the user show -as ``You``. ``whatsapp/state.json``, with mode 0600, keeps the batches -that are not in the chat yet, so they survive a restart, and the records -of the sends. A crash just after a batch goes to the chat can make the -batch repeat once. - -A batch of a private chat with a message of another person starts a -turn, and so does a batch of a group with a message that mentions the -user or replies to a message of the user. The other batches go to the -chat with no turn, and the agent sees them at its next turn. A batch -that starts a turn ends with a line that tells the agent to end its turn -with an empty reply when nothing in the batch needs the user. No client -shows an empty reply, so such a batch gives no message in Telegram. The -terminal client shows each batch, and Telegram shows none. - -The phone number of the user never goes to the chat: a mention of the -user shows as ``@you``, the number shows as ``OWN``, and the chat of the -user with itself is not read. But the IDs of the other chats go to the -chat and to the model, because the agent needs them to send. The ID of a -private chat holds the phone number of the contact, such as -``15550002222@s.whatsapp.net``, and the ID of a group can hold the -number of the person who made it. - -While the daemon is stopped or disconnected, the WhatsApp servers keep -the messages for the linked device and send them at the next connection. -They keep them only for a limited time, so a long stop can lose -messages. These messages form batches by their original times, as live -messages do. The daemon waits for the end of this catch-up, or one -minute at the most, and then gives all its batches to the agent in one -turn, not one turn for each chat. The batches of ``whatsapp/state.json`` -at the start wait for the first catch-up in the same way. - -The agent finds a chat with the tool ``whatsapp_find``: it gives the -contacts and the groups whose name holds a text, with no regard to case -and accents, with their IDs. The names of the contacts come from the -session, and the names of the groups from WhatsApp, kept in memory for -one hour. The names arrive from the phone some minutes after the link, -and until then the tool says that they are not ready. The agent sends a -text message with the tool ``whatsapp_send``, only when the user asks -for it. Each send has an ID that the agent gives: a repeated call with -the same ID gives the answer of the first call and sends nothing. The -state file keeps the last 1000 send records. When the daemon stops -during a send, the result is not known, and the tool tells the agent to -ask the user to look at the chat in WhatsApp. Without a link, both tools -tell the agent to ask the user to run ``/whatsapp pair PHONE``. Only the -agent has the WhatsApp tools, not its subagents. - -Each batch of a group goes to the chat, also with no turn, and the -memory tree summarizes it with model calls, which use the quota. - -When the session holds a linked device, the daemon connects at its -start, and after a network failure it connects again by itself. When a -disconnection lasts 10 minutes, all clients get one notice, and one more -when the connection comes back. When WhatsApp ends the connection for -good, for example because another program uses the same session, all -clients get a notice at once. When the user removes the device in the -phone, the daemon deletes the session and tells all clients one time. +Then start it, and let it run while you are logged out: -******* - Probe -******* +.. code:: console + + systemctl --user daemon-reload + systemctl --user enable --now onechat.service + loginctl enable-linger "$USER" -``onechat probe LOG COUNT [BUILDS]`` measures the compactor on a log. It -appends the first COUNT messages of LOG, one JSON object ``{"kind", -"text"}`` on each line, to a new chat in ``probe/`` in the data -directory, and builds its tree with real model calls. The calls use the -quota of the account. The run stops at the first failed call, after 660 -calls, or when the used percent of a quota window grows by more than 15 -points. At the end it reports the calls per message, the cached share of -the input tokens, the retries, the line sizes, and the quota at the -first and the last call. BUILDS, from 1 to 8, limits the model calls at -once in the run. +After a ``git pull``, ``bin/service-update`` builds and restarts it. +Never run two daemons on the same data. -A test writes a generated log of readable made-up messages: +Backups +======= .. code:: console - ONECHAT_PROBE_LOG=probe.jsonl go test -run TestProbeLogHasTheSyntheticMix + onechat backup BACKUP_DIR/onechat.tar.gz + ONECHAT_DIR=EMPTY_DIR onechat restore BACKUP_DIR/onechat.tar.gz -************* - Development -************* +A backup runs while the daemon runs. Restore needs a stopped daemon and +an empty directory. **An archive holds your logins, the bot token, and +the WhatsApp link**, so keep it private. -Nix supplies the Go toolchain and all tools. Enter the development shell -with direnv: +For a daily backup with the last 14 kept, save this script, with +``BACKUP_DIR`` replaced by a private directory outside the data +directory: + +.. code:: sh + + #!/bin/sh + set -eu + umask 077 + cd BACKUP_DIR + onechat backup "onechat-$(date +%F).tar.gz" + ls -1 onechat-????-??-??.tar.gz | sort -r | tail -n +15 | xargs -r rm -- + +and run it from a ``Type=oneshot`` user service with the same +``ONECHAT_DIR`` and an ``OnCalendar=daily`` timer. + +********* + Talking +********* .. code:: console - direnv allow + onechat + +``onechat`` connects to the running daemon. Each line is a message, and +replies show with Markdown styles and tables. These lines are commands: + +``/stop`` + Interrupts the running turn. + +``/stop NAME`` + Stops the subagent NAME. + +``/image PATH [caption]`` + Sends a JPEG, PNG, or WebP image. Quote a path with spaces. + +``/whatsapp ...`` + The WhatsApp commands above. + +In Telegram, write as in the client, also with photos and image files. +``onechat help`` lists all commands. -Or enter it directly: +********************* + What the agent does +********************* + +Memory + The agent remembers the whole chat as summaries that it can open + again, so it never starts over. ``agent/instructions.md`` holds your + lasting preferences. The agent can propose a change to it and makes + the change only after you agree. + +Tools + The agent runs shell commands as you, in its working directory, with + no sandbox. + +Subagents + For long tasks it starts subagents in the background, such as + ``GreenOtter``, and gets their reports as messages. + +Child agents + A child is a permanent agent for one area of your life, such as + ``House``, with its own memory. The root creates it, talks to it, and + hears from it only what needs you. Children never send WhatsApp + messages; they draft replies for you to approve. + +WhatsApp + The agent reads your chats in batches and wakes for a private message + or a mention of you. It finds chats and sends messages only when you + ask. It proposes to route a chat that is only about one area to the + child of that area, and sets the route after you agree. + +Images + Send a photo in Telegram, WhatsApp, or with ``/image`` to the agent. + Its memory keeps what the image shows, and it can look at an older + image again. + +Schedules + Ask for reminders and timed tasks, such as "every weekday at 8, tell + me what my groups talked about". Each run uses the quota, so runs are + at least 5 minutes apart. + +Import + Old WhatsApp chats can join the memory in date order. Stop the + daemon, export the chat as a ZIP, and run the plan before the import. + The import rebuilds the target's memory with model calls: .. code:: console - nix develop + onechat import -n [--agent NAME] CHAT.zip + onechat import [--agent NAME] CHAT.zip -The shell sets ``ONECHAT_DIR`` to ``.devenv/onechat`` in the checkout, -unless it is already set. Development data goes there, and Git ignores -``.devenv``. +***** + Log +***** -Build and run: +The daemon writes short events with ``key=value`` attributes to the user +journal, at three levels: info, warning for a failure that is retried or +loses nothing, and error for a failure that loses data or stops work. .. code:: console - nix build - ./result/bin/onechat + journalctl --user -u onechat -f + journalctl --user -u onechat -p warning -Format and check all files: +************* + Development +************* + +Nix supplies the Go toolchain and all tools. Enter the development shell +with ``direnv allow`` or ``nix develop``. Then: .. code:: console + nix build + ./result/bin/onechat version bin/format bin/check nix flake check -``bin/check`` and ``nix build`` run a short replay of the synthetic log. -``nix flake check`` also runs the full replay of 100,000 messages. +``onechat version`` prints ``yyyy-mm-dd-gitshort`` for a Nix build and +``dev`` for a direct Go build. ``bin/check`` runs the linters and a +short replay of a synthetic log, and ``nix flake check`` also runs the +full replay of 100,000 messages. ``onechat probe LOG COUNT`` measures +the memory summaries on a log with real model calls.