X-Git-Url: https://git.giacomolli.uk/onechat/blobdiff_plain/522d226230a2eae0a8a33ae0fbf32f96c6886b26..3391e4eaab1c20e3b172c443186b1947622a679e:/README.rst?ds=inline diff --git a/README.rst b/README.rst index c74a815..929272c 100644 --- a/README.rst +++ b/README.rst @@ -2,8 +2,9 @@ OneChat ######### -OneChat is a Go program. It prints its name. It is in early development. -Linux is supported. +OneChat is one AI agent in a single chat that never ends. A daemon owns +the agent and its files, and a terminal client and a Telegram bot talk +to it. It is in early development. Linux is supported. **************** Data directory @@ -32,7 +33,7 @@ field is an error. { "endpoint": "chatgpt", - "compactor": {"model": "gpt-6-luna", "effort": "high"} + "compactor": {"model": "gpt-6.1-sol", "effort": "high"} } ``endpoint`` @@ -43,7 +44,7 @@ field is an error. ``compactor`` The model and the reasoning effort of the summaries. The defaults are - ``gpt-6-luna`` and ``high``. + ``gpt-6.1-sol`` and ``high``. ******* Login @@ -63,18 +64,438 @@ before they expire. The file belongs to OneChat only. Do not share it with the Codex CLI, because each program rotates the refresh token and logs the other out. +******* + Agent +******* + +Start the daemon, and then the client in another terminal: + +.. code:: console + + onechat daemon + onechat + +``onechat`` with no command is the same as ``onechat chat``. ``onechat +help`` lists the commands. + +The daemon owns the root agent in ``agent/`` in the data directory and +listens on the Unix socket ``onechat.sock`` there. In the client, each +line is a message to the agent, the line ``/stop`` interrupts the +running turn, the line ``/stop NAME`` stops the subagent NAME, and +Ctrl-D ends the client. On terminal output, the client shows ``> `` +after each event while idle and clears it before the next event. An +event can interrupt a line while the user types. The client shows the +replies, the tool calls, and the first lines of each tool result as they +happen. Replies and subagent reports render Markdown with styles and +wrapped tables only when the output is a terminal, and stay unchanged in +a pipe or file. It also shows each message that the user sends from +another client, with the client as a mark, such as ``[telegram] text``. +When a subagent starts, the client shows ``[NAME] started``, and it +shows the whole report of the subagent, which starts with ``[NAME]``, +but not the tool calls of the subagent. The daemon is the only process +that uses the token file while it runs, so ``onechat login`` and +``onechat probe`` refuse to start until it stops. + +The agent runs commands with bash as the user in its working directory, +with no sandbox. It has the rights of the user, also on the token file. + +When the user asks for it, the agent starts subagents for long tasks. +Each subagent has a name, such as ``GreenOtter``, and runs in the +background while the agent answers other messages. It gets the view of +the agent and its task, and it runs its tools as the agent does, in +``agent/subagents/NAME/work``. A task in a repository gives the path of +the repository. The steps of a subagent go to its own log in +``agent/subagents/NAME``, not to the chat, and its final reply comes +back to the agent as one message that starts with ``[NAME]``. The agent +can send a subagent more messages: a running subagent reads them between +its tool calls, and a finished one goes on from its log, and its reply +comes back in the same way. When the user stops a subagent, the chat +gets a note, and the report of the subagent is logged, but it starts no +turn. Each model call of a subagent has the name in the ``agent`` field +of ``agent/usage.jsonl``. ``agent/work.json`` holds the names of the +running subagents. A subagent that runs when the daemon stops goes on at +the next start: the chat gets the note that the daemon restarted, and +the subagent gets the same note and goes on from its log. + +``agent/config.json`` holds the settings of the agent. The first start +writes it with the defaults and a random ID. Do not change the ID: the +prompt cache key comes from it. + +.. code:: json + + { + "name": "OneChat", + "id": "0123456789abcdef0123456789abcdef", + "model": "gpt-6.1-sol", + "effort": "medium", + "workdir": "/home/user" + } + +``workdir`` is an absolute path, and its default is the home directory. +``agent/instructions.md`` holds the user's instructions: who they are, +how their files are organized, and how they want work done. The prompt +ends with them, and a change applies at the next call. The agent can +propose a change to the file, and it makes the change only after the +user agrees. + +************** + User service +************** + +The Nix package includes ``share/systemd/user/onechat.service``. It runs +``onechat daemon`` from that package, starts at login when enabled, and +restarts 5 seconds after a failure. A stop sends SIGTERM to the daemon +and allows up to 120 seconds for cleanup. Its output goes to the user +journal. The service has the rights of the user, as the terminal daemon +does. + +Build a package with a separate result link. Keep this link so that Nix +does not collect the package while the service uses it: + +.. code:: console + + nix build --out-link result-service + systemctl --user link --force "$PWD/result-service/share/systemd/user/onechat.service" + systemctl --user edit onechat.service + +In the editor, set the data directory with this drop-in. Replace +``DATA_DIR`` with an absolute path to the data directory. Set the same +``ONECHAT_DIR`` for each terminal client. Without a drop-in, the data +directory rules above apply to the environment of the user manager, not +to the development shell. + +.. code:: ini + + [Service] + Environment="ONECHAT_DIR=DATA_DIR" + +The service does not read the environment of an open terminal. If the +agent needs tools from a shell, set ``Environment="PATH=TOOLS_PATH"`` in +the same drop-in, with ``TOOLS_PATH`` replaced by the complete search +path, including bash. For the API endpoint, use +``EnvironmentFile=ENV_FILE``, with ``ENV_FILE`` replaced by an absolute +path to a mode 0600 file that sets ``OPENAI_API_KEY``. Keep secrets out +of the unit and Nix sources. + +Use the existing data directory, or copy the development data before the +first service start. For a copy, first stop the terminal daemon with +Ctrl-C and wait for it to exit. Replace ``DATA_DIR`` below with a new, +empty destination. If the development shell uses a custom +``ONECHAT_DIR``, replace ``.devenv/onechat`` with that source. Copy the +whole directory so that ``auth.json``, ``telegram.json``, +``config.json``, ``agent/``, and ``whatsapp/`` stay together, including +the WhatsApp session and its database files. The copy preserves the file +modes and the source as a backup. Do not copy or use the same data from +two running daemons. + +.. code:: console + + install -d -m 0700 DATA_DIR + cp -a .devenv/onechat/. DATA_DIR/ + +After the terminal daemon exits and the data is ready, enable and start +the service: + +.. code:: console + + systemctl --user daemon-reload + systemctl --user enable --now onechat.service + systemctl --user status onechat.service + ONECHAT_DIR=DATA_DIR ./result-service/bin/onechat + journalctl --user -u onechat.service -f + +After a pull, update the package and unit with: + +.. code:: console + + git pull + bin/service-update + +The script builds with the same ``result-service`` link, reloads the +user manager, and restarts the service only if it is active. The unit +link follows ``result-service``, so no new link is needed. + +The service starts with the user manager at login, not at boot without a +login. To stop it, or also remove its start at login: + +.. code:: console + + systemctl --user stop onechat.service + systemctl --user disable onechat.service + +************ + Daemon log +************ + +The daemon writes short events with the part prefix and ordered +``key=value`` attributes. An error cause comes last. Normal events use +info, failures that can be retried or lose nothing use warning, and +failures that lose data or stop work use error. + +With ``JOURNAL_STREAM`` set, each line carries a syslog priority without +a time stamp or level word. The user journal can show only warnings and +errors: + +.. code:: console + + journalctl --user -u onechat -p warning + +The summary warning names its node. A schedule run has ``late`` only +when it ran late: + +.. code:: text + + <4>onechat: summary failed level=0 index=0: fake failure + <6>onechat: schedule ran id=1 + <6>onechat: schedule ran id=2 late=1m0s + +Outside the journal, each line starts with the time stamp and level. + +********** + Telegram +********** + +The daemon also connects the agent to a Telegram bot when +``telegram.json`` exists in the data directory. It polls the Bot API for +new messages, so it needs no open port. + +#. Make a bot with BotFather and keep its token. + +#. Find your numeric Telegram user ID. + +#. Write the token and the ID to ``telegram.json``, and give the file + mode 0600, because the token is a secret. The daemon refuses the file + with any other mode. + +#. Start the daemon, open a private chat with the bot, and write. + +.. code:: json + + {"token": "123456:ABC-token-from-BotFather", "owner": 123456789} + +The bot reads only the private chat of the owner and ignores all other +updates. Each text message goes to the agent as a user message, the line +``/stop`` interrupts the running turn, and the line ``/stop NAME`` stops +the subagent NAME. The bot sends the replies in parts of at most 4096 +characters, with their Markdown as Telegram formats: bold, italic, +strikethrough, code, code blocks, links, and headings as bold text. When +Telegram refuses the formats of a part, the bot sends the part again as +plain text. Each error goes as a short message in plain text. It shows +the typing action while a turn runs. It does not send the tool calls. +For a subagent it sends only ``[NAME] started`` and ``[NAME] finished``, +``[NAME] stopped``, or ``[NAME] failed``, because the reply of the agent +carries the content of the report. It sends each message that the user +types in the terminal client, with the mark ``[terminal]``. + +``agent/telegram-offset.json`` holds the ID of the next update. The bot +moves it past an update only after the message is durable in the log, so +a restart of the daemon does not lose a message. A crash just after the +write of a message can make the message repeat once. A network failure +or a rate limit makes the bot wait and try again, and does not stop the +daemon. + +********** + WhatsApp +********** + +The daemon connects to the WhatsApp account of the user as a linked +device. The terminal client and Telegram take three commands: + +``/whatsapp pair PHONE`` + Links the account of PHONE, the number with its country code, such as + ``+15550001111``. The daemon answers with the steps and then with a + pairing code in its own message: in WhatsApp on the phone, open + Linked devices, tap Link a device, then tap Link with phone number + instead, and enter the code. The code expires in about 2 minutes, and + the same command gives a new code. When the link succeeds, the daemon + answers with the linked number. Linked devices on the phone shows the + device as OneChat. The name applies at the pairing, so an existing + link keeps its old name until ``/whatsapp unlink`` and a new + ``/whatsapp pair PHONE``. + +``/whatsapp status`` + Shows the link: not linked, linked and connected, or linked and + disconnected, with the times of the connection. + +``/whatsapp unlink`` + Ends the link and deletes the session. When WhatsApp is disconnected, + the phone does not get the logout, so remove the device in Linked + devices on the phone. + +The answers go only to the client that asked, and the commands do not go +to the agent, so the agent never gets the phone number or the code. The +chat gets only the notes that WhatsApp is linked or unlinked. + +The session is ``whatsapp/session.db`` in the data directory, with mode +0600 in a directory with mode 0700. It gives full access to the WhatsApp +account, so the daemon refuses the file when its mode is not 0600. The +shell of the agent runs as the user, so it can read the session. If the +session leaks, remove the linked device in WhatsApp on the phone, and +the session stops working. Without the file, WhatsApp is not linked. + +The daemon reads the text messages of the private chats and the groups, +from other people and from the user. It does not read media, reactions, +edits, deletions, view-once and disappearing messages, or the history +from before the link. The messages of one chat form one batch until the +chat is quiet for 20 seconds, for 2 minutes or 50 messages at the most. +Each batch goes to the chat of the agent as one work message that starts +with ``[WhatsApp]``, with the name and the ID of the chat, and the time, +the sender, and the text of each message. The messages of the user show +as ``You``. ``whatsapp/state.json``, with mode 0600, keeps the batches +that are not in the chat yet, so they survive a restart, and the records +of the sends. A crash just after a batch goes to the chat can make the +batch repeat once. + +A batch of a private chat with a message of another person starts a +turn, and so does a batch of a group with a message that mentions the +user or replies to a message of the user. The other batches go to the +chat with no turn, and the agent sees them at its next turn. A batch +that starts a turn ends with a line that tells the agent to end its turn +with an empty reply when nothing in the batch needs the user. No client +shows an empty reply, so such a batch gives no message in Telegram. The +terminal client shows each batch, and Telegram shows none. + +The phone number of the user never goes to the chat: a mention of the +user shows as ``@you``, the number shows as ``OWN``, and the chat of the +user with itself is not read. But the IDs of the other chats go to the +chat and to the model, because the agent needs them to send. The ID of a +private chat holds the phone number of the contact, such as +``15550002222@s.whatsapp.net``, and the ID of a group can hold the +number of the person who made it. + +While the daemon is stopped or disconnected, the WhatsApp servers keep +the messages for the linked device and send them at the next connection. +They keep them only for a limited time, so a long stop can lose +messages. These messages form batches by their original times, as live +messages do. The daemon waits for the end of this catch-up, or one +minute at the most, and then gives all its batches to the agent in one +turn, not one turn for each chat. The batches of ``whatsapp/state.json`` +at the start wait for the first catch-up in the same way. + +The agent finds a chat with the tool ``whatsapp_find``: it gives the +contacts and the groups whose name holds a text, with no regard to case +and accents, with their IDs. The names of the contacts come from the +session, and the names of the groups from WhatsApp, kept in memory for +one hour. The names arrive from the phone some minutes after the link, +and until then the tool says that they are not ready. The agent sends a +text message with the tool ``whatsapp_send``, only when the user asks +for it. Each send has an ID that the agent gives: a repeated call with +the same ID gives the answer of the first call and sends nothing. The +state file keeps the last 1000 send records. When the daemon stops +during a send, the result is not known, and the tool tells the agent to +ask the user to look at the chat in WhatsApp. Without a link, both tools +tell the agent to ask the user to run ``/whatsapp pair PHONE``. Only the +agent has the WhatsApp tools, not its subagents. + +Each batch of a group goes to the chat, also with no turn, and the +memory tree summarizes it with model calls, which use the quota. + +When the session holds a linked device, the daemon connects at its +start. If the first connection fails, it tries again after 2 seconds, +then doubles the wait up to 5 minutes. The inbox and commands stay +active during each wait. An unlink or a logout stops these retries. +After a network failure of an open connection, the library connects +again by itself. When a disconnection lasts 10 minutes, all clients get +one notice, and one more when the connection comes back. When WhatsApp +ends the connection for good, for example because another program uses +the same session, all clients get a notice at once. When the user +removes the device in the phone, the daemon deletes the session and +tells all clients one time. + +************************* + Import WhatsApp history +************************* + +Stop the daemon and finish all subagents before an import. Export each +chat from WhatsApp as a ZIP with ``_chat.txt``. Keep the chat name in +the ZIP name, such as ``WhatsApp Chat - Example.zip``. Read the plan +first: + +.. code:: console + + onechat import -n CHAT.zip OTHER.zip + onechat import CHAT.zip OTHER.zip + +``-n`` makes no model call and prints the chats, chat-day batches, local +dates, cutoff, text bytes, merged message count, and estimated calls. +The import keeps multiline text, system lines, and omitted-media +markers. It maps the session's profile name to ``You``. Without that +name, it keeps the senders as written and says so in the plan. An exact, +unambiguous local contact or batch name supplies the chat ID when known. +No network lookup is made. + +Each chat and local day becomes one work message marked ``(imported)``. +Only messages before the first message of the earliest live WhatsApp +batch enter the import. Live times have minute precision, so the cutoff +excludes that whole minute. A chat-day already imported is skipped. +Overlapping chat-days in the supplied ZIPs are refused. Review the +exports before import: their text goes to the model as chat history. + +The import merges the full log in date order and builds its tree with +real compactor calls, the agent prompt, and the agent cache key. It uses +a limit of 1,500 calls and the probe's 15-quota-point limit, and prints +the same usage report. Its estimate is about 1.3 calls per merged +message, not a cost guarantee. A stopped or failed run leaves the agent +unchanged. A new run starts over. Success keeps the other agent files +and renames the old directory to ``agent-backup-TIMESTAMP`` before it +installs the new one. Keep the backup until the new memory is checked, +then start the daemon. Message IDs change to their positions in the +merged log. + +*********** + Schedules +*********** + +The agent sets reminders and timed tasks when the user asks for them, +such as "remind me tomorrow at 9" or "every weekday at 8, tell me what +my WhatsApp groups talked about". A schedule has a text, the task for +the agent, and either one time or a cron expression. The agent also +lists the schedules with their next times, and removes one when the user +asks. Only the agent has the schedule tools, not its subagents. + +A cron expression has five fields: minute, hour, day of month, month, +and day of week, where 0 and 7 are Sunday. A field is a comma list of +``*``, ``N``, and ``N-M``, and ``*`` and ``N-M`` can have a step ``/S``, +as in ``0 8 * * 1-5`` or ``*/15 9-17 * * *``. When both day fields start +with something other than ``*``, a day matches when either field +matches. Names such as ``MON``, macros such as ``@daily``, the +characters ``? L W #``, and ``N/S`` are refused. The runs must be at +least 5 minutes apart, because each run is a full turn that uses the +quota, so ``*/5 * * * *`` is the shortest schedule. + +The times use the time zone of the machine. A time that the clock skips +at the start of daylight saving time runs later by the length of the +skip. A time that the clock repeats at its end runs once, at its first +occurrence. + +At its time, a schedule goes to the chat as a work message that starts +with ``[Schedule ID]``, with its time and its text, and it starts a +turn. The message ends with a line that tells the agent to end its turn +with an empty reply when nothing needs the user. The terminal client +shows the message, and Telegram shows only the reply. After its run, a +one-time schedule goes, and a repeating schedule gets its next time. + +``agent/schedules.json``, with mode 0600, holds the schedules. Each +change writes it before the tool answers, so the schedules survive a +restart. A schedule that came due while the daemon was stopped runs once +at the next start, with its original time and a line that says that the +run is late. A repeating schedule that missed several times runs once. A +crash just after a run goes to the chat can make the run repeat once. + ******* Probe ******* -``onechat probe LOG COUNT`` measures the compactor on a log. It appends -the first COUNT messages of LOG, one JSON object ``{"kind", "text"}`` on -each line, to a new chat in ``probe/`` in the data directory, and builds -its tree with real model calls. The calls use the quota of the account. -The run stops at the first failed call, or when the used percent of a -quota window grows by more than 15 points. At the end it reports the -calls per message, the cached share of the input tokens, the retries, -the line sizes, and the quota at the first and the last call. +``onechat probe LOG COUNT [BUILDS]`` measures the compactor on a log. It +appends the first COUNT messages of LOG, one JSON object ``{"kind", +"text"}`` on each line, to a new chat in ``probe/`` in the data +directory, and builds its tree with real model calls. The calls use the +quota of the account. The run stops at the first failed call, after 660 +calls, or when the used percent of a quota window grows by more than 15 +points. At the end it reports the calls per message, the cached share of +the input tokens, the retries, the line sizes, and the quota at the +first and the last call. BUILDS, from 1 to 8, limits the model calls at +once in the run. A test writes a generated log of readable made-up messages: @@ -110,6 +531,9 @@ Build and run: nix build ./result/bin/onechat +``onechat version`` prints ``yyyy-mm-dd-gitshort`` for a Nix build, adds +``-dirty`` for a dirty tree, and prints ``dev`` for a direct Go build. + Format and check all files: .. code:: console @@ -117,3 +541,6 @@ Format and check all files: bin/format bin/check nix flake check + +``bin/check`` and ``nix build`` run a short replay of the synthetic log. +``nix flake check`` also runs the full replay of 100,000 messages.