Interactions & Tools

GeoSense supports embedded tool blocks for application interactions. Each tool defines its own data contract, described in full below.

Question Tool

Question blocks are interactive prompts emitted by the agent. Each question contains a stable id, a prompt in <header>, and one or more selectable <option> entries.

Each option includes a message attribute. This is a suggested response text for the user-selected path and can be used by the client as the default answer payload.

Some options include a context attribute (for example, context="visibility" or context="visibility position image"). In those cases, selecting the option indicates that the client should include the requested context types in the response message.

Generic textual question

<question id="1">
  <header>Would you like to proceed?</header>
  <option message="Yes, proceed with the aforementioned command.">Yes</option>
  <option message="Please abort the current execution.">No</option>
</question>

Clarification question

<question id="2">
  <header>Please clarify, do you mean A or B?</header>
  <option message="I mean A"><i>A</i></option>
  <option message="I mean B"><i>B</i></option>
  <option message="I don't actually mean either of those">Neither</option>
</question>

Context request question

<question id="3">
  <header>Could you tell me what your visibility settings are?</header>
  <option message="Of course, here you go." context="visibility">Allow</option>
  <option message="No, find some other way.">Deny</option>
</question>

Multi-context request question

<question id="4">
  <header>I'm not sure what you can see, let me take a look</header>
  <option
    message="Here's the viewport data."
    context="visibility position image"
  >
    Allow
  </option>
  <option message="No, I don't want to do that.">Deny</option>
</question>

Answering a question

When responding to a <question> from the agent, format your reply as follows.

The {{variable}} template syntax indicates what should be replaced with the relevant content. The index should be 0-based, and the message should, but mustn't necessarily, match the suggested message in the selected <option>. The questionId should be the ID of the question you are responding to.

<answer
  question="{{questionId}}"
  question-option-index="{{indexOf(selectedOption)}}"
>
  {{message}}
</answer>

If the question requests certain context types and you intend to satisfy the request, include those context types in the same response message.

The user can also write an answer of their own instead of choosing an option. That answer carries no question-option-index:

<answer question="{{questionId}}">{{message}}</answer>

Approvals

Some backend tool calls wait for the user's approval before they run: any call that costs credits or deletes something, and the first use of backend tools in a session that hasn't been granted them. The turn then pauses, and its final message carries a pending_approval instead of an answer. The app shows that message as an approval card, not as its text. The user approves or rejects it through /message/resume/stream, and a turn resumed that way can pause again on the next call. The user can also send a new message instead, which sets the approval aside.

The session's allow_backend_tools is the user's own toggle beside the send button. Approving a request for backend tool access turns it on as well.

Backend tool activity

The backend_tool_call and backend_thinking stream events are gathered into one collapsible activity log per turn: what the agent said it would do, and each call it made with whether that call failed. The log is not stored, so it is gone after a reload.

Highlight Tool

The highlight tool embeds a data payload into a tool block.

Use this tool when the agent needs the client to visually emphasize objects or relationships in the UI. The message body remains user-facing, while the data payload contains machine-readable instructions for highlighting logic.

In all highlight payloads:

• ids identifies the primary annotation set to highlight.
• Optional links and linkValues define pairwise relationships and relationship metrics.
• linkType defines how to interpret linked relationships (for example, distance or overlap).
• Optional attributeHints describes how additional attributes should be rendered (for example, as length, area, or credits).

When linkValues is present, it should be aligned index-by-index with ids and links.

Highlight tool type contract

export type HighlightToolData =
  | SimpleHighlightData
  | LinkedHighlightData;

type LinkedHighlightToData =
  | DistanceLinkedHighlightData
  | OverlapLinkedHighlightData;

type SimpleHighlightData = ExtraAttributes & {
  ids: number[]; // length: N, annotation IDs
}
type DistanceLinkedHighlightData = SimpleHighlightData & {
  linkType: 'distance';
  links: number[]; // length: N, annotation IDs
  linkValues?: number[]; // [optional] length: N, distances in meters
}

type OverlapLinkedHighlightData = SimpleHighlightData & {
  linkType: 'overlap';
  links: number[]; // length: N, annotation IDs
  linkValues?: number[]; // [optional] length: N, intersection %, [0-1]
}

type AttributeType = 'length' | 'area' | 'pixels' | 'credits' | 'position';
type ExtraAttributes = Record<string, any> & {
  attributeHints?: Record<string, AttributeType>
}

Highlight tool message example

<tool
  type="highlight"
  data='{
    "ids": [102, 103, 104, 201, 542],
    "links": [23, 23, 23, 51, 229],
    "linkValues": [4.3, 5.6, 5.8, 11.2, 18.9],
    "linkType": "distance"
  }'
>
  Showcase all trees that are within <distance value="20" /> of a house
</tool>

Find Similar Tool

Use the Find Similar tool when the agent wants the client to run a similarity search over geospatial annotations and return the located results.

The hint field provides optional semantic guidance (for example, pool), while samples can provide one or more explicit annotation exemplars. When v is included in a sample, it narrows the source vector for that sample.

Find similar type contract

type FindSimilarToolData = {
  hint?: string;
  samples?: AnnotationTarget[];
}
type AnnotationTarget = {
  id: number;
  v?: string;
}

Find similar message example

<tool
  type="find-similar"
  data='{
    "hint": "pool",
    "samples": [
      {"id": 25},
      {"id": 216, "v": "1234-abcd"}
    ]
  }'
>
  Find other swimming pools similar to <id>25</id> and
  <id v="1234-abcd">216</id>.
</tool>

Find similar interactive response format

The client should send this response format after the interactive request has been completed and results are available. The attributes are the stable fields for parsing (type, class-id, and request), while the text content is human-readable and may vary.

Include a SelectionContext in the response message with the annotations located by the find-similar operation.

<response
  type="find_similar"
  class-id={{classId}}
  request="{{requestId}}"
>
  Find Similar located {{results.count}}
  result{{results.count != 1 ? "s" : ""}}
  belonging to class {{className}}.
</response>

Linked Data Interpolation

Linked data interpolation allows message attributes to reference values from artifact-backed data contexts without inlining large payloads directly in the message text.

Interpolation placeholders use ${...} JSON-path-like addressing. During message parsing, placeholders are replaced textually before attribute-level parsing.

The first element of the JSON path is the artifact_id of one of the received data contexts. The rest (if present) are a path from the root of that artifact to some sub-object.

This pattern is useful for:

• large highlight payloads,
• repeated references to computed values,
• keeping message content concise while preserving determinism.

Interpolation in message attributes

The distance from the largest tree to the largest house is
<size value="${distances.261.24}" />. The furthest house is
<size value="${distances.261.101}" /> away.

Interpolation with tool payloads

<tool
  type="highlight"
  data="${artifact-1234-abcd}"
>
  Show Buildings
</tool>

The closest building (<ref id="${artifact-1234-abcd.ids.0}" />) is
<size value="${artifact-1234-abcd.linkValues.0}" /> away from
<ref id="${artifact-1234-abcd.links.0}" /> and has an area of
<area id="${artifact-1234-abcd.areas.0}" />.